Hi,
I was already trying to do it one by one, therefore reusing each
iteration the same objects, but I was not able yet to do it - all that
it wrote was NULL in all fields of each feature.
Anyway, you're code works for mysql too, except for the createSchema
that is not implemented yet. So, I run the same test you did, and I got
it done in half hour, avg. throughput of 2793 features/s in my
notebook. Thanks for the help, it was very nice.
Now, I have to compare this approach to the use of java persistent
objects. (My model is already made using only ram memory). If anyone
has already made this comparison I appreciate comments.
Thanks,
Rui
Quoting Gabriel Roldán <[EMAIL PROTECTED]>:
> Hi,
> problem seems to be in the approach. In general you should favor streaming
> over memory loading.
> Loading 3 million features in memory, then commiting them all at once is time
> and resource consuming for sure.
>
> I've tried the following modified version of your code over a postgis
> database
> in a 100Mb/s lan, with a throughput of ~880 Features/second (so it would take
> about 50+ minutes to complete), but its consuming a reasonable 30% CPU and a
> despreciable ammount of RAM.
>
> Gabriel.
>
>
> public class Main {
>
> private DataStore datastore;
>
> Main() {
> }
>
> public static void main(String[] args) {
> Main main = new Main();
> try {
> main.createDataStore();
> main.populateDB();
> } catch (Exception e) {
> e.printStackTrace();
> System.exit(-1);
> }
> }
>
> private void populateDB() throws Exception {
> FeatureType ftPatch = createFeatureType();
> datastore.createSchema(ftPatch);
>
> // instances
> final GeometryFactory gf = new GeometryFactory();
> final int NUM_FEATURES = 3000000;
>
> Transaction transaction = new DefaultTransaction();
> FeatureWriter featureWriter =
> datastore.getFeatureWriterAppend("patches", transaction);
>
> Feature tempF = null;
> Point locGeom = null;
> long totalTime = System.currentTimeMillis();
>
> // after how many inserts make a commit?
> final int commitInterval = 10000;
> try {
> for (int i = 0; i < NUM_FEATURES; i++) {
> tempF = featureWriter.next();
>
> locGeom = gf.createPoint(new Coordinate(i, i));
> tempF.setAttribute("width", new Integer(i));
> tempF.setAttribute("location", locGeom);
> tempF.setAttribute("elevation", new Integer(2 * i));
>
> featureWriter.write();
>
> if (i % commitInterval == 0) {
> System.out.println("commiting " + commitInterval + "
> features ("
> + (1 + (i / commitInterval) * commitInterval) + "
> to "
> + (((i / commitInterval) * commitInterval) +
> commitInterval) + ")");
> transaction.commit();
> }
> }
> transaction.commit();
> totalTime = System.currentTimeMillis() - totalTime;
> } finally {
> featureWriter.close();
> }
> System.out.println(NUM_FEATURES + " inserted in " + totalTime + "ms");
> System.out.println("Avg. throughput: " + (NUM_FEATURES / (totalTime /
> 1000))
> + " Features/s");
> }
>
> private FeatureType createFeatureType() throws SchemaException {
> AttributeType pWidth = AttributeTypeFactory.newAttributeType("width",
> Integer.class);
> GeometryAttributeType pLoc =
>
> (GeometryAttributeType)
> AttributeTypeFactory.newAttributeType("location", Point.class);
> AttributeType pElevation = AttributeTypeFactory
> .newAttributeType("elevation", Integer.class);
> FeatureType ftPatch = FeatureTypeFactory.newFeatureType(new
> AttributeType[] { pWidth, pLoc,
> pElevation }, "patches");
> return ftPatch;
> }
>
> private void createDataStore() throws IOException {
> Map<String, String> params = new HashMap<String, String>();
> params.put("dbtype", "postgis");
> params.put("host", "axolote");
> params.put("port", "5432");
> params.put("database", "udig_sample");
> params.put("user", "postgres");
> params.put("passwd", "****");
> params.put("wkb enabled", "true");
>
> this.datastore = DataStoreFinder.getDataStore(params);
> System.out.println("DataStore created!");
> }
>
> }
>
> On Wednesday 14 March 2007 21:22, [EMAIL PROTECTED] wrote:
>> Hi,
>>
>> Using geotools2 v2.4-M0, in kubuntu edgy, I was able to insert
>> dynamically features into a mysql database. I will send the code for
>> those who also need to do it.
>> For testing purposes I am using only one FeatureType, and I will store
>> 3000000 objects (3000000 rows in the mysql table). This is where I have
>> problems again...
>> My computer, a centrino dothan 1.6 with 1GB ram, has been working for
>> about one hour, then I killed the process and inspected my database.
>> There were only about 10000 rows inserted.
>> Does anyone worked with a db system and geotools with such a large
>> number of objects? The problem comes from mysql or from geotools?
>> Otherwise, is it my computer?
>>
>> Thanks,
>> Rui Lopes
>>
>>
>> CODE:
>>
>> public class Main {
>>
>> private MySQLDataStore datastore;
>>
>> Main(){}
>>
>> public static void main(String[] args) {
>> // TODO Auto-generated method stub
>> Main main = new Main();
>> main.createDataStore();
>> main.populateDB();
>> }
>>
>> private void populateDB() {
>> AttributeType pWidth =
>> AttributeTypeFactory.newAttributeType("width",
>> Integer.class);
>> GeometryAttributeType pLoc =
>>
>> (GeometryAttributeType)AttributeTypeFactory.newAttributeType("location",
>> Point.class);
>> AttributeType pElevation =
>> AttributeTypeFactory.newAttributeType("elevation", Integer.class);
>> try {
>> FeatureType ftPatch = FeatureTypeFactory.newFeatureType(
>> new AttributeType[] { pWidth, pLoc,
>> pElevation }, "patches");
>>
>> // instances
>> DefaultFeatureCollection fcol = new
>> DefaultFeatureCollection("patches", ftPatch);
>> Feature tempF = null;
>> GeometryFactory gf = new GeometryFactory();
>> Point locGeom = null;
>> for(int i = 1 ;i == 1 ;i++ /*i < patches.length; i++*/)
>> for(int j = 0 ; j < patches[0].length; j++){
>>
>> locGeom = gf.createPoint( new
>> Coordinate(i,j));
>> tempF = ftPatch.create(new Object[] { 1
>> ,locGeom, i+j },
>> "Patch-"+i+"-"+j);
>> fcol.add(tempF);
>> }
>>
>> JDBCFeatureSource fsrcPatches =
>> (JDBCFeatureSource)datastore.getFeatureSource("patches");
>> JDBCFeatureStore fsPatches = new
>> JDBCFeatureStore(fsrcPatches.getJDBCDataStore(), ftPatch );
>> //FeatureReader aReader = DataUtilities.reader( altemp
>> );
>> fsPatches.addFeatures( fcol );
>>
>> } catch (FactoryConfigurationError e) {
>> // TODO Auto-generated catch block
>> e.printStackTrace();
>> } catch (SchemaException e) {
>> // TODO Auto-generated catch block
>> e.printStackTrace();
>> } catch (IllegalAttributeException e) {
>> // TODO Auto-generated catch block
>> e.printStackTrace();
>> } catch (IOException e) {
>> // TODO Auto-generated catch block
>> e.printStackTrace();
>> }
>>
>>
>>
>>
>> }
>>
>> private void createDataStore() {
>>
>> MySQLDataStoreFactory factory = new MySQLDataStoreFactory();
>> Map<String,String> params = new HashMap<String,String>();
>> params.put( "database", "mater" );
>> params.put( "dbtype", "mysql");
>> params.put( "host", "localhost");
>> params.put( "port", "3306");
>> params.put( "user", "ruya");
>> params.put( "passwd", "");
>>
>> try {
>> this.datastore = (MySQLDataStore)
>> factory.createDataStore( params );
>> } catch (IOException e) {
>> // TODO Auto-generated catch block
>> e.printStackTrace();
>> }
>> System.out.println("DataStore created!");
>> }
>>
>> private void loadGISData( Patch[][] patches ) {
>> loadTopo( patches, _TOPO_PATH );
>> loadHidro( patches, _HIDRO_PATH );
>> /*
>> for( int i = 0 ; i < getWorldXSize(); i++ ){
>> for( int j = 0 ; j < getWorldYSize(); j++ ){
>> //patches[i][j].setAgentCapacity();
>> space.putObjectAt( i , j , patches[i][j] );
>> }
>> }
>> */
>> }
>>
>> ----------------------------------------------------------------
>> This message was sent using IMP, the Internet Messaging Program.
>>
>>
>> -------------------------------------------------------------------------
>> Take Surveys. Earn Cash. Influence the Future of IT
>> Join SourceForge.net's Techsay panel and you'll get the chance to share
>> your opinions on IT & business topics through brief surveys-and earn cash
>> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
>> _______________________________________________
>> Geotools-gt2-users mailing list
>> [email protected]
>> https://lists.sourceforge.net/lists/listinfo/geotools-gt2-users
>
----------------------------------------------------------------
This message was sent using IMP, the Internet Messaging Program.
-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
Geotools-gt2-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/geotools-gt2-users
