Hi,
problem seems to be in the approach. In general you should favor streaming
over memory loading.
Loading 3 million features in memory, then commiting them all at once is time
and resource consuming for sure.
I've tried the following modified version of your code over a postgis database
in a 100Mb/s lan, with a throughput of ~880 Features/second (so it would take
about 50+ minutes to complete), but its consuming a reasonable 30% CPU and a
despreciable ammount of RAM.
Gabriel.
public class Main {
private DataStore datastore;
Main() {
}
public static void main(String[] args) {
Main main = new Main();
try {
main.createDataStore();
main.populateDB();
} catch (Exception e) {
e.printStackTrace();
System.exit(-1);
}
}
private void populateDB() throws Exception {
FeatureType ftPatch = createFeatureType();
datastore.createSchema(ftPatch);
// instances
final GeometryFactory gf = new GeometryFactory();
final int NUM_FEATURES = 3000000;
Transaction transaction = new DefaultTransaction();
FeatureWriter featureWriter =
datastore.getFeatureWriterAppend("patches", transaction);
Feature tempF = null;
Point locGeom = null;
long totalTime = System.currentTimeMillis();
// after how many inserts make a commit?
final int commitInterval = 10000;
try {
for (int i = 0; i < NUM_FEATURES; i++) {
tempF = featureWriter.next();
locGeom = gf.createPoint(new Coordinate(i, i));
tempF.setAttribute("width", new Integer(i));
tempF.setAttribute("location", locGeom);
tempF.setAttribute("elevation", new Integer(2 * i));
featureWriter.write();
if (i % commitInterval == 0) {
System.out.println("commiting " + commitInterval + "
features ("
+ (1 + (i / commitInterval) * commitInterval) + "
to "
+ (((i / commitInterval) * commitInterval) +
commitInterval) + ")");
transaction.commit();
}
}
transaction.commit();
totalTime = System.currentTimeMillis() - totalTime;
} finally {
featureWriter.close();
}
System.out.println(NUM_FEATURES + " inserted in " + totalTime + "ms");
System.out.println("Avg. throughput: " + (NUM_FEATURES / (totalTime /
1000))
+ " Features/s");
}
private FeatureType createFeatureType() throws SchemaException {
AttributeType pWidth = AttributeTypeFactory.newAttributeType("width",
Integer.class);
GeometryAttributeType pLoc =
(GeometryAttributeType)
AttributeTypeFactory.newAttributeType("location", Point.class);
AttributeType pElevation = AttributeTypeFactory
.newAttributeType("elevation", Integer.class);
FeatureType ftPatch = FeatureTypeFactory.newFeatureType(new
AttributeType[] { pWidth, pLoc,
pElevation }, "patches");
return ftPatch;
}
private void createDataStore() throws IOException {
Map<String, String> params = new HashMap<String, String>();
params.put("dbtype", "postgis");
params.put("host", "axolote");
params.put("port", "5432");
params.put("database", "udig_sample");
params.put("user", "postgres");
params.put("passwd", "****");
params.put("wkb enabled", "true");
this.datastore = DataStoreFinder.getDataStore(params);
System.out.println("DataStore created!");
}
}
On Wednesday 14 March 2007 21:22, [EMAIL PROTECTED] wrote:
> Hi,
>
> Using geotools2 v2.4-M0, in kubuntu edgy, I was able to insert
> dynamically features into a mysql database. I will send the code for
> those who also need to do it.
> For testing purposes I am using only one FeatureType, and I will store
> 3000000 objects (3000000 rows in the mysql table). This is where I have
> problems again...
> My computer, a centrino dothan 1.6 with 1GB ram, has been working for
> about one hour, then I killed the process and inspected my database.
> There were only about 10000 rows inserted.
> Does anyone worked with a db system and geotools with such a large
> number of objects? The problem comes from mysql or from geotools?
> Otherwise, is it my computer?
>
> Thanks,
> Rui Lopes
>
>
> CODE:
>
> public class Main {
>
> private MySQLDataStore datastore;
>
> Main(){}
>
> public static void main(String[] args) {
> // TODO Auto-generated method stub
> Main main = new Main();
> main.createDataStore();
> main.populateDB();
> }
>
> private void populateDB() {
> AttributeType pWidth =
> AttributeTypeFactory.newAttributeType("width",
> Integer.class);
> GeometryAttributeType pLoc =
>
> (GeometryAttributeType)AttributeTypeFactory.newAttributeType("location",
> Point.class);
> AttributeType pElevation =
> AttributeTypeFactory.newAttributeType("elevation", Integer.class);
> try {
> FeatureType ftPatch = FeatureTypeFactory.newFeatureType(
> new AttributeType[] { pWidth, pLoc,
> pElevation }, "patches");
>
> // instances
> DefaultFeatureCollection fcol = new
> DefaultFeatureCollection("patches", ftPatch);
> Feature tempF = null;
> GeometryFactory gf = new GeometryFactory();
> Point locGeom = null;
> for(int i = 1 ;i == 1 ;i++ /*i < patches.length; i++*/)
> for(int j = 0 ; j < patches[0].length; j++){
>
> locGeom = gf.createPoint( new
> Coordinate(i,j));
> tempF = ftPatch.create(new Object[] { 1
> ,locGeom, i+j },
> "Patch-"+i+"-"+j);
> fcol.add(tempF);
> }
>
> JDBCFeatureSource fsrcPatches =
> (JDBCFeatureSource)datastore.getFeatureSource("patches");
> JDBCFeatureStore fsPatches = new
> JDBCFeatureStore(fsrcPatches.getJDBCDataStore(), ftPatch );
> //FeatureReader aReader = DataUtilities.reader( altemp
> );
> fsPatches.addFeatures( fcol );
>
> } catch (FactoryConfigurationError e) {
> // TODO Auto-generated catch block
> e.printStackTrace();
> } catch (SchemaException e) {
> // TODO Auto-generated catch block
> e.printStackTrace();
> } catch (IllegalAttributeException e) {
> // TODO Auto-generated catch block
> e.printStackTrace();
> } catch (IOException e) {
> // TODO Auto-generated catch block
> e.printStackTrace();
> }
>
>
>
>
> }
>
> private void createDataStore() {
>
> MySQLDataStoreFactory factory = new MySQLDataStoreFactory();
> Map<String,String> params = new HashMap<String,String>();
> params.put( "database", "mater" );
> params.put( "dbtype", "mysql");
> params.put( "host", "localhost");
> params.put( "port", "3306");
> params.put( "user", "ruya");
> params.put( "passwd", "");
>
> try {
> this.datastore = (MySQLDataStore)
> factory.createDataStore( params );
> } catch (IOException e) {
> // TODO Auto-generated catch block
> e.printStackTrace();
> }
> System.out.println("DataStore created!");
> }
>
> private void loadGISData( Patch[][] patches ) {
> loadTopo( patches, _TOPO_PATH );
> loadHidro( patches, _HIDRO_PATH );
> /*
> for( int i = 0 ; i < getWorldXSize(); i++ ){
> for( int j = 0 ; j < getWorldYSize(); j++ ){
> //patches[i][j].setAgentCapacity();
> space.putObjectAt( i , j , patches[i][j] );
> }
> }
> */
> }
>
> ----------------------------------------------------------------
> This message was sent using IMP, the Internet Messaging Program.
>
>
> -------------------------------------------------------------------------
> Take Surveys. Earn Cash. Influence the Future of IT
> Join SourceForge.net's Techsay panel and you'll get the chance to share
> your opinions on IT & business topics through brief surveys-and earn cash
> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
> _______________________________________________
> Geotools-gt2-users mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/geotools-gt2-users
-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
Geotools-gt2-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/geotools-gt2-users
