(sorry I didn't notice the "Reply" sends not to the mailing listaddress... not intended )
yeah I did use the memory mapped buffer option, but somehow it does notconsume much memory (only using about 40MB after running >10000querieson 6 shape files totalling about 2GB ), so I assume there is much more potential to be explored Thanks Yang On Thu, Jan 12, 2012 at 10:57 PM, Jody Garnett <[email protected]> wrote: > Please keep Questions on the user list; so others can offer advice and take > part in the conversation. > > If your employer does ask that you stay off public email lists I would be > happy to arrange a support contract (which would allow me to answer your > private emails in a timely and responsible fashion). > > You can experiment with the memory mapped setting for different shape file > performance (for supporting operating systems). > -- > Jody Garnett > > On Friday, 13 January 2012 at 4:42 PM, Yang wrote: > > thanks, > > while I think most likely the memory-based version will be faster than > my current choice of IndexedShapeFileDataStore, the memory approach > has a limitation in that if it's run on a machine with a smaller > memory, it may fail at all, instead of degrading to a slower > performance. I wonder on the IndexedShapeFileDataStore side, how is it > possible to let it be smarter and use as much memory as possible ?? > > On Thu, Jan 12, 2012 at 8:47 PM, Jody Garnett <[email protected]> > wrote: > > No iris not. > > One uses an index on disk. And one used an index in memory. > > Jody > > > > On 13/01/2012, at 11:13 AM, Yang <[email protected]> wrote: > > since I need to look up a latitude,longitude point, I need spatial > index, i.e. SpatialIndexFeatureCollection, but isn't this the > underlying structure utilized by IndexedShapeFileDataSource (which I'm > already using) ? > > thanks > Yang > > On Thu, Jan 12, 2012 at 2:01 PM, Jody Garnett <[email protected]> > wrote: > > There are three implementations available for an in memory > FeatureCollection. > > http://docs.geotools.org/latest/userguide/library/main/collection.html#performance-options > > -- > Jody Garnett > > On Friday, 13 January 2012 at 6:50 AM, Mark Prins wrote: > > Op 12-Jan-12 21:10, Yang schreef: > > thanks, I did utilize the useMemoryMappedBuffer option. let me dig up > the first one. > > by "copy all of the shapefile into a DataStore or feature store that > uses a memory stucture " do you mean something like "MemoryDataStore" > ( > http://udig.refractions.net/files/docs/api-geotools/org/geotools/data/memory/MemoryDataStore.html > ) ? so basically I need to iterate through all the features in my > existing ShapeFileDataStore, and insert them into MemoryDataStore, and > hopefully the later queries would be faster , right? sounds like a > possible approach. > > > yes, this would save using file io. i've never used memoryDataStore and > the docs > (http://docs.geotools.org/latest/userguide/library/data/memory.html) say > it may be slow, so you may want to cook up one yourself. > Maybe one of the SimpleFeatureCollection implementations together with > subCollection(Filter) may also fit your need. > > > actually I did a later test and put all my indexedShapeFileDataStore > on a tmpfs mount (basically memory disk), and surprisingly the > performance did not change much from direct disk access. most of the > time is still spent on reading file buffers and opening index file > readers > > > this opening and closing of file handles is probably the most costly > part. I try to steer away from shapefiles whenever I can. > > maybe one of the devs has something to say about this, but it's a bit > difficult without you code. > > > Yang > > On Thu, Jan 12, 2012 at 11:37 AM, Mark Prins<[email protected]> wrote: > > Op 12-Jan-12 01:36, Yang schreef: > > my code looks up features from a IndexedShapeFileDataStore > > basically: > > for(Point p : given points ) { > dataStore.getFeature(query(p)); > } > > > I can see from "top" command that the RES memory consumption is only 40MB, > I have about 8GB on the box, and have allocated -Xmx4000M to jvm. > is there any way I could let geotools load more content into memory, > so hopefully it would query faster? > > > you could try to copy all of the shapefile into a DataStore or feature > store that uses a memory stucture such as eg. a hashtable to create an > in memory cache of the data, of course this will only work reliably in a > read-only scenario. it depends on you data if it's worth the tradeoff > with longer startup time. > > btw. don't the "cacheMemoryMaps" and "useMemoryMappedBuffer" options on > IndexedShapeFileDataStore() do anything? > > > ------------------------------------------------------------------------------ > RSA(R) Conference 2012 > Mar 27 - Feb 2 > Save $400 by Jan. 27 > Register now! > http://p.sf.net/sfu/rsa-sfdev2dev2 > _______________________________________________ > Geotools-gt2-users mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/geotools-gt2-users > > > > ------------------------------------------------------------------------------ > RSA(R) Conference 2012 > Mar 27 - Feb 2 > Save $400 by Jan. 27 > Register now! > http://p.sf.net/sfu/rsa-sfdev2dev2 > _______________________________________________ > Geotools-gt2-users mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/geotools-gt2-users > > > > ------------------------------------------------------------------------------ > RSA(R) Conference 2012 > Mar 27 - Feb 2 > Save $400 by Jan. 27 > Register now! > http://p.sf.net/sfu/rsa-sfdev2dev2 > _______________________________________________ > Geotools-gt2-users mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/geotools-gt2-users > > ------------------------------------------------------------------------------ RSA(R) Conference 2012 Mar 27 - Feb 2 Save $400 by Jan. 27 Register now! http://p.sf.net/sfu/rsa-sfdev2dev2 _______________________________________________ Geotools-gt2-users mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/geotools-gt2-users
