Re: [Neo4j] OSMImporter: Is there a way to do incremental imports?
thanks for the response(s)! The hardware I'm testing on is not the best and only 4G of ram so I'm limited, but this seems the best opportunity for me to learn this...that being said... For incremental imports, stitching osm files together, we re-activate the old code that tests the lucene index before adding nodes and relations. There might be some subtle edge cases to consider, but a set of tests with overlapping and non-overlapping osm files should flush them out. I'd love to play with this. Is the old code there for me to re-enable in testing? Or can you point me to where this might be put in? Thx, Greg -- View this message in context: http://neo4j-community-discussions.438527.n3.nabble.com/OSMImporter-Is-there-a-way-to-do-incremental-imports-tp3526941p3527995.html Sent from the Neo4j Community Discussions mailing list archive at Nabble.com. ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
[Neo4j] OSMImporter: Is there a way to do incremental imports?
I've been playing with OSMImporter; tried batch and native java. I've had mixed success trying to import the planet, but since it's of considerable size, the job usually blows up or grinds to a halt about half way. I think the most I've made it to is 651M nodes and that's not even the ways or relations. I just don't know enough about it and thought I would ask before I try to dive in to it, but what would I have to do to so that I could restart the job ( where it left off ) when it blows? -- View this message in context: http://neo4j-community-discussions.438527.n3.nabble.com/OSMImporter-Is-there-a-way-to-do-incremental-imports-tp3526941p3526941.html Sent from the Neo4j Community Discussions mailing list archive at Nabble.com. ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] osm_import.rb
I ended up trying again with just java (but still running with batchInserter), adjusting my memory settings and max heap, it's currently working on the americas.osm file from cloudmade - http://downloads.cloudmade.com/americas#downloads_breadcrumbs. The file is about 99 GB when assembled. I'm running on ubuntu 11.10 Core 2 Duo 2.Ghz with 4G ram (not very fast, but what I have available right now), Java Heap -- -Xmx=3072M config settings: neostore.nodestore.db.mapped_memory=1000M neostore.relationshipstore.db.mapped_memory=300M neostore.propertystore.db.mapped_memory=400M neostore.propertystore.db.strings.mapped_memory=800M neostore.propertystore.db.arrays.mapped_memory=100M My code is essentially from the test suite that you suggested but I am using the batchImporter instead. I'm about 1/3 of the way through and don't want to interrupt the process, but when it's done I'll try it without the batch importer. It runs at about 4500 nodes/second. Is that reasonable? I haven't looked at performance numbers from anyone else. Would the non batch performance be better? Is is better to 'includePoints' or not? One questions I had was, once I get this imported via this method ( neo4j embedded ), is it possible to move the imported db to a neo4j server? I'm hoping it is. If so, what would that process be? -- View this message in context: http://neo4j-community-discussions.438527.n3.nabble.com/osm-import-rb-tp3493463p3496760.html Sent from the Neo4j Community Discussions mailing list archive at Nabble.com. ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
[Neo4j] osm_import.rb
I'm new to neo4j, and neo4j-spatial. I've been able to import a small OSM file using the java OSMImporter class and BatchInserter (based on the mvn test code) but this blows up out-of-memory on larger files. In looking for other methods I came across instructions for using jruby. I installed jruby and neo4j-spatial. I tried to import an OSM file using jruby using these instructions: git clone g...@github.com:craigtaverner/neo4j-spatial.rb.git cd neo4j-spatial.rb/examples jruby osm_import.rb map2.osm ... but when I got to the 1st import statement I got an error back: Error opening script file: /home/gregm/Downloads/neo4j-spatial.rb/examples/osm_import.rb (No such file or directory) I can find an 'osm_import' in the bin directory of this repository but I'm not sure how to use it. I'm new to jruby ( ruby as well ) and may not have installed correctly. I believe I did this: sudo jruby -S gem install neo4j-spatial and it seemed to install. If I run this command ( from the upper directory in neo4j-spatial.rb ): jruby bin/osm_import examples/map2.osm I get this: /usr/lib/jruby//lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `require': no such file to load -- neo4j/spatial/jars/neo4j-spatial-0.6-SNAPSHOT (LoadError) from /usr/lib/jruby//lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `require' It looks, to me, like the code has changed but the instructions haven't, but I'm at a loss for how to resolve this. I'm just trying to import a larger OSM file. Any ideas? -- View this message in context: http://neo4j-community-discussions.438527.n3.nabble.com/osm-import-rb-tp3493463p3493463.html Sent from the Neo4j Community Discussions mailing list archive at Nabble.com. ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
[Neo4j] version conflicts
Hi, I'm just getting started with Neo4j, spatial, gremlin and Neoclipse. I can successfully run (via eclipse) an import of 'map.osm' file using OSMImporter by linking to libraries that come with the spatial distribution. When I try to use gremlin or neoclipse to attach to the data I get a db version error. I'm wondering if there is a repository that I can download from ( or build ) where all these components are in sync. Or maybe someone has a suggestion for how I can proceed. both gremlin and neoclipse report a version problem, here is gremlins output: Caused by: org.neo4j.kernel.impl.nioneo.store.IllegalStoreVersionException: Store version [NeoStore v0.A.0]. Please make sure you are not running old Neo4j kernel on a store that has been created by newer version of Neo4j. -- View this message in context: http://neo4j-community-discussions.438527.n3.nabble.com/version-conflicts-tp3488028p3488028.html Sent from the Neo4j Community Discussions mailing list archive at Nabble.com. ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] version conflicts
i managed to get this working by taking the neo4j- jars from my gremlin installation and use those in the dependency folder of spatial, but I'm still not sure in what way I can fix the dependencies. It seems like the spatial distribution is has a newer kernel, though they both seem to be 1.5 -- View this message in context: http://neo4j-community-discussions.438527.n3.nabble.com/version-conflicts-tp3488028p3488122.html Sent from the Neo4j Community Discussions mailing list archive at Nabble.com. ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
