Re: [Neo4j] OSMImporter: Is there a way to do incremental imports?
There was only a method ending in 'WithCheck', or something like that, lying unused in the code from last year. Nothing more than that. Except for thinking about it, which is why I wrote the previous mail. On Dec 2, 2011 12:50 PM, Peter Neubauer pe...@neubauer.se wrote: Not sure, Craig, do you have the code somewhere? /peter On Tue, Nov 22, 2011 at 4:17 PM, grimace macegh...@gmail.com wrote: thanks for the response(s)! The hardware I'm testing on is not the best and only 4G of ram so I'm limited, but this seems the best opportunity for me to learn this...that being said... For incremental imports, stitching osm files together, we re-activate the old code that tests the lucene index before adding nodes and relations. There might be some subtle edge cases to consider, but a set of tests with overlapping and non-overlapping osm files should flush them out. I'd love to play with this. Is the old code there for me to re-enable in testing? Or can you point me to where this might be put in? Thx, Greg -- View this message in context: http://neo4j-community-discussions.438527.n3.nabble.com/OSMImporter-Is-there-a-way-to-do-incremental-imports-tp3526941p3527995.html Sent from the Neo4j Community Discussions mailing list archive at Nabble.com. ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] OSMImporter: Is there a way to do incremental imports?
Not sure, Craig, do you have the code somewhere? /peter On Tue, Nov 22, 2011 at 4:17 PM, grimace macegh...@gmail.com wrote: thanks for the response(s)! The hardware I'm testing on is not the best and only 4G of ram so I'm limited, but this seems the best opportunity for me to learn this...that being said... For incremental imports, stitching osm files together, we re-activate the old code that tests the lucene index before adding nodes and relations. There might be some subtle edge cases to consider, but a set of tests with overlapping and non-overlapping osm files should flush them out. I'd love to play with this. Is the old code there for me to re-enable in testing? Or can you point me to where this might be put in? Thx, Greg -- View this message in context: http://neo4j-community-discussions.438527.n3.nabble.com/OSMImporter-Is-there-a-way-to-do-incremental-imports-tp3526941p3527995.html Sent from the Neo4j Community Discussions mailing list archive at Nabble.com. ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] OSMImporter: Is there a way to do incremental imports?
Gregory, incremental loads (and thus, restarts of OSM imports) are a feature we want to add later on, but it's not in there yet. This would also mean we could stitch in other areas on demand, and support submitting changesets back to OSM or at least capture them, so you as an OSM based app can contribute to OSM automagically. I know it's much to ask, but help here would be greatly appreciated. I hope to lab with Michael Hunger on import of data into OSM (and others) this Friday and hope to get somewhere :) Cheers, /peter neubauer GTalk: neubauer.peter Skype peter.neubauer Phone +46 704 106975 LinkedIn http://www.linkedin.com/in/neubauer Twitter http://twitter.com/peterneubauer http://www.neo4j.org - NOSQL for the Enterprise. http://startupbootcamp.org/ - Öresund - Innovation happens HERE. On Tue, Nov 22, 2011 at 7:15 AM, grimace macegh...@gmail.com wrote: I've been playing with OSMImporter; tried batch and native java. I've had mixed success trying to import the planet, but since it's of considerable size, the job usually blows up or grinds to a halt about half way. I think the most I've made it to is 651M nodes and that's not even the ways or relations. I just don't know enough about it and thought I would ask before I try to dive in to it, but what would I have to do to so that I could restart the job ( where it left off ) when it blows? -- View this message in context: http://neo4j-community-discussions.438527.n3.nabble.com/OSMImporter-Is-there-a-way-to-do-incremental-imports-tp3526941p3526941.html Sent from the Neo4j Community Discussions mailing list archive at Nabble.com. ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] OSMImporter: Is there a way to do incremental imports?
I did some initial work on incremental imports back in 2010, but stopped due to some complications: - We needed to mix lucene reads and writes during the import (read to check if the node already exists, so we don't import twice) and this performs very badly in the batch inserter. We decided to first code a non-batch insert mode before re-starting the incremental import work. Now Peter and I did code a non-batch importer in early 2011, but never went back to complete the incremental import. - We wanted to support both the case of importing multiple OSM files that could be stitched together by resolving overlaps, as well as the case of applying changesets to the existing OSM model. This increased the complexity of the work just enough to ensure it got dropped. In early 2011 we also added support to changesets in the model (but only as a data structure, not in terms of importing changesets). So we are one step closer to this also. Since we now have non-batch importing, and changeset data structures, the opportunity to re-start the incremental import and importing changesets is there. It should not be too hard. For incremental imports, stitching osm files together, we re-activate the old code that tests the lucene index before adding nodes and relations. There might be some subtle edge cases to consider, but a set of tests with overlapping and non-overlapping osm files should flush them out. For applying changesets, more thinking is still required. Do we want to support history in the model, or only the latest version? Should we verify that only newer changesets are applied and in the right order, or rely on the user to get it right? I can say that we did some thinking this summer on the data structures required to support a complete change history. This relies on the fact that we already support multiple possible ways on the same nodes, so we can also, in principle, support multiple possible 'versions' of ways on the same nodes. More thinking is required, but I have a suspicion that we should actually go ahead and do this properly will full history, because that might be the only way to make sure the user never messes things up by importing in the wrong order. On Tue, Nov 22, 2011 at 9:58 AM, Peter Neubauer peter.neuba...@neotechnology.com wrote: Gregory, incremental loads (and thus, restarts of OSM imports) are a feature we want to add later on, but it's not in there yet. This would also mean we could stitch in other areas on demand, and support submitting changesets back to OSM or at least capture them, so you as an OSM based app can contribute to OSM automagically. I know it's much to ask, but help here would be greatly appreciated. I hope to lab with Michael Hunger on import of data into OSM (and others) this Friday and hope to get somewhere :) Cheers, /peter neubauer GTalk: neubauer.peter Skype peter.neubauer Phone +46 704 106975 LinkedIn http://www.linkedin.com/in/neubauer Twitter http://twitter.com/peterneubauer http://www.neo4j.org - NOSQL for the Enterprise. http://startupbootcamp.org/- Öresund - Innovation happens HERE. On Tue, Nov 22, 2011 at 7:15 AM, grimace macegh...@gmail.com wrote: I've been playing with OSMImporter; tried batch and native java. I've had mixed success trying to import the planet, but since it's of considerable size, the job usually blows up or grinds to a halt about half way. I think the most I've made it to is 651M nodes and that's not even the ways or relations. I just don't know enough about it and thought I would ask before I try to dive in to it, but what would I have to do to so that I could restart the job ( where it left off ) when it blows? -- View this message in context: http://neo4j-community-discussions.438527.n3.nabble.com/OSMImporter-Is-there-a-way-to-do-incremental-imports-tp3526941p3526941.html Sent from the Neo4j Community Discussions mailing list archive at Nabble.com. ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] OSMImporter: Is there a way to do incremental imports?
thanks for the response(s)! The hardware I'm testing on is not the best and only 4G of ram so I'm limited, but this seems the best opportunity for me to learn this...that being said... For incremental imports, stitching osm files together, we re-activate the old code that tests the lucene index before adding nodes and relations. There might be some subtle edge cases to consider, but a set of tests with overlapping and non-overlapping osm files should flush them out. I'd love to play with this. Is the old code there for me to re-enable in testing? Or can you point me to where this might be put in? Thx, Greg -- View this message in context: http://neo4j-community-discussions.438527.n3.nabble.com/OSMImporter-Is-there-a-way-to-do-incremental-imports-tp3526941p3527995.html Sent from the Neo4j Community Discussions mailing list archive at Nabble.com. ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
[Neo4j] OSMImporter: Is there a way to do incremental imports?
I've been playing with OSMImporter; tried batch and native java. I've had mixed success trying to import the planet, but since it's of considerable size, the job usually blows up or grinds to a halt about half way. I think the most I've made it to is 651M nodes and that's not even the ways or relations. I just don't know enough about it and thought I would ask before I try to dive in to it, but what would I have to do to so that I could restart the job ( where it left off ) when it blows? -- View this message in context: http://neo4j-community-discussions.438527.n3.nabble.com/OSMImporter-Is-there-a-way-to-do-incremental-imports-tp3526941p3526941.html Sent from the Neo4j Community Discussions mailing list archive at Nabble.com. ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
