So basically I can:
create nodes and relationships by using the normal Neo4j batchInserters;
then, once i finished to create nodes and relationships i need to add to
the spatial index what i want (basically all the created nodes); did I get
the point?
In this case....I'm wondering how much it will be perfomant by considering
1 million of nodes and iterate on them....Btw I'll try and I'll let you know
Thank you
Angelo
Il giorno giovedì 1 maggio 2014 10:44:10 UTC+2, Angelo Immediata ha scritto:
>
> Hi there
> This is my scenario: we are building a routing system by using neo4j and
> the spatial plugin. We start from the OSM file and we read this file and
> import nodes and relationships in our graph (a custom graph model)
> Now, if we don't use the batch inserter of neo4j, in order to import a
> compressed OSM file (with compressed dimension of around 140MB, and normal
> dimensions around 2GB) it takes around 3 days on a dedicated server with
> the following characteristics: CentOS 6.5 64bit, quad core, 8GB RAM; pease
> note that the most time is related to the Neo4J Nodes and relationships
> creation; in-fact if we read the same file without doing anything with
> neo4j, the file is read in around 7 minutes (i'm sure about this becouse in
> our process we first read the file in order to store the correct osm nodes
> ids and then we read again the file in order to create the neo4j graph)
>
> Obviously we need to improve the import proces so we are trying to use the
> batchInserter. So far, so good (I need to check how much it will perform by
> using the batchInserter but I guess it will be faster); so the first thing
> I did was: let's try to use the batch inserter in a simple test case (very
> similar to our code, but without modifying our code directly)
>
> I list my software versions:
>
> - Neo4j: 2.0.2
> - Neo4jSpatial: 0.13-neo4j-2.0.1
> - Neo4jGraphCollections: 0.7.1-neo4j-2.0.1
> - Osmosis: 0.43.1
>
> Since I'm using osmosis in order to read the osm file, I wrote the
> following Sink implementation:
>
> package it.graph.batch.test;
>
>
> import java.io.File;
> import java.util.HashMap;
> import java.util.Map;
>
>
> import org.neo4j.gis.spatial.SimplePointLayer;
> import org.neo4j.gis.spatial.SpatialDatabaseService;
> import org.neo4j.graphdb.GraphDatabaseService;
> import org.neo4j.unsafe.batchinsert.BatchInserter;
> import org.neo4j.unsafe.batchinsert.BatchInserters;
> import org.neo4j.unsafe.batchinsert.SpatialBatchGraphDatabaseService;
> import org.openstreetmap.osmosis.core.container.v0_6.EntityContainer;
> import org.openstreetmap.osmosis.core.domain.v0_6.Entity;
> import org.openstreetmap.osmosis.core.domain.v0_6.Node;
> import org.openstreetmap.osmosis.core.domain.v0_6.Relation;
> import org.openstreetmap.osmosis.core.domain.v0_6.Way;
> import org.openstreetmap.osmosis.core.task.v0_6.Sink;
>
>
>
>
> public class BatchInserterSinkTest implements Sink
> {
> public static final Map<String, String> NEO4J_CFG = new HashMap<String,
> String>();
> private static File basePath = new File("/home/angelo/Scrivania/neo4j");
> private static File dbPath = new File(basePath, "db");
> private GraphDatabaseService graphDb;
> private BatchInserter batchInserter;
> // private BatchInserterIndexProvider batchIndexService;
> private SpatialDatabaseService spatialDb;
> private SimplePointLayer spl;
> static
> {
> NEO4J_CFG.put( "neostore.nodestore.db.mapped_memory", "100M" );
> NEO4J_CFG.put( "neostore.relationshipstore.db.mapped_memory",
> "300M" );
> NEO4J_CFG.put( "neostore.propertystore.db.mapped_memory", "400M"
> );
> NEO4J_CFG.put( "neostore.propertystore.db.strings.mapped_memory",
> "800M" );
> NEO4J_CFG.put( "neostore.propertystore.db.arrays.mapped_memory",
> "10M" );
> NEO4J_CFG.put( "dump_configuration", "true" );
> }
> @Override
> public void initialize(Map<String, Object> arg0)
> {
> batchInserter = BatchInserters.inserter(dbPath.getAbsolutePath(),NEO4J_CFG
> );
> graphDb = new SpatialBatchGraphDatabaseService(batchInserter);
> spatialDb = new SpatialDatabaseService(graphDb);
> spl = spatialDb.createSimplePointLayer("testBatch", "latitudine",
> "longitudine");
> //batchIndexService = new
> LuceneBatchInserterIndexProvider(batchInserter);
>
> }
>
>
> @Override
> public void complete()
> {
> // TODO Auto-generated method stub
>
>
> }
>
>
> @Override
> public void release()
> {
> // TODO Auto-generated method stub
>
>
> }
>
>
> @Override
> public void process(EntityContainer ec)
> {
> Entity entity = ec.getEntity();
> if (entity instanceof Node) {
>
> Node osmNodo = (Node)entity;
> org.neo4j.graphdb.Node graphNode = graphDb.createNode();
> graphNode.setProperty("osmId", osmNodo.getId());
> graphNode.setProperty("latitudine", osmNodo.getLatitude());
> graphNode.setProperty("longitudine", osmNodo.getLongitude());
> spl.add(graphNode);
>
> } else if (entity <span style="colo
> ...
--
You received this message because you are subscribed to the Google Groups
"Neo4j" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/d/optout.
