Hi Mattias,
here is some more details:
- The function to create the graph
public void createGraphForMassiveLoad(String dbPath) {
System.out.println("Creating Neo4j Graph Database for massive load . . .
.");
Map<String, String> config = new HashMap<String, String>();
config.put("cache_type", "none");
config.put("use_memory_mapped_buffers", "true");
config.put("neostore.nodestore.db.mapped_memory", "200M");
config.put("neostore.relationshipstore.db.mapped_memory", "1000M");
config.put("neostore.propertystore.db.mapped_memory", "250M");
config.put("neostore.propertystore.db.strings.mapped_memory", "250M");
inserter = BatchInserters.inserter(dbPath, config);
indexProvider = new LuceneBatchInserterIndexProvider(inserter);
nodes = indexProvider.nodeIndex("nodes", MapUtil.stringMap("type",
"exact"));
}
- The code to load the data
public void createGraph(String datasetDir) {
System.out.println("Loading data in massive mode in Neo4j database . . .
.");
inserter.createDeferredSchemaIndex(Neo4jGraphDatabase.NODE_LABEL).on("nodeId").create();
try {
BufferedReader reader = new BufferedReader(new InputStreamReader(new
FileInputStream(datasetDir)));
String line;
int lineCounter = 1;
// Map<String, Object> properties;
// IndexHits<Long> cache;
long srcNode, dstNode;
while((line = reader.readLine()) != null) {
if(lineCounter > 4) {
String[] parts = line.split("\t");
srcNode = getOrCreate(parts[0]);
dstNode = getOrCreate(parts[1]);
inserter.createRelationship(srcNode, dstNode,
Neo4jGraphDatabase.RelTypes.SIMILAR, null);
}
lineCounter++;
}
reader.close();
}
catch (IOException e) {
e.printStackTrace();
}
nodes.flush();
}
private long getOrCreate(String value) {
Long id = cache.get(Long.valueOf(value));
if(id == null) {
Map<String, Object> properties = MapUtil.map("nodeId", value);
id = inserter.createNode(properties, Neo4jGraphDatabase.NODE_LABEL);
cache.put(Long.valueOf(value), id);
nodes.add(id, properties);
}
return id;
}
- The dataset I made the tests:
http://snap.stanford.edu/data/email-Enron.html
Unfortunately the database is not available right now, but it's pretty easy
to reproduce with the above code.
Thanks,
Sotiris
Τη Κυριακή, 16 Μαρτίου 2014 4:48:16 μ.μ. UTC+2, ο χρήστης Mattias Persson
έγραψε:
>
> Yup, it's probably a regression introduced in the recent store format
> changes. I'd love to track this and be able to reproduce it. How were the
> relationships added to the node? Distribution of types/directions and also
> in which order they were added. Could you provide detailed information
> about that, or provide the database zipped up to me directly (
> [email protected] <javascript:>) ?
>
> Thanks in advance
>
>
> 2014-03-13 12:26 GMT+01:00 Sotiris Beis <[email protected] <javascript:>>:
>
>> Ok, can you suggest me another temporary solution?
>>
>>
>> On 03/13/2014 01:24 PM, Michael Hunger wrote:
>>
>> Thanks for the feedback. Could be related to the changes in the store
>> format for heavily connected nodes.
>>
>> We'll investigate.
>>
>> Cheers,
>>
>> Michael
>>
>> ----
>> (michael <http://twitter.com/mesirii>)-[:SUPPORTS]->(*YOU*)-[:USE]->(
>> Neo4j <http://neo4j.org>)
>> Learn Online <http://neo4j.org/learn/online_course>,
>> Offline<http://www.neo4j.org/events> or
>> Read a Book <http://graphdatabases.com> (in Deutsch<http://bit.ly/das-buch>
>> )
>> We're trading T-shirts for cool Graph Models <http://bit.ly/graphgist>
>>
>>
>>
>> Am 13.03.2014 um 12:22 schrieb Sotiris Beis <[email protected]<javascript:>
>> >:
>>
>> Which version of Neo4j are you using?
>>
>> I use neo4j-2.1.0-M01
>>
>> You use a Set which elminates duplicates. You probably have duplicate
>> neighbourId's that are only 100 distinct ones.
>>
>> That was my first thought, but I cheched it. There are no dublicates.
>> Why do you think the result is different when I use this test line
>>
>>
>> System.out.println(IteratorUtil.count(n.getRelationships(Direction.OUTGOING)));
>>
>> before the iteration of the relationships?
>>
>> n 03/13/2014 01:00 PM, Michael Hunger wrote:
>>
>> Which version of Neo4j are you using?
>>
>> You use a Set which elminates duplicates. You probably have duplicate
>> neighbourId's that are only 100 distinct ones.
>>
>> And you close the transaction twice. It is an auto-closable resource so
>> you can remove your manual tx.close() line.
>>
>> Cheers,
>>
>> Michael
>>
>> ----
>> (michael <http://twitter.com/mesirii>)-[:SUPPORTS]->(*YOU*)-[:USE]->(
>> Neo4j <http://neo4j.org/>)
>> Learn Online <http://neo4j.org/learn/online_course>,
>> Offline<http://www.neo4j.org/events> or
>> Read a Book <http://graphdatabases.com/> (in Deutsch<http://bit.ly/das-buch>
>> )
>> We're trading T-shirts for cool Graph Models <http://bit.ly/graphgist>
>>
>>
>>
>>
>>
>>
>>
>> Am 13.03.2014 um 11:21 schrieb Sotiris Beis <[email protected]<javascript:>
>> >:
>>
>> I have the following case study. I want to get the neighbors of the
>> certain node. My function is this one:
>> public Set<Integer> getNeighborsIds(int nodeId) {
>> Set<Integer> neighbours = new HashSet<Integer>();
>> try (Transaction tx = neo4jGraph.beginTx()) {
>> Node n = nodeIndex.get("nodeId", nodeId).getSingle();
>> for(Relationship relationship : n.getRelationships(Direction.OUTGOING))
>> {
>> Node neighbour = relationship.getEndNode();
>> String neighbourId = (String)neighbour.getProperty("nodeId");
>> neighbours.add(Integer.valueOf(neighbourId));
>> }
>> tx.success();
>> tx.close();
>> }
>> return neighbours;
>> }
>>
>> I know that this node has 255 neighbors but this function return only 100
>> nodes. When I put this line
>>
>> System.out.println(IteratorUtil.count(n.getRelationships(Direction.OUTGOING)));
>>
>>
>> 4 and 5 line the function returns 255 neighbors. The most strange is
>> that the above function prints out the value 100. Is this a bug or can
>> anyone explain this to me?
>>
>> Thanks,
>> Sotiris
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Neo4j" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to [email protected] <javascript:>.
>> For more options, visit https://groups.google.com/d/optout.
>>
>>
>> --
>> You received this message because you are subscribed to a topic in the
>> Google Groups "Neo4j" group.
>> To unsubscribe from this topic, visit
>> https://groups.google.com/d/topic/neo4j/stHamJpQSBk/unsubscribe.
>> To unsubscribe from this group and all its topics, send an email to
>> [email protected] <javascript:>.
>> For more options, visit https://groups.google.com/d/optout.
>>
>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Neo4j" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to [email protected] <javascript:>.
>> For more options, visit https://groups.google.com/d/optout.
>>
>>
>> --
>> You received this message because you are subscribed to a topic in the
>> Google Groups "Neo4j" group.
>> To unsubscribe from this topic, visit
>> https://groups.google.com/d/topic/neo4j/stHamJpQSBk/unsubscribe.
>> To unsubscribe from this group and all its topics, send an email to
>> [email protected] <javascript:>.
>> For more options, visit https://groups.google.com/d/optout.
>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Neo4j" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to [email protected] <javascript:>.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
>
> --
> Mattias Persson, [[email protected] <javascript:>]
> Hacker, Neo Technology
> www.neotechnology.com
>
--
You received this message because you are subscribed to the Google Groups
"Neo4j" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/d/optout.
