Well the data is there. I does not matter what is in there. And no I have no index or anything, just testing the row mass insertion. Ussing a good SSD. Well the RAM could be a reason, will test this soon.
Am Freitag, 29. August 2014 23:57:12 UTC+2 schrieb Michael Hunger: > > Cypher won't be faster than the Java API, only the batch-inserter is > faster than the java API > > Cypher is just more convenient. > > Do you have any long property strings? > Most of the time is spent in the label store and in the index updates, if > you run the same without labels and indexes it will be faster (but again > less convenient to use). > > If you only create nodes you can increase the periodic commit limit to 50k > or 100k > > What kind of disk do you use? SSD? > > How much RAM does your shell have? Do you run it against server or > standalone against a store-directory ? > > > Michael > > Am 29.08.2014 um 23:50 schrieb 'Curtis Mosters' via Neo4j < > [email protected] <javascript:>>: > > neo4j-sh (?)$ USING PERIODIC COMMIT 10000 > > LOAD CSV WITH HEADERS FROM "file:///C:/test/tls206_part01.txt" AS > csvLine WITH > csvLine LIMIT 100000 > > CREATE (p:Person { person_id: toInt(csvLine.person_id), doc_std_name_id: > csvLi > ne.doc_std_name_id , person_name: csvLine.person_name }); > +-------------------+ > | No data returned. | > +-------------------+ > Nodes created: 100000 > Properties set: 300000 > Labels added: 100000 > 16969 ms > > Well too slow, the same result I'm getting with the Java API =/ > > Am Freitag, 29. August 2014 10:14:09 UTC+2 schrieb Curtis Mosters: >> >> Ohh didn't know that this was a command. Sorry. Could you maybe let it be >> code next time. So it's much easier too see. Will test it this evening. >> Thanks. >> >> Am Donnerstag, 28. August 2014 22:15:55 UTC+2 schrieb Michael Hunger: >>> >>> did you see the _WITH csvLine_ before the limit? >>> >>> Am 28.08.2014 um 19:03 schrieb 'Curtis Mosters' via Neo4j < >>> [email protected]>: >>> >>> C:\Program Files\Neo4j Community\neo4j-community-2.1.3\bin>Neo4jShell.bat >>> start >>> Welcome to the Neo4j Shell! Enter 'help' for a list of commands >>> NOTE: Remote Neo4j graph database service 'shell' at port 1337 >>> >>> neo4j-sh (?)$ USING PERIODIC COMMIT 10000 >>> > LOAD CSV WITH HEADERS FROM "file:///C:/test/tls206_part01.txt" AS >>> csvLine LIMI >>> T 10 >>> > CREATE (p:Person { person_id: toInt(csvLine.person_id), >>> doc_std_name_id: csvLi >>> ne.doc_std_name_id , person_name: csvLine.person_name }); >>> SyntaxException: Invalid input 'I': expected 'o/O' (line 2, column 76) >>> "LOAD CSV WITH HEADERS FROM "file:///C:/test/tls206_part01.txt" AS >>> csvLine LIMIT >>> 10" >>> >>> ^ >>> >>> >>> >>> Am Donnerstag, 28. August 2014 18:37:14 UTC+2 schrieb Michael Hunger: >>>> >>>> What happen in your shell? >>>> Do you have the index? >>>> >>>> can you try >>>> >>>> USING PERIODIC COMMIT 10000 >>>>>>> LOAD CSV WITH HEADERS FROM "file:C:/test/tls206_part01.txt" AS >>>>>>> csvLine >>>>>>> >>>>>>> WITH csvLine LIMIT 10 >>>> >>>> CREATE (p:Person { person_id: toInt(csvLine.person_id), doc_std_name_id >>>>>>> : csvLine.doc_std_name_id , person_name: csvLine.person_name }); >>>>>>> >>>>>>> >>>>>>> >>>> Am 28.08.2014 um 18:14 schrieb 'Curtis Mosters' via Neo4j <ne...@ >>>> googlegroups.com>: >>>> >>>> Already tried this out. Does not work in my Shell. >>>> >>>> Am Donnerstag, 28. August 2014 17:50:24 UTC+2 schrieb david fauth: >>>>> >>>>> No semi-colon after the first line: >>>>> >>>>> >>>>> USING PERIODIC COMMIT 10000 >>>>> LOAD CSV WITH HEADERS FROM "file:C:/test/tls206_part01.txt" AS csvLine >>>>> CREATE (p:Person { person_id: toInt(csvLine.person_id), >>>>> doc_std_name_id: csvLine.doc_std_name_id , person_name: csvLine. >>>>> person_name}); >>>>> >>>>> On Thursday, August 28, 2014 10:33:27 AM UTC-4, Curtis Mosters wrote: >>>>>> >>>>>> As you see in all lines above there is actually already a comma? >>>>>> >>>>>> What do you mean with comma after CREATE, Michael? >>>>>> >>>>>> Am Donnerstag, 28. August 2014 16:11:13 UTC+2 schrieb Michael Hunger: >>>>>>> >>>>>>> Nope no semicolon after 10000 >>>>>>> >>>>>>> Only after the create >>>>>>> >>>>>>> Sent from mobile device >>>>>>> >>>>>>> Am 28.08.2014 um 16:01 schrieb "'Curtis Mosters' via Neo4j" <ne...@ >>>>>>> googlegroups.com>: >>>>>>> >>>>>>> That was it. Now I have: >>>>>>> >>>>>>> USING PERIODIC COMMIT 10000; >>>>>>> LOAD CSV WITH HEADERS FROM "file:C:/test/tls206_part01.txt" AS >>>>>>> csvLine >>>>>>> CREATE (p:Person { person_id: toInt(csvLine.person_id), >>>>>>> doc_std_name_id: csvLine.doc_std_name_id , person_name: csvLine. >>>>>>> person_name }); >>>>>>> >>>>>>> Well it seems to do something. My CPU is at 100%. But looking into >>>>>>> the graph folder. There isn't happening anything. >>>>>>> >>>>>>> What's going on? >>>>>>> >>>>>>> Btw here the state: >>>>>>> >>>>>>> neo4j-sh (?)$ USING PERIODIC COMMIT 10000; >>>>>>> SyntaxException: Unexpected end of input: expected an identifier >>>>>>> character, whit >>>>>>> espace or LoadCSVQuery (line 1, column 28) >>>>>>> "USING PERIODIC COMMIT 10000" >>>>>>> ^ >>>>>>> neo4j-sh (?)$ LOAD CSV WITH HEADERS FROM >>>>>>> "file:C:/test/tls206_part01.txt" AS csv >>>>>>> Line >>>>>>> > CREATE (p:Person { person_id: toInt(csvLine.person_id), >>>>>>> doc_std_name_id: csvLi >>>>>>> ne.doc_std_name_id , person_name: csvLine.person_name }); >>>>>>> >>>>>>> >>>>>>> >>>>>>> Am Donnerstag, 28. August 2014 15:37:44 UTC+2 schrieb Rik Van >>>>>>> Bruggen: >>>>>>>> >>>>>>>> Did you forget a semi-colon at the end? >>>>>>>> >>>>>>>> In the shell all commands need to end with ; ... >>>>>>>> >>>>>>>> Rik >>>>>>>> >>>>>>>> On Thursday, August 28, 2014 2:04:22 PM UTC+1, Curtis Mosters wrote: >>>>>>>>> >>>>>>>>> Well it works under Browser Cypher. So it should be fin. But I >>>>>>>>> changed it: >>>>>>>>> >>>>>>>>> USING PERIODIC COMMIT 10000 >>>>>>>>> LOAD CSV WITH HEADERS FROM "file:///C:/test/tls206_part01.txt" AS >>>>>>>>> csvLine >>>>>>>>> CREATE (p:Person { person_id: toInt(csvLine.person_id), >>>>>>>>> doc_std_name_id: csvLine.doc_std_name_id , person_name: csvLine. >>>>>>>>> person_name }) >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> And still nothing happening. Another idea? >>>>>>>>> >>>>>>>>> Am Donnerstag, 28. August 2014 14:37:15 UTC+2 schrieb Aru Sahni: >>>>>>>>>> >>>>>>>>>> Your file URI seems to be malformed. Here's the correct syntax >>>>>>>>>> for Windows: >>>>>>>>>> http://blogs.msdn.com/b/ie/archive/2006/12/06/file-uris-in-windows.aspx >>>>>>>>>> >>>>>>>>>> ~A >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Thu, Aug 28, 2014 at 8:34 AM, 'Curtis Mosters' via Neo4j < >>>>>>>>>> [email protected]> wrote: >>>>>>>>>> >>>>>>>>>>> Well I will definitily test this. But I don't know how to start >>>>>>>>>>> this command with the shell: >>>>>>>>>>> >>>>>>>>>>> http://i.imgur.com/uiYbq0O.png >>>>>>>>>>> >>>>>>>>>>> Any tips for windows user? Nothing happening. >>>>>>>>>>> >>>>>>>>>>> Am Donnerstag, 28. August 2014 11:39:59 UTC+2 schrieb Michael >>>>>>>>>>> Hunger: >>>>>>>>>>>> >>>>>>>>>>>> Hi Curtis, >>>>>>>>>>>> >>>>>>>>>>>> if you do this: >>>>>>>>>>>> >>>>>>>>>>>> USING PERIODIC COMMIT 10000 >>>>>>>>>>>> LOAD CSV WITH HEADERS FROM "file:C:/test.txt" AS csvLine >>>>>>>>>>>> CREATE (:Person { person_id: toInt(csvLine.person_id), name: >>>>>>>>>>>> csvLine.name }); >>>>>>>>>>>> >>>>>>>>>>>> It should do between 10k-30k nodes per second. >>>>>>>>>>>> It will be slower if you have a unique constraint in place. >>>>>>>>>>>> >>>>>>>>>>>> Please run it in the Neo4j-Shell, it is much easier to handle >>>>>>>>>>>> there. >>>>>>>>>>>> >>>>>>>>>>>> As you are running on windows, with 4GB in total, please also >>>>>>>>>>>> make sure that your mmio config in neo4j.properties is not too >>>>>>>>>>>> large. >>>>>>>>>>>> >>>>>>>>>>>> Perhaps 100MB for nodes and 500MB for rels and another 250MB >>>>>>>>>>>> for properties. >>>>>>>>>>>> >>>>>>>>>>>> On windows the memory mapping memory is taken from the heap. >>>>>>>>>>>> >>>>>>>>>>>> You should see some info about that in in your >>>>>>>>>>>> graph.db/messages.log files after the startup, feel free to share >>>>>>>>>>>> that file >>>>>>>>>>>> with us then we can help you to set up your config. >>>>>>>>>>>> >>>>>>>>>>>> Michael >>>>>>>>>>>> >>>>>>>>>>>> Am 28.08.2014 um 11:11 schrieb 'Curtis Mosters' via Neo4j < >>>>>>>>>>>> [email protected]>: >>>>>>>>>>>> >>>>>>>>>>>> Yesterday evening I was running it 52 minutes. And then I got >>>>>>>>>>>> an "Unknown Error". >>>>>>>>>>>> >>>>>>>>>>>> So now I tested it now with the "USING PERIODIC COMMIT 10000" >>>>>>>>>>>> and now its about 60 minutes. Then the same error. >>>>>>>>>>>> >>>>>>>>>>>> I now looked into the graph.db folder and its overall 1,75 GB >>>>>>>>>>>> big. The propertystore file ist 370 mb e.g. >>>>>>>>>>>> >>>>>>>>>>>> So what else can I do do get it running in the browser? Or >>>>>>>>>>>> could I run this task in the Neo4jShell? >>>>>>>>>>>> >>>>>>>>>>>> Am Donnerstag, 28. August 2014 00:16:07 UTC+2 schrieb Chris >>>>>>>>>>>> Vest: >>>>>>>>>>>>> >>>>>>>>>>>>> All transaction state is currently kept in memory on the java >>>>>>>>>>>>> heap, and 20+ mio. changes is too much to fit in a 4 GB heap. >>>>>>>>>>>>> When you have too much stuff on the heap, it will manifest >>>>>>>>>>>>> with those “GC overhead limit exceeded” and the database will run >>>>>>>>>>>>> slow, >>>>>>>>>>>>> though there are other things that can produce similar symptoms. >>>>>>>>>>>>> >>>>>>>>>>>>> Try putting USING PERIODIC COMMIT 10000 in front of your LOAD >>>>>>>>>>>>> CSV query. This will periodically commit the transaction, >>>>>>>>>>>>> thus limiting the transaction state kept in memory. Unfortunately >>>>>>>>>>>>> it will >>>>>>>>>>>>> also break the atomicity of the transaction. >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> Chris Vest >>>>>>>>>>>>> System Engineer, Neo Technology >>>>>>>>>>>>> [ skype: mr.chrisvest, twitter: chvest ] >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On 27 Aug 2014, at 22:31, 'Curtis Mosters' via Neo4j < >>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> Let's say I have: >>>>>>>>>>>>> >>>>>>>>>>>>> LOAD CSV WITH HEADERS FROM "file:C:/test.txt" AS csvLine >>>>>>>>>>>>> CREATE (p:Person { person_id: toInt(csvLine.person_id), name: >>>>>>>>>>>>> csvLine.name }) >>>>>>>>>>>>> >>>>>>>>>>>>> I run this query in the browser. I know that it's not the >>>>>>>>>>>>> fastest way and I should think about using the batch importer. >>>>>>>>>>>>> But I really >>>>>>>>>>>>> like that way somehow and want to speed it up. >>>>>>>>>>>>> >>>>>>>>>>>>> So when I ran this the first time, after like 2 or 3 minutes I >>>>>>>>>>>>> got an erro saying "GC overhead limit exceeded". So It set >>>>>>>>>>>>> >>>>>>>>>>>>> wrapper.java.initmemory=4096 >>>>>>>>>>>>> wrapper.java.maxmemory=4096 >>>>>>>>>>>>> >>>>>>>>>>>>> Now the error does not come up. But it's still slow and I >>>>>>>>>>>>> can't see how much time is still needed. So if you have tips on >>>>>>>>>>>>> doing this, >>>>>>>>>>>>> I would be very thankful. =) >>>>>>>>>>>>> >>>>>>>>>>>>> PS: the file is 2 gb big and has like 20 mio entries >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>>>>> Google Groups "Neo4j" group. >>>>>>>>>>>>> To unsubscribe from this group and stop receiving emails from >>>>>>>>>>>>> it, send an email to [email protected]. >>>>>>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>>>> Google Groups "Neo4j" group. >>>>>>>>>>>> To unsubscribe from this group and stop receiving emails from >>>>>>>>>>>> it, send an email to [email protected]. >>>>>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>>> Google Groups "Neo4j" group. >>>>>>>>>>> To unsubscribe from this group and stop receiving emails from >>>>>>>>>>> it, send an email to [email protected]. >>>>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "Neo4j" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>> send an email to [email protected]. >>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>> >>>>>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Neo4j" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>>> >>>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Neo4j" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> For more options, visit https://groups.google.com/d/optout. >>> >>> >>> > -- > You received this message because you are subscribed to the Google Groups > "Neo4j" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] <javascript:>. > For more options, visit https://groups.google.com/d/optout. > > > -- You received this message because you are subscribed to the Google Groups "Neo4j" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
