That's VERY cool, great work, especially regarding Gremlin and Batching! Now, if there was any good docs we could pull out, I would like to at least point to it form the manual, otherwise - what are the requirements to build this in a CI system?
Cheers, /peter neubauer GTalk: neubauer.peter Skype peter.neubauer Phone +46 704 106975 LinkedIn http://www.linkedin.com/in/neubauer Twitter http://twitter.com/peterneubauer http://www.neo4j.org - Your high performance graph database. http://startupbootcamp.org/ - Öresund - Innovation happens HERE. http://www.thoughtmade.com - Scandinavia's coolest Bring-a-Thing party. On Mon, Sep 26, 2011 at 11:47 PM, Tatham Oddie <[email protected]> wrote: > Some more usage examples: > > http://hg.readify.net/neo4jclient/src/tip/Test/ApiUsageIdeas.cs > > > -- Tatham > > > -----Original Message----- > From: Tatham Oddie > Sent: Tuesday, 27 September 2011 7:41 AM > To: 'Peter Neubauer'; Neo4j user discussions > Subject: .NET client for neo4j > > Hi Peter, > > We publish our client on NuGet: http://nuget.org/List/Packages/Neo4jClient > > (NuGet is .NET's package manager.) > > The source is at http://hg.readify.net/neo4jclient > > It supports: > > * all the basic CRUD operations > > * index management > > * Gremlin queries via a fluent interface: you use C# lambdas and get full > code completion in the IDE as you type; we only compile it down to Gremlin > and send it over the wire when you enumerate it or call a scalar method like > .Count(). See http://dl.dropbox.com/u/5216304/FluentGremlin.png > > This week we moved to parametized Gremlin queries over the wire (eg, > g.v(p0).out[[label:p1]]) and saw a good performance improvement in our > downstream app. This change was implemented completely within Neo4jClient's > query processor so downstream apps just got the improvement automatically. > > We've also started optimizing some of our internal methods to use batching. > For example, creating, relating and indexing a node all happens in a single > REST call. We'll probably end up exposing the batching API outside of the > client too via some type of unit-of-work pattern. > > As for build, our CI triggers on a code push and publishes a new package to > NuGet upon a successful build. If the tests pass, it gets published. (Our > unit tests mock out all the REST calls so we have a high level of confidence > in them. Risk is low anyway, as NuGet makes it easy to grab any previous > version of the package.) > > > -- Tatham > > -----Original Message----- > From: [email protected] [mailto:[email protected]] On Behalf Of > Peter Neubauer > Sent: Tuesday, 27 September 2011 2:19 AM > To: Neo4j user discussions > Cc: Tatham Oddie > Subject: Re: [Neo4j] Turning Off or Configuring log rotation > > Very cool, > glad to hear things work out. Got any code for the .NET bindings? I think we > should start pulling client libraries into our build system, so there are > stable versions of the most used clients for Neo4j available as part of the > QA process. > > WDYT? > > Cheers, > > /peter neubauer > > GTalk: neubauer.peter > Skype peter.neubauer > Phone +46 704 106975 > LinkedIn http://www.linkedin.com/in/neubauer > Twitter http://twitter.com/peterneubauer > > http://www.neo4j.org - Your high performance graph database. > http://startupbootcamp.org/ - Öresund - Innovation happens HERE. > http://www.thoughtmade.com - Scandinavia's coolest Bring-a-Thing party. > > > > On Mon, Sep 26, 2011 at 1:24 PM, Romiko Derbynew > <[email protected]> wrote: >> Hi Peter, >> >> Indeed our desired result is now achieved. Tatham added batching to our .net >> client as well. We are more than happy to assist with HA as this is one of >> the last hurdles we need to overcome. If you can let us know in advance when >> the release is scheduled then we will incorporate it into our sprint for HA >> featuring. >> >> Much appreciated and thx for 1.5 release, could not have timed it better. >> >> Sent from my iPhone >> >> On 26/09/2011, at 5:33 PM, "Peter Neubauer" >> <[email protected]> wrote: >> >>> Romiko, >>> soudns good. Is this solving your current problems or is there stuff >>> left to take care of for the moment? Also, there will be some code >>> coming out on deploying Neo4j HA on Azure. Would you be interested in >>> trying that out and maybe take it a bit further? >>> >>> Cheers, >>> >>> /peter neubauer >>> >>> GTalk: neubauer.peter >>> Skype peter.neubauer >>> Phone +46 704 106975 >>> LinkedIn http://www.linkedin.com/in/neubauer >>> Twitter http://twitter.com/peterneubauer >>> >>> http://www.neo4j.org - Your high performance graph database. >>> http://startupbootcamp.org/ - Öresund - Innovation happens HERE. >>> http://www.thoughtmade.com - Scandinavia's coolest Bring-a-Thing party. >>> >>> >>> >>> On Mon, Sep 26, 2011 at 4:16 AM, Romiko Derbynew >>> <[email protected]> wrote: >>>> Hi Guys, >>>> >>>> We managed to fix the performance issue. >>>> >>>> * Used Paramterised Queries - increased performance by 1 extra node >>>> per second, reduced frequency usage of heap thus no jvm memory >>>> errors >>>> * Configured our .Net client to use 70 threads for rest calls (I chose >>>> this number as Neo4j reserves 80, so I decided 70) 5 TIMES INCREASE IN >>>> PERFORMANCE!!!! >>>> >>>> >>>> We are now going to implement the batching to improve performance >>>> even more :) >>>> >>>> I have updated the blog post with these findings. >>>> http://romikoderbynew.com/2011/09/23/neo4j-java-jvm-garbage-collecti >>>> on-profilingheap-sizelog-rotationlucene-rest-api-performance-analysi >>>> s/ >>>> >>>> >>>> I will write a separate post on the improvements from the batching. >>>> >>>> MyStory Welcome, Temporary from MyStory Administration. Logout? >>>> Agency Import Job 301 >>>> >>>> Log output: >>>> >>>> [01:44:58] [Thread 10] Migration message received at: 26/09/2011 >>>> 1:44:58 AM +00:00 >>>> >>>> [01:44:59] [Thread 10] Migration message payload: JobUniqueId: >>>> 301, AgencyUniqueId: 201 >>>> >>>> [01:44:59] [Thread 10] Executing sql script: >>>> e:\AzureTemp\Resources\4bb24cfd-0b35-4c91-83de-97f335159a49\temp\tem >>>> p\RoleTemp\634526342992784034\output.sql >>>> >>>> [01:45:01] [Thread 10] Processing script command line 10000/37620 >>>> >>>> [01:45:03] [Thread 10] Processing script command line 20000/37620 >>>> >>>> [01:45:05] [Thread 10] Processing script command line 30000/37620 >>>> >>>> [01:45:06] [Thread 10] All sql script commands executed >>>> 37620/37620 >>>> >>>> [01:45:13] [Thread 10] Importing 37619 clients into the database >>>> >>>> [01:45:44] [Thread 12] Processed 169/37619 clients, 5.62 clients >>>> per second, 01:51:06 remaining >>>> >>>> [01:46:16] [Thread 29] Processed 360/37619 clients, 5.80 clients >>>> per second, 01:47:02 remaining >>>> >>>> [01:46:29] [Thread 69] Processed 1000/37619 clients, 13.30 clients >>>> per second, 00:45:52 remaining >>>> >>>> [01:46:50] [Thread 28] Processed 2000/37619 clients, 20.78 clients >>>> per second, 00:28:34 remaining >>>> >>>> [01:47:11] [Thread 30] Processed 3000/37619 clients, 25.47 clients >>>> per second, 00:22:39 remaining >>>> >>>> [01:47:33] [Thread 32] Processed 4000/37619 clients, 28.71 clients >>>> per second, 00:19:31 remaining >>>> >>>> [01:47:55] [Thread 76] Processed 5000/37619 clients, 30.95 clients >>>> per second, 00:17:33 remaining >>>> >>>> [01:48:18] [Thread 87] Processed 6000/37619 clients, 32.52 clients >>>> per second, 00:16:12 remaining >>>> >>>> [01:48:46] [Thread 13] Processed 7000/37619 clients, 32.95 clients >>>> per second, 00:15:29 remaining >>>> >>>> [01:49:10] [Thread 70] Processed 8000/37619 clients, 33.84 clients >>>> per second, 00:14:35 remaining >>>> >>>> [01:49:36] [Thread 54] Processed 9000/37619 clients, 34.25 clients >>>> per second, 00:13:55 remaining >>>> >>>> [01:50:02] [Thread 64] Processed 10000/37619 clients, 34.69 >>>> clients per second, 00:13:16 remaining >>>> >>>> [01:50:29] [Thread 50] Processed 11000/37619 clients, 34.85 >>>> clients per second, 00:12:43 remaining >>>> >>>> [01:50:57] [Thread 82] Processed 12000/37619 clients, 34.97 >>>> clients per second, 00:12:12 remaining >>>> >>>> [01:51:25] [Thread 27] Processed 13000/37619 clients, 35.02 >>>> clients per second, 00:11:42 remaining >>>> >>>> [01:51:54] [Thread 10] Processed 14000/37619 clients, 34.99 >>>> clients per second, 00:11:15 remaining >>>> >>>> [01:52:24] [Thread 76] Processed 15000/37619 clients, 34.89 >>>> clients per second, 00:10:48 remaining >>>> >>>> [01:52:54] [Thread 68] Processed 15970/37619 clients, 34.70 >>>> clients per second, 00:10:23 remaining >>>> >>>> [01:52:55] [Thread 17] Processed 16000/37619 clients, 34.69 >>>> clients per second, 00:10:23 remaining >>>> >>>> [01:53:25] [Thread 59] Processed 16918/37619 clients, 34.43 >>>> clients per second, 00:10:01 remaining >>>> >>>> [01:53:27] [Thread 46] Processed 17000/37619 clients, 34.42 >>>> clients per second, 00:09:59 remaining >>>> >>>> [01:53:58] [Thread 58] Processed 17998/37619 clients, 34.33 >>>> clients per second, 00:09:31 remaining >>>> >>>> [01:53:58] [Thread 34] Processed 18000/37619 clients, 34.32 >>>> clients per second, 00:09:31 remaining >>>> >>>> [01:54:28] [Thread 21] Processed 18900/37619 clients, 34.07 >>>> clients per second, 00:09:09 remaining >>>> >>>> [01:54:34] [Thread 13] Processed 19000/37619 clients, 33.90 >>>> clients per second, 00:09:09 remaining >>>> >>>> [01:55:04] [Thread 35] Processed 19973/37619 clients, 33.81 >>>> clients per second, 00:08:41 remaining >>>> >>>> [01:55:05] [Thread 86] Processed 20000/37619 clients, 33.80 >>>> clients per second, 00:08:41 remaining >>>> >>>> [01:55:36] [Thread 93] Processed 20910/37619 clients, 33.61 >>>> clients per second, 00:08:17 remaining >>>> >>>> [01:55:40] [Thread 42] Processed 21000/37619 clients, 33.54 >>>> clients per second, 00:08:15 remaining >>>> >>>> [01:56:10] [Thread 31] Processed 21886/37619 clients, 33.35 >>>> clients per second, 00:07:51 remaining >>>> >>>> [01:56:14] [Thread 81] Processed 22000/37619 clients, 33.32 >>>> clients per second, 00:07:48 remaining >>>> >>>> [01:56:44] [Thread 17] Processed 22809/37619 clients, 33.02 >>>> clients per second, 00:07:28 remaining >>>> >>>> [01:56:51] [Thread 77] Processed 23000/37619 clients, 32.96 >>>> clients per second, 00:07:23 remaining >>>> >>>> [01:57:22] [Thread 42] Processed 23770/37619 clients, 32.63 >>>> clients per second, 00:07:04 remaining >>>> >>>> [01:57:30] [Thread 71] Processed 24000/37619 clients, 32.57 >>>> clients per second, 00:06:58 remaining >>>> >>>> [01:58:01] [Thread 44] Processed 24652/37619 clients, 32.13 >>>> clients per second, 00:06:43 remaining >>>> >>>> [01:58:16] [Thread 42] Processed 25000/37619 clients, 31.94 >>>> clients per second, 00:06:35 remaining >>>> >>>> [01:58:47] [Thread 9] Processed 25691/37619 clients, 31.60 >>>> clients per second, 00:06:17 remaining >>>> >>>> [01:59:03] [Thread 84] Processed 26000/37619 clients, 31.33 >>>> clients per second, 00:06:10 remaining >>>> >>>> [01:59:36] [Thread 91] Processed 26625/37619 clients, 30.88 >>>> clients per second, 00:05:56 remaining >>>> >>>> [01:59:57] [Thread 74] Processed 27000/37619 clients, 30.57 >>>> clients per second, 00:05:47 remaining >>>> >>>> [02:00:27] [Thread 79] Processed 27583/37619 clients, 30.20 >>>> clients per second, 00:05:32 remaining >>>> >>>> [02:00:47] [Thread 82] Processed 28000/37619 clients, 30.01 >>>> clients per second, 00:05:20 remaining >>>> >>>> [02:01:17] [Thread 64] Processed 28591/37619 clients, 29.67 >>>> clients per second, 00:05:04 remaining >>>> >>>> [02:01:38] [Thread 17] Processed 29000/37619 clients, 29.46 >>>> clients per second, 00:04:52 remaining >>>> >>>> [02:02:08] [Thread 24] Processed 29610/37619 clients, 29.18 >>>> clients per second, 00:04:34 remaining >>>> >>>> [02:02:29] [Thread 58] Processed 30000/37619 clients, 28.98 >>>> clients per second, 00:04:22 remaining >>>> >>>> [02:02:59] [Thread 69] Processed 30566/37619 clients, 28.68 >>>> clients per second, 00:04:05 remaining >>>> >>>> [02:03:23] [Thread 89] Processed 31000/37619 clients, 28.46 >>>> clients per second, 00:03:52 remaining >>>> >>>> [02:03:23] [Thread 88] Processed 31000/37619 clients, 28.46 >>>> clients per second, 00:03:52 remaining >>>> >>>> [02:03:53] [Thread 30] Processed 31556/37619 clients, 28.18 >>>> clients per second, 00:03:35 remaining >>>> >>>> [02:04:16] [Thread 61] Processed 32000/37619 clients, 28.00 >>>> clients per second, 00:03:20 remaining >>>> >>>> [02:04:48] [Thread 66] Processed 32568/37619 clients, 27.73 >>>> clients per second, 00:03:02 remaining >>>> >>>> [02:05:18] [Thread 20] Processed 33000/37619 clients, 27.41 >>>> clients per second, 00:02:48 remaining >>>> >>>> [02:05:48] [Thread 13] Processed 33529/37619 clients, 27.16 >>>> clients per second, 00:02:30 remaining >>>> >>>> [02:06:15] [Thread 56] Processed 34000/37619 clients, 26.95 >>>> clients per second, 00:02:14 remaining >>>> >>>> [02:06:45] [Thread 92] Processed 34442/37619 clients, 26.66 >>>> clients per second, 00:01:59 remaining >>>> >>>> [02:07:16] [Thread 95] Processed 34959/37619 clients, 26.44 >>>> clients per second, 00:01:40 remaining >>>> >>>> [02:07:17] [Thread 91] Processed 35000/37619 clients, 26.44 >>>> clients per second, 00:01:39 remaining >>>> >>>> [02:07:49] [Thread 95] Processed 35369/37619 clients, 26.09 >>>> clients per second, 00:01:26 remaining >>>> >>>> [02:08:19] [Thread 41] Processed 35867/37619 clients, 25.88 >>>> clients per second, 00:01:07 remaining >>>> >>>> [02:08:28] [Thread 66] Processed 36000/37619 clients, 25.81 >>>> clients per second, 00:01:02 remaining >>>> >>>> [02:08:59] [Thread 45] Processed 36485/37619 clients, 25.60 >>>> clients per second, 00:00:44 remaining >>>> >>>> [02:09:29] [Thread 78] Processed 36922/37619 clients, 25.37 >>>> clients per second, 00:00:27 remaining >>>> >>>> [02:09:34] [Thread 81] Processed 37000/37619 clients, 25.34 >>>> clients per second, 00:00:24 remaining >>>> >>>> [02:10:04] [Thread 12] Processed 37287/37619 clients, 25.01 >>>> clients per second, 00:00:13 remaining >>>> >>>> [02:10:27] [Thread 10] All 37619 clients imported >>>> >>>> >>>> -----Original Message----- >>>> From: [email protected] >>>> [mailto:[email protected]] On Behalf Of Romiko Derbynew >>>> Sent: Friday, 23 September 2011 1:06 PM >>>> To: Neo4j user discussions >>>> Cc: Tatham Oddie >>>> Subject: Re: [Neo4j] Turning Off or Configuring log rotation >>>> >>>> Apologise. >>>> Issue Number: https://github.com/neo4j/community/issues/28 >>>> Blog: http://bit.ly/r9hBZC >>>> >>>> Cheers >>>> >>>> -----Original Message----- >>>> From: [email protected] >>>> [mailto:[email protected]] On Behalf Of Romiko Derbynew >>>> Sent: Friday, 23 September 2011 12:52 PM >>>> To: Neo4j user discussions >>>> Cc: Tatham Oddie >>>> Subject: Re: [Neo4j] Turning Off or Configuring log rotation >>>> >>>> Hi Peter, >>>> >>>> We solved the issue by disable index creations when importing 1000's of >>>> nodes. So we will need to figure out why Lucene Index creations via the >>>> rest api causes so much instability. Below is the history of work done. I >>>> have also published a blog report on this here: >>>> >>>> >>>> I have created an issue with this email as the content. Issue number >>>> is: INSERT ISSUE NUMBER http://bit.ly/r9hBZC >>>> >>>> >>>> Also the circuit breaker did not get triggered at all, which means the >>>> REST API is responding with 100% uptime when just doing the insert >>>> nodes/relationships. I guess we then have a performance issue with the >>>> Lucene side. I am more than willing to try collect information on this for >>>> you, if I enable FINEST log setting, will we be able to get details from >>>> neo4j about the lucene issue and why it causes such problems? >>>> >>>> >>>> Troubleshoot log: >>>> >>>> I managed to disable the tx logging via the console in the emebedded web >>>> server, and then ran an import. >>>> >>>> * Each Node that gets inserted - does a query, insert node, insert >>>> index entries >>>> * At around 3000 node mark, the rest api became unresponsive >>>> >>>> I have optimised the JVM by: >>>> Increases stack size to 2048k >>>> Increase Heap size to 512m -> 1024m >>>> Set JVM to Server Mode >>>> The GC is now optimized (In GCViewer, very few second generaton >>>> collections occur) Optmised the Gremlin code in the neo4jClient to used >>>> paramterised queries. >>>> Reduced the number concurrent threads from the .net application to 10 >>>> instead of unlimited to the rest api. >>>> Disabled LUCENE Indexes >>>> >>>> So with all the optimisation techniques from the performance guide being >>>> followed, we still get a slow response. So I guess this is not the flush >>>> to disk, as I notice my disk is very fast (SSD) and the %disk time and >>>> queue length is very low. >>>> >>>> Primary Cause Of Performance Issue >>>> Now, with the Lucene Index Calls via rest turned off, it is running >>>> much faster!! :) >>>> >>>> So the issue for us was all the Lucene Index rest Api calls. I guess this >>>> is the case as for each node we need to add +- 4-8 per node. This means, >>>> if we can reduce lucene index calls as Tatham logged by parsing a >>>> dictionary of key values in a json post, then we can reduce 4-8 rest calls >>>> to 1 call. >>>> >>>> So to CONFIRM, the REST API is STABLE when not creating the indexes >>>> for each node we create. Note all the calls are multithreaded. See >>>> below >>>> >>>> Log output: >>>> MyStory Welcome, Temporary from MyStory Administration. Logout? >>>> Agency Import Job 281 >>>> >>>> Log output: >>>> >>>> [02:07:36] [Thread 11] Migration message received at: 23/09/2011 >>>> 2:07:36 AM +00:00 >>>> >>>> [02:07:36] [Thread 11] Migration message payload: JobUniqueId: 281, >>>> AgencyUniqueId: 181 >>>> >>>> [02:07:37] [Thread 11] Executing sql script: >>>> e:\AzureTemp\Resources\483c6036-8050-4387-a806-6e880d655b58\temp\tem >>>> p\RoleTemp\634523764566141006\output.sql >>>> >>>> [02:07:39] [Thread 11] Processing script command line 10000/37620 >>>> >>>> [02:07:40] [Thread 11] Processing script command line 20000/37620 >>>> >>>> [02:07:42] [Thread 11] Processing script command line 30000/37620 >>>> >>>> [02:07:43] [Thread 11] All sql script commands executed 37620/37620 >>>> >>>> [02:07:51] [Thread 11] Importing 37619 clients into the database >>>> >>>> [02:08:22] [Thread 18] Processed 146/37619 clients, 4.72 clients >>>> per second, 02:12:25 remaining >>>> >>>> [02:08:53] [Thread 35] Processed 302/37619 clients, 4.87 clients >>>> per second, 02:07:43 remaining >>>> >>>> [02:09:24] [Thread 82] Processed 457/37619 clients, 4.92 clients >>>> per second, 02:05:58 remaining >>>> >>>> [02:09:58] [Thread 55] Processed 624/37619 clients, 4.93 clients >>>> per second, 02:05:02 remaining >>>> >>>> [02:10:29] [Thread 66] Processed 779/37619 clients, 4.93 clients >>>> per second, 02:04:32 remaining >>>> >>>> [02:11:01] [Thread 36] Processed 933/37619 clients, 4.92 clients >>>> per second, 02:04:12 remaining >>>> >>>> [02:11:13] [Thread 35] Processed 1000/37619 clients, 4.95 clients >>>> per second, 02:03:21 remaining >>>> >>>> [02:11:14] [Thread 40] Processed 1000/37619 clients, 4.95 clients >>>> per second, 02:03:21 remaining >>>> >>>> [02:11:48] [Thread 13] Processed 1147/37619 clients, 4.84 clients >>>> per second, 02:05:40 remaining >>>> >>>> [02:12:21] [Thread 9] Processed 1291/37619 clients, 4.78 clients >>>> per second, 02:06:40 remaining >>>> >>>> [02:12:52] [Thread 79] Processed 1437/37619 clients, 4.77 clients >>>> per second, 02:06:24 remaining >>>> >>>> [02:13:23] [Thread 167] Processed 1565/37619 clients, 4.71 clients >>>> per second, 02:07:33 remaining >>>> >>>> [02:13:56] [Thread 212] Processed 1708/37619 clients, 4.68 clients >>>> per second, 02:08:00 remaining >>>> >>>> [02:14:27] [Thread 26] Processed 1839/37619 clients, 4.64 clients >>>> per second, 02:08:31 remaining >>>> >>>> [02:14:58] [Thread 203] Processed 1971/37619 clients, 4.61 clients >>>> per second, 02:08:50 remaining >>>> >>>> [02:15:02] [Thread 91] Processed 2000/37619 clients, 4.64 clients >>>> per second, 02:08:04 remaining >>>> >>>> [02:15:04] [Thread 243] Processed 2000/37619 clients, 4.64 clients >>>> per second, 02:08:04 remaining >>>> >>>> [02:15:06] [Thread 13] Processed 2000/37619 clients, 4.64 clients >>>> per second, 02:08:04 remaining >>>> >>>> [02:15:09] [Thread 39] Processed 2000/37619 clients, 4.63 clients >>>> per second, 02:08:04 remaining >>>> >>>> [02:15:42] [Thread 71] Processed 2161/37619 clients, 4.59 clients >>>> per second, 02:08:47 remaining >>>> >>>> [02:16:15] [Thread 235] Processed 2305/37619 clients, 4.57 clients >>>> per second, 02:08:45 remaining >>>> >>>> [02:16:48] [Thread 179] Processed 2448/37619 clients, 4.56 clients >>>> per second, 02:08:34 remaining >>>> >>>> [02:17:20] [Thread 371] Processed 2591/37619 clients, 4.55 clients >>>> per second, 02:08:11 remaining >>>> >>>> [02:17:50] [Thread 327] Processed 2725/37619 clients, 4.55 clients >>>> per second, 02:07:52 remaining >>>> >>>> [02:18:23] [Thread 408] Processed 2865/37619 clients, 4.54 clients >>>> per second, 02:07:43 remaining >>>> >>>> [02:18:52] [Thread 111] Processed 3000/37619 clients, 4.54 clients >>>> per second, 02:07:04 remaining >>>> >>>> [02:18:53] [Thread 72] Processed 3000/37619 clients, 4.54 clients >>>> per second, 02:07:04 remaining >>>> >>>> [02:18:56] [Thread 424] Processed 3000/37619 clients, 4.54 clients >>>> per second, 02:07:05 remaining >>>> >>>> [02:18:58] [Thread 45] Processed 3000/37619 clients, 4.54 clients >>>> per second, 02:07:05 remaining >>>> >>>> [02:19:00] [Thread 254] Processed 3000/37619 clients, 4.54 clients >>>> per second, 02:07:05 remaining >>>> >>>> [02:19:02] [Thread 425] Processed 3000/37619 clients, 4.54 clients >>>> per second, 02:07:05 remaining >>>> >>>> [02:19:36] [Thread 320] Processed 3207/37619 clients, 4.55 clients >>>> per second, 02:06:02 remaining >>>> >>>> [02:20:06] [Thread 170] Processed 3350/37619 clients, 4.56 clients >>>> per second, 02:05:18 remaining >>>> >>>> [02:20:39] [Thread 46] Processed 3506/37619 clients, 4.57 clients >>>> per second, 02:04:31 remaining >>>> >>>> [02:21:11] [Thread 98] Processed 3638/37619 clients, 4.55 clients >>>> per second, 02:04:27 remaining >>>> >>>> [02:21:44] [Thread 447] Processed 3783/37619 clients, 4.54 clients >>>> per second, 02:04:11 remaining >>>> >>>> [02:22:18] [Thread 144] Processed 3927/37619 clients, 4.53 clients >>>> per second, 02:03:56 remaining >>>> >>>> [02:22:34] [Thread 200] Processed 4000/37619 clients, 4.53 clients >>>> per second, 02:03:38 remaining >>>> >>>> [02:22:34] [Thread 398] Processed 4000/37619 clients, 4.53 clients >>>> per second, 02:03:38 remaining >>>> >>>> [02:22:36] [Thread 41] Processed 4000/37619 clients, 4.53 clients >>>> per second, 02:03:38 remaining >>>> >>>> [02:22:38] [Thread 470] Processed 4000/37619 clients, 4.53 clients >>>> per second, 02:03:38 remaining >>>> >>>> [02:22:39] [Thread 471] Processed 4000/37619 clients, 4.53 clients >>>> per second, 02:03:38 remaining >>>> >>>> [02:22:42] [Thread 335] Processed 4000/37619 clients, 4.53 clients >>>> per second, 02:03:38 remaining >>>> >>>> [02:22:44] [Thread 558] Processed 4000/37619 clients, 4.53 clients >>>> per second, 02:03:39 remaining >>>> >>>> [02:22:46] [Thread 557] Processed 4000/37619 clients, 4.53 clients >>>> per second, 02:03:39 remaining >>>> >>>> [02:23:19] [Thread 587] Processed 4210/37619 clients, 4.54 clients >>>> per second, 02:02:41 remaining >>>> >>>> [02:23:52] [Thread 159] Processed 4353/37619 clients, 4.53 clients >>>> per second, 02:02:22 remaining >>>> >>>> [02:24:24] [Thread 311] Processed 4497/37619 clients, 4.53 clients >>>> per second, 02:01:53 remaining >>>> >>>> [02:24:55] [Thread 229] Processed 4631/37619 clients, 4.52 clients >>>> per second, 02:01:34 remaining >>>> >>>> [02:25:28] [Thread 200] Processed 4784/37619 clients, 4.53 clients >>>> per second, 02:00:54 remaining >>>> >>>> [02:26:00] [Thread 245] Processed 4936/37619 clients, 4.53 clients >>>> per second, 02:00:10 remaining >>>> >>>> [02:26:14] [Thread 693] Processed 5000/37619 clients, 4.53 clients >>>> per second, 01:59:53 remaining >>>> >>>> [02:26:15] [Thread 581] Processed 5000/37619 clients, 4.53 clients >>>> per second, 01:59:53 remaining >>>> >>>> [02:26:16] [Thread 694] Processed 5000/37619 clients, 4.53 clients >>>> per second, 01:59:53 remaining >>>> >>>> [02:26:18] [Thread 584] Processed 5000/37619 clients, 4.53 clients >>>> per second, 01:59:53 remaining >>>> >>>> [02:26:20] [Thread 416] Processed 5000/37619 clients, 4.53 clients >>>> per second, 01:59:53 remaining >>>> >>>> [02:26:22] [Thread 68] Processed 5000/37619 clients, 4.53 clients >>>> per second, 01:59:54 remaining >>>> >>>> [02:26:24] [Thread 46] Processed 5000/37619 clients, 4.53 clients >>>> per second, 01:59:54 remaining >>>> >>>> [02:26:57] [Thread 431] Processed 5224/37619 clients, 4.56 clients >>>> per second, 01:58:23 remaining >>>> >>>> [02:27:29] [Thread 627] Processed 5367/37619 clients, 4.56 clients >>>> per second, 01:57:56 remaining >>>> >>>> [02:28:00] [Thread 766] Processed 5522/37619 clients, 4.57 clients >>>> per second, 01:57:05 remaining >>>> >>>> [02:28:32] [Thread 659] Processed 5665/37619 clients, 4.57 clients >>>> per second, 01:56:38 remaining >>>> >>>> [02:29:06] [Thread 32] Processed 5821/37619 clients, 4.57 clients >>>> per second, 01:56:05 remaining >>>> >>>> [02:29:38] [Thread 693] Processed 5978/37619 clients, 4.57 clients >>>> per second, 01:55:19 remaining >>>> >>>> [02:30:11] [Thread 358] Processed 6157/37619 clients, 4.59 clients >>>> per second, 01:54:08 remaining >>>> >>>> [02:30:42] [Thread 615] Processed 6301/37619 clients, 4.59 clients >>>> per second, 01:53:35 remaining >>>> >>>> [02:31:14] [Thread 631] Processed 6458/37619 clients, 4.60 clients >>>> per second, 01:52:49 remaining >>>> >>>> [02:31:47] [Thread 229] Processed 6625/37619 clients, 4.61 clients >>>> per second, 01:51:57 remaining >>>> >>>> [02:32:20] [Thread 663] Processed 6806/37619 clients, 4.63 clients >>>> per second, 01:50:50 remaining >>>> >>>> [02:32:53] [Thread 202] Processed 6962/37619 clients, 4.63 clients >>>> per second, 01:50:14 remaining >>>> >>>> [02:33:00] [Thread 965] Processed 7000/37619 clients, 4.64 clients >>>> per second, 01:50:00 remaining >>>> >>>> [02:33:03] [Thread 569] Processed 7000/37619 clients, 4.64 clients >>>> per second, 01:50:00 remaining >>>> >>>> [02:33:04] [Thread 481] Processed 7000/37619 clients, 4.64 clients >>>> per second, 01:50:01 remaining >>>> >>>> [02:33:06] [Thread 966] Processed 7000/37619 clients, 4.64 clients >>>> per second, 01:50:01 remaining >>>> >>>> [02:33:08] [Thread 342] Processed 7000/37619 clients, 4.64 clients >>>> per second, 01:50:01 remaining >>>> >>>> [02:33:11] [Thread 483] Processed 7000/37619 clients, 4.64 clients >>>> per second, 01:50:01 remaining >>>> >>>> [02:33:13] [Thread 484] Processed 7000/37619 clients, 4.64 clients >>>> per second, 01:50:01 remaining >>>> >>>> [02:33:48] [Thread 68] Processed 7223/37619 clients, 4.64 clients >>>> per second, 01:49:11 remaining >>>> >>>> [02:34:19] [Thread 131] Processed 7367/37619 clients, 4.64 clients >>>> per second, 01:48:41 remaining >>>> >>>> [02:34:51] [Thread 869] Processed 7534/37619 clients, 4.65 clients >>>> per second, 01:47:49 remaining >>>> >>>> [02:35:22] [Thread 376] Processed 7689/37619 clients, 4.66 clients >>>> per second, 01:47:07 remaining >>>> >>>> [02:35:54] [Thread 1086] Processed 7857/37619 clients, 4.67 clients >>>> per second, 01:46:16 remaining >>>> >>>> [02:36:27] [Thread 123] Processed 8048/37619 clients, 4.69 clients >>>> per second, 01:45:06 remaining >>>> >>>> [02:37:01] [Thread 201] Processed 8227/37619 clients, 4.70 clients >>>> per second, 01:44:11 remaining >>>> >>>> [02:37:33] [Thread 1161] Processed 8407/37619 clients, 4.72 clients >>>> per second, 01:43:11 remaining >>>> >>>> [02:38:06] [Thread 59] Processed 8586/37619 clients, 4.73 clients >>>> per second, 01:42:16 remaining >>>> >>>> [02:38:38] [Thread 300] Processed 8755/37619 clients, 4.74 clients >>>> per second, 01:41:29 remaining >>>> >>>> [02:39:12] [Thread 862] Processed 8922/37619 clients, 4.74 clients >>>> per second, 01:40:49 remaining >>>> >>>> [02:39:25] [Thread 1239] Processed 9000/37619 clients, 4.75 clients >>>> per second, 01:40:21 remaining >>>> >>>> [02:39:26] [Thread 437] Processed 9000/37619 clients, 4.75 clients >>>> per second, 01:40:21 remaining >>>> >>>> [02:39:28] [Thread 1240] Processed 9000/37619 clients, 4.75 clients >>>> per second, 01:40:21 remaining >>>> >>>> [02:39:31] [Thread 23] Processed 9000/37619 clients, 4.75 clients >>>> per second, 01:40:21 remaining >>>> >>>> [02:40:05] [Thread 888] Processed 9208/37619 clients, 4.76 clients >>>> per second, 01:39:26 remaining >>>> >>>> [02:40:37] [Thread 320] Processed 9398/37619 clients, 4.78 clients >>>> per second, 01:38:23 remaining >>>> >>>> [02:41:10] [Thread 921] Processed 9566/37619 clients, 4.79 clients >>>> per second, 01:37:40 remaining >>>> >>>> [02:41:44] [Thread 792] Processed 9756/37619 clients, 4.80 clients >>>> per second, 01:36:46 remaining >>>> >>>> [02:42:17] [Thread 958] Processed 9936/37619 clients, 4.81 clients >>>> per second, 01:35:55 remaining >>>> >>>> [02:42:28] [Thread 1155] Processed 10000/37619 clients, 4.81 clients >>>> per second, 01:35:37 remaining >>>> >>>> [02:42:30] [Thread 67] Processed 10000/37619 clients, 4.81 clients >>>> per second, 01:35:37 remaining >>>> >>>> [02:42:33] [Thread 485] Processed 10000/37619 clients, 4.81 clients >>>> per second, 01:35:37 remaining >>>> >>>> [02:42:35] [Thread 406] Processed 10000/37619 clients, 4.81 clients >>>> per second, 01:35:37 remaining >>>> >>>> [02:42:37] [Thread 808] Processed 10000/37619 clients, 4.81 clients >>>> per second, 01:35:37 remaining >>>> >>>> [02:42:39] [Thread 1378] Processed 10000/37619 clients, 4.81 clients >>>> per second, 01:35:37 remaining >>>> >>>> [02:43:11] [Thread 1411] Processed 10234/37619 clients, 4.83 clients >>>> per second, 01:34:34 remaining >>>> >>>> [02:43:45] [Thread 298] Processed 10391/37619 clients, 4.82 clients >>>> per second, 01:34:03 remaining >>>> >>>> [02:44:16] [Thread 1459] Processed 10582/37619 clients, 4.84 clients >>>> per second, 01:33:01 remaining >>>> >>>> [02:44:49] [Thread 1041] Processed 10761/37619 clients, 4.85 clients >>>> per second, 01:32:16 remaining >>>> >>>> [02:45:22] [Thread 1057] Processed 10940/37619 clients, 4.86 clients >>>> per second, 01:31:29 remaining >>>> >>>> [02:45:53] [Thread 753] Processed 11108/37619 clients, 4.87 clients >>>> per second, 01:30:47 remaining >>>> >>>> [02:46:27] [Thread 910] Processed 11288/37619 clients, 4.87 clients >>>> per second, 01:30:03 remaining >>>> >>>> [02:47:00] [Thread 1580] Processed 11477/37619 clients, 4.89 clients >>>> per second, 01:29:10 remaining >>>> >>>> [02:47:32] [Thread 125] Processed 11645/37619 clients, 4.89 clients >>>> per second, 01:28:30 remaining >>>> >>>> [02:48:06] [Thread 1628] Processed 11824/37619 clients, 4.90 clients >>>> per second, 01:27:48 remaining >>>> >>>> [02:48:40] [Thread 1385] Processed 12003/37619 clients, 4.90 clients >>>> per second, 01:27:06 remaining >>>> >>>> [02:49:12] [Thread 1401] Processed 12171/37619 clients, 4.91 clients >>>> per second, 01:26:27 remaining >>>> >>>> [02:49:44] [Thread 1192] Processed 12331/37619 clients, 4.91 clients >>>> per second, 01:25:53 remaining >>>> >>>> [02:50:15] [Thread 1444] Processed 12508/37619 clients, 4.92 clients >>>> per second, 01:25:07 remaining >>>> >>>> [02:50:47] [Thread 1463] Processed 12684/37619 clients, 4.92 clients >>>> per second, 01:24:24 remaining >>>> >>>> -----Original Message----- >>>> From: [email protected] >>>> [mailto:[email protected]] On Behalf Of Romiko Derbynew >>>> Sent: Friday, 23 September 2011 9:37 AM >>>> To: Neo4j user discussions >>>> Cc: Tatham Oddie >>>> Subject: Re: [Neo4j] Turning Off or Configuring log rotation >>>> >>>> Hi Peter, >>>> >>>> Thank you so much for the tips, will look into this today and see >>>> what I can come up with. After my musings will definitely file for >>>> new features e.g >>>> * Batching via rest api (Being on .Net we rely solely on the Rest >>>> API) >>>> * Configuration level control of transaction log default sizes e.g. >>>> 10mb -> 100mb, disabled >>>> * Explore ways to optimise flushing to disk without affecting the >>>> rest api >>>> >>>> Regards >>>> >>>> -----Original Message----- >>>> From: [email protected] >>>> [mailto:[email protected]] On Behalf Of Peter Neubauer >>>> Sent: Friday, 23 September 2011 7:28 AM >>>> To: Neo4j user discussions >>>> Subject: Re: [Neo4j] Turning Off or Configuring log rotation >>>> >>>> Romiko, >>>> Gremlin is basically executing Java on the server side and thus can do >>>> anything java can do. >>>> http://docs.neo4j.org/chunked/snapshot/gremlin-plugin.html#rest-api- >>>> send-an-arbitrary-groovy-script---lucene-sorting >>>> has an example of mixing native Neo4j API calls (g.createNode() etc) >>>> after obtaining the injected Neo4j DB instance with >>>> >>>> g.getRawGraph(), in this case using Neo4j Index sorting. >>>> >>>> After that, just call the log rotation code. Let me know how it works out, >>>> otherwise I can provide an example tomorrow? >>>> >>>> Cheers, >>>> >>>> /peter neubauer >>>> >>>> GTalk: neubauer.peter >>>> Skype peter.neubauer >>>> Phone +46 704 106975 >>>> LinkedIn http://www.linkedin.com/in/neubauer >>>> Twitter http://twitter.com/peterneubauer >>>> >>>> http://www.neo4j.org - Your high performance graph database. >>>> http://startupbootcamp.org/ - Öresund - Innovation happens HERE. >>>> http://www.thoughtmade.com - Scandinavia's coolest Bring-a-Thing party. >>>> >>>> >>>> >>>> On Thu, Sep 22, 2011 at 11:17 PM, Romiko Derbynew >>>> <[email protected]> wrote: >>>>> Hi Peter, >>>>> >>>>> I think this will be ok via gremlin. Have you got any sample gremlin code >>>>> that can do this, that would be much appreciated. Also in the long term I >>>>> think a combo of tx logs + rest API batching will be the solution. >>>>> >>>>> Sent from my iPhone >>>>> >>>>> On 22/09/2011, at 10:08 PM, "Peter Neubauer" >>>>> <[email protected]> wrote: >>>>> >>>>>> Romiko, >>>>>> I would suggest to do the TX log config via Gremlin for the time >>>>>> being, so you can resolve it via scripting. For the exposure of >>>>>> the Tx log rotate size as a configuration property (there are logs >>>>>> for every XA resource participating in a transaction), please >>>>>> raise an issue at https://github.com/neo4j/community/issues - good input! >>>>>> >>>>>> For the flushing to disk, we will merge a Deferred Write feature >>>>>> back into master soonish (after 1.5) which lets the OS decide when >>>>>> to flush, if you can stand the risk of a lost flush if things go down. >>>>>> The REST interface is waiting for the TX to be finished, which is >>>>>> why it is slowing down when you have big flushes. See >>>>>> https://github.com/neo4j/community/issues/18 for the ticket on >>>>>> this >>>>>> :) >>>>>> >>>>>> Would that be ok? >>>>>> >>>>>> Cheers, >>>>>> >>>>>> /peter neubauer >>>>>> >>>>>> GTalk: neubauer.peter >>>>>> Skype peter.neubauer >>>>>> Phone +46 704 106975 >>>>>> LinkedIn http://www.linkedin.com/in/neubauer >>>>>> Twitter http://twitter.com/peterneubauer >>>>>> >>>>>> http://www.neo4j.org - Your high performance graph >>>>>> database. >>>>>> http://startupbootcamp.org/ - Öresund - Innovation happens HERE. >>>>>> http://www.thoughtmade.com - Scandinavia's coolest Bring-a-Thing party. >>>>>> >>>>>> >>>>>> >>>>>> On Thu, Sep 22, 2011 at 1:52 PM, Romiko Derbynew >>>>>> <[email protected]> wrote: >>>>>>> Hi, >>>>>>> >>>>>>> It's the database logging, I'm finding the transaction logs are causing >>>>>>> the rest API to become unresponsive when it flushes to disk. Basically >>>>>>> just want to turn it off or set the tlog size by configuration and not >>>>>>> via code sample as it is demonstrated in the performance guide. >>>>>>> >>>>>>> The current work around without this setting is a circuit breaker >>>>>>> pattern. But I am very concerned as the DB flush should not slow >>>>>>> down further rest calls :( >>>>>>> >>>>>>> Sent from my iPhone >>>>>>> >>>>>>> On 22/09/2011, at 6:36 PM, "Peter Neubauer" >>>>>>> <[email protected]> wrote: >>>>>>> >>>>>>>> Romiko, >>>>>>>> which log files are you referring to? For the database >>>>>>>> transaction log rotation, look at the neo4j.properties file to >>>>>>>> tune things, see >>>>>>>> http://docs.neo4j.org/chunked/snapshot/server-configuration.html >>>>>>>> >>>>>>>> For the server, there is logging.properties to tweak stuff. Note >>>>>>>> that these settings require a restart to take effect. >>>>>>>> >>>>>>>> HTH >>>>>>>> >>>>>>>> Cheers, >>>>>>>> >>>>>>>> /peter neubauer >>>>>>>> >>>>>>>> GTalk: neubauer.peter >>>>>>>> Skype peter.neubauer >>>>>>>> Phone +46 704 106975 >>>>>>>> LinkedIn http://www.linkedin.com/in/neubauer >>>>>>>> Twitter http://twitter.com/peterneubauer >>>>>>>> >>>>>>>> http://www.neo4j.org - Your high performance graph >>>>>>>> database. >>>>>>>> http://startupbootcamp.org/ - Öresund - Innovation happens HERE. >>>>>>>> http://www.thoughtmade.com - Scandinavia's coolest Bring-a-Thing party. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Thu, Sep 22, 2011 at 1:38 AM, Romiko Derbynew >>>>>>>> <[email protected]> wrote: >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> I am using Neo4j on a Windows Server, is there any way I can >>>>>>>>> configure log rotation via the configuration files? I would like to >>>>>>>>> turn it off or try a different log rotation size. Since I am on >>>>>>>>> windows, I am not sure how I would do it at runtime. >>>>>>>>> >>>>>>>>> Thanks! >>>>>>>>> _______________________________________________ >>>>>>>>> Neo4j mailing list >>>>>>>>> [email protected] >>>>>>>>> https://lists.neo4j.org/mailman/listinfo/user >>>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Neo4j mailing list >>>>>>>> [email protected] >>>>>>>> https://lists.neo4j.org/mailman/listinfo/user >>>>>>>> >>>>>>> _______________________________________________ >>>>>>> Neo4j mailing list >>>>>>> [email protected] >>>>>>> https://lists.neo4j.org/mailman/listinfo/user >>>>>>> >>>>>> _______________________________________________ >>>>>> Neo4j mailing list >>>>>> [email protected] >>>>>> https://lists.neo4j.org/mailman/listinfo/user >>>>>> >>>>> _______________________________________________ >>>>> Neo4j mailing list >>>>> [email protected] >>>>> https://lists.neo4j.org/mailman/listinfo/user >>>>> >>>> _______________________________________________ >>>> Neo4j mailing list >>>> [email protected] >>>> https://lists.neo4j.org/mailman/listinfo/user >>>> >>>> >>>> _______________________________________________ >>>> Neo4j mailing list >>>> [email protected] >>>> https://lists.neo4j.org/mailman/listinfo/user >>>> >>>> >>>> _______________________________________________ >>>> Neo4j mailing list >>>> [email protected] >>>> https://lists.neo4j.org/mailman/listinfo/user >>>> >>>> >>>> _______________________________________________ >>>> Neo4j mailing list >>>> [email protected] >>>> https://lists.neo4j.org/mailman/listinfo/user >>>> >>>> >>>> _______________________________________________ >>>> Neo4j mailing list >>>> [email protected] >>>> https://lists.neo4j.org/mailman/listinfo/user >>>> >>> _______________________________________________ >>> Neo4j mailing list >>> [email protected] >>> https://lists.neo4j.org/mailman/listinfo/user >>> >> _______________________________________________ >> Neo4j mailing list >> [email protected] >> https://lists.neo4j.org/mailman/listinfo/user >> > > _______________________________________________ Neo4j mailing list [email protected] https://lists.neo4j.org/mailman/listinfo/user
