Yes, this is most likely a disk fragmentation problem: https://support.microsoft.com/en-us/kb/967351 <https://support.microsoft.com/en-us/kb/967351>
The message in the exception is from the file system. If the file system can do online defragmentation at only the cost of performance (I don’t know if NTFS can do that), then yes, Neo4j should work like normal except slower. Defragmentation involves a lot of disk seeks, so if you are running a cluster, then the increased IO latencies could destabilise the cluster, because the cluster stability relies on the instances being able to send and reply to network messages within certain timeouts. -- Chris Vest System Engineer, Neo Technology [ skype: mr.chrisvest, twitter: chvest ] > On 14 Oct 2015, at 16:34, Avner Shahar-Kashtan <[email protected]> wrote: > > We've been having strange crashes in Neo4j at a client site that have been > worrying us, and I'd love to hear if they can be mitigated. > > System Spec > Neo4j 2.1.7 on JDK 1.7.0_71, running on Windows Server 2012 R2 (32GB RAM, 1 > QuadCore CPU E3-11220 V2 @ 3.10Ghz) > > Course of Events > Our system runs continuously, receiving data every several minutes, parsing > the items and feeding them into Neo4j for later querying. Most nodes have a > very large number of relationships and properties: for 4 million nodes, we > had about 390 million relationships and 1.8 billion properties. After a few > weeks of operation, the graph.db weighed about 150Gb - ~75Gb for the .strings > file, ~70Gb for the propertystore and ~10Gb for the relationship store file. > > Slowly we noticed that performance degraded for most of our queries. Then, > after a few days, we started getting errors when writing to the graph.db. > They would be TransactionFailedExceptions, caused by > UnderlyingStorageExceptions, which in turn are caused by java.io.IOException: > The requested operation could not be completed due to a file system > limitation. (Sorry for not bringing the full stack trace, but the log files > are at a client site and I can't remove them). This happened sporadically at > first (followed by logfile recovery), and then started happening on startup, > preventing the service from starting. > > Possible Causes > > I checked the usual suspects - there was plenty of room on the disk (~230Gb > free) and on the system drive (~70Gb), none of the files were close to being > too big for the filesystem and there weren't any quotas or permissions or > anything else like that defined. The same drive also hosts heavy loads - > there's a SQL Server database and a drop folder where the incoming data > arrives (~50-100Mb/minute). > > One forum post (which I can't find now, unfortunately) suggested that disk > fragmentation could be an issue. I checked and saw that Windows' scheduled > defrag task was never run, probably because the server is never idle long > enough for it to start. On running it manually, it reported 100% > fragmentation of the disk, but optimizing the 1TB disk would take >10 hours, > which was more downtime than we were willing to have, so we simply erased the > graph.db and started a new (which allowed Neo4j to start). However, I'm > worried that the problem will return in a few days/weeks, since we never > addressed the root cause. > > My questions > > 1. Is disk fragmentation a possible cause for these errors, and for > preventing Neo4j from starting? > 2. Would regular defrag operations prevent it from reoccuring? Other than > performance slowdown, will the defrag operation interfere with Neo4j > operations? > > > Thanks in advance, > Avner > > -- > You received this message because you are subscribed to the Google Groups > "Neo4j" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] > <mailto:[email protected]>. > For more options, visit https://groups.google.com/d/optout > <https://groups.google.com/d/optout>. -- You received this message because you are subscribed to the Google Groups "Neo4j" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
