Looks like Btrfs has a history of running into ENOSPC problems. Google turns up a lot of results where Btrfs runs out of space in situations where most other file systems would not.
Also try enabling the `enospc_debug` mount option and see if anything useful gets written to the syslog next time you run into the problem. -- Chris Vest System Engineer, Neo Technology [ skype: mr.chrisvest, twitter: chvest ] > On 14 Apr 2015, at 16:00, [email protected] wrote: > > Thanks Christian. > The OS is ubuntu 14.04 and file system for the neo4j storage partition is > btrfs. > The partition size is 1.1TB and 625GB is used. > > > On Tuesday, April 14, 2015 at 2:24:37 AM UTC-5, Christian Morgner wrote: > The "no space left on device" error can occur when there are many very > small files on your disk, since a file will always consume a certain > min. amount of space, but the disk usage utilities will report the > actual size, not the inode size. > > What operating system and filesystem type do you use? > > > Best, > Christian > > > > > Am 13.04.2015 um 16:41 schrieb [email protected] <javascript:>: > > It is really strange. The disk or the partition is nowhere near being > > full. The open file limits on the system is set to a really high number > > as well. Any other quota I should check? > > Any chance when the current logical log is approaching its size limit > > 25MB, if the last commit is too big, the error is raise? > > Thanks. > > > > On Friday, April 10, 2015 at 8:08:22 AM UTC-5, Chris Vest wrote: > > > > “No space left on device” is what the operating system gives us as > > reason for failing a write. Maybe you are running into quota limits, > > or the particular partition of the drive is getting full? Also note > > that Neo4j wants to create the next log file and complete the > > rotation, before it looks for old log files to delete. > > > > -- > > Chris Vest > > System Engineer, Neo Technology > > [ skype: mr.chrisvest, twitter: chvest ] > > > > > >> On 09 Apr 2015, at 18:27, [email protected] <> <javascript:> wrote: > >> > >> I am running neo4j 2.1.6 embedded. Under load, I ran into > >> "org.neo4j.graphdb.TransactionFailureException: Unable to write > >> command to logical log" several times. The root cause says 'No > >> space left on device' while there is plenty of disk space available. > >> The logical log file size is the default 25MB and 10 files are > >> kept. I noticed that file rotation occurred every 2 minutes. > >> Any idea why this error occurred? Any recommendations for > >> configure change/tuning? > >> > >> Caused by: javax.transaction.xa.XAException: null > >> at > >> > >> org.neo4j.kernel.impl.transaction.TransactionImpl.doCommit(TransactionImpl.java:560) > >> > >> ~[neo4j-kernel-2.1.6.jar:2.1.6] > >> at > >> org.neo4j.kernel.impl.transaction.TxManager.commit(TxManager.java:448) > >> ~[neo4j-kernel-2.1.6.jar:2.1.6] > >> at > >> org.neo4j.kernel.impl.transaction.TxManager.commit(TxManager.java:385) > >> ~[neo4j-kernel-2.1.6.jar:2.1.6] > >> at > >> > >> org.neo4j.kernel.impl.transaction.TransactionImpl.commit(TransactionImpl.java:123) > >> > >> ~[neo4j-kernel-2.1.6.jar:2.1.6] > >> at > >> > >> org.neo4j.kernel.impl.transaction.BaseSpringTransactionImpl.commit(BaseSpringTransactionImpl.java:66) > >> > >> ~[neo4j-kernel-2.1.6.jar:2.1.6] > >> at > >> > >> org.springframework.transaction.jta.JtaTransactionManager.doCommit(JtaTransactionManager.java:1011) > >> > >> ~[spring-tx-3.2.4.RELEASE.jar:3.2.4.RELEASE] > >> at > >> > >> org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:755) > >> > >> ~[spring-tx-3.2.4.RELEASE.jar:3.2.4.RELEASE] > >> at > >> > >> org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:724) > >> > >> ~[spring-tx-3.2.4.RELEASE.jar:3.2.4.RELEASE] > >> at > >> > >> org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:475) > >> > >> ~[spring-tx-3.2.4.RELEASE.jar:3.2.4.RELEASE] > >> at > >> > >> org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:270) > >> > >> ~[spring-tx-3.2.4.RELEASE.jar:3.2.4.RELEASE] > >> at > >> > >> org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:94) > >> > >> ~[spring-tx-3.2.4.RELEASE.jar:3.2.4.RELEASE] > >> at > >> > >> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) > >> > >> ~[spring-aop-3.2.4.RELEASE.jar:3.2.4.RELEASE] > >> at > >> > >> com.ryantenney.metrics.spring.MeteredMethodInterceptor.invoke(MeteredMethodInterceptor.java:45) > >> > >> ~[metrics-spring-3.0.1.jar:na] > >> at > >> > >> com.ryantenney.metrics.spring.MeteredMethodInterceptor.invoke(MeteredMethodInterceptor.java:32) > >> > >> ~[metrics-spring-3.0.1.jar:na] > >> at > >> > >> com.ryantenney.metrics.spring.AbstractMetricMethodInterceptor.invoke(AbstractMetricMethodInterceptor.java:59) > >> > >> ~[metrics-spring-3.0.1.jar:na] > >> at > >> > >> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) > >> > >> ~[spring-aop-3.2.4.RELEASE.jar:3.2.4.RELEASE] > >> at > >> > >> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) > >> > >> ~[spring-aop-3.2.4.RELEASE.jar:3.2.4.RELEASE] > >> at com.sun.proxy.$Proxy68.setIntrinsicState(Unknown Source) ~[na:na] > >> at > >> > >> com.zenoss.dsa.impl.StateChangeProcessorImpl.processStateEvents(StateChangeProcessorImpl.java:51) > >> > >> ~[classes/:na] > >> ... 23 common frames omitted > >> Caused by: org.neo4j.graphdb.TransactionFailureException: Unable > >> to write command to logical log. > >> at > >> > >> org.neo4j.kernel.impl.transaction.xaframework.XaTransaction.addCommand(XaTransaction.java:254) > >> > >> ~[neo4j-kernel-2.1.6.jar:2.1.6] > >> at > >> > >> org.neo4j.kernel.impl.nioneo.xa.NeoStoreTransaction.doPrepare(NeoStoreTransaction.java:322) > >> > >> ~[neo4j-kernel-2.1.6.jar:2.1.6] > >> at > >> > >> org.neo4j.kernel.impl.transaction.xaframework.XaTransaction.prepare(XaTransaction.java:307) > >> > >> ~[neo4j-kernel-2.1.6.jar:2.1.6] > >> at > >> > >> org.neo4j.kernel.impl.transaction.xaframework.XaResourceManager.commitWriteTx(XaResourceManager.java:588) > >> > >> ~[neo4j-kernel-2.1.6.jar:2.1.6] > >> at > >> > >> org.neo4j.kernel.impl.transaction.xaframework.XaResourceManager.commit(XaResourceManager.java:533) > >> > >> ~[neo4j-kernel-2.1.6.jar:2.1.6] > >> at > >> > >> org.neo4j.kernel.impl.transaction.xaframework.XaResourceHelpImpl.commit(XaResourceHelpImpl.java:64) > >> > >> ~[neo4j-kernel-2.1.6.jar:2.1.6] > >> at > >> > >> org.neo4j.kernel.impl.transaction.TransactionImpl.doCommit(TransactionImpl.java:548) > >> > >> ~[neo4j-kernel-2.1.6.jar:2.1.6] > >> ... 41 common frames omitted > >> Caused by: java.io.IOException: No space left on device > >> at java.nio.MappedByteBuffer.force0(Native Method) ~[na:1.7.0_75] > >> at java.nio.MappedByteBuffer.force(MappedByteBuffer.java:203) > >> ~[na:1.7.0_75] > >> at > >> > >> org.neo4j.kernel.impl.nioneo.store.MappedPersistenceWindow.force(MappedPersistenceWindow.java:91) > >> > >> ~[neo4j-kernel-2.1.6.jar:2.1.6] > >> at > >> > >> org.neo4j.kernel.impl.nioneo.store.PersistenceWindowPool.flushAll(PersistenceWindowPool.java:319) > >> > >> ~[neo4j-kernel-2.1.6.jar:2.1.6] > >> at > >> > >> org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.flushAll(CommonAbstractStore.java:562) > >> > >> ~[neo4j-kernel-2.1.6.jar:2.1.6] > >> at > >> > >> org.neo4j.kernel.impl.nioneo.store.PropertyStore.flushAll(PropertyStore.java:146) > >> > >> ~[neo4j-kernel-2.1.6.jar:2.1.6] > >> at > >> > >> org.neo4j.kernel.impl.nioneo.store.NeoStore.flushAll(NeoStore.java:315) > >> ~[neo4j-kernel-2.1.6.jar:2.1.6] > >> at > >> > >> org.neo4j.kernel.impl.nioneo.xa.NeoStoreXaDataSource.forceEverything(NeoStoreXaDataSource.java:522) > >> > >> ~[neo4j-kernel-2.1.6.jar:2.1.6] > >> at > >> > >> org.neo4j.kernel.impl.nioneo.xa.NeoStoreXaDataSource.access$1200(NeoStoreXaDataSource.java:119) > >> > >> ~[neo4j-kernel-2.1.6.jar:2.1.6] > >> at > >> > >> org.neo4j.kernel.impl.nioneo.xa.NeoStoreXaDataSource$TransactionFactory.flushAll(NeoStoreXaDataSource.java:592) > >> > >> ~[neo4j-kernel-2.1.6.jar:2.1.6] > >> at > >> > >> org.neo4j.kernel.impl.transaction.xaframework.XaLogicalLog.rotate(XaLogicalLog.java:1167) > >> > >> ~[neo4j-kernel-2.1.6.jar:2.1.6] > >> at > >> > >> org.neo4j.kernel.impl.transaction.xaframework.XaLogicalLog.checkLogRotation(XaLogicalLog.java:532) > >> > >> ~[neo4j-kernel-2.1.6.jar:2.1.6] > >> at > >> > >> org.neo4j.kernel.impl.transaction.xaframework.XaLogicalLog.writeCommand(XaLogicalLog.java:501) > >> > >> ~[neo4j-kernel-2.1.6.jar:2.1.6] > >> at > >> > >> org.neo4j.kernel.impl.transaction.xaframework.XaTransaction.addCommand(XaTransaction.java:250) > >> > >> ~[neo4j-kernel-2.1.6.jar:2.1.6] > >> > >> -- > >> You received this message because you are subscribed to the Google > >> Groups "Neo4j" group. > >> To unsubscribe from this group and stop receiving emails from it, > >> send an email to [email protected] <> <javascript:>. > >> For more options, visit https://groups.google.com/d/optout > >> <https://groups.google.com/d/optout> > >> <https://groups.google.com/d/optout > >> <https://groups.google.com/d/optout>>. > > > > -- > > You received this message because you are subscribed to the Google > > Groups "Neo4j" group. > > To unsubscribe from this group and stop receiving emails from it, send > > an email to [email protected] <javascript:> > > <mailto:[email protected] <javascript:>>. > > For more options, visit https://groups.google.com/d/optout > > <https://groups.google.com/d/optout>. > > > -- > You received this message because you are subscribed to the Google Groups > "Neo4j" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] > <mailto:[email protected]>. > For more options, visit https://groups.google.com/d/optout > <https://groups.google.com/d/optout>. -- You received this message because you are subscribed to the Google Groups "Neo4j" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
