I think you are right. Most likely it is btrfs. Will add the mount option you recommended. Thanks!
On Tuesday, April 14, 2015 at 9:59:59 AM UTC-5, Chris Vest wrote: > > Looks like Btrfs has a history of running into ENOSPC problems. Google > turns up a lot of results where Btrfs runs out of space in situations where > most other file systems would not. > > Also try enabling the `enospc_debug` mount option and see if anything > useful gets written to the syslog next time you run into the problem. > > -- > Chris Vest > System Engineer, Neo Technology > [ skype: mr.chrisvest, twitter: chvest ] > > > On 14 Apr 2015, at 16:00, [email protected] <javascript:> wrote: > > Thanks Christian. > The OS is ubuntu 14.04 and file system for the neo4j storage partition is > btrfs. > The partition size is 1.1TB and 625GB is used. > > > On Tuesday, April 14, 2015 at 2:24:37 AM UTC-5, Christian Morgner wrote: >> >> The "no space left on device" error can occur when there are many very >> small files on your disk, since a file will always consume a certain >> min. amount of space, but the disk usage utilities will report the >> actual size, not the inode size. >> >> What operating system and filesystem type do you use? >> >> >> Best, >> Christian >> >> >> >> >> Am 13.04.2015 um 16:41 schrieb [email protected]: >> > It is really strange. The disk or the partition is nowhere near being >> > full. The open file limits on the system is set to a really high number >> > as well. Any other quota I should check? >> > Any chance when the current logical log is approaching its size limit >> > 25MB, if the last commit is too big, the error is raise? >> > Thanks. >> > >> > On Friday, April 10, 2015 at 8:08:22 AM UTC-5, Chris Vest wrote: >> > >> > “No space left on device” is what the operating system gives us as >> > reason for failing a write. Maybe you are running into quota >> limits, >> > or the particular partition of the drive is getting full? Also note >> > that Neo4j wants to create the next log file and complete the >> > rotation, before it looks for old log files to delete. >> > >> > -- >> > Chris Vest >> > System Engineer, Neo Technology >> > [ skype: mr.chrisvest, twitter: chvest ] >> > >> > >> >> On 09 Apr 2015, at 18:27, [email protected] <javascript:> wrote: >> >> >> >> I am running neo4j 2.1.6 embedded. Under load, I ran into >> >> "org.neo4j.graphdb.TransactionFailureException: Unable to write >> >> command to logical log" several times. The root cause says 'No >> >> space left on device' while there is plenty of disk space >> available. >> >> The logical log file size is the default 25MB and 10 files are >> >> kept. I noticed that file rotation occurred every 2 minutes. >> >> Any idea why this error occurred? Any recommendations for >> >> configure change/tuning? >> >> >> >> Caused by: javax.transaction.xa.XAException: null >> >> at >> >> >> org.neo4j.kernel.impl.transaction.TransactionImpl.doCommit(TransactionImpl.java:560) >> >> >> >> ~[neo4j-kernel-2.1.6.jar:2.1.6] >> >> at >> >> >> org.neo4j.kernel.impl.transaction.TxManager.commit(TxManager.java:448) >> >> ~[neo4j-kernel-2.1.6.jar:2.1.6] >> >> at >> >> >> org.neo4j.kernel.impl.transaction.TxManager.commit(TxManager.java:385) >> >> ~[neo4j-kernel-2.1.6.jar:2.1.6] >> >> at >> >> >> org.neo4j.kernel.impl.transaction.TransactionImpl.commit(TransactionImpl.java:123) >> >> >> >> ~[neo4j-kernel-2.1.6.jar:2.1.6] >> >> at >> >> >> org.neo4j.kernel.impl.transaction.BaseSpringTransactionImpl.commit(BaseSpringTransactionImpl.java:66) >> >> >> >> ~[neo4j-kernel-2.1.6.jar:2.1.6] >> >> at >> >> >> org.springframework.transaction.jta.JtaTransactionManager.doCommit(JtaTransactionManager.java:1011) >> >> >> >> ~[spring-tx-3.2.4.RELEASE.jar:3.2.4.RELEASE] >> >> at >> >> >> org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:755) >> >> >> >> ~[spring-tx-3.2.4.RELEASE.jar:3.2.4.RELEASE] >> >> at >> >> >> org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:724) >> >> >> >> ~[spring-tx-3.2.4.RELEASE.jar:3.2.4.RELEASE] >> >> at >> >> >> org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:475) >> >> >> >> ~[spring-tx-3.2.4.RELEASE.jar:3.2.4.RELEASE] >> >> at >> >> >> org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:270) >> >> >> >> ~[spring-tx-3.2.4.RELEASE.jar:3.2.4.RELEASE] >> >> at >> >> >> org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:94) >> >> >> >> ~[spring-tx-3.2.4.RELEASE.jar:3.2.4.RELEASE] >> >> at >> >> >> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) >> >> >> >> ~[spring-aop-3.2.4.RELEASE.jar:3.2.4.RELEASE] >> >> at >> >> >> com.ryantenney.metrics.spring.MeteredMethodInterceptor.invoke(MeteredMethodInterceptor.java:45) >> >> >> >> ~[metrics-spring-3.0.1.jar:na] >> >> at >> >> >> com.ryantenney.metrics.spring.MeteredMethodInterceptor.invoke(MeteredMethodInterceptor.java:32) >> >> >> >> ~[metrics-spring-3.0.1.jar:na] >> >> at >> >> >> com.ryantenney.metrics.spring.AbstractMetricMethodInterceptor.invoke(AbstractMetricMethodInterceptor.java:59) >> >> >> >> ~[metrics-spring-3.0.1.jar:na] >> >> at >> >> >> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) >> >> >> >> ~[spring-aop-3.2.4.RELEASE.jar:3.2.4.RELEASE] >> >> at >> >> >> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) >> >> >> >> ~[spring-aop-3.2.4.RELEASE.jar:3.2.4.RELEASE] >> >> at com.sun.proxy.$Proxy68.setIntrinsicState(Unknown Source) >> ~[na:na] >> >> at >> >> >> com.zenoss.dsa.impl.StateChangeProcessorImpl.processStateEvents(StateChangeProcessorImpl.java:51) >> >> >> >> ~[classes/:na] >> >> ... 23 common frames omitted >> >> Caused by: org.neo4j.graphdb.TransactionFailureException: Unable >> >> to write command to logical log. >> >> at >> >> >> org.neo4j.kernel.impl.transaction.xaframework.XaTransaction.addCommand(XaTransaction.java:254) >> >> >> >> ~[neo4j-kernel-2.1.6.jar:2.1.6] >> >> at >> >> >> org.neo4j.kernel.impl.nioneo.xa.NeoStoreTransaction.doPrepare(NeoStoreTransaction.java:322) >> >> >> >> ~[neo4j-kernel-2.1.6.jar:2.1.6] >> >> at >> >> >> org.neo4j.kernel.impl.transaction.xaframework.XaTransaction.prepare(XaTransaction.java:307) >> >> >> >> ~[neo4j-kernel-2.1.6.jar:2.1.6] >> >> at >> >> >> org.neo4j.kernel.impl.transaction.xaframework.XaResourceManager.commitWriteTx(XaResourceManager.java:588) >> >> >> >> ~[neo4j-kernel-2.1.6.jar:2.1.6] >> >> at >> >> >> org.neo4j.kernel.impl.transaction.xaframework.XaResourceManager.commit(XaResourceManager.java:533) >> >> >> >> ~[neo4j-kernel-2.1.6.jar:2.1.6] >> >> at >> >> >> org.neo4j.kernel.impl.transaction.xaframework.XaResourceHelpImpl.commit(XaResourceHelpImpl.java:64) >> >> >> >> ~[neo4j-kernel-2.1.6.jar:2.1.6] >> >> at >> >> >> org.neo4j.kernel.impl.transaction.TransactionImpl.doCommit(TransactionImpl.java:548) >> >> >> >> ~[neo4j-kernel-2.1.6.jar:2.1.6] >> >> ... 41 common frames omitted >> >> Caused by: java.io.IOException: No space left on device >> >> at java.nio.MappedByteBuffer.force0(Native Method) ~[na:1.7.0_75] >> >> at java.nio.MappedByteBuffer.force(MappedByteBuffer.java:203) >> >> ~[na:1.7.0_75] >> >> at >> >> >> org.neo4j.kernel.impl.nioneo.store.MappedPersistenceWindow.force(MappedPersistenceWindow.java:91) >> >> >> >> ~[neo4j-kernel-2.1.6.jar:2.1.6] >> >> at >> >> >> org.neo4j.kernel.impl.nioneo.store.PersistenceWindowPool.flushAll(PersistenceWindowPool.java:319) >> >> >> >> ~[neo4j-kernel-2.1.6.jar:2.1.6] >> >> at >> >> >> org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.flushAll(CommonAbstractStore.java:562) >> >> >> >> ~[neo4j-kernel-2.1.6.jar:2.1.6] >> >> at >> >> >> org.neo4j.kernel.impl.nioneo.store.PropertyStore.flushAll(PropertyStore.java:146) >> >> >> >> ~[neo4j-kernel-2.1.6.jar:2.1.6] >> >> at >> >> >> org.neo4j.kernel.impl.nioneo.store.NeoStore.flushAll(NeoStore.java:315) >> >> ~[neo4j-kernel-2.1.6.jar:2.1.6] >> >> at >> >> >> org.neo4j.kernel.impl.nioneo.xa.NeoStoreXaDataSource.forceEverything(NeoStoreXaDataSource.java:522) >> >> >> >> ~[neo4j-kernel-2.1.6.jar:2.1.6] >> >> at >> >> >> org.neo4j.kernel.impl.nioneo.xa.NeoStoreXaDataSource.access$1200(NeoStoreXaDataSource.java:119) >> >> >> >> ~[neo4j-kernel-2.1.6.jar:2.1.6] >> >> at >> >> >> org.neo4j.kernel.impl.nioneo.xa.NeoStoreXaDataSource$TransactionFactory.flushAll(NeoStoreXaDataSource.java:592) >> >> >> >> ~[neo4j-kernel-2.1.6.jar:2.1.6] >> >> at >> >> >> org.neo4j.kernel.impl.transaction.xaframework.XaLogicalLog.rotate(XaLogicalLog.java:1167) >> >> >> >> ~[neo4j-kernel-2.1.6.jar:2.1.6] >> >> at >> >> >> org.neo4j.kernel.impl.transaction.xaframework.XaLogicalLog.checkLogRotation(XaLogicalLog.java:532) >> >> >> >> ~[neo4j-kernel-2.1.6.jar:2.1.6] >> >> at >> >> >> org.neo4j.kernel.impl.transaction.xaframework.XaLogicalLog.writeCommand(XaLogicalLog.java:501) >> >> >> >> ~[neo4j-kernel-2.1.6.jar:2.1.6] >> >> at >> >> >> org.neo4j.kernel.impl.transaction.xaframework.XaTransaction.addCommand(XaTransaction.java:250) >> >> >> >> ~[neo4j-kernel-2.1.6.jar:2.1.6] >> >> >> >> -- >> >> You received this message because you are subscribed to the Google >> >> Groups "Neo4j" group. >> >> To unsubscribe from this group and stop receiving emails from it, >> >> send an email to [email protected] <javascript:>. >> >> For more options, visit https://groups.google.com/d/optout >> >> <https://groups.google.com/d/optout>. >> > >> > -- >> > You received this message because you are subscribed to the Google >> > Groups "Neo4j" group. >> > To unsubscribe from this group and stop receiving emails from it, send >> > an email to [email protected] >> > <mailto:[email protected]>. >> > For more options, visit https://groups.google.com/d/optout. >> >> > -- > You received this message because you are subscribed to the Google Groups > "Neo4j" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] <javascript:>. > For more options, visit https://groups.google.com/d/optout. > > > -- You received this message because you are subscribed to the Google Groups "Neo4j" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
