[ 
https://issues.apache.org/jira/browse/CASSANDRA-14380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16436357#comment-16436357
 ] 

Ariel Weisberg commented on CASSANDRA-14380:
--------------------------------------------

This looks like a bad FD, but I just wanted to drop a note here that we can't 
catch fsync errors and continue so we shouldn't add code that does that in C*. 
There was a Postgres bug where fsync returned an error and the FS lost data, 
but subsequent fsyncs succeeded.

The LastErrorException code in NativeLibrary looks a little janky. What's up 
with that? When would trySync be something we would merely try? If try is good 
enough why do it at all considering try is the default behavior of a series of 
unsynced filesystem operations.

> Cassandra crashes after fsync exception
> ---------------------------------------
>
>                 Key: CASSANDRA-14380
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-14380
>             Project: Cassandra
>          Issue Type: Bug
>            Reporter: Adam Geiger
>            Priority: Critical
>         Attachments: debug.log, debug.log.1.zip, 
> logs-from-cassandra-in-r97bb66e967-apiconnect-cc-0.txt
>
>
> Running Cassandra with a Rook Ceph filesystem within Kubernetes.  During the 
> startup, the following Warnings in the debug log pop up and then Cassandra 
> crashes shortly after and restarts.  It looks like before hitting this error, 
> it is doing a lot of writing and flushing
> WARN [MemtableFlushWriter:2] 2018-04-11 14:34:42,748 NativeLibrary.java:328 - 
> fsync(666) failed, errorno (22) {}
> com.sun.jna.LastErrorException: [22] Invalid argument
>  at org.apache.cassandra.utils.NativeLibraryLinux.fsync(Native Method) 
> ~[apache-cassandra-3.11.0.jar:3.11.0]
>  at 
> org.apache.cassandra.utils.NativeLibraryLinux.callFsync(NativeLibraryLinux.java:107)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
>  at org.apache.cassandra.utils.NativeLibrary.trySync(NativeLibrary.java:317) 
> ~[apache-cassandra-3.11.0.jar:3.11.0]
>  at org.apache.cassandra.utils.SyncUtil.trySync(SyncUtil.java:179) 
> [apache-cassandra-3.11.0.jar:3.11.0]
>  at org.apache.cassandra.utils.SyncUtil.trySyncDir(SyncUtil.java:190) 
> [apache-cassandra-3.11.0.jar:3.11.0]
>  at 
> org.apache.cassandra.io.util.SequentialWriter.openChannel(SequentialWriter.java:107)
>  [apache-cassandra-3.11.0.jar:3.11.0]
>  at 
> org.apache.cassandra.io.util.SequentialWriter.<init>(SequentialWriter.java:141)
>  [apache-cassandra-3.11.0.jar:3.11.0]
>  at 
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.writeMetadata(BigTableWriter.java:402)
>  [apache-cassandra-3.11.0.jar:3.11.0]
>  at 
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.access$300(BigTableWriter.java:53)
>  [apache-cassandra-3.11.0.jar:3.11.0]
>  at 
> org.apache.cassandra.io.sstable.format.big.BigTableWriter$TransactionalProxy.doPrepare(BigTableWriter.java:368)
>  [apache-cassandra-3.11.0.jar:3.11.0]
>  at 
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173)
>  [apache-cassandra-3.11.0.jar:3.11.0]
>  at 
> org.apache.cassandra.io.sstable.format.SSTableWriter.prepareToCommit(SSTableWriter.java:281)
>  [apache-cassandra-3.11.0.jar:3.11.0]
>  at 
> org.apache.cassandra.io.sstable.SimpleSSTableMultiWriter.prepareToCommit(SimpleSSTableMultiWriter.java:101)
>  [apache-cassandra-3.11.0.jar:3.11.0]
>  at 
> org.apache.cassandra.db.ColumnFamilyStore$Flush.flushMemtable(ColumnFamilyStore.java:1153)
>  [apache-cassandra-3.11.0.jar:3.11.0]
>  at 
> org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:1086)
>  [apache-cassandra-3.11.0.jar:3.11.0]
>  at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1160)
>  [na:1.8.0]
>  at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
>  [na:1.8.0]
>  at 
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81)
>  [apache-cassandra-3.11.0.jar:3.11.0]
>  at 
> org.apache.cassandra.concurrent.NamedThreadFactory$$Lambda$12.00000000BCF32600.run(Unknown
>  Source) ~[na:na]
>  at java.lang.Thread.run(Thread.java:811) ~[na:2.9 (12-15-2017)]
>  
> Syslog shows the following 
> (logs-from-cassandra-in-r97bb66e967-apiconnect-cc-0.txt):
> INFO  [main] 2018-04-11 14:49:01,848 ColumnFamilyStore.java:406 - 
> Initializing apim.ur_to_op_by_op
> INFO  [MemoryMXBean notification dispatcher] 2018-04-11 14:49:25,889 
> GCInspector.java:284 - global GC in 206ms.  class storage: 28700680 -> 
> 28692744; miscellaneous non-heap storage: 49871216 -> 53570176; 
> nursery-allocate: 1296878920 -> 149116672; tenured-SOA: 140321968 -> 139143760
> #0: /opt/ibm/java/jre/lib/amd64/compressedrefs/libj9jit29.so(+0x302a94) 
> [0x7f17e4f10a94]
> #1: /opt/ibm/java/jre/lib/amd64/compressedrefs/libj9jit29.so(+0x306b2d) 
> [0x7f17e4f14b2d]
> #2: /opt/ibm/java/jre/lib/amd64/compressedrefs/libj9jit29.so(+0xc82da) 
> [0x7f17e4cd62da]
> #3: /opt/ibm/java/jre/lib/amd64/compressedrefs/libj9prt29.so(+0x22056) 
> [0x7f17e6531056]
> #4: /lib/x86_64-linux-gnu/libpthread.so.0(+0x11390) [0x7f17ed0de390]
> #5: /opt/ibm/java/jre/lib/amd64/compressedrefs/libj9jit29.so(+0x2c4e1f) 
> [0x7f17e4ed2e1f]
> #6: /opt/ibm/java/jre/lib/amd64/compressedrefs/libj9jit29.so(+0x158c04) 
> [0x7f17e4d66c04]
> #7: /opt/ibm/java/jre/lib/amd64/compressedrefs/libj9jit29.so(+0x542d24) 
> [0x7f17e5150d24]
> #8: /opt/ibm/java/jre/lib/amd64/compressedrefs/libj9jit29.so(+0x542e0b) 
> [0x7f17e5150e0b]
> #9: /opt/ibm/java/jre/lib/amd64/compressedrefs/libj9jit29.so(+0x54981a) 
> [0x7f17e515781a]
> #10: /opt/ibm/java/jre/lib/amd64/compressedrefs/libj9jit29.so(+0x5494c8) 
> [0x7f17e51574c8]
> #11: /opt/ibm/java/jre/lib/amd64/compressedrefs/libj9jit29.so(+0x548dd2) 
> [0x7f17e5156dd2]
> #12: /opt/ibm/java/jre/lib/amd64/compressedrefs/libj9jit29.so(+0x2d7019) 
> [0x7f17e4ee5019]
> #13: /opt/ibm/java/jre/lib/amd64/compressedrefs/libj9jit29.so(+0xd31ee) 
> [0x7f17e4ce11ee]
> #14: /opt/ibm/java/jre/lib/amd64/compressedrefs/libj9jit29.so(+0xd3e51) 
> [0x7f17e4ce1e51]
> #15: /opt/ibm/java/jre/lib/amd64/compressedrefs/libj9prt29.so(+0x223e5) 
> [0x7f17e65313e5]
> #16: /opt/ibm/java/jre/lib/amd64/compressedrefs/libj9jit29.so(+0xd5ce5) 
> [0x7f17e4ce3ce5]
> #17: /opt/ibm/java/jre/lib/amd64/compressedrefs/libj9jit29.so(+0xd668b) 
> [0x7f17e4ce468b]
> #18: /opt/ibm/java/jre/lib/amd64/compressedrefs/libj9jit29.so(+0xd6d20) 
> [0x7f17e4ce4d20]
> #19: /opt/ibm/java/jre/lib/amd64/compressedrefs/libj9jit29.so(+0xd6fea) 
> [0x7f17e4ce4fea]
> #20: /opt/ibm/java/jre/lib/amd64/compressedrefs/libj9jit29.so(+0xd70b4) 
> [0x7f17e4ce50b4]
> #21: /opt/ibm/java/jre/lib/amd64/compressedrefs/libj9prt29.so(+0x223e5) 
> [0x7f17e65313e5]
> #22: /opt/ibm/java/jre/lib/amd64/compressedrefs/libj9jit29.so(+0xd743f) 
> [0x7f17e4ce543f]
> #23: /opt/ibm/java/jre/lib/amd64/compressedrefs/libj9thr29.so(+0xa38e) 
> [0x7f17e699738e]
> #24: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba) [0x7f17ed0d46ba]
> #25: function clone+0x6d [0x7f17efdd382d]
> Unhandled exception
> Type=Segmentation error vmState=0x0005ff01
> J9Generic_Signal_Number=00000004 Signal_Number=0000000b Error_Value=00000000 
> Signal_Code=00000001
> Handler1=00007F17E6C26460 Handler2=00007F17E6530E40 
> InaccessibleAddress=000000000000001A
> RDI=00007F17B50070B0 RSI=00007F17C8E51100 RAX=0000000000000012 
> RBX=00007F17B5007120
> RCX=0000000000000000 RDX=00007F17E577BC90 R8=00007F17B5070670 
> R9=0000000000000000
> R10=0000000000000011 R11=00007F17E5459071 R12=00007F17C8E536D8 
> R13=00007F17B5006420
> R14=00007F17B5006290 R15=00007F17C4016CB0
> RIP=00007F17E4ED2E1F GS=0000 FS=0000 RSP=00007F17C8E51150
> EFlags=0000000000010216 CS=E033 RBP=00007F17B5000000 ERR=0000000000000004
> TRAPNO=000000000000000E OLDMASK=0000000000000000 CR2=000000000000001A
> xmm0 0000000000000000 (f: 0.000000, d: 0.000000e+00)
> xmm1 0000000000000000 (f: 0.000000, d: 0.000000e+00)
> xmm2 0000000000000000 (f: 0.000000, d: 0.000000e+00)
> xmm3 0000000000000000 (f: 0.000000, d: 0.000000e+00)
> xmm4 00000000000000ff (f: 255.000000, d: 1.259867e-321)
> xmm5 0000000000000000 (f: 0.000000, d: 0.000000e+00)
> xmm6 0000000000000000 (f: 0.000000, d: 0.000000e+00)
> xmm7 0000000000000000 (f: 0.000000, d: 0.000000e+00)
> xmm8 0000000000000000 (f: 0.000000, d: 0.000000e+00)
> xmm9 0000000000000000 (f: 0.000000, d: 0.000000e+00)
> xmm10 0000000000000000 (f: 0.000000, d: 0.000000e+00)
> xmm11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
> xmm12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
> xmm13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
> xmm14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
> xmm15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
> Module=/opt/ibm/java/jre/lib/amd64/compressedrefs/libj9jit29.so
> Module_base_address=00007F17E4C0E000
> Method_being_compiled=org/apache/cassandra/utils/concurrent/Locks.monitorExitUnsafe(Ljava/lang/Object;)V
> Target=2_90_20171215_373586 (Linux 4.4.0-116-generic)
> CPU=amd64 (8 logical CPUs) (0xbc2d25000 RAM)
> ----------- Stack Backtrace -----------
> (0x00007F17E4ED2E1F [libj9jit29.so+0x2c4e1f])
> (0x00007F17E4D66C04 [libj9jit29.so+0x158c04])
> (0x00007F17E5150D24 [libj9jit29.so+0x542d24])
> (0x00007F17E5150E0B [libj9jit29.so+0x542e0b])
> (0x00007F17E515781A [libj9jit29.so+0x54981a])
> (0x00007F17E51574C8 [libj9jit29.so+0x5494c8])
> (0x00007F17E5156DD2 [libj9jit29.so+0x548dd2])
> (0x00007F17E4EE5019 [libj9jit29.so+0x2d7019])
> (0x00007F17E4CE11EE [libj9jit29.so+0xd31ee])
> (0x00007F17E4CE1E51 [libj9jit29.so+0xd3e51])
> (0x00007F17E65313E5 [libj9prt29.so+0x223e5])
> (0x00007F17E4CE3CE5 [libj9jit29.so+0xd5ce5])
> (0x00007F17E4CE468B [libj9jit29.so+0xd668b])
> (0x00007F17E4CE4D20 [libj9jit29.so+0xd6d20])
> (0x00007F17E4CE4FEA [libj9jit29.so+0xd6fea])
> (0x00007F17E4CE50B4 [libj9jit29.so+0xd70b4])
> (0x00007F17E65313E5 [libj9prt29.so+0x223e5])
> (0x00007F17E4CE543F [libj9jit29.so+0xd743f])
> (0x00007F17E699738E [libj9thr29.so+0xa38e])
> (0x00007F17ED0D46BA [libpthread.so.0+0x76ba])
> clone+0x6d (0x00007F17EFDD382D [libc.so.6+0x10682d])
> ---------------------------------------
> JVMDUMP039I Processing dump event "gpf", detail "" at 2018/04/11 14:49:42 - 
> please wait.
> JVMDUMP032I JVM requested System dump using 
> '//core.20180411.144942.84.0001.dmp' in response to an event
> JVMDUMP030W Cannot write dump to file //core.20180411.144942.84.0001.dmp: 
> Permission denied
> #0: /opt/ibm/java/jre/lib/amd64/compressedrefs/libj9jit29.so(+0x302a94) 
> [0x7f17e4f10a94]
> #1: /opt/ibm/java/jre/lib/amd64/compressedrefs/libj9jit29.so(+0x306b2d) 
> [0x7f17e4f14b2d]
> #2: /opt/ibm/java/jre/lib/amd64/compressedrefs/libj9jit29.so(+0xc82da) 
> [0x7f17e4cd62da]
> #3: /opt/ibm/java/jre/lib/amd64/compressedrefs/libj9prt29.so(+0x22056) 
> [0x7f17e6531056]
> #4: /lib/x86_64-linux-gnu/libpthread.so.0(+0x11390) [0x7f17ed0de390]
> #5: /opt/ibm/java/jre/lib/amd64/compressedrefs/libj9jit29.so(+0x2c4e1f) 
> [0x7f17e4ed2e1f]
> #6: /opt/ibm/java/jre/lib/amd64/compressedrefs/libj9jit29.so(+0x158c04) 
> [0x7f17e4d66c04]
> #7: /opt/ibm/java/jre/lib/amd64/compressedrefs/libj9jit29.so(+0x542d24) 
> [0x7f17e5150d24]
> #8: /opt/ibm/java/jre/lib/amd64/compressedrefs/libj9jit29.so(+0x542e0b) 
> [0x7f17e5150e0b]
> #9: /opt/ibm/java/jre/lib/amd64/compressedrefs/libj9jit29.so(+0x54981a) 
> [0x7f17e515781a]
> #10: /opt/ibm/java/jre/lib/amd64/compressedrefs/libj9jit29.so(+0x5494c8) 
> [0x7f17e51574c8]
> #11: /opt/ibm/java/jre/lib/amd64/compressedrefs/libj9jit29.so(+0x548dd2) 
> [0x7f17e5156dd2]
> #12: /opt/ibm/java/jre/lib/amd64/compressedrefs/libj9jit29.so(+0x2d7019) 
> [0x7f17e4ee5019]
> #13: /opt/ibm/java/jre/lib/amd64/compressedrefs/libj9jit29.so(+0xd31ee) 
> [0x7f17e4ce11ee]
> #14: /opt/ibm/java/jre/lib/amd64/compressedrefs/libj9jit29.so(+0xd3e51) 
> [0x7f17e4ce1e51]
> #15: /opt/ibm/java/jre/lib/amd64/compressedrefs/libj9prt29.so(+0x223e5) 
> [0x7f17e65313e5]
> #16: /opt/ibm/java/jre/lib/amd64/compressedrefs/libj9jit29.so(+0xd5ce5) 
> [0x7f17e4ce3ce5]
> #17: /opt/ibm/java/jre/lib/amd64/compressedrefs/libj9jit29.so(+0xd668b) 
> [0x7f17e4ce468b]
> #18: /opt/ibm/java/jre/lib/amd64/compressedrefs/libj9jit29.so(+0xd6d20) 
> [0x7f17e4ce4d20]
> #19: /opt/ibm/java/jre/lib/amd64/compressedrefs/libj9jit29.so(+0xd6fea) 
> [0x7f17e4ce4fea]
> #20: /opt/ibm/java/jre/lib/amd64/compressedrefs/libj9jit29.so(+0xd70b4) 
> [0x7f17e4ce50b4]
> #21: /opt/ibm/java/jre/lib/amd64/compressedrefs/libj9prt29.so(+0x223e5) 
> [0x7f17e65313e5]
> #22: /opt/ibm/java/jre/lib/amd64/compressedrefs/libj9jit29.so(+0xd743f) 
> [0x7f17e4ce543f]
> #23: /opt/ibm/java/jre/lib/amd64/compressedrefs/libj9thr29.so(+0xa38e) 
> [0x7f17e699738e]
> #24: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba) [0x7f17ed0d46ba]
> #25: function clone+0x6d [0x7f17efdd382d]
> Unhandled exception
> Type=Segmentation error vmState=0x0005ff01
> J9Generic_Signal_Number=00000004 Signal_Number=0000000b Error_Value=00000000 
> Signal_Code=00000001
> Handler1=00007F17E6C26460 Handler2=00007F17E6530E40 
> InaccessibleAddress=000000000000001A
> RDI=00007F17B1BF70B0 RSI=00007F17C8D50100 RAX=0000000000000012 
> RBX=00007F17B1BF7120
> RCX=0000000000000000 RDX=00007F17E577BC90 R8=00007F17B1C60670 
> R9=00007F17C8D50300
> R10=0000000000000011 R11=00007F17E5459071 R12=00007F17C8D526D8 
> R13=00007F17B1BF6420
> R14=00007F17B1BF6290 R15=00007F17BC0270A0
> RIP=00007F17E4ED2E1F GS=0000 FS=0000 RSP=00007F17C8D50150
> EFlags=0000000000010216 CS=E033 RBP=00007F17B1BF0000 ERR=0000000000000004
> TRAPNO=000000000000000E OLDMASK=0000000000000000 CR2=000000000000001A
> xmm0 0000000000000000 (f: 0.000000, d: 0.000000e+00)
> xmm1 0000000000000000 (f: 0.000000, d: 0.000000e+00)
> xmm2 0000000000000000 (f: 0.000000, d: 0.000000e+00)
> xmm3 0000000000000000 (f: 0.000000, d: 0.000000e+00)
> xmm4 00000000000000ff (f: 255.000000, d: 1.259867e-321)
> xmm5 0000000000000000 (f: 0.000000, d: 0.000000e+00)
> xmm6 0000000000000000 (f: 0.000000, d: 0.000000e+00)
> xmm7 0000000000000000 (f: 0.000000, d: 0.000000e+00)
> xmm8 0000000000000000 (f: 0.000000, d: 0.000000e+00)
> xmm9 0000000000000000 (f: 0.000000, d: 0.000000e+00)
> xmm10 0000000000000000 (f: 0.000000, d: 0.000000e+00)
> xmm11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
> xmm12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
> xmm13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
> xmm14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
> xmm15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
> Module=/opt/ibm/java/jre/lib/amd64/compressedrefs/libj9jit29.so
> Module_base_address=00007F17E4C0E000
> Method_being_compiled=org/apache/cassandra/utils/concurrent/Locks.monitorEnterUnsafe(Ljava/lang/Object;)V
> Target=2_90_20171215_373586 (Linux 4.4.0-116-generic)
> CPU=amd64 (8 logical CPUs) (0xbc2d25000 RAM)
> ----------- Stack Backtrace -----------
> (0x00007F17E4ED2E1F [libj9jit29.so+0x2c4e1f])
> (0x00007F17E4D66C04 [libj9jit29.so+0x158c04])
> (0x00007F17E5150D24 [libj9jit29.so+0x542d24])
> (0x00007F17E5150E0B [libj9jit29.so+0x542e0b])
> (0x00007F17E515781A [libj9jit29.so+0x54981a])
> (0x00007F17E51574C8 [libj9jit29.so+0x5494c8])
> (0x00007F17E5156DD2 [libj9jit29.so+0x548dd2])
> (0x00007F17E4EE5019 [libj9jit29.so+0x2d7019])
> (0x00007F17E4CE11EE [libj9jit29.so+0xd31ee])
> (0x00007F17E4CE1E51 [libj9jit29.so+0xd3e51])
> (0x00007F17E65313E5 [libj9prt29.so+0x223e5])
> (0x00007F17E4CE3CE5 [libj9jit29.so+0xd5ce5])
> (0x00007F17E4CE468B [libj9jit29.so+0xd668b])
> (0x00007F17E4CE4D20 [libj9jit29.so+0xd6d20])
> (0x00007F17E4CE4FEA [libj9jit29.so+0xd6fea])
> (0x00007F17E4CE50B4 [libj9jit29.so+0xd70b4])
> (0x00007F17E65313E5 [libj9prt29.so+0x223e5])
> (0x00007F17E4CE543F [libj9jit29.so+0xd743f])
> (0x00007F17E699738E [libj9thr29.so+0xa38e])
> (0x00007F17ED0D46BA [libpthread.so.0+0x76ba])
> clone+0x6d (0x00007F17EFDD382D [libc.so.6+0x10682d])
> ---------------------------------------
> JVMDUMP039I Processing dump event "gpf", detail "" at 2018/04/11 14:49:44 - 
> please wait.
> JVMPORT030W /proc/sys/kernel/core_pattern setting "|/usr/share/apport/apport 
> %p %s %c %d %P" specifies that the core dump is to be piped to an external 
> program.  Attempting to rename either core or core.1770.
> JVMDUMP012E Error in System dump: The core file created by child process with 
> pid = 1770 was not found. Expected to find core file with name "/tmp/core"
> JVMDUMP030W Cannot write dump to file //javacore.20180411.144942.84.0003.txt: 
> Permission denied
> JVMDUMP032I JVM requested Java dump using 
> '/tmp/javacore.20180411.144942.84.0003.txt' in response to an event
> JVMDUMP032I JVM requested System dump using 
> '//core.20180411.144944.84.0002.dmp' in response to an event
> JVMDUMP030W Cannot write dump to file //core.20180411.144944.84.0002.dmp: 
> Permission denied
> JVMDUMP010I Java dump written to /tmp/javacore.20180411.144942.84.0003.txt
> ERROR [MemtableFlushWriter:1] 2018-04-11 14:49:51,747 LogTransaction.java:257 
> - Transaction log [mc_txn_flush_95f3d0a0-3d97-11e8-9d33-2f3add552a16.log in 
> /var/db/data/data/apim/org_by_owner_url-e1f6b6b03ccb11e887040d4467579f5c] 
> indicates txn was not completed, trying to abort it now
> ERROR [MemtablePostFlush:1] 2018-04-11 14:49:51,816 CassandraDaemon.java:228 
> - Exception in thread Thread[MemtablePostFlush:1,5,main]
> java.lang.RuntimeException: java.io.IOException: Bad file descriptor
>       at org.apache.cassandra.io.util.FileUtils.write(FileUtils.java:588) 
> ~[apache-cassandra-3.11.0.jar:3.11.0]
>       at 
> org.apache.cassandra.io.util.FileUtils.appendAndSync(FileUtils.java:567) 
> ~[apache-cassandra-3.11.0.jar:3.11.0]
>       at 
> org.apache.cassandra.db.lifecycle.LogReplica.append(LogReplica.java:85) 
> ~[apache-cassandra-3.11.0.jar:3.11.0]
>       at 
> org.apache.cassandra.db.lifecycle.LogReplicaSet.lambda$null$5(LogReplicaSet.java:210)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
>       at 
> org.apache.cassandra.db.lifecycle.LogReplicaSet$$Lambda$155.00000000BC4477D0.perform(Unknown
>  Source) ~[na:na]
>       at org.apache.cassandra.utils.Throwables.perform(Throwables.java:113) 
> ~[apache-cassandra-3.11.0.jar:3.11.0]
>       at org.apache.cassandra.utils.Throwables.perform(Throwables.java:103) 
> ~[apache-cassandra-3.11.0.jar:3.11.0]
>       at 
> org.apache.cassandra.db.lifecycle.LogReplicaSet.append(LogReplicaSet.java:210)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
>       at 
> org.apache.cassandra.db.lifecycle.LogFile.addRecord(LogFile.java:303) 
> ~[apache-cassandra-3.11.0.jar:3.11.0]
>       at org.apache.cassandra.db.lifecycle.LogFile.commit(LogFile.java:250) 
> ~[apache-cassandra-3.11.0.jar:3.11.0]
>       at 
> org.apache.cassandra.db.lifecycle.LogTransaction$$Lambda$169.00000000BC44FA10.perform(Unknown
>  Source) ~[na:na]
>       at org.apache.cassandra.utils.Throwables.perform(Throwables.java:113) 
> ~[apache-cassandra-3.11.0.jar:3.11.0]
>       at org.apache.cassandra.utils.Throwables.perform(Throwables.java:103) 
> ~[apache-cassandra-3.11.0.jar:3.11.0]
>       at org.apache.cassandra.utils.Throwables.perform(Throwables.java:98) 
> ~[apache-cassandra-3.11.0.jar:3.11.0]
>       at 
> org.apache.cassandra.db.lifecycle.LogTransaction.doCommit(LogTransaction.java:389)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
>       at 
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.commit(Transactional.java:116)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
>       at 
> org.apache.cassandra.db.lifecycle.LifecycleTransaction.doCommit(LifecycleTransaction.java:218)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
>       at 
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.commit(Transactional.java:116)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
>       at 
> org.apache.cassandra.db.ColumnFamilyStore$Flush.flushMemtable(ColumnFamilyStore.java:1176)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
>       at 
> org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:1086)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
>       at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1160)
>  ~[na:1.8.0]
>       at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
>  [na:1.8.0]
>       at 
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81)
>  [apache-cassandra-3.11.0.jar:3.11.0]
>       at 
> org.apache.cassandra.concurrent.NamedThreadFactory$$Lambda$12.00000000E0F32EF0.run(Unknown
>  Source) ~[na:na]
>       at java.lang.Thread.run(Thread.java:811) ~[na:2.9 (12-15-2017)]
>       Suppressed: java.lang.IllegalStateException: Attempted to abort a 
> committed operation
>               at 
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.abort(Transactional.java:133)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
>               at 
> org.apache.cassandra.db.lifecycle.LifecycleTransaction.doAbort(LifecycleTransaction.java:243)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
>               at 
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.abort(Transactional.java:144)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
>               at 
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.abort(Transactional.java:193)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
>               at 
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.close(Transactional.java:158)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
>               at 
> org.apache.cassandra.db.ColumnFamilyStore$Flush.flushMemtable(ColumnFamilyStore.java:1193)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
>               ... 6 common frames omitted
> Caused by: java.io.IOException: Bad file descriptor
>       at sun.nio.ch.FileDispatcherImpl.close0(Native Method) ~[na:1.8.0]
>       at sun.nio.ch.FileDispatcherImpl.close(FileDispatcherImpl.java:127) 
> ~[na:1.8.0]
>       at 
> sun.nio.ch.FileChannelImpl.implCloseChannel(FileChannelImpl.java:153) 
> ~[na:1.8.0]
>       at 
> java.nio.channels.spi.AbstractInterruptibleChannel.close(AbstractInterruptibleChannel.java:126)
>  ~[na:1.8.0]
>       at java.nio.channels.Channels$1.close(Channels.java:189) ~[na:1.8.0]
>       at sun.nio.cs.StreamEncoder.implClose(StreamEncoder.java:332) 
> ~[na:1.8.0]
>       at sun.nio.cs.StreamEncoder.close(StreamEncoder.java:161) ~[na:1.8.0]
>       at java.io.OutputStreamWriter.close(OutputStreamWriter.java:295) 
> ~[na:1.8.0]
>       at java.io.BufferedWriter.close(BufferedWriter.java:277) ~[na:1.8.0]
>       at java.nio.file.Files.write(Files.java:3368) ~[na:1.8.0]
>       at org.apache.cassandra.io.util.FileUtils.write(FileUtils.java:581) 
> ~[apache-cassandra-3.11.0.jar:3.11.0]
>       ... 24 common frames omitted
> JVMPORT030W /proc/sys/kernel/core_pattern setting "|/usr/share/apport/apport 
> %p %s %c %d %P" specifies that the core dump is to be piped to an external 
> program.  Attempting to rename either core or core.1781.
> JVMDUMP012E Error in System dump: The core file created by child process with 
> pid = 1781 was not found. Expected to find core file with name "/tmp/core"
> JVMDUMP030W Cannot write dump to file //javacore.20180411.144944.84.0005.txt: 
> Permission denied
> JVMDUMP032I JVM requested Java dump using 
> '/tmp/javacore.20180411.144944.84.0005.txt' in response to an event
> JVMDUMP032I JVM requested Snap dump using 
> '//Snap.20180411.144942.84.0004.trc' in response to an event
> JVMDUMP030W Cannot write dump to file //Snap.20180411.144942.84.0004.trc: 
> Permission denied
> JVMDUMP010I Snap dump written to /tmp/Snap.20180411.144942.84.0004.trc
> JVMDUMP030W Cannot write dump to file //jitdump.20180411.144942.84.0006.dmp: 
> Permission denied
> JVMDUMP007I 
> #JITDUMP:  vmThread=0000000001B3B800 Recursive crash occurred. Aborting JIT 
> dump.JVMDUMP010I JIT dump written to /tmp/jitdump.20180411.144942.84.0006.dmp
> WARN  [MemtableFlushWriter:1] 2018-04-11 14:49:58,241 NativeLibrary.java:328 
> - fsync(3084) failed, errorno (22) {}
> com.sun.jna.LastErrorException: [22] Invalid argument
>       at org.apache.cassandra.utils.NativeLibraryLinux.fsync(Native Method) 
> ~[apache-cassandra-3.11.0.jar:3.11.0]
>       at 
> org.apache.cassandra.utils.NativeLibraryLinux.callFsync(NativeLibraryLinux.java:107)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
>       at 
> org.apache.cassandra.utils.NativeLibrary.trySync(NativeLibrary.java:317) 
> ~[apache-cassandra-3.11.0.jar:3.11.0]
>       at 
> org.apache.cassandra.db.lifecycle.LogReplica.syncDirectory(LogReplica.java:96)
>  [apache-cassandra-3.11.0.jar:3.11.0]
>       at 
> org.apache.cassandra.db.lifecycle.LogReplica.delete(LogReplica.java:102) 
> [apache-cassandra-3.11.0.jar:3.11.0]
>       at 
> org.apache.cassandra.db.lifecycle.LogReplicaSet$$Lambda$175.00000000BC451FC0.perform(Unknown
>  Source) ~[na:na]
>       at org.apache.cassandra.utils.Throwables.perform(Throwables.java:113) 
> ~[apache-cassandra-3.11.0.jar:3.11.0]
>       at org.apache.cassandra.utils.Throwables.perform(Throwables.java:103) 
> ~[apache-cassandra-3.11.0.jar:3.11.0]
>       at 
> org.apache.cassandra.db.lifecycle.LogReplicaSet.delete(LogReplicaSet.java:92) 
> ~[apache-cassandra-3.11.0.jar:3.11.0]
>       at 
> org.apache.cassandra.db.lifecycle.LogFile.removeUnfinishedLeftovers(LogFile.java:128)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
>       at 
> org.apache.cassandra.db.lifecycle.LogTransaction$TransactionTidier.run(LogTransaction.java:263)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
>       at 
> org.apache.cassandra.db.lifecycle.LogTransaction$TransactionTidier.tidy(LogTransaction.java:242)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
>       at 
> org.apache.cassandra.utils.concurrent.Ref$GlobalState.release(Ref.java:321) 
> ~[apache-cassandra-3.11.0.jar:3.11.0]
>       at 
> org.apache.cassandra.utils.concurrent.Ref$State.ensureReleased(Ref.java:199) 
> ~[apache-cassandra-3.11.0.jar:3.11.0]
>       at 
> org.apache.cassandra.utils.concurrent.Ref.ensureReleased(Ref.java:119) 
> ~[apache-cassandra-3.11.0.jar:3.11.0]
>       at 
> org.apache.cassandra.db.lifecycle.LogTransaction.complete(LogTransaction.java:377)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
>       at 
> org.apache.cassandra.db.lifecycle.LogTransaction.doCommit(LogTransaction.java:389)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
>       at 
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.commit(Transactional.java:116)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
>       at 
> org.apache.cassandra.db.lifecycle.LifecycleTransaction.doCommit(LifecycleTransaction.java:218)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
>       at 
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.commit(Transactional.java:116)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
>       at 
> org.apache.cassandra.db.ColumnFamilyStore$Flush.flushMemtable(ColumnFamilyStore.java:1176)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
>       at 
> org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:1086)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
>       at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1160)
>  ~[na:1.8.0]
>       at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
>  ~[na:1.8.0]
>       at 
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81)
>  ~[apache-cassandra-3.11.0.jar:3.11.0]
>       at 
> org.apache.cassandra.concurrent.NamedThreadFactory$$Lambda$12.00000000E0F32EF0.run(Unknown
>  Source) ~[na:na]
>       at java.lang.Thread.run(Thread.java:811) ~[na:2.9 (12-15-2017)]
> JVMDUMP013I Processed dump event "gpf", detail "".
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

Reply via email to