[
https://issues.apache.org/jira/browse/CASSANDRA-5059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13529830#comment-13529830
]
Jason Harvey edited comment on CASSANDRA-5059 at 12/12/12 11:07 AM:
--------------------------------------------------------------------
I was just able to reproduce this in testing by upgrading a node with a single
size-tiered compressed CF copied from our production ring. I also attempted to
run upgradesstables on that CF after upgrading, and the following exception
occurred:
{code}
Error occured while upgrading the sstables for keyspace reddit
java.util.concurrent.ExecutionException: java.io.IOError: java.io.IOException:
Bad file descriptor
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
at java.util.concurrent.FutureTask.get(FutureTask.java:83)
at
org.apache.cassandra.db.compaction.CompactionManager.performAllSSTableOperation(CompactionManager.java:218)
at
org.apache.cassandra.db.compaction.CompactionManager.performSSTableRewrite(CompactionManager.java:234)
at
org.apache.cassandra.db.ColumnFamilyStore.sstablesRewrite(ColumnFamilyStore.java:983)
at
org.apache.cassandra.service.StorageService.upgradeSSTables(StorageService.java:1788)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93)
at
com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27)
at
com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208)
at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:120)
at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:262)
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836)
at
com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761)
at
javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1427)
at
javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72)
at
javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1265)
at
javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1360)
at
javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:788)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:305)
at sun.rmi.transport.Transport$1.run(Transport.java:159)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
at
sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOError: java.io.IOException: Bad file descriptor
at org.apache.cassandra.utils.MergeIterator.close(MergeIterator.java:65)
at
org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:195)
at
org.apache.cassandra.db.compaction.CompactionManager$4.perform(CompactionManager.java:246)
at
org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:197)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
... 3 more
Caused by: java.io.IOException: Bad file descriptor
at sun.nio.ch.FileDispatcher.preClose0(Native Method)
at sun.nio.ch.FileDispatcher.preClose(FileDispatcher.java:59)
at sun.nio.ch.FileChannelImpl.implCloseChannel(FileChannelImpl.java:96)
at
java.nio.channels.spi.AbstractInterruptibleChannel.close(AbstractInterruptibleChannel.java:97)
at java.io.FileInputStream.close(FileInputStream.java:258)
at
org.apache.cassandra.io.compress.CompressedRandomAccessReader.close(CompressedRandomAccessReader.java:131)
at sun.nio.ch.FileChannelImpl.implCloseChannel(FileChannelImpl.java:121)
at
java.nio.channels.spi.AbstractInterruptibleChannel.close(AbstractInterruptibleChannel.java:97)
at java.io.RandomAccessFile.close(RandomAccessFile.java:541)
at
org.apache.cassandra.io.util.RandomAccessReader.close(RandomAccessReader.java:224)
at
org.apache.cassandra.io.compress.CompressedRandomAccessReader.close(CompressedRandomAccessReader.java:130)
at
org.apache.cassandra.io.sstable.SSTableScanner.close(SSTableScanner.java:89)
at org.apache.cassandra.utils.MergeIterator.close(MergeIterator.java:61)
... 8 more
{code}
was (Author: alienth):
I was just able to reproduce this in testing by upgrading a node with a
single size-tiered compressed CF. I also attempted to run upgradesstables on
that CF after upgrading, and the following exception occurred:
{code}
Error occured while upgrading the sstables for keyspace reddit
java.util.concurrent.ExecutionException: java.io.IOError: java.io.IOException:
Bad file descriptor
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
at java.util.concurrent.FutureTask.get(FutureTask.java:83)
at
org.apache.cassandra.db.compaction.CompactionManager.performAllSSTableOperation(CompactionManager.java:218)
at
org.apache.cassandra.db.compaction.CompactionManager.performSSTableRewrite(CompactionManager.java:234)
at
org.apache.cassandra.db.ColumnFamilyStore.sstablesRewrite(ColumnFamilyStore.java:983)
at
org.apache.cassandra.service.StorageService.upgradeSSTables(StorageService.java:1788)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93)
at
com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27)
at
com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208)
at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:120)
at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:262)
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836)
at
com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761)
at
javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1427)
at
javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72)
at
javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1265)
at
javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1360)
at
javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:788)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:305)
at sun.rmi.transport.Transport$1.run(Transport.java:159)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
at
sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOError: java.io.IOException: Bad file descriptor
at org.apache.cassandra.utils.MergeIterator.close(MergeIterator.java:65)
at
org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:195)
at
org.apache.cassandra.db.compaction.CompactionManager$4.perform(CompactionManager.java:246)
at
org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:197)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
... 3 more
Caused by: java.io.IOException: Bad file descriptor
at sun.nio.ch.FileDispatcher.preClose0(Native Method)
at sun.nio.ch.FileDispatcher.preClose(FileDispatcher.java:59)
at sun.nio.ch.FileChannelImpl.implCloseChannel(FileChannelImpl.java:96)
at
java.nio.channels.spi.AbstractInterruptibleChannel.close(AbstractInterruptibleChannel.java:97)
at java.io.FileInputStream.close(FileInputStream.java:258)
at
org.apache.cassandra.io.compress.CompressedRandomAccessReader.close(CompressedRandomAccessReader.java:131)
at sun.nio.ch.FileChannelImpl.implCloseChannel(FileChannelImpl.java:121)
at
java.nio.channels.spi.AbstractInterruptibleChannel.close(AbstractInterruptibleChannel.java:97)
at java.io.RandomAccessFile.close(RandomAccessFile.java:541)
at
org.apache.cassandra.io.util.RandomAccessReader.close(RandomAccessReader.java:224)
at
org.apache.cassandra.io.compress.CompressedRandomAccessReader.close(CompressedRandomAccessReader.java:130)
at
org.apache.cassandra.io.sstable.SSTableScanner.close(SSTableScanner.java:89)
at org.apache.cassandra.utils.MergeIterator.close(MergeIterator.java:61)
... 8 more
{code}
> 1.0.11 -> 1.1.7 upgrade results in Bad file descriptor exception
> ----------------------------------------------------------------
>
> Key: CASSANDRA-5059
> URL: https://issues.apache.org/jira/browse/CASSANDRA-5059
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.1.7
> Environment: ubuntu
> sun-java6 6.24-1build0.10.10.1
> Reporter: Jason Harvey
>
> Upgraded a single node in my ring to 1.1.7. Upgrade process went normally
> with no errors. However, as soon as the node joined the ring, it started
> spewing this exception hundreds of times a second:
> {code}
> WARN [ReadStage:22] 2012-12-12 02:00:56,181 FileUtils.java (line 116) Failed
> closing org.apache.cassandra.db.columniterator.SSTableSliceIterator@5959baa2
> java.io.IOException: Bad file descriptor
> at sun.nio.ch.FileDispatcher.preClose0(Native Method)
> at sun.nio.ch.FileDispatcher.preClose(FileDispatcher.java:59)
> at
> sun.nio.ch.FileChannelImpl.implCloseChannel(FileChannelImpl.java:96)
> at
> java.nio.channels.spi.AbstractInterruptibleChannel.close(AbstractInterruptibleChannel.java:97)
> at java.io.FileInputStream.close(FileInputStream.java:258)
> at
> org.apache.cassandra.io.compress.CompressedRandomAccessReader.close(CompressedRandomAccessReader.java:131)
> at
> sun.nio.ch.FileChannelImpl.implCloseChannel(FileChannelImpl.java:121)
> at
> java.nio.channels.spi.AbstractInterruptibleChannel.close(AbstractInterruptibleChannel.java:97)
> at java.io.RandomAccessFile.close(RandomAccessFile.java:541)
> at
> org.apache.cassandra.io.util.RandomAccessReader.close(RandomAccessReader.java:224)
> at
> org.apache.cassandra.io.compress.CompressedRandomAccessReader.close(CompressedRandomAccessReader.java:130)
> at
> org.apache.cassandra.db.columniterator.SSTableSliceIterator.close(SSTableSliceIterator.java:132)
> at
> org.apache.cassandra.io.util.FileUtils.closeQuietly(FileUtils.java:112)
> at
> org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:300)
> at
> org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:64)
> at
> org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1347)
> at
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1209)
> at
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1144)
> at org.apache.cassandra.db.Table.getRow(Table.java:378)
> at
> org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:69)
> at
> org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:51)
> at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {code}
> The node was not responding to reads on any CFs, so I was forced to do an
> emergency roll-back and abandon the upgrade.
> Node has roughly 3800 sstables. Both LCS and SizeTiered, as well as
> compressed and uncompressed CFs. Looks like the exception might have
> something to do with compression?
> Verified that the service was not bumping into any open file descriptor
> limitations.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
