[jira] [Updated] (CASSANDRA-14308) Remove invalid SSTables from interrupted compaction

2018-03-11 Thread Jay Zhuang (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jay Zhuang updated CASSANDRA-14308:
---
Description: 
If the JVM crash while compaction is in progress, the incompleted SSTable won't 
be cleaned up, which causes "{{Stats component is missing for sstable}}" error 
in the startup log:
{noformat}
ERROR [SSTableBatchOpen:3] 2018-03-11 00:17:35,597 CassandraDaemon.java:207 - 
Exception in thread Thread[SSTableBatchOpen:3,5,main]
java.lang.AssertionError: Stats component is missing for sstable 
/cassandra/data/keyspace/table-id/mc-12345-big
at 
org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:458)
 ~[apache-cassandra-3.0.14.jar:3.0.14]
at 
org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:374)
 ~[apache-cassandra-3.0.14.jar:3.0.14]
at 
org.apache.cassandra.io.sstable.format.SSTableReader$4.run(SSTableReader.java:533)
 ~[apache-cassandra-3.0.14.jar:3.0.14]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
~[na:1.8.0_121]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_121]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
~[na:1.8.0_121]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
[na:1.8.0_121]
at 
org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79)
 [apache-cassandra-3.0.14.jar:3.0.14]
at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_121]
{noformat}

The accumulated incompleted SSTables could take lots of space, especially for 
STCS, which could have very large SSTables.
Here is the script we use to delete the SSTables after node is restarted:
{noformat}
grep 'Stats component is missing for sstable' $SYSTEM_LOG | awk '{print $8}' > 
~/invalid_sstables ; for ss in `cat ~/invalid_sstables`; do echo == $ss; ll 
$ss*; sudo rm $ss* ; done
{noformat}
I would suggest to remove these incompleted SSTables while startup.

  was:
If the JVM crash while compaction is in progress, the incompleted SSTable won't 
be cleaned up, which causes "{{Stats component is missing for sstable}}" error 
in the start up log:
{noformat}
ERROR [SSTableBatchOpen:3] 2018-03-11 00:17:35,597 CassandraDaemon.java:207 - 
Exception in thread Thread[SSTableBatchOpen:3,5,main]
java.lang.AssertionError: Stats component is missing for sstable 
/cassandra/data/keyspace/table-id/mc-12345-big
at 
org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:458)
 ~[apache-cassandra-3.0.14.jar:3.0.14]
at 
org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:374)
 ~[apache-cassandra-3.0.14.jar:3.0.14]
at 
org.apache.cassandra.io.sstable.format.SSTableReader$4.run(SSTableReader.java:533)
 ~[apache-cassandra-3.0.14.jar:3.0.14]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
~[na:1.8.0_121]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_121]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
~[na:1.8.0_121]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
[na:1.8.0_121]
at 
org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79)
 [apache-cassandra-3.0.14.jar:3.0.14]
at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_121]
{noformat}

The accumulated incompleted SSTables could take lots of space, especially for 
STCS, which could have very large SSTables.
Here is the script we use to delete the SSTables after node is restarted:
{noformat}
grep 'Stats component is missing for sstable' $SYSTEM_LOG | awk '{print $8}' > 
~/invalid_sstables ; for ss in `cat ~/invalid_sstables`; do echo == $ss; ll 
$ss*; sudo rm $ss* ; done
{noformat}
I would suggest to remove these incompleted SSTables while startup.


> Remove invalid SSTables from interrupted compaction
> ---
>
> Key: CASSANDRA-14308
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14308
> Project: Cassandra
>  Issue Type: Bug
>  Components: Compaction
>Reporter: Jay Zhuang
>Assignee: Jay Zhuang
>Priority: Minor
>
> If the JVM crash while compaction is in progress, the incompleted SSTable 
> won't be cleaned up, which causes "{{Stats component is missing for 
> sstable}}" error in the startup log:
> {noformat}
> ERROR [SSTableBatchOpen:3] 2018-03-11 00:17:35,597 CassandraDaemon.java:207 - 
> Exception in thread Thread[SSTableBatchOpen:3,5,main]
> java.lang.AssertionError: Stats component is missing for sstable 
> /cassandra/data/keyspace/table-id/mc-12345-big
> at 
>

[jira] [Updated] (CASSANDRA-14308) Remove invalid SSTables from interrupted compaction

2018-03-11 Thread Jay Zhuang (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jay Zhuang updated CASSANDRA-14308:
---
Description: 
If the JVM crash while compaction is in progress, the incompleted SSTable won't 
be cleaned up, which causes "{{Stats component is missing for sstable}}" error 
in the startup log:
{noformat}
ERROR [SSTableBatchOpen:3] 2018-03-11 00:17:35,597 CassandraDaemon.java:207 - 
Exception in thread Thread[SSTableBatchOpen:3,5,main]
java.lang.AssertionError: Stats component is missing for sstable 
/cassandra/data/keyspace/table-id/mc-12345-big
at 
org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:458)
 ~[apache-cassandra-3.0.14.jar:3.0.14]
at 
org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:374)
 ~[apache-cassandra-3.0.14.jar:3.0.14]
at 
org.apache.cassandra.io.sstable.format.SSTableReader$4.run(SSTableReader.java:533)
 ~[apache-cassandra-3.0.14.jar:3.0.14]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
~[na:1.8.0_121]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_121]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
~[na:1.8.0_121]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
[na:1.8.0_121]
at 
org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79)
 [apache-cassandra-3.0.14.jar:3.0.14]
at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_121]
{noformat}

The accumulated incompleted SSTables could take lots of space, especially for 
STCS which could have very large SSTables.
Here is the script we use to delete the SSTables after node is restarted:
{noformat}
grep 'Stats component is missing for sstable' $SYSTEM_LOG | awk '{print $8}' > 
~/invalid_sstables ; for ss in `cat ~/invalid_sstables`; do echo == $ss; ll 
$ss*; sudo rm $ss* ; done
{noformat}
I would suggest to remove these incompleted SSTables while startup.

  was:
If the JVM crash while compaction is in progress, the incompleted SSTable won't 
be cleaned up, which causes "{{Stats component is missing for sstable}}" error 
in the startup log:
{noformat}
ERROR [SSTableBatchOpen:3] 2018-03-11 00:17:35,597 CassandraDaemon.java:207 - 
Exception in thread Thread[SSTableBatchOpen:3,5,main]
java.lang.AssertionError: Stats component is missing for sstable 
/cassandra/data/keyspace/table-id/mc-12345-big
at 
org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:458)
 ~[apache-cassandra-3.0.14.jar:3.0.14]
at 
org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:374)
 ~[apache-cassandra-3.0.14.jar:3.0.14]
at 
org.apache.cassandra.io.sstable.format.SSTableReader$4.run(SSTableReader.java:533)
 ~[apache-cassandra-3.0.14.jar:3.0.14]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
~[na:1.8.0_121]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_121]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
~[na:1.8.0_121]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
[na:1.8.0_121]
at 
org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79)
 [apache-cassandra-3.0.14.jar:3.0.14]
at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_121]
{noformat}

The accumulated incompleted SSTables could take lots of space, especially for 
STCS, which could have very large SSTables.
Here is the script we use to delete the SSTables after node is restarted:
{noformat}
grep 'Stats component is missing for sstable' $SYSTEM_LOG | awk '{print $8}' > 
~/invalid_sstables ; for ss in `cat ~/invalid_sstables`; do echo == $ss; ll 
$ss*; sudo rm $ss* ; done
{noformat}
I would suggest to remove these incompleted SSTables while startup.


> Remove invalid SSTables from interrupted compaction
> ---
>
> Key: CASSANDRA-14308
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14308
> Project: Cassandra
>  Issue Type: Bug
>  Components: Compaction
>Reporter: Jay Zhuang
>Assignee: Jay Zhuang
>Priority: Minor
>
> If the JVM crash while compaction is in progress, the incompleted SSTable 
> won't be cleaned up, which causes "{{Stats component is missing for 
> sstable}}" error in the startup log:
> {noformat}
> ERROR [SSTableBatchOpen:3] 2018-03-11 00:17:35,597 CassandraDaemon.java:207 - 
> Exception in thread Thread[SSTableBatchOpen:3,5,main]
> java.lang.AssertionError: Stats component is missing for sstable 
> /cassandra/data/keyspace/table-id/mc-12345-big
> at 
>

[jira] [Updated] (CASSANDRA-14308) Remove invalid SSTables from interrupted compaction

2018-03-11 Thread Jay Zhuang (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jay Zhuang updated CASSANDRA-14308:
---
Description: 
If the JVM crash while compaction is in progress, the incompleted SSTable won't 
be cleaned up, which causes "{{Stats component is missing for sstable}}" error 
in the start up log:
{noformat}
ERROR [SSTableBatchOpen:3] 2018-03-11 00:17:35,597 CassandraDaemon.java:207 - 
Exception in thread Thread[SSTableBatchOpen:3,5,main]
java.lang.AssertionError: Stats component is missing for sstable 
/cassandra/data/keyspace/table-id/mc-12345-big
at 
org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:458)
 ~[apache-cassandra-3.0.14.jar:3.0.14]
at 
org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:374)
 ~[apache-cassandra-3.0.14.jar:3.0.14]
at 
org.apache.cassandra.io.sstable.format.SSTableReader$4.run(SSTableReader.java:533)
 ~[apache-cassandra-3.0.14.jar:3.0.14]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
~[na:1.8.0_121]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_121]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
~[na:1.8.0_121]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
[na:1.8.0_121]
at 
org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79)
 [apache-cassandra-3.0.14.jar:3.0.14]
at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_121]
{noformat}

The accumulated incompleted SSTables could take lots of space, especially for 
STCS, which could have very large SSTables.
Here is the script we use to delete the SSTables after node is restarted:
{noformat}
grep 'Stats component is missing for sstable' $SYSTEM_LOG | awk '{print $8}' > 
~/invalid_sstables ; for ss in `cat ~/invalid_sstables`; do echo == $ss; ll 
$ss*; sudo rm $ss* ; done
{noformat}
I would suggest to remove these incompleted SSTables while startup.

  was:
If the JVM crash while compaction is in progress, the incompleted SSTable won't 
be cleaned up, which causes {{Stats component is missing for sstable}} error in 
the start up log:
{noformat}
ERROR [SSTableBatchOpen:3] 2018-03-11 00:17:35,597 CassandraDaemon.java:207 - 
Exception in thread Thread[SSTableBatchOpen:3,5,main]
java.lang.AssertionError: Stats component is missing for sstable 
/cassandra/data/keyspace/table-id/mc-12345-big
at 
org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:458)
 ~[apache-cassandra-3.0.14.jar:3.0.14]
at 
org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:374)
 ~[apache-cassandra-3.0.14.jar:3.0.14]
at 
org.apache.cassandra.io.sstable.format.SSTableReader$4.run(SSTableReader.java:533)
 ~[apache-cassandra-3.0.14.jar:3.0.14]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
~[na:1.8.0_121]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_121]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
~[na:1.8.0_121]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
[na:1.8.0_121]
at 
org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79)
 [apache-cassandra-3.0.14.jar:3.0.14]
at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_121]
{noformat}

The accumulated incompleted SSTables could take lots of space, especially for 
STCS, which could have very large SSTables.
Here is the script we use to delete the SSTables after node is restarted:
{noformat}
grep 'Stats component is missing for sstable' $SYSTEM_LOG | awk '{print $8}' > 
~/invalid_sstables ; for ss in `cat ~/invalid_sstables`; do echo == $ss; ll 
$ss*; sudo rm $ss* ; done
{noformat}
I would suggest to remove these incompleted SSTables while startup.


> Remove invalid SSTables from interrupted compaction
> ---
>
> Key: CASSANDRA-14308
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14308
> Project: Cassandra
>  Issue Type: Bug
>  Components: Compaction
>Reporter: Jay Zhuang
>Assignee: Jay Zhuang
>Priority: Minor
>
> If the JVM crash while compaction is in progress, the incompleted SSTable 
> won't be cleaned up, which causes "{{Stats component is missing for 
> sstable}}" error in the start up log:
> {noformat}
> ERROR [SSTableBatchOpen:3] 2018-03-11 00:17:35,597 CassandraDaemon.java:207 - 
> Exception in thread Thread[SSTableBatchOpen:3,5,main]
> java.lang.AssertionError: Stats component is missing for sstable 
> /cassandra/data/keyspace/table-id/mc-12345-big
> at 
>