RE: Exception in logs using LCS .

2016-06-30 Thread Prakash Chauhan 
Hi  Paulo,

Thanks for the reply. Running scrub every time the exception occurs is not a 
possible solution for us. We don’t have a log patrolling system in place that 
verify the log on regular basis.

I have some queries based on your reply:

1.   You mentioned race condition. Can you please elaborate a little more 
what actual race condition may be happening in this scenario ?

2.   What if this happens in production ? What would be the impact? Can we 
live with it?

3.   Test system where the problem was reported has been scrapped so we 
can’t run user defined compaction now. Any suggestions to reproduce it on a new 
system?



From: Paulo Motta [mailto:pauloricard...@gmail.com]
Sent: Tuesday, June 28, 2016 5:43 PM
To: user@cassandra.apache.org
Subject: Re: Exception in logs using LCS .

1. Not necessarily data corruption, but it seems compaction is trying to write 
data in the wrong order most likely due to a temporary race condition/bug a la 
#9935, but since the compaction fails your original data is probably safe (you 
can try running scrub to verify/fix corruptions).
2. This is pretty tricky to reproduce because it will depend on which sstables 
were picked for compaction at a particular instant, but you could try running a 
user-defined compaction or scrub on the sstables that contain this key, see 
CASSANDRA-11337and https://gist.github.com/jeromatron/e238e5795b3e79866b83
3. clone the cassandra repository of your current version, git-cherry-pick the 
commit of CASSANDRA-9935, ant jar, replace the cassandra jar with the generated 
SNAPSHOT.jar and restart the node`

2016-06-28 7:55 GMT-03:00 Prakash Chauhan 
<prakash.chau...@ericsson.com<mailto:prakash.chau...@ericsson.com>>:
Hello,

Recently we changed compaction strategy fora table to LCS from the default 
STCS. While bootstrapping a node , we are getting following Exception in the 
logs:

ERROR [CompactionExecutor:81] 2016-05-11 13:48:54,694 CassandraDaemon.java 
(line 258) Exception in thread Thread[CompactionExecutor:81,1,main]
java.lang.RuntimeException: Last written key DecoratedKey(-2711050270696519088, 
623330333a313a35) >= current key DecoratedKey(-8631371593982690738, 
623437393a313a30) writing into /cassandra/data/main/myCF/myCF-tmp-jb-326-Data.db
at 
org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:143)
at 
org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:166)
at 
org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:167)
at 
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at 
org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60)
at 
org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
at 
org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:198)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
INFO [CompactionExecutor:68] 2016-05-11 14:10:28,957 ColumnFamilyStore.java 
(line 795) Enqueuing flush of Memtable-compactions_in_progress@541886922(0/0 
serialized/live bytes, 1 ops)

Questions:

1.   Are these exceptions due to data corruption ?

2.   I am unable to reproduce the problem. How can I reproduce the 
Exception? Is there any specific case when such exceptions are raised?

3.   Without reproducing the Exception, how can I test the patch available 
at related JIRA : https://issues.apache.org/jira/browse/CASSANDRA-9935



Thanks,
Prakash Chauhan.

Re: Exception in logs using LCS .

2016-06-28 Thread Paulo Motta 
1. Not necessarily data corruption, but it seems compaction is trying to
write data in the wrong order most likely due to a temporary race
condition/bug a la #9935, but since the compaction fails your original data
is probably safe (you can try running scrub to verify/fix corruptions).
2. This is pretty tricky to reproduce because it will depend on which
sstables were picked for compaction at a particular instant, but you could
try running a user-defined compaction or scrub on the sstables that contain
this key, see CASSANDRA-11337and
https://gist.github.com/jeromatron/e238e5795b3e79866b83
3. clone the cassandra repository of your current version, git-cherry-pick
the commit of CASSANDRA-9935, ant jar, replace the cassandra jar with the
generated SNAPSHOT.jar and restart the node`

2016-06-28 7:55 GMT-03:00 Prakash Chauhan :

> Hello,
>
>
>
> Recently we changed compaction strategy fora table to LCS from the default
> STCS. While bootstrapping a node , we are getting following Exception in
> the logs:
>
>
>
> ERROR [CompactionExecutor:81] 2016-05-11 13:48:54,694 CassandraDaemon.java
> (line 258) Exception in thread Thread[CompactionExecutor:81,1,main]
>
> java.lang.RuntimeException: Last written key
> DecoratedKey(-2711050270696519088, 623330333a313a35) >= current key
> DecoratedKey(-8631371593982690738, 623437393a313a30) writing into
> /cassandra/data/main/myCF/myCF-tmp-jb-326-Data.db
>
> at
> org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:143)
>
> at
> org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:166)
>
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:167)
>
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
>
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60)
>
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
>
> at
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:198)
>
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>
> at java.lang.Thread.run(Thread.java:745)
>
> INFO [CompactionExecutor:68] 2016-05-11 14:10:28,957
> ColumnFamilyStore.java (line 795) Enqueuing flush of
> Memtable-compactions_in_progress@541886922(0/0 serialized/live bytes, 1
> ops)
>
>
>
> Questions:
>
> *1.   *Are these exceptions due to data corruption ?
>
> *2.   *I am unable to reproduce the problem. How can I reproduce the
> Exception? Is there any specific case when such exceptions are raised?
>
> *3.   *Without reproducing the Exception, how can I test the patch
> available at related JIRA :
> https://issues.apache.org/jira/browse/CASSANDRA-9935
>
>
>
>
>
> Thanks,
>
> Prakash Chauhan.
>
>
>
>
>
>
>
>
>
>
>