C* 3.0.7 - Compactions pending after TRUNCATE

[Stefano Ortolani]

Hi all,

I've just updated to C* 3.0.7, and I am now seeing some compactions getting
stuck: nodetool compationstats has been showing the same output for the
last 6 hours, and all compactions are stuck at 0.00% progress.

What is interesting is that the stuck CFs have been truncated exactly 6
hours ago.
I am fairly confident this issue was not there in C* 3.0.5.

Any idea?

Regards,
Stefano Ortolani

Re: C* 3.0.7 - Compactions pending after TRUNCATE

[Stefano Ortolani]

I am updating the following ticket
https://issues.apache.org/jira/browse/CASSANDRA-12100 as I discover new
bits.

Regards,
Stefano

On Tue, Jun 28, 2016 at 9:37 AM, Stefano Ortolani 
wrote:

> Hi all,
>
> I've just updated to C* 3.0.7, and I am now seeing some compactions
> getting stuck: nodetool compationstats has been showing the same output for
> the last 6 hours, and all compactions are stuck at 0.00% progress.
>
> What is interesting is that the stuck CFs have been truncated exactly 6
> hours ago.
> I am fairly confident this issue was not there in C* 3.0.5.
>
> Any idea?
>
> Regards,
> Stefano Ortolani
>
>
>

Exception in logs using LCS .

[Prakash Chauhan]

Hello,

Recently we changed compaction strategy fora table to LCS from the default 
STCS. While bootstrapping a node , we are getting following Exception in the 
logs:

ERROR [CompactionExecutor:81] 2016-05-11 13:48:54,694 CassandraDaemon.java 
(line 258) Exception in thread Thread[CompactionExecutor:81,1,main]
java.lang.RuntimeException: Last written key DecoratedKey(-2711050270696519088, 
623330333a313a35) >= current key DecoratedKey(-8631371593982690738, 
623437393a313a30) writing into /cassandra/data/main/myCF/myCF-tmp-jb-326-Data.db
at 
org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:143)
at 
org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:166)
at 
org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:167)
at 
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at 
org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60)
at 
org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
at 
org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:198)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
INFO [CompactionExecutor:68] 2016-05-11 14:10:28,957 ColumnFamilyStore.java 
(line 795) Enqueuing flush of Memtable-compactions_in_progress@541886922(0/0 
serialized/live bytes, 1 ops)

Questions:

1.   Are these exceptions due to data corruption ?

2.   I am unable to reproduce the problem. How can I reproduce the 
Exception? Is there any specific case when such exceptions are raised?

3.   Without reproducing the Exception, how can I test the patch available 
at related JIRA : https://issues.apache.org/jira/browse/CASSANDRA-9935



Thanks,
Prakash Chauhan.

Re: Exception in logs using LCS .

[Paulo Motta]

1. Not necessarily data corruption, but it seems compaction is trying to
write data in the wrong order most likely due to a temporary race
condition/bug a la #9935, but since the compaction fails your original data
is probably safe (you can try running scrub to verify/fix corruptions).
2. This is pretty tricky to reproduce because it will depend on which
sstables were picked for compaction at a particular instant, but you could
try running a user-defined compaction or scrub on the sstables that contain
this key, see CASSANDRA-11337and
https://gist.github.com/jeromatron/e238e5795b3e79866b83
3. clone the cassandra repository of your current version, git-cherry-pick
the commit of CASSANDRA-9935, ant jar, replace the cassandra jar with the
generated SNAPSHOT.jar and restart the node`

2016-06-28 7:55 GMT-03:00 Prakash Chauhan :

> Hello,
>
>
>
> Recently we changed compaction strategy fora table to LCS from the default
> STCS. While bootstrapping a node , we are getting following Exception in
> the logs:
>
>
>
> ERROR [CompactionExecutor:81] 2016-05-11 13:48:54,694 CassandraDaemon.java
> (line 258) Exception in thread Thread[CompactionExecutor:81,1,main]
>
> java.lang.RuntimeException: Last written key
> DecoratedKey(-2711050270696519088, 623330333a313a35) >= current key
> DecoratedKey(-8631371593982690738, 623437393a313a30) writing into
> /cassandra/data/main/myCF/myCF-tmp-jb-326-Data.db
>
> at
> org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:143)
>
> at
> org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:166)
>
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:167)
>
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
>
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60)
>
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
>
> at
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:198)
>
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>
> at java.lang.Thread.run(Thread.java:745)
>
> INFO [CompactionExecutor:68] 2016-05-11 14:10:28,957
> ColumnFamilyStore.java (line 795) Enqueuing flush of
> Memtable-compactions_in_progress@541886922(0/0 serialized/live bytes, 1
> ops)
>
>
>
> Questions:
>
> *1.   *Are these exceptions due to data corruption ?
>
> *2.   *I am unable to reproduce the problem. How can I reproduce the
> Exception? Is there any specific case when such exceptions are raised?
>
> *3.   *Without reproducing the Exception, how can I test the patch
> available at related JIRA :
> https://issues.apache.org/jira/browse/CASSANDRA-9935
>
>
>
>
>
> Thanks,
>
> Prakash Chauhan.
>
>
>
>
>
>
>
>
>
>
>

Motivation for a DHT ring

[jean paul]

Hi all,

Please, What is the motivation for choosing a DHT ring in cassandra? Why
not use a normal parallel or distributed file system that supports
replication?

Thank you so much for clarification.

Kind regards.

Re: Adding column to materialized view

[Tyler Hobbs]

This is expected.  It's something we plan to support, but it hasn't been
done yet: https://issues.apache.org/jira/browse/CASSANDRA-9736

On Mon, Jun 27, 2016 at 4:25 PM, Jason J. W. Williams <
jasonjwwilli...@gmail.com> wrote:

> Hey Guys,
>
> Running Cassandra 3.0.5. Needed to add a column to a materialized view,
> but ALTER MATERIALIZED VIEW doesn't seem to allow that. So we ended up
> dropping the view and recreating it. Is that expected or did I miss
> something in the docs?
>
> -J
>



-- 
Tyler Hobbs
DataStax 

Re: Read operation can read uncommitted data?

[Tyler Hobbs]

Reads at CL.SERIAL will complete any in-progress paxos writes, so the
behavior you're seeing is expected.

On Mon, Jun 27, 2016 at 1:55 AM, Yuji Ito  wrote:

> Hi,
>
> I'm testing Cassandra CAS operation.
>
> Can a read operation read uncommitted data which is being updated by CAS
> in the following case?
>
> I use Cassandra 2.2.6.
> There are 3 nodes (A, B and C) in a cluster.
> Replication factor of keyspace is 3.
> CAS operation on node A starts to update row X (updating the value in row
> from 0 to 1).
>
> 1. prepare/promise phase succeeds on node A
> 2. node C is down
> 3. read/results phase in node A sends read requests to node B and C and
> waits for read responses from them.
> 4. (unrelated) read operation (CL: SERIAL) reads the same row X and gets
> the value "1" in the row!!
> 5. read/results phase fails by ReadTimeoutException caused by failure of
> node C
>
> Thanks,
> Yuji Ito
>



-- 
Tyler Hobbs
DataStax 

Changing a cluster name

[Surbhi Gupta]

system.local uses local strategy . You need to update on all nodes .


On 28 June 2016 at 14:51, Tyler Hobbs > wrote:

> First, make sure that you call nodetool flush after modifying the system
> table.  That's probably why it's not surviving the restart.
>
> Second, I believe you will have to do this across all nodes and restart
> them at the same time.  Otherwise, cluster name mismatches will prevent the
> nodes from communicating with each other.
>
> On Fri, Jun 24, 2016 at 3:51 PM, Paul Fife  > wrote:
>
>> I am following the instructions here to attempt to change the name of a
>> cluster: https://wiki.apache.org/cassandra/FAQ#clustername_mismatch
>> or at least the more up to date advice:
>> http://stackoverflow.com/questions/22006887/cassandra-saved-cluster-name-test-cluster-configured-name
>>
>> I am able to query the system.local to verify the clusterName is
>> modified, but when I restart Cassandra it fails, and the value is back at
>> the original cluster name. Is this still possible, or are there changes
>> preventing this from working anymore?
>>
>> I have attempted this several times and it did actually work the first
>> time, but when I moved around to the other nodes it no longer worked.
>>
>> Thanks,
>> Paul Fife
>>
>>
>
>
> --
> Tyler Hobbs
> DataStax 
>