Re: Single sstable file compaction issue

[wxn...@zjqunshuo.com]

Thanks Jeff for the reply. Answers inlined.

Tombstones probably aren't clearing because the same partition exists with 
older timestamps in other files (this is the "sstableexpiredblockers" problem, 
or "overlaps"). 
>>The RF is 2, so there is two copies of one partition in two node. So my 
>>method to clear expired data doesn't work because of the "overlaps" you 
>>mentioned. Is my understanding corrent? One more question, nodetool cleanup 
>>may work for me, but how cleanup deal with the sstable files in TWCS mode? I 
>>have large sstable files before changing from STCS to TWCS, and newer ones 
>>with time buckets with TWCS. How does the command deal with them? Compact all 
>>of them into large sstable files?

If you're certain you are ok losing that data, then you could stop the node, 
remove lb-143951-big-* , and start the node. This is usually a bad idea in data 
models that aren't ttl-only time-series, but if you KNOW the data is all 
expired, and you didnt manually delete any other data, it may work for you.
>>My data model is indeed ttl-only time-series.

Cheers,
-Simon
 
From: Jeff Jirsa
Date: 2018-03-27 11:52
To: cassandra
Subject: Re: Single sstable file compaction issue
Tombstones probably aren't clearing because the same partition exists with 
older timestamps in other files (this is the "sstableexpiredblockers" problem, 
or "overlaps"). 

If you're certain you are ok losing that data, then you could stop the node, 
remove lb-143951-big-* , and start the node. This is usually a bad idea in data 
models that aren't ttl-only time-series, but if you KNOW the data is all 
expired, and you didnt manually delete any other data, it may work for you.



On Mon, Mar 26, 2018 at 8:03 PM, wxn...@zjqunshuo.com  
wrote:
Hi All,
I changed STCS to TWCS months ago and left some old sstable files. Some are 
almost tombstones. To release disk space, I issued compaction command on one 
file by JMX. After the compaction is done, I got one new file with almost the 
same size of the old one. Seems no tombstones are cleaned during the 
compaction. 

Before compaciton:
Max: 01/12/2017 Min: 11/25/2016 Estimated droppable tombstones: 
0.9115440366604225 53G Jan 16 00:36 lb-124337-big-Data.db
After compaction:
Max: 01/12/2017 Min: 11/25/2016 Estimated droppable tombstones: 
0.9114708007586322 53G Mar 27 00:17 lb-143951-big-Data.db

Questions:
1. Why the compaction didn't clean the tombstones?
2. If one file are all tombstones and I want to it manually(including data, 
index, filter, etc), do I need to shutdown the node?

Cheers,
-Simon

答复: 答复: A node down every day in a 6 nodes cluster

[Xiangfei Ni]

I have checked the dmesg and message logs ,there is no eth* content in it.so I 
think there was no network connection issue.

Best Regards,

倪项菲/ David Ni
中移德电网络科技有限公司
Virtue Intelligent Network Ltd, co.
Add: 2003,20F No.35 Luojia creative city,Luoyu Road,Wuhan,HuBei
Mob: +86 13797007811|Tel: + 86 27 5024 2516

发件人: daemeon reiydelle 
发送时间: 2018年3月27日 11:42
收件人: user 
主题: Re: 答复: A node down every day in a 6 nodes cluster

Look for errors on your network interface. I think you have periodic errors in 
your network connectivity


<==>
"Who do you think made the first stone spear? The Asperger guy.
If you get rid of the autism genetics, there would be no Silicon Valley"
Temple Grandin
Daemeon C.M. Reiydelle
San Francisco 1.415.501.0198
London 44 020 8144 9872

On Mon, Mar 26, 2018 at 8:26 PM, Xiangfei Ni 
> wrote:
Hi Jeff,
I need to restart the node manually every time,only one node has this 
problem.
I have attached the nodetool output,thanks.

Best Regards,

倪项菲/ David Ni
中移德电网络科技有限公司
Virtue Intelligent Network Ltd, co.
Add: 2003,20F No.35 Luojia creative city,Luoyu Road,Wuhan,HuBei
Mob: +86 13797007811|Tel: + 86 27 5024 
2516

发件人: Jeff Jirsa >
发送时间: 2018年3月27日 11:03
收件人: user@cassandra.apache.org
主题: Re: A node down every day in a 6 nodes cluster

That warning isn’t sufficient to understand why the node is going down


Cassandra 3.9 has some pretty serious known issues - upgrading to 3.11.3 is 
likely a good idea

Are the nodes coming up on their own? Or are you restarting them?

Paste the output of nodetool tpstats and nodetool cfstats



--
Jeff Jirsa


On Mar 26, 2018, at 7:56 PM, Xiangfei Ni 
> wrote:
Hi Cassandra experts,
  I am facing an issue,a node downs every day in a 6 nodes cluster,the cluster 
is just in one DC,
  Every node has 4C 16G,and the heap configuration is MAX_HEAP_SIZE=8192m 
HEAP_NEWSIZE=512m,every node load about 200G data,the RF for the business CF is 
3,a node downs one time every day,the system.log shows below info:
WARN  [Native-Transport-Requests-19] 2018-03-26 18:53:17,128 
CassandraAuthorizer.java:101 - CassandraAuthorizer failed to authorize # for 
ERROR [Native-Transport-Requests-19] 2018-03-26 18:53:17,129 
QueryMessage.java:128 - Unexpected error during query
com.google.common.util.concurrent.UncheckedExecutionException: 
java.lang.RuntimeException: 
org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - 
received only 0 responses.
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203) 
~[guava-18.0.jar:na]
at com.google.common.cache.LocalCache.get(LocalCache.java:3937) 
~[guava-18.0.jar:na]
at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941) 
~[guava-18.0.jar:na]
at 
com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824) 
~[guava-18.0.jar:na]
at org.apache.cassandra.auth.AuthCache.get(AuthCache.java:108) 
~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.auth.PermissionsCache.getPermissions(PermissionsCache.java:45)
 ~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.auth.AuthenticatedUser.getPermissions(AuthenticatedUser.java:104)
 ~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.service.ClientState.authorize(ClientState.java:419) 
~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.service.ClientState.checkPermissionOnResourceChain(ClientState.java:352)
 ~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:329)
 ~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:316) 
~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:300)
 ~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.cql3.statements.ModificationStatement.checkAccess(ModificationStatement.java:211)
 ~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:185)
 ~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:219) 
~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:204) 
~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:115)
 ~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:513)
 [apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:407)
 

Re: LWT on data mutated by non-LWT operation is valid ?

[Hiroyuki Yamada]

Thank you, JH.
I took a look and probably understand it.

Maybe, in summary,
if all the operations are LWT, there is no issue even if clocks are
drifted, because it's ballot based.
But, if some of the operations are non-LWT and clocks are drifted,
there might be causing issues.
(like overwriting data with previous insert with future time or something)

Anyways, thank you.

Thanks,
Hiro

On Mon, Mar 26, 2018 at 5:42 PM, Jacques-Henri Berthemet
 wrote:
> If you check the Jira issue I linked you'll see a recent comment describing a 
> potential explanation for my mixed LWT/non-LWT problem. So, it looks like 
> there can be some edge cases.
>
> I'd say that if data was inserted a while ago (seconds) there should be no 
> problems.
>
> --
> Jacques-Henri Berthemet
>
> -Original Message-
> From: Hiroyuki Yamada [mailto:mogwa...@gmail.com]
> Sent: Sunday, March 25, 2018 1:10 AM
> To: user@cassandra.apache.org
> Subject: Re: LWT on data mutated by non-LWT operation is valid ?
>
> Thank you JH.
> But, it's still a little bit unclear to me
>
> Let me clarify the question.
> What I wanted to know is whether or not linearizability is sustained by doing 
> LWT  (Consistency: QUORUM, Serial Consistency: SERIAL) on data previously 
> mutated by non-LWT (Consistency: QUORUM).
> I think It should be OK if non-LWT surely correctly happened before the LWT, 
> but I'm wondering if there is a corner case where it's not OK.
>
> For example, to test LWT (Update) operations, initial data should be inserted 
> by LWT operations ? or it can be non-LWT operations ?
>
> Thanks,
> Hiroyuki
>
> On Sat, Mar 24, 2018 at 8:27 PM, Jacques-Henri Berthemet 
>  wrote:
>> Hi Hiroyuki,
>>
>>
>> For both operations you'll have to provide partition key so "conflict"
>> at DB level can always be resolved.
>>
>> But if two operations, LWT and non-LWT, are racing against each others
>> the result is unpredictable, if non-LWT is applied after LWT the
>> result will be overwritten.
>>
>>
>> It seems mixing LWT and non-LWT can result in strange results, we
>> recently opened a bug on non-working delete after LWT insert:
>> https://issues.apache.org/jira/browse/CASSANDRA-14304
>> .apache.org
>>
>>
>> Regards,
>>
>> JH
>>
>> 
>> From: Hiroyuki Yamada 
>> Sent: Saturday, March 24, 2018 4:38:15 AM
>> To: user@cassandra.apache.org
>> Subject: LWT on data mutated by non-LWT operation is valid ?
>>
>> Hi all,
>>
>> I have some question about LWT.
>>
>> I am wondering if LWT works only for data mutated by LWT or not.
>> In other words, doing LWT on some data mutated by non-LWT operations
>> is still valid ?
>> I don't fully understand how system.paxos table works in LWT, but
>> row_key should be empty for a data mutated by non-LWT operation, so
>> conflict resolution seems impossible.
>> It works only if a previous non-LWT operation is completely finished ?
>>
>> Thanks in advance.
>>
>> Best regards,
>> Hiroyuki
>>
>> -
>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
>> For additional commands, e-mail: user-h...@cassandra.apache.org
>>
>
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
>
>
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org

-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: Single sstable file compaction issue

[Jeff Jirsa]

Tombstones probably aren't clearing because the same partition exists with
older timestamps in other files (this is the "sstableexpiredblockers"
problem, or "overlaps").

If you're certain you are ok losing that data, then you could stop the
node, remove lb-143951-big-* , and start the node. This is usually a bad
idea in data models that aren't ttl-only time-series, but if you KNOW the
data is all expired, and you didnt manually delete any other data, it may
work for you.



On Mon, Mar 26, 2018 at 8:03 PM, wxn...@zjqunshuo.com 
wrote:

> Hi All,
> I changed STCS to TWCS months ago and left some old sstable files. Some
> are almost tombstones. To release disk space, I issued compaction command
> on one file by JMX. After the compaction is done, I got one new file with
> almost the same size of the old one. Seems no tombstones are cleaned during
> the compaction.
>
> Before compaciton:
> Max: 01/12/2017 Min: 11/25/2016 Estimated droppable tombstones: 0.
> 9115440366604225 53G Jan 16 00:36 lb-124337-big-Data.db
> After compaction:
> Max: 01/12/2017 Min: 11/25/2016 Estimated droppable tombstones: 0.
> 9114708007586322 53G Mar 27 00:17 lb-143951-big-Data.db
>
> Questions:
> 1. Why the compaction didn't clean the tombstones?
> 2. If one file are all tombstones and I want to it manually(including
> data, index, filter, etc), do I need to shutdown the node?
>
> Cheers,
> -Simon
>

Re: 答复: A node down every day in a 6 nodes cluster

[Jeff Jirsa]

Only one node having the problem is suspicious. May be that your
application is improperly pooling connections, or you have a hardware
problem.

I dont see anything in nodetool that explains it, though you certainly have
a data model likely to cause problems over time (the cardinality of

rt_ac_stat.idx_rt_ac_stat_prot_verrt_ac_stat.idx_rt_ac_stat_prot_ver
is such that you have very wide partitions and it'll be difficult to
read).




On Mon, Mar 26, 2018 at 8:26 PM, Xiangfei Ni  wrote:

> Hi Jeff,
>
> I need to restart the node manually every time,only one node has this
> problem.
>
> I have attached the nodetool output,thanks.
>
>
>
> Best Regards,
>
>
>
> 倪项菲*/ **David Ni*
>
> 中移德电网络科技有限公司
>
> Virtue Intelligent Network Ltd, co.
>
> Add: 2003,20F No.35 Luojia creative city,Luoyu Road,Wuhan,HuBei
>
> Mob: +86 13797007811 <+86%20137%209700%207811>|Tel: + 86 27 5024 2516
> <+86%2027%205024%202516>
>
>
>
> *发件人:* Jeff Jirsa 
> *发送时间:* 2018年3月27日 11:03
> *收件人:* user@cassandra.apache.org
> *主题:* Re: A node down every day in a 6 nodes cluster
>
>
>
> That warning isn’t sufficient to understand why the node is going down
>
>
>
>
>
> Cassandra 3.9 has some pretty serious known issues - upgrading to 3.11.3
> is likely a good idea
>
>
>
> Are the nodes coming up on their own? Or are you restarting them?
>
>
>
> Paste the output of nodetool tpstats and nodetool cfstats
>
>
>
>
>
>
>
> --
>
> Jeff Jirsa
>
>
>
>
> On Mar 26, 2018, at 7:56 PM, Xiangfei Ni  wrote:
>
> Hi Cassandra experts,
>
>   I am facing an issue,a node downs every day in a 6 nodes cluster,the
> cluster is just in one DC,
>
>   Every node has 4C 16G,and the heap configuration is MAX_HEAP_SIZE=8192m
> HEAP_NEWSIZE=512m,every node load about 200G data,the RF for the business
> CF is 3,a node downs one time every day,the system.log shows below info:
>
> WARN  [Native-Transport-Requests-19] 2018-03-26 18:53:17,128
> CassandraAuthorizer.java:101 - CassandraAuthorizer failed to authorize
> # for 
>
> ERROR [Native-Transport-Requests-19] 2018-03-26 18:53:17,129
> QueryMessage.java:128 - Unexpected error during query
>
> com.google.common.util.concurrent.UncheckedExecutionException:
> java.lang.RuntimeException: 
> org.apache.cassandra.exceptions.ReadTimeoutException:
> Operation timed out - received only 0 responses.
>
> at 
> com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203)
> ~[guava-18.0.jar:na]
>
> at com.google.common.cache.LocalCache.get(LocalCache.java:3937)
> ~[guava-18.0.jar:na]
>
> at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941)
> ~[guava-18.0.jar:na]
>
> at 
> com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)
> ~[guava-18.0.jar:na]
>
> at org.apache.cassandra.auth.AuthCache.get(AuthCache.java:108)
> ~[apache-cassandra-3.9.jar:3.9]
>
> at 
> org.apache.cassandra.auth.PermissionsCache.getPermissions(PermissionsCache.java:45)
> ~[apache-cassandra-3.9.jar:3.9]
>
> at 
> org.apache.cassandra.auth.AuthenticatedUser.getPermissions(AuthenticatedUser.java:104)
> ~[apache-cassandra-3.9.jar:3.9]
>
> at 
> org.apache.cassandra.service.ClientState.authorize(ClientState.java:419)
> ~[apache-cassandra-3.9.jar:3.9]
>
> at org.apache.cassandra.service.ClientState.
> checkPermissionOnResourceChain(ClientState.java:352)
> ~[apache-cassandra-3.9.jar:3.9]
>
> at 
> org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:329)
> ~[apache-cassandra-3.9.jar:3.9]
>
> at 
> org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:316)
> ~[apache-cassandra-3.9.jar:3.9]
>
> at 
> org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:300)
> ~[apache-cassandra-3.9.jar:3.9]
>
> at org.apache.cassandra.cql3.statements.ModificationStatement.
> checkAccess(ModificationStatement.java:211) ~[apache-cassandra-3.9.jar:3.
> 9]
>
> at 
> org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:185)
> ~[apache-cassandra-3.9.jar:3.9]
>
> at 
> org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:219)
> ~[apache-cassandra-3.9.jar:3.9]
>
> at 
> org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:204)
> ~[apache-cassandra-3.9.jar:3.9]
>
> at 
> org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:115)
> ~[apache-cassandra-3.9.jar:3.9]
>
> at 
> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:513)
> [apache-cassandra-3.9.jar:3.9]
>
> at 
> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:407)
> [apache-cassandra-3.9.jar:3.9]
>
> at io.netty.channel.SimpleChannelInboundHandler.channelRead(
> SimpleChannelInboundHandler.java:105) [netty-all-4.0.39.Final.jar:4.
> 0.39.Final]
>
> at 

Re: 答复: A node down every day in a 6 nodes cluster

[daemeon reiydelle]

Look for errors on your network interface. I think you have periodic errors
in your network connectivity


<==>
"Who do you think made the first stone spear? The Asperger guy.
If you get rid of the autism genetics, there would be no Silicon Valley"
Temple Grandin


*Daemeon C.M. ReiydelleSan Francisco 1.415.501.0198London 44 020 8144 9872*


On Mon, Mar 26, 2018 at 8:26 PM, Xiangfei Ni  wrote:

> Hi Jeff,
>
> I need to restart the node manually every time,only one node has this
> problem.
>
> I have attached the nodetool output,thanks.
>
>
>
> Best Regards,
>
>
>
> 倪项菲*/ **David Ni*
>
> 中移德电网络科技有限公司
>
> Virtue Intelligent Network Ltd, co.
>
> Add: 2003,20F No.35 Luojia creative city,Luoyu Road,Wuhan,HuBei
>
> Mob: +86 13797007811 <+86%20137%209700%207811>|Tel: + 86 27 5024 2516
> <+86%2027%205024%202516>
>
>
>
> *发件人:* Jeff Jirsa 
> *发送时间:* 2018年3月27日 11:03
> *收件人:* user@cassandra.apache.org
> *主题:* Re: A node down every day in a 6 nodes cluster
>
>
>
> That warning isn’t sufficient to understand why the node is going down
>
>
>
>
>
> Cassandra 3.9 has some pretty serious known issues - upgrading to 3.11.3
> is likely a good idea
>
>
>
> Are the nodes coming up on their own? Or are you restarting them?
>
>
>
> Paste the output of nodetool tpstats and nodetool cfstats
>
>
>
>
>
>
>
> --
>
> Jeff Jirsa
>
>
>
>
> On Mar 26, 2018, at 7:56 PM, Xiangfei Ni  wrote:
>
> Hi Cassandra experts,
>
>   I am facing an issue,a node downs every day in a 6 nodes cluster,the
> cluster is just in one DC,
>
>   Every node has 4C 16G,and the heap configuration is MAX_HEAP_SIZE=8192m
> HEAP_NEWSIZE=512m,every node load about 200G data,the RF for the business
> CF is 3,a node downs one time every day,the system.log shows below info:
>
> WARN  [Native-Transport-Requests-19] 2018-03-26 18:53:17,128
> CassandraAuthorizer.java:101 - CassandraAuthorizer failed to authorize
> # for 
>
> ERROR [Native-Transport-Requests-19] 2018-03-26 18:53:17,129
> QueryMessage.java:128 - Unexpected error during query
>
> com.google.common.util.concurrent.UncheckedExecutionException:
> java.lang.RuntimeException: 
> org.apache.cassandra.exceptions.ReadTimeoutException:
> Operation timed out - received only 0 responses.
>
> at 
> com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203)
> ~[guava-18.0.jar:na]
>
> at com.google.common.cache.LocalCache.get(LocalCache.java:3937)
> ~[guava-18.0.jar:na]
>
> at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941)
> ~[guava-18.0.jar:na]
>
> at 
> com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)
> ~[guava-18.0.jar:na]
>
> at org.apache.cassandra.auth.AuthCache.get(AuthCache.java:108)
> ~[apache-cassandra-3.9.jar:3.9]
>
> at 
> org.apache.cassandra.auth.PermissionsCache.getPermissions(PermissionsCache.java:45)
> ~[apache-cassandra-3.9.jar:3.9]
>
> at 
> org.apache.cassandra.auth.AuthenticatedUser.getPermissions(AuthenticatedUser.java:104)
> ~[apache-cassandra-3.9.jar:3.9]
>
> at 
> org.apache.cassandra.service.ClientState.authorize(ClientState.java:419)
> ~[apache-cassandra-3.9.jar:3.9]
>
> at org.apache.cassandra.service.ClientState.
> checkPermissionOnResourceChain(ClientState.java:352)
> ~[apache-cassandra-3.9.jar:3.9]
>
> at 
> org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:329)
> ~[apache-cassandra-3.9.jar:3.9]
>
> at 
> org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:316)
> ~[apache-cassandra-3.9.jar:3.9]
>
> at 
> org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:300)
> ~[apache-cassandra-3.9.jar:3.9]
>
> at org.apache.cassandra.cql3.statements.ModificationStatement.
> checkAccess(ModificationStatement.java:211) ~[apache-cassandra-3.9.jar:3.
> 9]
>
> at 
> org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:185)
> ~[apache-cassandra-3.9.jar:3.9]
>
> at 
> org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:219)
> ~[apache-cassandra-3.9.jar:3.9]
>
> at 
> org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:204)
> ~[apache-cassandra-3.9.jar:3.9]
>
> at 
> org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:115)
> ~[apache-cassandra-3.9.jar:3.9]
>
> at 
> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:513)
> [apache-cassandra-3.9.jar:3.9]
>
> at 
> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:407)
> [apache-cassandra-3.9.jar:3.9]
>
> at io.netty.channel.SimpleChannelInboundHandler.channelRead(
> SimpleChannelInboundHandler.java:105) [netty-all-4.0.39.Final.jar:4.
> 0.39.Final]
>
> at io.netty.channel.AbstractChannelHandlerContext.
> invokeChannelRead(AbstractChannelHandlerContext.java:366)

Single sstable file compaction issue

[wxn...@zjqunshuo.com]

Hi All,
I changed STCS to TWCS months ago and left some old sstable files. Some are 
almost tombstones. To release disk space, I issued compaction command on one 
file by JMX. After the compaction is done, I got one new file with almost the 
same size of the old one. Seems no tombstones are cleaned during the 
compaction. 

Before compaciton:
Max: 01/12/2017 Min: 11/25/2016 Estimated droppable tombstones: 
0.9115440366604225 53G Jan 16 00:36 lb-124337-big-Data.db
After compaction:
Max: 01/12/2017 Min: 11/25/2016 Estimated droppable tombstones: 
0.9114708007586322 53G Mar 27 00:17 lb-143951-big-Data.db

Questions:
1. Why the compaction didn't clean the tombstones?
2. If one file are all tombstones and I want to it manually(including data, 
index, filter, etc), do I need to shutdown the node?

Cheers,
-Simon

Re: A node down every day in a 6 nodes cluster

[Jeff Jirsa]

That warning isn’t sufficient to understand why the node is going down


Cassandra 3.9 has some pretty serious known issues - upgrading to 3.11.3 is 
likely a good idea

Are the nodes coming up on their own? Or are you restarting them?

Paste the output of nodetool tpstats and nodetool cfstats




-- 
Jeff Jirsa


> On Mar 26, 2018, at 7:56 PM, Xiangfei Ni  wrote:
> 
> Hi Cassandra experts,
>   I am facing an issue,a node downs every day in a 6 nodes cluster,the 
> cluster is just in one DC,
>   Every node has 4C 16G,and the heap configuration is MAX_HEAP_SIZE=8192m 
> HEAP_NEWSIZE=512m,every node load about 200G data,the RF for the business CF 
> is 3,a node downs one time every day,the system.log shows below info:
> WARN  [Native-Transport-Requests-19] 2018-03-26 18:53:17,128 
> CassandraAuthorizer.java:101 - CassandraAuthorizer failed to authorize # nev_tsp_sa> for 
> ERROR [Native-Transport-Requests-19] 2018-03-26 18:53:17,129 
> QueryMessage.java:128 - Unexpected error during query
> com.google.common.util.concurrent.UncheckedExecutionException: 
> java.lang.RuntimeException: 
> org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - 
> received only 0 responses.
> at 
> com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203) 
> ~[guava-18.0.jar:na]
> at com.google.common.cache.LocalCache.get(LocalCache.java:3937) 
> ~[guava-18.0.jar:na]
> at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941) 
> ~[guava-18.0.jar:na]
> at 
> com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)
>  ~[guava-18.0.jar:na]
> at org.apache.cassandra.auth.AuthCache.get(AuthCache.java:108) 
> ~[apache-cassandra-3.9.jar:3.9]
> at 
> org.apache.cassandra.auth.PermissionsCache.getPermissions(PermissionsCache.java:45)
>  ~[apache-cassandra-3.9.jar:3.9]
> at 
> org.apache.cassandra.auth.AuthenticatedUser.getPermissions(AuthenticatedUser.java:104)
>  ~[apache-cassandra-3.9.jar:3.9]
> at 
> org.apache.cassandra.service.ClientState.authorize(ClientState.java:419) 
> ~[apache-cassandra-3.9.jar:3.9]
> at 
> org.apache.cassandra.service.ClientState.checkPermissionOnResourceChain(ClientState.java:352)
>  ~[apache-cassandra-3.9.jar:3.9]
> at 
> org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:329)
>  ~[apache-cassandra-3.9.jar:3.9]
> at 
> org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:316) 
> ~[apache-cassandra-3.9.jar:3.9]
> at 
> org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:300)
>  ~[apache-cassandra-3.9.jar:3.9]
> at 
> org.apache.cassandra.cql3.statements.ModificationStatement.checkAccess(ModificationStatement.java:211)
>  ~[apache-cassandra-3.9.jar:3.9]
> at 
> org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:185)
>  ~[apache-cassandra-3.9.jar:3.9]
> at 
> org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:219) 
> ~[apache-cassandra-3.9.jar:3.9]
> at 
> org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:204) 
> ~[apache-cassandra-3.9.jar:3.9]
> at 
> org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:115)
>  ~[apache-cassandra-3.9.jar:3.9]
> at 
> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:513)
>  [apache-cassandra-3.9.jar:3.9]
> at 
> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:407)
>  [apache-cassandra-3.9.jar:3.9]
> at 
> io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
>  [netty-all-4.0.39.Final.jar:4.0.39.Final]
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:366)
>  [netty-all-4.0.39.Final.jar:4.0.39.Final]
> at 
> io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:35)
>  [netty-all-4.0.39.Final.jar:4.0.39.Final]
> at 
> io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:357)
>  [netty-all-4.0.39.Final.jar:4.0.39.Final]
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
> [na:1.8.0_91]
> at 
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
>  [apache-cassandra-3.9.jar:3.9]
> at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109) 
> [apache-cassandra-3.9.jar:3.9]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
> Caused by: java.lang.RuntimeException: 
> org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - 
> received only 0 responses.
> at 
> org.apache.cassandra.auth.CassandraAuthorizer.authorize(CassandraAuthorizer.java:102)
>  

A node down every day in a 6 nodes cluster

[Xiangfei Ni]

Hi Cassandra experts,
  I am facing an issue,a node downs every day in a 6 nodes cluster,the cluster 
is just in one DC,
  Every node has 4C 16G,and the heap configuration is MAX_HEAP_SIZE=8192m 
HEAP_NEWSIZE=512m,every node load about 200G data,the RF for the business CF is 
3,a node downs one time every day,the system.log shows below info:
WARN  [Native-Transport-Requests-19] 2018-03-26 18:53:17,128 
CassandraAuthorizer.java:101 - CassandraAuthorizer failed to authorize # for 
ERROR [Native-Transport-Requests-19] 2018-03-26 18:53:17,129 
QueryMessage.java:128 - Unexpected error during query
com.google.common.util.concurrent.UncheckedExecutionException: 
java.lang.RuntimeException: 
org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - 
received only 0 responses.
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203) 
~[guava-18.0.jar:na]
at com.google.common.cache.LocalCache.get(LocalCache.java:3937) 
~[guava-18.0.jar:na]
at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941) 
~[guava-18.0.jar:na]
at 
com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824) 
~[guava-18.0.jar:na]
at org.apache.cassandra.auth.AuthCache.get(AuthCache.java:108) 
~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.auth.PermissionsCache.getPermissions(PermissionsCache.java:45)
 ~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.auth.AuthenticatedUser.getPermissions(AuthenticatedUser.java:104)
 ~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.service.ClientState.authorize(ClientState.java:419) 
~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.service.ClientState.checkPermissionOnResourceChain(ClientState.java:352)
 ~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:329)
 ~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:316) 
~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:300)
 ~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.cql3.statements.ModificationStatement.checkAccess(ModificationStatement.java:211)
 ~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:185)
 ~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:219) 
~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:204) 
~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:115)
 ~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:513)
 [apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:407)
 [apache-cassandra-3.9.jar:3.9]
at 
io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
 [netty-all-4.0.39.Final.jar:4.0.39.Final]
at 
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:366)
 [netty-all-4.0.39.Final.jar:4.0.39.Final]
at 
io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:35)
 [netty-all-4.0.39.Final.jar:4.0.39.Final]
at 
io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:357)
 [netty-all-4.0.39.Final.jar:4.0.39.Final]
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
[na:1.8.0_91]
at 
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
 [apache-cassandra-3.9.jar:3.9]
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109) 
[apache-cassandra-3.9.jar:3.9]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
Caused by: java.lang.RuntimeException: 
org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - 
received only 0 responses.
at 
org.apache.cassandra.auth.CassandraAuthorizer.authorize(CassandraAuthorizer.java:102)
 ~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.auth.PermissionsCache.lambda$new$0(PermissionsCache.java:37)
 ~[apache-cassandra-3.9.jar:3.9]
at org.apache.cassandra.auth.AuthCache$1.load(AuthCache.java:183) 
~[apache-cassandra-3.9.jar:3.9]
at 
com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
 ~[guava-18.0.jar:na]
at 
com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319) 
~[guava-18.0.jar:na]
at 
com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
 

Re: Measuring eventual consistency latency

[Jeronimo de A. Barros]

Jeff and Christophe,

Thank you very much ! I'll take a look.

Jero

On Sun, Mar 25, 2018 at 10:21 PM, Jeff Jirsa  wrote:

> Probably closer to https://issues.apache.org/jira/browse/CASSANDRA-13289
>
>
> Will be in 4.0
> --
> Jeff Jirsa
>
>
> On Mar 25, 2018, at 4:44 PM, Christophe Schmitz <
> christo...@instaclustr.com> wrote:
>
> Hi Jeronimo,
>
> I am not sure that will address your exact request, but did you look at
> this issue (resolved in 3.8) which adds a ned latency across DCs metrics?
> https://issues.apache.org/jira/browse/CASSANDRA-11569
>
> Cheers,
>
> Christophe
>
> On 26 March 2018 at 10:01, Jeronimo de A. Barros <
> jeronimo.bar...@gmail.com> wrote:
>
>> I'd like to know if there is a reasonable method to measure how long take
>> to have the data available across all replica nodes in a multi DC
>> environment using LOCAL_ONE or LOCAL_QUORUM consistency levels.
>>
>> If already there be a study about this topic in some place and someone
>> could point me the direction, it will be of great help.
>>
>> Thanks !
>>
>
>
>
> --
>
> *Christophe Schmitz - **VP Consulting*
>
> AU: +61 4 03751980 / FR: +33 7 82022899
>
>    
>
>
> Read our latest technical blog posts here
> . This email has been sent on behalf
> of Instaclustr Pty. Limited (Australia) and Instaclustr Inc (USA). This
> email and any attachments may contain confidential and legally
> privileged information.  If you are not the intended recipient, do not copy
> or disclose its content, but please reply to this email immediately and
> highlight the error to the sender and then immediately delete the message.
>
>

RE: LWT on data mutated by non-LWT operation is valid ?

[Jacques-Henri Berthemet]

If you check the Jira issue I linked you'll see a recent comment describing a 
potential explanation for my mixed LWT/non-LWT problem. So, it looks like there 
can be some edge cases.

I'd say that if data was inserted a while ago (seconds) there should be no 
problems.

--
Jacques-Henri Berthemet

-Original Message-
From: Hiroyuki Yamada [mailto:mogwa...@gmail.com] 
Sent: Sunday, March 25, 2018 1:10 AM
To: user@cassandra.apache.org
Subject: Re: LWT on data mutated by non-LWT operation is valid ?

Thank you JH.
But, it's still a little bit unclear to me

Let me clarify the question.
What I wanted to know is whether or not linearizability is sustained by doing 
LWT  (Consistency: QUORUM, Serial Consistency: SERIAL) on data previously 
mutated by non-LWT (Consistency: QUORUM).
I think It should be OK if non-LWT surely correctly happened before the LWT, 
but I'm wondering if there is a corner case where it's not OK.

For example, to test LWT (Update) operations, initial data should be inserted 
by LWT operations ? or it can be non-LWT operations ?

Thanks,
Hiroyuki

On Sat, Mar 24, 2018 at 8:27 PM, Jacques-Henri Berthemet 
 wrote:
> Hi Hiroyuki,
>
>
> For both operations you'll have to provide partition key so "conflict" 
> at DB level can always be resolved.
>
> But if two operations, LWT and non-LWT, are racing against each others 
> the result is unpredictable, if non-LWT is applied after LWT the 
> result will be overwritten.
>
>
> It seems mixing LWT and non-LWT can result in strange results, we 
> recently opened a bug on non-working delete after LWT insert:
> https://issues.apache.org/jira/browse/CASSANDRA-14304
> .apache.org
>
>
> Regards,
>
> JH
>
> 
> From: Hiroyuki Yamada 
> Sent: Saturday, March 24, 2018 4:38:15 AM
> To: user@cassandra.apache.org
> Subject: LWT on data mutated by non-LWT operation is valid ?
>
> Hi all,
>
> I have some question about LWT.
>
> I am wondering if LWT works only for data mutated by LWT or not.
> In other words, doing LWT on some data mutated by non-LWT operations 
> is still valid ?
> I don't fully understand how system.paxos table works in LWT, but 
> row_key should be empty for a data mutated by non-LWT operation, so 
> conflict resolution seems impossible.
> It works only if a previous non-LWT operation is completely finished ?
>
> Thanks in advance.
>
> Best regards,
> Hiroyuki
>
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
>

-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org


-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org