Commit Log sync problems

[Meg Mara]

Hello all,

I recently upgraded from C* 3.0.10 to 3.0.16 and have been receiving these 
warnings about Commit-Log durations being longer than the configured interval. 
I don't understand what the problem is, why is the system complaining about 
such small sync durations? Please advice.

Here are some warnings:


3:WARN  [PERIODIC-COMMIT-LOG-SYNCER] 2019-03-07 13:32:23,785 
NoSpamLogger.java:94 - Out of 2750 commit log syncs over the past 274s with 
average duration of 0.66ms, 4 have exceeded the configured commit interval by 
an average of 22.50ms



2:WARN  [PERIODIC-COMMIT-LOG-SYNCER] 2019-03-07 13:27:01,214 
NoSpamLogger.java:94 - Out of 1 commit log syncs over the past 0s with average 
duration of 113.00ms, 1 have exceeded the configured commit interval by an 
average of 13.00ms


Node's cassandra.yaml setting related to Commit-Log:

commit_failure_policy: stop
commitlog_directory: /cassandra/log
commitlog_segment_size_in_mb: 32
commitlog_sync: periodic
commitlog_sync_period_in_ms: 1
commitlog_total_space_in_mb: 4096

Thank you,
Meg Mara

RE: TWCS: Repair create new buckets with old data

[Meg Mara]

Hey Jon,

About table level TTL -> It wasn’t for optimization, just  a suggestion. The 
user had no table level TTL set. It was at default 0. So if an insert comes in 
with no TTL, that row would never expire. There is no default TTL to fall back 
on in his case. Just thinking about possible problems with that setup and data 
never expiring.

About inc repair -> Yup, I agree, I did some extensive research into that 
earlier this year. We’re just waiting for 4.0 as well. But for some of our 
larger clusters, doing a full repair, even if it’s just with the –pr option is 
just not feasible. It takes weeks to repair and is a huge load on the cluster.

So I’m continuing to use Inc repair and keeping an eye out for any issues. And 
I’m using –pr for the smaller clusters. But for tables with TWCS, I wouldn’t do 
a Full repair. Because it would include all the old and newer sstables in the 
operation and it is bound to cause new sstables with mixed data and may 
eventually lead to sstable blockers. It also kind of defeats the purpose of 
separating our data into neat TWCS windows. And therefore would take a while 
for tombstone cleanup too.

Thank you,
Meg


From: Jonathan Haddad [mailto:j...@jonhaddad.com]
Sent: Wednesday, October 24, 2018 1:42 PM
To: user 
Subject: Re: TWCS: Repair create new buckets with old data

Hey Meg, a couple thoughts.

>   Set a table level TTL with TWCS, and stop setting it with inserts/updates 
> (insert TTL overrides table level TTL). So, that your entire sstable expires 
> at the same time, as opposed to each insert expiring at its own pace. So that 
> for tombstone clean up, the system can just drop the entire sstable at once.

Setting the TTL on a table or the query gives you the same exact result.  
Setting it on the table is just there for convenience.  If it's not set at the 
query level, it uses the default value.  See 
org.apache.cassandra.cql3.Attributes#getTimeToLive.  Generally speaking I'd 
rather set it at the table level as well, but it's just to avoid weird 
application bugs, not as an optimization.

>   I’d suggest removing the -pr. Running incremental repair with TWCS is 
> better.

If incremental repair worked correctly I would agree with you, but 
unfortunately it doesn't.  Incremental repair has some subtle bugs that can 
result in massive overstreaming due to the fact that it will sometimes not 
correctly mark data as repaired.  My coworker Alex wrote up a good summary on 
the changes to incremental going into 4.0 to fix these issues, it's worth a 
read.  
http://thelastpickle.com/blog/2018/09/10/incremental-repair-improvements-in-cassandra-4.html.

Reaper (OSS, maintained by us @ TLP, see http://cassandra-reaper.io/) has the 
ability to schedule subrange repairs on one or more tables, or all tables 
except those in a blacklist.  Doing frequent subrange repairs will limit the 
amount of data that will get streamed in and should help keep things pretty 
consistent unless you're dropping a lot of mutations.  It's not perfect but 
should cause less headache than incremental repair will.

Hope this helps.
Jon



On Thu, Oct 25, 2018 at 4:21 AM Meg Mara 
mailto:mm...@digitalriver.com>> wrote:
Hi Maik,

I have a similar Cassandra env, with similar table requirements. So these would 
be my suggestions:


•   Set a table level TTL with TWCS, and stop setting it with 
inserts/updates (insert TTL overrides table level TTL). So, that your entire 
sstable expires at the same time, as opposed to each insert expiring at its own 
pace. So that for tombstone clean up, the system can just drop the entire 
sstable at once.

•   Since you’re on v3.0.9, nodetool repair command runs incremental repair 
by default. And with inc repair, -pr option is not recommended. (ref. link 
below)

•   I’d suggest removing the -pr. Running incremental repair with TWCS is 
better.

•   Here’s why I think so -> Full repair and Full repair with –PR option 
would include all  the sstables in the repair process, which means the chance 
of your oldest and newest data mixing is very high.

•   Whereas, if you run incremental repair every 5 days for example, only 
the last five days of data would be included in that repair operation. So, the 
maximum ‘damage’ it would do is mixing 5 day old data in a new sstable.

•   Your table level TTL would then tombstone this data on 4 month + 5 day 
mark instead of on the 4 month mark. Which shouldn’t be a big concern. At least 
in our case it isn’t!

•   I wouldn’t stop running repairs on our TWCS tables, because we are too 
concerned with data consistency.


Please read the note here:
https://docs.datastax.com/en/cassandra/3.0/cassandra/tools/toolsRepair.html


Thank you,
Meg


From: Caesar, Maik [mailto:maik.cae...@dxc.com<mailto:maik.cae...@dxc.com>]
Sent: Wednesday, October 24, 2018 2:17 AM
To: user@cassandra.apache.org<mailto:user@cassandra.apache.org>
Subject: RE: TWCS: Repair create 

RE: TWCS: Repair create new buckets with old data

[Meg Mara]

Hi Maik,

I have a similar Cassandra env, with similar table requirements. So these would 
be my suggestions:


·   Set a table level TTL with TWCS, and stop setting it with 
inserts/updates (insert TTL overrides table level TTL). So, that your entire 
sstable expires at the same time, as opposed to each insert expiring at its own 
pace. So that for tombstone clean up, the system can just drop the entire 
sstable at once.

·   Since you’re on v3.0.9, nodetool repair command runs incremental repair 
by default. And with inc repair, -pr option is not recommended. (ref. link 
below)

·   I’d suggest removing the -pr. Running incremental repair with TWCS is 
better.

·   Here’s why I think so -> Full repair and Full repair with –PR option 
would include all  the sstables in the repair process, which means the chance 
of your oldest and newest data mixing is very high.

·   Whereas, if you run incremental repair every 5 days for example, only 
the last five days of data would be included in that repair operation. So, the 
maximum ‘damage’ it would do is mixing 5 day old data in a new sstable.

·   Your table level TTL would then tombstone this data on 4 month + 5 day 
mark instead of on the 4 month mark. Which shouldn’t be a big concern. At least 
in our case it isn’t!

·   I wouldn’t stop running repairs on our TWCS tables, because we are too 
concerned with data consistency.


Please read the note here:
https://docs.datastax.com/en/cassandra/3.0/cassandra/tools/toolsRepair.html


Thank you,
Meg


From: Caesar, Maik [mailto:maik.cae...@dxc.com]
Sent: Wednesday, October 24, 2018 2:17 AM
To: user@cassandra.apache.org
Subject: RE: TWCS: Repair create new buckets with old data

Hi Meg,
the ttl (4 month) is set during insert via insert statement with the 
application.
The repair is started each day on one of ten hosts with command : nodetool 
--host hostname_# repair –pr

Regards
Maik

From: Meg Mara [mailto:mm...@digitalriver.com]
Sent: Dienstag, 23. Oktober 2018 17:05
To: user@cassandra.apache.org<mailto:user@cassandra.apache.org>
Subject: RE: TWCS: Repair create new buckets with old data

Hi Maik,

I noticed in your table description that your default_time_to_live = 0, so 
where is your TTL property set? At what point do your sstables get tombstoned?

Also, could you please mention what kind of repair you performed on this table? 
(Incremental, Full, Full repair with -pr option)

Thank you,
Meg


From: Caesar, Maik [mailto:maik.cae...@dxc.com]
Sent: Monday, October 22, 2018 10:17 AM
To: user@cassandra.apache.org<mailto:user@cassandra.apache.org>
Subject: RE: TWCS: Repair create new buckets with old data

Ok, thanks.
My conclusion:

1.   I will set unchecked_tombstone_compaction to true to get old data with 
tombstones removed

2.   I will exclude TWCS tables from repair

Regarding exclude table from repair, is there any easy way to do this?  
Nodetool repaire do not support excludes.

Regards
Maik

From: wxn...@zjqunshuo.com<mailto:wxn...@zjqunshuo.com> 
[mailto:wxn...@zjqunshuo.com]
Sent: Freitag, 19. Oktober 2018 03:58
To: user mailto:user@cassandra.apache.org>>
Subject: RE: TWCS: Repair create new buckets with old data

> Is the repair not necessary to get data files remove from filesystem ?
The answer is no. IMO, Cassandra will remove sstable files automatically if it 
can make sure the sstable files are 100% of tombstones and safe to do deletion. 
If you use TWCS and you have only insertion and no update, you don't need run 
repair manually.

-Simon

From: Caesar, Maik<mailto:maik.cae...@dxc.com>
Date: 2018-10-18 20:30
To: user@cassandra.apache.org<mailto:user@cassandra.apache.org>
Subject: RE: TWCS: Repair create new buckets with old data
Hello Simon,
Is the repair not necessary to get data files remove from filesystem ? My 
assumption was, that only repaired data will removed after TTL is reached.

Regards
Maik

From: wxn...@zjqunshuo.com<mailto:wxn...@zjqunshuo.com> 
[mailto:wxn...@zjqunshuo.com]
Sent: Mittwoch, 17. Oktober 2018 09:02
To: user mailto:user@cassandra.apache.org>>
Subject: Re: TWCS: Repair create new buckets with old data

Hi Maik,
IMO, when using TWCS, you had better not run repair. The behaviour of TWCS is 
same with STCS for repair when merging sstables and the result is leaving 
sstables spanning multiple time buckets, but maybe I'm wrong. In my use case, I 
don't do repair with table using TWCS.

-Simon

From: Caesar, Maik<mailto:maik.cae...@dxc.com>
Date: 2018-10-16 17:46
To: user@cassandra.apache.org<mailto:user@cassandra.apache.org>
Subject: TWCS: Repair create new buckets with old data
Hallo,
we work with Cassandra version 3.0.9 and have a problem in a table with TWCS. 
The command “nodetool repair” create always new files with old data. This avoid 
the delete of the old data.
The layout of the Table is following:
cqlsh> desc stat.spa

CREATE TABLE stat.spa (
region in

RE: TWCS: Repair create new buckets with old data

[Meg Mara]

Hi Maik,

I noticed in your table description that your default_time_to_live = 0, so 
where is your TTL property set? At what point do your sstables get tombstoned?

Also, could you please mention what kind of repair you performed on this table? 
(Incremental, Full, Full repair with -pr option)

Thank you,
Meg


From: Caesar, Maik [mailto:maik.cae...@dxc.com]
Sent: Monday, October 22, 2018 10:17 AM
To: user@cassandra.apache.org
Subject: RE: TWCS: Repair create new buckets with old data

Ok, thanks.
My conclusion:

1.   I will set unchecked_tombstone_compaction to true to get old data with 
tombstones removed

2.   I will exclude TWCS tables from repair

Regarding exclude table from repair, is there any easy way to do this?  
Nodetool repaire do not support excludes.

Regards
Maik

From: wxn...@zjqunshuo.com 
[mailto:wxn...@zjqunshuo.com]
Sent: Freitag, 19. Oktober 2018 03:58
To: user mailto:user@cassandra.apache.org>>
Subject: RE: TWCS: Repair create new buckets with old data

> Is the repair not necessary to get data files remove from filesystem ?
The answer is no. IMO, Cassandra will remove sstable files automatically if it 
can make sure the sstable files are 100% of tombstones and safe to do deletion. 
If you use TWCS and you have only insertion and no update, you don't need run 
repair manually.

-Simon

From: Caesar, Maik
Date: 2018-10-18 20:30
To: user@cassandra.apache.org
Subject: RE: TWCS: Repair create new buckets with old data
Hello Simon,
Is the repair not necessary to get data files remove from filesystem ? My 
assumption was, that only repaired data will removed after TTL is reached.

Regards
Maik

From: wxn...@zjqunshuo.com 
[mailto:wxn...@zjqunshuo.com]
Sent: Mittwoch, 17. Oktober 2018 09:02
To: user mailto:user@cassandra.apache.org>>
Subject: Re: TWCS: Repair create new buckets with old data

Hi Maik,
IMO, when using TWCS, you had better not run repair. The behaviour of TWCS is 
same with STCS for repair when merging sstables and the result is leaving 
sstables spanning multiple time buckets, but maybe I'm wrong. In my use case, I 
don't do repair with table using TWCS.

-Simon

From: Caesar, Maik
Date: 2018-10-16 17:46
To: user@cassandra.apache.org
Subject: TWCS: Repair create new buckets with old data
Hallo,
we work with Cassandra version 3.0.9 and have a problem in a table with TWCS. 
The command “nodetool repair” create always new files with old data. This avoid 
the delete of the old data.
The layout of the Table is following:
cqlsh> desc stat.spa

CREATE TABLE stat.spa (
region int,
id int,
date text,
hour int,
zippedjsonstring blob,
PRIMARY KEY ((region, id), date, hour)
) WITH CLUSTERING ORDER BY (date ASC, hour ASC)
AND bloom_filter_fp_chance = 0.01
AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
AND comment = ''
AND compaction = {'class': 
'org.apache.cassandra.db.compaction.TimeWindowCompactionStrategy', 
'compaction_window_size': '1', 'compaction_window_unit': 'DAYS', 
'max_threshold': '100', 'min_threshold': '4', 'tombstone_compaction_interval': 
'86460'}
AND compression = {'chunk_length_in_kb': '64', 'class': 
'org.apache.cassandra.io.compress.LZ4Compressor'}
AND crc_check_chance = 1.0
AND dclocal_read_repair_chance = 0.0
AND default_time_to_live = 0
AND gc_grace_seconds = 864000
AND max_index_interval = 2048
AND memtable_flush_period_in_ms = 0
AND min_index_interval = 128
AND read_repair_chance = 0.0
AND speculative_retry = '99PERCENTILE';

Actual the oldest data are from 2017/04/15 and will not remove:

$ for f in *Data.db; do meta=$(sudo sstablemetadata $f); echo -e "Max:" $(date 
--date=@$(echo "$meta" | grep Maximum\ time | cut -d" "  -f3| cut -c 1-10) 
'+%Y/%m/%d %H:%M') "Min:" $(date --date=@$(echo "$meta" | grep Minimum\ time | 
cut -d" "  -f3| cut -c 1-10) '+%Y/%m/%d %H:%M') $(echo "$meta" | grep 
droppable) $(echo "$meta" | grep "Repaired at") ' \t ' $(ls -lh $f | awk 
'{print $5" "$6" "$7" "$8" "$9}'); done | sort
Max: 2017/04/15 12:08 Min: 2017/03/31 13:09 Estimated droppable tombstones: 
1.7731048805815162 Repaired at: 1525685601400 42K May 7 19:56 
mc-22922-big-Data.db
Max: 2017/04/17 13:49 Min: 2017/03/31 13:09 Estimated droppable tombstones: 
1.9600207684319835 Repaired at: 1525685601400 116M May 7 13:31 
mc-15096-big-Data.db
Max: 2017/04/21 13:43 Min: 2017/04/15 13:34 Estimated droppable tombstones: 
1.9090909090909092 Repaired at: 1525685601400 11K May 7 19:56 
mc-22921-big-Data.db
Max: 2017/05/23 21:45 Min: 2017/04/21 14:00 Estimated droppable tombstones: 
1.8360655737704918 Repaired at: 1525685601400 21M May 7 19:56 
mc-22919-big-Data.db
Max: 2017/06/12 15:19 Min: 2017/04/25 14:45 Estimated droppable tombstones: 
1.8091397849462365 Repaired at: 

RE: how to check C* partition size

[Meg Mara]

Nodetool tablestats gives you a general idea.

Meg Mara


From: Peng Xiao [mailto:2535...@qq.com]
Sent: Sunday, January 07, 2018 9:26 AM
To: user <user@cassandra.apache.org>
Subject: how to check C* partition size

Hi guys,

Could anyone please help on this simple question?
How to check C* partition size and related information.
looks nodetool ring only shows the token distribution.

Thanks

RE: Tombstone warnings in log file

[Meg Mara]

Simon,

It means that in processing your queries, Cassandra is going through that many 
tombstone cells in order to return your results. It is because some of the 
partitions that you are querying for have already expired. The warning is just 
cassandra's way of letting you know that your reads are less efficient because 
you are reading a lot of expired data.

You could tune this by altering your tombstone parameters in Cassandra yaml 
file. But a better solution would be to reduce your GC grace seconds for that 
table to a smaller value (as opposed to default of 10 days) so that the TTLed 
data will be purged sooner.

You could also consider drafting more efficient queries which won't hit TTLed 
partitions.

Thanks,
Meg


From: wxn...@zjqunshuo.com [mailto:wxn...@zjqunshuo.com]
Sent: Sunday, December 03, 2017 7:49 PM
To: user 
Subject: Tombstone warnings in log file

Hi,
My cluster is running 2.2.8, no update and deletion, only insertion with TTL.  
I saw below warnings reacently. What's the meaning of them and what's the 
impact?

WARN  [SharedPool-Worker-2] 2017-12-04 09:32:48,833 SliceQueryFilter.java:308 - 
Read 2461 live and 1978 tombstone cells in cargts.eventdata for key: 
129762:20171202 (see tombstone_warn_threshold). 5000 columns were requested, 
slices=[-]


Best regards,
-Simon

WARN  [SharedPool-Worker-2] 2017-12-04 09:32:48,833 SliceQueryFilter.java:308 - 
Read 2461 live and 1978 tombstone cells in cargts.eventdata for key: 
129762:20171202 (see tombstone_warn_threshold). 5000 columns were requested, 
slices=[-]

inc repairs aren't updating repairedat timestamp

[Meg Mara]

Hello,

I have v3.0.10. Cassandra incremental repair has not been updating the 
repairedat timestamp on the sstables. Because of this jira : 
https://issues.apache.org/jira/browse/CASSANDRA-9143. Going back to repair 
documentation, Datastax now seems to have changed their recommendation from 
incremental to full repairs. 
https://docs.datastax.com/en/cassandra/3.0/cassandra/tools/toolsRepair.html

But full repairs are much less performant than incremental. Is anyone else 
dealing with this issue? How are you verifying that incremental repairs are 
functioning inspite of the repaired-at timestamp never getting updated? What is 
the solution to this apart from moving to Cassandra 3.0.14 (not feasible for me 
at the moment.)? I'd appreciate any answers.

Thank you,
Meg Mara

RE: how to identify the root cause of cassandra hang

[Meg Mara]

Hello,

It could happen if your GC pauses are too long and/or too frequent. If your 
heap sizes are not large enough. When a long GC happens, Cassandra node 
effectively behaves like a dead node (unresponsive). Other nodes start 
collecting hints for it etc. Maybe you should look into your logs to see if 
your GC pauses are happening too often. Grep for GCInspector in system.log. 
Could be a possibility.

Meg Mara


From: Peng Xiao [mailto:2535...@qq.com]
Sent: Thursday, October 26, 2017 9:24 AM
To: user <user@cassandra.apache.org>
Subject: how to identify the root cause of cassandra hang

Hi,

We have a cluster with 48 nodes configured with RACK,sometimes it's hang for 
even 2 minutes.the response time jump from 300ms to 15s.
Could anyone please advise how to identified the root cause ?

The following is from the system log

INFO  [Service Thread] 2017-10-26 21:45:46,796 GCInspector.java:258 - G1 Young 
Generation GC in 222ms.  G1 Eden Space: 939524096 -> 0; G1 Old Gen: 6652738584 
-> 6662878232; G1 Survivor Space: 134217728 -> 109051904;
INFO  [Service Thread] 2017-10-26 21:45:46,796 StatusLogger.java:51 - Pool Name 
   Active   Pending  Completed   Blocked  All Time Blocked
INFO  [Service Thread] 2017-10-26 21:45:46,796 StatusLogger.java:66 - 
MutationStage 0 3 3612475121 0  
   0
INFO  [Service Thread] 2017-10-26 21:45:46,796 StatusLogger.java:66 - 
RequestResponseStage  0 0 6333593550 0  
   0
INFO  [Service Thread] 2017-10-26 21:45:46,796 StatusLogger.java:66 - 
ReadRepairStage   0 02773154 0  
   0
INFO  [Service Thread] 2017-10-26 21:45:46,796 StatusLogger.java:66 - 
CounterMutationStage  0 0  0 0  
   0
INFO  [Service Thread] 2017-10-26 21:45:46,796 StatusLogger.java:66 - ReadStage 
0 4  417419357 0 0

Thanks.

RE: cassandra non-super user login fails but super user works

[Meg Mara]

You should probably verify if the ‘can_login’ field of the non-superuser role 
is set to true. You can query the column family system_auth.roles to find out.

Thanks,
Meg Mara

From: Justin Cameron [mailto:jus...@instaclustr.com]
Sent: Sunday, October 22, 2017 6:21 PM
To: user@cassandra.apache.org
Subject: Re: cassandra non-super user login fails but super user works

Try setting the replication factor of the system_auth keyspace to the number of 
nodes in your cluster.

ALTER KEYSPACE system_auth WITH replication = {'class': 
'NetworkTopologyStrategy', '': ''};

On Sun, 22 Oct 2017 at 20:06 Who Dadddy 
<qwerty15...@gmail.com<mailto:qwerty15...@gmail.com>> wrote:
Anyone seen this before? Pretty basic setup, super user can login fine but 
non-super user can’t?

Any pointers appreciated.



-
To unsubscribe, e-mail: 
user-unsubscr...@cassandra.apache.org<mailto:user-unsubscr...@cassandra.apache.org>
For additional commands, e-mail: 
user-h...@cassandra.apache.org<mailto:user-h...@cassandra.apache.org>
--
Justin Cameron
Senior Software Engineer

[https://cdn2.hubspot.net/hubfs/2549680/Instaclustr-Navy-logo-new.png]<https://www.instaclustr.com/>

This email has been sent on behalf of Instaclustr Pty. Limited (Australia) and 
Instaclustr Inc (USA).

This email and any attachments may contain confidential and legally privileged 
information.  If you are not the intended recipient, do not copy or disclose 
its content, but please reply to this email immediately and highlight the error 
to the sender and then immediately delete the message.

Error Exception in Repair Thread

[Meg Mara]

Hello Cassandra Experts!

I have seen the following errors in the system log when a scheduled nodetool 
repair operation runs on the cluster. Cassandra Version 3.0.10. Any thoughts or 
suggestions are welcome!

ERROR [Repair#3794:3] 2017-07-27 17:47:22,000 CassandraDaemon.java:207 - 
Exception in thread Thread[Repair#3794:3,5,RMI Runtime]
java.lang.AssertionError: java.lang.InterruptedException
at 
org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.extractThrowable(DebuggableThreadPoolExecutor.java:265)
 ~[apache-cassandra-3.0.10.jar:3.0.10]
at 
org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.logExceptionsAfterExecute(DebuggableThreadPoolExecutor.java:225)
 ~[apache-cassandra-3.0.10.jar:3.0.10]
at 
org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.afterExecute(DebuggableThreadPoolExecutor.java:196)
 ~[apache-cassandra-3.0.10.jar:3.0.10]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1150) 
~[na:1.8.0_101]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
~[na:1.8.0_101]
at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_101]
Caused by: java.lang.InterruptedException: null
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1302)
 ~[na:1.8.0_101]
at 
com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:285)
 ~[guava-18.0.jar:na]
at 
com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116) 
~[guava-18.0.jar:na]
at 
org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.extractThrowable(DebuggableThreadPoolExecutor.java:261)
 ~[apache-cassandra-3.0.10.jar:3.0.10]
... 5 common frames omitted

Thank you,
Meg Mara

RE: Hints files are not getting truncated

[Meg Mara]

Anuj,

Thank you for your explanation. I had misread the documentation and was under 
the impression all these days that Hints_window refers to ‘time until 
deletion’. Also, thanks for the information about TCP settings, I will have to 
recheck the existing configuration. I will also test the performance 
improvement of increasing the threads for the hint delivery.

Thank you,
Meg Mara

From: Anuj Wadehra [mailto:anujw_2...@yahoo.co.in.INVALID]
Sent: Tuesday, June 27, 2017 11:56 AM
To: user@cassandra.apache.org
Subject: Re: Hints files are not getting truncated

Hi Meg,

max_hint_window_in_ms =3 hrs means that if a node is down/unresponsive for more 
than 3 hrs, hints would not be stored for it any further until it becomes 
responsive again. It should not mean that already stored hints would be 
truncated after 3 hours.

Regarding connection timeouts between DCs, please check your firewall settings 
and tcp settings on node. Firewall between the DC must not kill an idle 
connection which is still considered to be usable by Cassandra. Please see 
http://docs.datastax.com/en/archived/cassandra/2.0/cassandra/troubleshooting/trblshootIdleFirewall.html
 .

In multi dc setup, documentation recommends to increase number of threads. You 
can try increasing it and check whether it improves the situation.

Thanks
Anuj


On Tue, Jun 27, 2017 at 9:47 PM, Meg Mara
<mm...@digitalriver.com<mailto:mm...@digitalriver.com>> wrote:

Hello,



I am facing an issue with Hinted Handoff files in Cassandra v3.0.10. A DC1 node 
is storing large number of hints for DC2 nodes (we are facing connection 
timeout issues). The problem is that the hint files which are created on DC1 
are not getting deleted after the 3 hour window. Hints are now being stored as 
flat files in the Cassandra home directory and I can see that old hints are 
being deleted but at a very slow pace. It still contains hints from May.

max_hint_window_in_ms: 1080

max_hints_delivery_threads: 2



Why do you suppose this is happening? Any suggestions or recommendations would 
be much appreciated.



Thanks for your time.

Meg Mara

RE: Hints files are not getting truncated

[Meg Mara]

Hannu,

Thanks for your response. Yes our gc grace period is at default setting, so you 
are right, I do not need hints which are a month old. Especially since I have 
scheduled repairs running on the cluster almost every day and cross DC repair 
running every week. The network connection is slow but we have never faced any 
major issues apart from some dropped mutation messages from time to time.

I will consider your advice about turning off hints.

Thank you,
Meg Mara

From: Hannu Kröger [mailto:hkro...@gmail.com]
Sent: Tuesday, June 27, 2017 11:37 AM
To: Meg Mara <mm...@digitalriver.com>; user@cassandra.apache.org
Subject: Re: Hints files are not getting truncated

Hi,

First of all, I don’t know why they get delivered so slowly.

However, if your gc grace seconds is the default 10 days then those hints from 
May are not needed and could/should be truncated. If the hint delivery is 
causing problems, then one option is that you could just disable it and rely on 
periodic repair doing its job and getting the data synchronized.

Now my question to you: Do you have repairs running periodically across the 
cluster and actually succeeding if the connection is flaky?

Hannu


On 27 June 2017 at 19:17:26, Meg Mara 
(mm...@digitalriver.com<mailto:mm...@digitalriver.com>) wrote:
Hello,

I am facing an issue with Hinted Handoff files in Cassandra v3.0.10. A DC1 node 
is storing large number of hints for DC2 nodes (we are facing connection 
timeout issues). The problem is that the hint files which are created on DC1 
are not getting deleted after the 3 hour window. Hints are now being stored as 
flat files in the Cassandra home directory and I can see that old hints are 
being deleted but at a very slow pace. It still contains hints from May.
max_hint_window_in_ms: 1080
max_hints_delivery_threads: 2

Why do you suppose this is happening? Any suggestions or recommendations would 
be much appreciated.

Thanks for your time.
Meg Mara

Hints files are not getting truncated

[Meg Mara]

Hello,

I am facing an issue with Hinted Handoff files in Cassandra v3.0.10. A DC1 node 
is storing large number of hints for DC2 nodes (we are facing connection 
timeout issues). The problem is that the hint files which are created on DC1 
are not getting deleted after the 3 hour window. Hints are now being stored as 
flat files in the Cassandra home directory and I can see that old hints are 
being deleted but at a very slow pace. It still contains hints from May.
max_hint_window_in_ms: 1080
max_hints_delivery_threads: 2

Why do you suppose this is happening? Any suggestions or recommendations would 
be much appreciated.

Thanks for your time.
Meg Mara

RE: 2.1 to 3.0 upgrade problems

[Meg Mara]

Hi Eric,

I am not sure why the commit logs were growing so much but I do have some 
suggestions regarding the Unexpected exception during request and about 
upgradesstables. I had recently upgraded from 2.2.5 to 3.0.10 and faced the 
same issue with nodetool upgradesstables not converting all SStables. To 
convert them all, you can run "nodetool upgradesstables -a" and then do a check 
to see which sstables are still in old format. In your /cassandra/data/ 
directory use the find command to search for older format sstables named as 
-la- . This command should work "find . -name "la*" | grep -v snapshots".

Then run "nodetool upgradesstables keyspace_name" for each of the keyspaces 
listed. For me, primarily the system columnfamilies were left in older format.

And about that connection request exceptions, they could be caused if the 
connecting client is on an older incompatible java driver. Upgrade the java 
driver to be compatible with your Cassandra version. Hope that helps.

Thanks,
Meg Mara 

-Original Message-
From: Eric Yablonowitz [mailto:er...@jana.com] 
Sent: Tuesday, March 28, 2017 9:48 PM
To: user@cassandra.apache.org
Subject: 2.1 to 3.0 upgrade problems

We just upgraded our 33 node cluster from 2.0.10 to 2.1.15 to 3.0.12.  The 
upgrade from 2.0.10 to 2.1.15 went very smoothly - both the rolling software 
update and the subsequent ‘nodetool upgradesstables’.  However we have had a 
number of issues with the 3.0.12 upgrade:

The first issue we noticed (well into the rolling upgrade) was that the schema 
versions for the upgraded nodes were changing with each new node added.  
Perhaps related to https://issues.apache.org/jira/browse/CASSANDRA-13274 ?  
This seemed innocuous until we noticed that commit logs seemed to be growing 
without bound.  We attempted to force replay of commit logs by restarting an 
upgraded node.  However upon restart we hit:

ERROR [main] 2017-03-27 19:22:38,929 CommitLogReplayer.java:677 - Ignoring 
commit log replay error
org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException: 
Unexpected error deserializing mutation; saved to 
/tmp/mutation1621984759842154734dat.  This may be caused by replaying a 
mutation against a table with the same name but incompatible schema.   
Exception follows: java.io.IOError: java.io.IOException: Corrupt empty row 
found in unfiltered partition

We were up against the wall with disks filling and had to get nodes restarted 
or face an outage so so we used the cassandra.commitlog.ignorereplayerrors=true 
JVM option to get Cassandra nodes restarted.  Once we had all nodes upgraded, 
the schema version stabilized and we stopped seeing any issues with commitlog 
replay.

However we are still getting a small number of these seemingly related errors 
(3-4 per hour):

ERROR [SharedPool-Worker-3] 2017-03-28 23:50:33,998 Message.java:621 - 
Unexpected exception during request; channel = [id: 0x2366241f, 
L:/10.7.150.165:9042 - R:/10.179.229.62:32119]
java.io.IOError: java.io.IOException: Corrupt empty row found in unfiltered 
partition
at 
org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer$1.computeNext(UnfilteredRowIteratorSerializer.java:222)
 ~[apache-cassandra-3.0.12.jar:3.0.12]
at 
org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer$1.computeNext(UnfilteredRowIteratorSerializer.java:210)
 ~[apache-cassandra-3.0.12.jar:3.0.12]
at 
org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) 
~[apache-cassandra-3.0.12.jar:3.0.12]
at 
org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:129) 
~[apache-cassandra-3.0.12.jar:3.0.12]
at 
org.apache.cassandra.db.transform.FilteredRows.isEmpty(FilteredRows.java:50) 
~[apache-cassandra-3.0.12.jar:3.0.12]
at 
org.apache.cassandra.db.transform.Filter.closeIfEmpty(Filter.java:73) 
~[apache-cassandra-3.0.12.jar:3.0.12]
at 
org.apache.cassandra.db.transform.Filter.applyToPartition(Filter.java:43) 
~[apache-cassandra-3.0.12.jar:3.0.12]
at 
org.apache.cassandra.db.transform.Filter.applyToPartition(Filter.java:26) 
~[apache-cassandra-3.0.12.jar:3.0.12]
at 
org.apache.cassandra.db.transform.BasePartitions.hasNext(BasePartitions.java:96)
 ~[apache-cassandra-3.0.12.jar:3.0.12]
at 
org.apache.cassandra.cql3.statements.SelectStatement.process(SelectStatement.java:707)
 ~[apache-cassandra-3.0.12.jar:3.0.12]
at 
org.apache.cassandra.cql3.statements.SelectStatement.processResults(SelectStatement.java:400)
 ~[apache-cassandra-3.0.12.jar:3.0.12]
at 
org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:353)
 ~[apache-cassandra-3.0.12.jar:3.0.12]
at 
org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:227)
 ~[apache-cassandra-3.0.12.jar:3.0.12]
at 
org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:7

Read exceptions after upgrading to 3.0.10

[Meg Mara]

Hello,

We have a 6 node cluster which was recently upgraded from 2.2.5 to 3.0.10. 
Since then, we have been facing a lot of these exceptions. What could be the 
possible cause for this exception?

INFO  [SharedPool-Worker-2] 2017-02-23 20:14:04,984 Message.java:611 - 
Unexpected exception during request; channel = [id: 0x034e0914, 
/xx.xx.117.34:19836 :> /xx.xx.9.119:9042]
java.io.IOException: Error while read(...): Connection reset by peer
  at io.netty.channel.epoll.Native.readAddress(Native Method) 
~[netty-all-4.0.23.Final.jar:4.0.23.Final]
  at 
io.netty.channel.epoll.EpollSocketChannel$EpollSocketUnsafe.doReadBytes(EpollSocketChannel.java:675)
 ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
  at 
io.netty.channel.epoll.EpollSocketChannel$EpollSocketUnsafe.epollInReady(EpollSocketChannel.java:714)
 ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
  at 
io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:326) 
~[netty-all-4.0.23.Final.jar:4.0.23.Final]
  at 
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:264) 
~[netty-all-4.0.23.Final.jar:4.0.23.Final]
  at 
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
 ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
  at 
io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
 ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
  at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101]

INFO  [ScheduledTasks:1] 2017-02-23 20:13:12,330 StatusLogger.java:52 - Pool 
NameActive   Pending  Completed   Blocked  All Time 
Blocked
INFO  [ScheduledTasks:1] 2017-02-23 20:13:12,340 StatusLogger.java:56 - 
MutationStage 0 0  88103 0  
   0
INFO  [ScheduledTasks:1] 2017-02-23 20:13:12,340 StatusLogger.java:56 - 
ViewMutationStage 0 0  0 0  
   0
INFO  [ScheduledTasks:1] 2017-02-23 20:13:12,340 StatusLogger.java:56 - 
ReadStage 0 0 155903 0  
   0
INFO  [ScheduledTasks:1] 2017-02-23 20:13:12,340 StatusLogger.java:56 - 
RequestResponseStage  0 0 188450 0  
   0
INFO  [ScheduledTasks:1] 2017-02-23 20:13:12,341 StatusLogger.java:56 - 
ReadRepairStage   0 0   4118 0  
   0
INFO  [ScheduledTasks:1] 2017-02-23 20:13:12,341 StatusLogger.java:56 - 
CounterMutationStage  0 0  0 0  
   0
INFO  [ScheduledTasks:1] 2017-02-23 20:13:12,341 StatusLogger.java:56 - 
MiscStage 0 0  0 0  
   0
INFO  [ScheduledTasks:1] 2017-02-23 20:13:12,341 StatusLogger.java:56 - 
CompactionExecutor0 0  20443 0  
   0
INFO  [ScheduledTasks:1] 2017-02-23 20:13:12,342 StatusLogger.java:56 - 
MemtableReclaimMemory 0 0 52 0  
   0
INFO  [ScheduledTasks:1] 2017-02-23 20:13:12,342 StatusLogger.java:56 - 
PendingRangeCalculator0 0  8 0  
   0
INFO  [ScheduledTasks:1] 2017-02-23 20:13:12,342 StatusLogger.java:56 - 
GossipStage   0 0  21308 0  
   0
INFO  [ScheduledTasks:1] 2017-02-23 20:13:12,345 StatusLogger.java:56 - 
Native-Transport-Requests 0 0 155489 0  
   4

INFO  [ScheduledTasks:1] 2017-02-23 20:13:12,346 StatusLogger.java:103 - Table  
 Memtable ops,data
INFO  [ScheduledTasks:1] 2017-02-23 20:13:12,346 StatusLogger.java:106 - look   
300,21900
INFO  [ScheduledTasks:1] 2017-02-23 20:13:12,346 StatusLogger.java:106 - 
segment   8,336
INFO  [ScheduledTasks:1] 2017-02-23 20:13:12,347 StatusLogger.java:106 - 
sys_spend 13,5678
INFO  [ScheduledTasks:1] 2017-02-23 20:13:12,347 StatusLogger.java:106 - list   
  0,0
INFO  [ScheduledTasks:1] 2017-02-23 20:13:12,347 StatusLogger.java:106 - 
a_count   14,392
INFO  [ScheduledTasks:1] 2017-02-23 20:13:12,347 StatusLogger.java:106 - 
r_count   618,79787
INFO  [ScheduledTasks:1] 2017-02-23 20:13:12,347 StatusLogger.java:106 - 
sys_prd 642,1393965

Thank you in advance,
Meghana Mara