Re: hanging validation compaction

oh yes .. you are absolutely right thank you! i will provide all the necessary info in the created jira issue On Thu, 2017-04-13 at 15:09 +0200, benjamin roth wrote: you should be able to find that out by scrubbing the corresponding table(s) and see wich one hangs? i guess the debuglog tells

Re: hanging validation compaction

you should be able to find that out by scrubbing the corresponding table(s) and see wich one hangs? i guess the debuglog tells you which sstable is being scrubbed. 2017-04-13 15:07 GMT+02:00 Roland Otta : > i made a copy and also have the permission to upload sstables

Re: hanging validation compaction

i made a copy and also have the permission to upload sstables for that particular column_family is it possible to track down which sstable of that cf is affected or should i upload all of them? br, roland On Thu, 2017-04-13 at 13:57 +0200, benjamin roth wrote: I think thats a good

Re: hanging validation compaction

I think thats a good reproduction case for the issue - you should copy the sstable away for further testing. Are you allowed to upload the broken sstable to JIRA? 2017-04-13 13:15 GMT+02:00 Roland Otta : > sorry .. i have to correct myself .. the problem still persists.

Re: hanging validation compaction

sorry .. i have to correct myself .. the problem still persists. tried nodetool scrub now for the table ... but scrub is also stuck at the same percentage id compaction type keyspace table completed total unit progress

Re: hanging validation compaction

What if you run it again with cache enabled? Am 13.04.2017 12:04 schrieb "Roland Otta" : > i did 2 restarts before which did not help > > after that i have set for testing purposes file_cache_size_in_mb: 0 and > buffer_pool_use_heap_if_exhausted: false and restarted

Re: hanging validation compaction

i did 2 restarts before which did not help after that i have set for testing purposes file_cache_size_in_mb: 0 and buffer_pool_use_heap_if_exhausted: false and restarted again after that it worked ... but it also could be that it just worked by accident after the last restart and is not

Re: hanging validation compaction

If you restart the server the same validation completes successfully? If not, have you tries scrubbing the affected sstables? 2017-04-13 11:43 GMT+02:00 Roland Otta : > thank you guys ... i will > > i just wanted to make sure that i am not doing something completely

Re: hanging validation compaction

thank you guys ... i will i just wanted to make sure that i am not doing something completely wrong before opening an issue br, roland On Thu, 2017-04-13 at 21:35 +1200, Nate McCall wrote: Not sure what is going on there either. Roland - can you open an issue with the information above:

Re: hanging validation compaction

Not sure what is going on there either. Roland - can you open an issue with the information above: https://issues.apache.org/jira/browse/CASSANDRA On Thu, Apr 13, 2017 at 7:49 PM, benjamin roth wrote: > What I can tell you from that trace - given that this is the correct >

Re: hanging validation compaction

What I can tell you from that trace - given that this is the correct thread and it really hangs there: The validation is stuck when reading from an SSTable. Unfortunately I am no caffeine expert. It looks like the read is cached and after the read caffeine tries to drain the cache and this is

Re: hanging validation compaction

i had a closer look at the validation executor thread (i hope thats what you meant) it seems the thread is always repeating stuff in org.apache.cassandra.cache.ChunkCache$CachingRebufferer.rebuffer(ChunkCache.java:235) here is the full stack trace ... i am sorry .. but i have no clue whats

Re: hanging validation compaction

You should connect to the node with JConsole and see where the compaction thread is stuck 2017-04-13 8:34 GMT+02:00 Roland Otta : > hi, > > we have the following issue on our 3.10 development cluster. > > we are doing regular repairs with thelastpickle's fork of

hanging validation compaction

hi, we have the following issue on our 3.10 development cluster. we are doing regular repairs with thelastpickle's fork of creaper. sometimes the repair (it is a full repair in that case) hangs because of a stuck validation compaction nodetool compactionstats gives me