hanging validation compaction

hi, we have the following issue on our 3.10 development cluster. we are doing regular repairs with thelastpickle's fork of creaper. sometimes the repair (it is a full repair in that case) hangs because of a stuck validation compaction nodetool compactionstats gives me

Re: hanging validation compaction

You should connect to the node with JConsole and see where the compaction thread is stuck 2017-04-13 8:34 GMT+02:00 Roland Otta : > hi, > > we have the following issue on our 3.10 development cluster. > > we are doing regular repairs with thelastpickle's fork of

Re: hanging validation compaction

i did 2 restarts before which did not help after that i have set for testing purposes file_cache_size_in_mb: 0 and buffer_pool_use_heap_if_exhausted: false and restarted again after that it worked ... but it also could be that it just worked by accident after the last restart and is not

Re: hanging validation compaction

What if you run it again with cache enabled? Am 13.04.2017 12:04 schrieb "Roland Otta" : > i did 2 restarts before which did not help > > after that i have set for testing purposes file_cache_size_in_mb: 0 and > buffer_pool_use_heap_if_exhausted: false and restarted

Re: IPv6-only host, can't seem to get Cassandra to bind to a public port

From my original email: http://www.mail-archive.com/user@cassandra.apache.org/msg51736.html: > My configuration changes: > > listen_address: > listen_interface_prefer_ipv6: true listen_interface is commented out. I've just now tried again with "# listen_interface_prefer_ipv6: false" (option

Re: force processing of pending hinted handoffs

i figured out that there is an mbean org.apache.cassandra.db.type=HintedHandoffManager with the operation scheduleHintDelivery i guess thats what i would need in that case. at least the docs let me think so http://javadox.com/org.apache.cassandra/cassandra-all/3.0.0/or

Re: hanging validation compaction

What I can tell you from that trace - given that this is the correct thread and it really hangs there: The validation is stuck when reading from an SSTable. Unfortunately I am no caffeine expert. It looks like the read is cached and after the read caffeine tries to drain the cache and this is

Re: force processing of pending hinted handoffs

There is a nodetool command to resume hints. Maybe that helps? Am 13.04.2017 09:42 schrieb "Roland Otta" : > oh ... the operation is deprecated according to the docs ... > > > On Thu, 2017-04-13 at 07:40 +, Roland Otta wrote: > > i figured out that there is an mbean

Re: force processing of pending hinted handoffs

I encountered this situation also once or twice but didn't succeed. I just deleted the old hints and ran a repair  Am 13.04.2017 10:35 schrieb "Roland Otta" : > unfortunately it does not. > > i guess this is intended for resuming hinted handoff handling in case it >

Re: hanging validation compaction

Not sure what is going on there either. Roland - can you open an issue with the information above: https://issues.apache.org/jira/browse/CASSANDRA On Thu, Apr 13, 2017 at 7:49 PM, benjamin roth wrote: > What I can tell you from that trace - given that this is the correct >

Re: hanging validation compaction

i had a closer look at the validation executor thread (i hope thats what you meant) it seems the thread is always repeating stuff in org.apache.cassandra.cache.ChunkCache$CachingRebufferer.rebuffer(ChunkCache.java:235) here is the full stack trace ... i am sorry .. but i have no clue whats

Re: force processing of pending hinted handoffs

unfortunately it does not. i guess this is intended for resuming hinted handoff handling in case it hase been paused with the pausehandoff before. i have tested it (resuming .. pausing & resuming) but it has no effect on those old hints On Thu, 2017-04-13 at 10:27 +0200, benjamin roth wrote:

Re: hanging validation compaction

thank you guys ... i will i just wanted to make sure that i am not doing something completely wrong before opening an issue br, roland On Thu, 2017-04-13 at 21:35 +1200, Nate McCall wrote: Not sure what is going on there either. Roland - can you open an issue with the information above:

Re: hanging validation compaction

oh yes .. you are absolutely right thank you! i will provide all the necessary info in the created jira issue On Thu, 2017-04-13 at 15:09 +0200, benjamin roth wrote: you should be able to find that out by scrubbing the corresponding table(s) and see wich one hangs? i guess the debuglog tells

Re: IPv6-only host, can't seem to get Cassandra to bind to a public port

Hi Martjin, Looking at your original message: http://www.mail-archive.com/user@cassandra.apache.org/msg51736.html I see you edited etc/cassandra/cassandra-env.sh, by changing: +#JVM_OPTS="$JVM_OPTS -Djava.net.preferIPv4Stack=true" +JVM_OPTS="$JVM_OPTS -Djava.net.preferIPv6Addresses=true"

Re: force processing of pending hinted handoffs

oh ... the operation is deprecated according to the docs ... On Thu, 2017-04-13 at 07:40 +, Roland Otta wrote: > i figured out that there is an mbean > org.apache.cassandra.db.type=HintedHandoffManager with the operation > scheduleHintDelivery > > i guess thats what i would need in that

Re: hanging validation compaction

If you restart the server the same validation completes successfully? If not, have you tries scrubbing the affected sstables? 2017-04-13 11:43 GMT+02:00 Roland Otta : > thank you guys ... i will > > i just wanted to make sure that i am not doing something completely

Re: hanging validation compaction

i made a copy and also have the permission to upload sstables for that particular column_family is it possible to track down which sstable of that cf is affected or should i upload all of them? br, roland On Thu, 2017-04-13 at 13:57 +0200, benjamin roth wrote: I think thats a good

How to stress test collections in Cassandra Stress

Hi, I'm trying to do a stress test on a a table with a collection column, but cannot figure out how to do that. I tried table_definition: | CREATE TABLE list ( customer_id bigint, items list, PRIMARY KEY (customer_id)); columnspec: - name: customer_id size: fixed(64)

Re: hanging validation compaction

you should be able to find that out by scrubbing the corresponding table(s) and see wich one hangs? i guess the debuglog tells you which sstable is being scrubbed. 2017-04-13 15:07 GMT+02:00 Roland Otta : > i made a copy and also have the permission to upload sstables

Re: hanging validation compaction

sorry .. i have to correct myself .. the problem still persists. tried nodetool scrub now for the table ... but scrub is also stuck at the same percentage id compaction type keyspace table completed total unit progress

Migrating to LCS : Disk Size recommendation clashes

Hi All, We are in process of migrating from STCS to LCS and was just doing few reads on line . Below is the excerpt from Datastax recommendation on data size : Doc link : https://docs.datastax.com/en/landing_page/doc/landing_page/planning/planningHardware.html

Re: WriteTimeoutException with LWT after few milliseconds

I found out that if the WTEs occur, there was already another process inserting the same primary key because I found duplicates in some places that perfectly match the WTE logs. Does anybody know, why this throws a WTE instead of returning [applied]' = false ? This is quite confusing! 2017-04-12

Re: hanging validation compaction

I think thats a good reproduction case for the issue - you should copy the sstable away for further testing. Are you allowed to upload the broken sstable to JIRA? 2017-04-13 13:15 GMT+02:00 Roland Otta : > sorry .. i have to correct myself .. the problem still persists.

Re: IPv6-only host, can't seem to get Cassandra to bind to a public port

You are welcome Martjin! Glad to have been able to help ( Best, -Razi On 4/13/17, 12:13 PM, "Martijn Pieters" wrote: On 13/04/2017, 15:06, "Khaja, Raziuddin (NIH/NLM/NCBI) [C]" wrote: > Looking at your original message:

Re: IPv6-only host, can't seem to get Cassandra to bind to a public port

On 13/04/2017, 15:06, "Khaja, Raziuddin (NIH/NLM/NCBI) [C]" wrote: > Looking at your original message: >