Re: cassandra user request log

2017-02-10 Thread Benjamin Roth
-10 9:46 GMT+01:00 Benjamin Roth : > >> On a cluster with just a little bit load, that would cause zillions of >> petabytes of logs (just roughly ;)). I don't think this is viable. >> There are many many JMX metrics on an aggregated level. But none per >> authed used.

Re: cassandra user request log

2017-02-10 Thread Benjamin Roth
If you want to audit write operations only, you could maybe use CDC, this is a quite new feature in 3.x (I think it was introduced in 3.9 or 3.10) 2017-02-10 10:10 GMT+01:00 vincent gromakowski < vincent.gromakow...@gmail.com>: > tx > > 2017-02-10 10:01 GMT+01:00 Benjamin Roth :

Re: How does cassandra achieve Linearizability?

2017-02-10 Thread Benjamin Roth
servers are good enough for most >>>> > uses. https://www.instaclustr.com/bl >>>> og/2015/11/05/apache-cassandra-synchronization/ >>>> > <https://www.instaclustr.com/blog/2015/11/05/apache-cassand >>>> ra-synchronization/> >>>> > >>>> > Cheers, >>>> > Justin >>>> > >>>> > On Thu, 9 Feb 2017 at 16:09 Kant Kodali >>> > <mailto:k...@peernova.com>> wrote: >>>> > >>>> > How does Cassandra achieve Linearizability with “Last write >>>> > wins” (conflict resolution methods based on time-of-day >>>> clocks) ? >>>> > >>>> > Relying on synchronized clocks are almost certainly >>>> > non-linearizable, because clock timestamps cannot be >>>> guaranteed >>>> > to be consistent with actual event ordering due to clock skew. >>>> > isn't it? >>>> > >>>> > Thanks! >>>> > >>>> > -- >>>> > >>>> > Justin Cameron >>>> > >>>> > Senior Software Engineer | Instaclustr >>>> > >>>> > >>>> > >>>> > >>>> > This email has been sent on behalf of Instaclustr Pty Ltd >>>> > (Australia) and Instaclustr Inc (USA). >>>> > >>>> > This email and any attachments may contain confidential and >>>> legally >>>> > privileged information. If you are not the intended recipient, do >>>> > not copy or disclose its content, but please reply to this email >>>> > immediately and highlight the error to the sender and then >>>> > immediately delete the message. >>>> > >>>> > >>>> >>>> >>> >>> >> > -- Benjamin Roth Prokurist Jaumo GmbH · www.jaumo.com Wehrstraße 46 · 73035 Göppingen · Germany Phone +49 7161 304880-6 · Fax +49 7161 304880-1 AG Ulm · HRB 731058 · Managing Director: Jens Kammerer

High disk io read load

2017-02-15 Thread Benjamin Roth
tency due to the wait states. Has anybody experienced similar situations? Any hints how to analyze or optimize this - I mean 128GB cache for 460GB payload is not that few. I am pretty sure that not the whole dataset of 460GB is "hot". -- Benjamin Roth Prokurist Jaumo GmbH · www.jaumo.co

Re: High disk io read load

2017-02-15 Thread Benjamin Roth
ore read > repair chances. More load can swing it's way during node failures etc. > > On Wednesday, February 15, 2017, Benjamin Roth > wrote: > >> Hi there, >> >> Following situation in cluster with 10 nodes: >> Node A's disk read IO is ~20 times higher t

Re: High disk io read load

2017-02-15 Thread Benjamin Roth
Erm sorry, forgot to mention. In this case "cas10" is Node A with 512 tokens and "cas9" Node B with 256 tokens. 2017-02-16 6:38 GMT+01:00 Benjamin Roth : > It doesn't really look like that: > https://cl.ly/2c3Z1u2k0u2I > > Thats the ReadLatency.count metri

Re: sasi index question (read timeout on many selects)

2017-02-16 Thread Benjamin Roth
0, DuyHai Doan wrote: >> > Why indexing BLOB data ? It does not make any sense >> >> My partition key is a secure hash sum, I don't index a blob. >> >> >> >> >> > -- Benjamin Roth Prokurist Jaumo GmbH · www.jaumo.com Wehrstraße 46 · 7303

Re: High disk io read load

2017-02-17 Thread Benjamin Roth
n, NZ > @zznate > > CTO > Apache Cassandra Consulting > http://www.thelastpickle.com > -- Benjamin Roth Prokurist Jaumo GmbH · www.jaumo.com Wehrstraße 46 · 73035 Göppingen · Germany Phone +49 7161 304880-6 · Fax +49 7161 304880-1 AG Ulm · HRB 731058 · Managing Director: Jens Kammerer

Re: sasi index question (read timeout on many selects)

2017-02-17 Thread Benjamin Roth
e issues that have been filed in > Jira. Benjamin Roth is one of the only people talking about his MV usage, > and has filed a few JIRAs discussing their problems when bootstrapping new > nodes, as well as issues repairing. > > https://issues.apache.org/jira/browse/CASSANDRA-127

Re: High disk io read load

2017-02-18 Thread Benjamin Roth
greaves : > what's the Owns % for the relevant keyspace from nodetool status? > -- Benjamin Roth Prokurist Jaumo GmbH · www.jaumo.com Wehrstraße 46 · 73035 Göppingen · Germany Phone +49 7161 304880-6 · Fax +49 7161 304880-1 AG Ulm · HRB 731058 · Managing Director: Jens Kammerer

Re: High disk io read load

2017-02-18 Thread Benjamin Roth
ith puppet, they share the same roles and no manual "optimizations" are applied. So I can't imagine, a different configuration is responsible for it. 2017-02-18 21:28 GMT+01:00 Benjamin Roth : > This is status of the largest KS of these both nodes: > UN 10.23.71.10 437.9

Re: High disk io read load

2017-02-18 Thread Benjamin Roth
cat /sys/block/sda/queue/read_ahead_kb => 8 On all CS nodes. Is that what you mean? 2017-02-18 21:32 GMT+01:00 Bhuvan Rawal : > Hi Benjamin, > > What is the disk read ahead on both nodes? > > Regards, > Bhuvan > > On Sun, Feb 19, 2017 at 1:58 AM, Benjamin Roth >

Re: High disk io read load

2017-02-18 Thread Benjamin Roth
and 512 token > node. > > Regards, > > On Sun, Feb 19, 2017 at 2:07 AM, Benjamin Roth > wrote: > >> cat /sys/block/sda/queue/read_ahead_kb >> => 8 >> >> On all CS nodes. Is that what you mean? >> >> 2017-02-18 21:32 GMT+01:00 Bhu

Re: High disk io read load

2017-02-18 Thread Benjamin Roth
S has started up and accepts requests. I also diffed sysctl of the both machines. No significant differences. Only CPU-related, random values and some hashes differ. 2017-02-18 21:49 GMT+01:00 Benjamin Roth : > 256 tokens: > > root@cas9:/sys/block/dm-0# blockdev --report > RORA SSZ

Re: High disk io read load

2017-02-19 Thread Benjamin Roth
> > Regards, > > On Sun, Feb 19, 2017 at 7:47 PM, Edward Capriolo > wrote: > >> >> >> On Sat, Feb 18, 2017 at 3:35 PM, Benjamin Roth >> wrote: >> >>> We are talking about a read IO increase of over 2000% with 512 tokens >>> compared t

Re: Cassandra blob vs base64 text

2017-02-20 Thread Benjamin Roth
You could save space when storing your data (base64-)decoded as blobs. 2017-02-20 13:38 GMT+01:00 Oskar Kjellin : > We currently have some cases where we store base64 as a text field instead > of a blob (running version 2.0.17). > I would like to move these to blob but wondering what benefits and

Re: Count(*) is not working

2017-02-20 Thread Benjamin Roth
gt;>>>select count(*) from tablename; >>>>> >>>>> and i have got the below output >>>>> >>>>> Read 100 live rows and 1423 tombstone cells for query SELECT * FROM >>>>> keysace.table WHERE token(id) > token(test:ODP0144-0883E-022R-002/04

Re: High disk io read load

2017-02-20 Thread Benjamin Roth
Hah! Found the problem! After setting read_ahead to 0 and compression chunk size to 4kb on all CFs, the situation was PERFECT (nearly, please see below)! I scrubbed some CFs but not the whole dataset, yet. I knew it was not too few RAM. Some stats: - Latency of a quite large CF: https://cl.ly/1r3

Re: Does C* coordinator writes to replicas in same order or different order?

2017-02-21 Thread Benjamin Roth
gt;>> may >>>>> choose is fine but I want the same order across all replicas and with >>>>> async >>>>> replication I am not sure how that is possible ? for example, if a >>>>> request >>>>> arrives with timestamp t1 and another request arrives with a timestamp t2 >>>>> where t1 < t2...with async replication what if one replica chooses to >>>>> execute t2 first and then t1 simply because t1 is slow while another >>>>> replica choose to execute t1 first and then t2..how would that work? )* >>>>> >>>>>> >>>>>> Note that C* each node can be a coordinator (one per request) and its >>>>>> the desired case in order to load balance the incoming requests. Once >>>>>> again, >>>>>> timestamps determine the order among the requests. >>>>>> >>>>>> Cheers, >>>>>> Dor >>>>>> >>>>>> On Mon, Feb 20, 2017 at 4:12 PM, Kant Kodali >>>>>> wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> when C* coordinator writes to replicas does it write it in same >>>>>>> order or >>>>>>> different order? other words, Does the replication happen >>>>>>> synchronously or >>>>>>> asynchrnoulsy ? Also does this depend sync or async client? What >>>>>>> happens in >>>>>>> the case of concurrent writes to a coordinator ? >>>>>>> >>>>>>> Thanks, >>>>>>> kant >>>>>>> >>>>>> >>>>>> >>>>>> >>>>> >>>> >>> >> > -- Benjamin Roth Prokurist Jaumo GmbH · www.jaumo.com Wehrstraße 46 · 73035 Göppingen · Germany Phone +49 7161 304880-6 · Fax +49 7161 304880-1 AG Ulm · HRB 731058 · Managing Director: Jens Kammerer

Re: Does C* coordinator writes to replicas in same order or different order?

2017-02-21 Thread Benjamin Roth
ch update." > > http://cassandra.apache.org/ > > 2. Is it Guaranteed that C* coordinator writes data in the same order to > all the replicas (either sync or async)? > > Thanks, > kant > > On Tue, Feb 21, 2017 at 1:23 AM, Benjamin Roth > wrote: > >> T

Re: High disk io read load

2017-02-24 Thread Benjamin Roth
It was only the schema change. 2017-02-24 19:18 GMT+01:00 kurt greaves : > How many CFs are we talking about here? Also, did the script also kick off > the scrubs or was this purely from changing the schemas? > ​ > -- Benjamin Roth Prokurist Jaumo GmbH · www.jaumo.com Wehrstraß

Re: Understanding of proliferation of sstables during a repair

2017-02-26 Thread Benjamin Roth
to achieve consistency instead. Is > this generally true? > > > Thanks! > -- Benjamin Roth Prokurist Jaumo GmbH · www.jaumo.com Wehrstraße 46 · 73035 Göppingen · Germany Phone +49 7161 304880-6 · Fax +49 7161 304880-1 AG Ulm · HRB 731058 · Managing Director: Jens Kammerer

Re: Understanding of proliferation of sstables during a repair

2017-02-26 Thread Benjamin Roth
Too many open files. Which is 100k by default and we had >40k sstables. Normally the are around 500-1000. Am 27.02.2017 02:40 schrieb "Seth Edwards" : > This makes a lot more sense. What does TMOF stand for? > > On Sun, Feb 26, 2017 at 1:01 PM, Benjamin Roth > wrote: &g

Re: Which compaction strategy when modeling a dumb set

2017-02-27 Thread Benjamin Roth
t; It works well enough, but I'm wondering which compaction strategy I should > use. I expect maybe 1% or less of events will end up duplicated (thus not > generating an insert), so the workload will probably be 50% writes 50% read. > > Is LCS a good strategy here or should I stick

Re: Is periodic manual repair necessary?

2017-02-27 Thread Benjamin Roth
cally purged because of TTL. > > > > In such a situation, do I need to have a periodic (weekly?) manual/batch > read_repair process? > > > > Thanks, > > Jayesh Thakrar > > -- Benjamin Roth Prokurist Jaumo GmbH · www.jaumo.com Wehrstraße 46 · 73035 Göppingen · Germany Phone +49 7161 304880-6 · Fax +49 7161 304880-1 AG Ulm · HRB 731058 · Managing Director: Jens Kammerer

unsubscribe

2017-02-28 Thread Benjamin Roth
-- Benjamin Roth Prokurist Jaumo GmbH · www.jaumo.com Wehrstraße 46 · 73035 Göppingen · Germany Phone +49 7161 304880-6 · Fax +49 7161 304880-1 AG Ulm · HRB 731058 · Managing Director: Jens Kammerer

Re: Is periodic manual repair necessary?

2017-02-28 Thread benjamin roth
; > > > Among other things, this advice also gives an impression to people not > familiar with C* (e.g. me) that it is too fragile and needs substantial > manual intervention. > > > > Appreciate all the feedback and details that you have been sharing. > > > > *From: *Ed

Re: Resources for fire drills

2017-03-01 Thread benjamin roth
Could you specify it a little bit? There are really a lot of things that can go wrong. 2017-03-01 10:59 GMT+01:00 Malte Pickhan : > Hi Cassandra users, > > I am looking for some resources/guides for firedrill scenarios with apache > cassandra. > > Do you know anything like that? > > Best, > > Mal

Re: Resources for fire drills

2017-03-01 Thread benjamin roth
am. > > Best > > > On 1 Mar 2017, at 11:01, benjamin roth wrote: > > Could you specify it a little bit? There are really a lot of things that > can go wrong. > > 2017-03-01 10:59 GMT+01:00 Malte Pickhan : > >> Hi Cassandra users, >> >> I am looking f

Re: Resources for fire drills

2017-03-01 Thread benjamin roth
are some overview for basic scenarios for firedrills, so that > you can exercise them with your team. > > Best > > > On 1 Mar 2017, at 11:01, benjamin roth wrote: > > Could you specify it a little bit? There are really a lot of things that > can go wrong. > > 2017-03-

Re: Resources for fire drills

2017-03-01 Thread benjamin roth
@Doc: http://cassandra.apache.org/doc/latest/ is built from the git repo. So you can add documentation in doc/source and submit a patch. I personally think that is not the very best place or way to build a knowledge DB but thats what we have. 2017-03-01 13:39 GMT+01:00 Malte Pickhan : > Hi, > >

Re: Non-zero nodes are marked as down after restarting cassandra process

2017-03-01 Thread benjamin roth
You should always drain nodes before stopping the daemon whenever possible. This avoids commitlog replay on startup. This can take a while. But according to your description commit log replay seems not to be the cause. I once had a similar effect. Some nodes appeared down for some other nodes and

Rebuild / removenode with MV is inconsistent

2017-03-01 Thread benjamin roth
Hi there, Today I come up with the following thesis: A rebuild / removenode may break the base-table <> MV contract. I'd even claim that a rebuild / removenode requires rebuilding all MVs to guarantee MV consistency. Reason: A node can have base tables with MVs. This is no problem. If these are

Re: Limit on number of keyspaces/tables

2017-03-05 Thread benjamin roth
I ask back: what's your intention Am 05.03.2017 11:58 schrieb "Lata Kannan" : > Hi > > I just wanted to check if there is any known limit to the number of > keyspaces one can create in a Cassandra cluster? Alternatively is there a > max on the number of tables that can be created in a cluster? >

Re: Limit on number of keyspaces/tables

2017-03-05 Thread benjamin roth
No seriously. Am 05.03.2017 2:54 nachm. schrieb "Rakesh Kumar" : > > I ask back: what's your intention > > May be documenting the limitations of Cassandra to show Oracle is better > :-) > > Am 05.03.2017 11:58 schrieb "Lata Kannan" lata.kan...@oracle.com>>: >

Re: Limit on number of keyspaces/tables

2017-03-05 Thread benjamin roth
Why do you think 1 table consumes 1m?? Am 05.03.2017 20:36 schrieb "Vladimir Yudovin" : > Hi, > > there is no such hard limit, but each table consume at least 1M memory, so > 1000 tables takes at least 1G. > > Best regards, Vladimir Yudovin, > *Winguzone - Cloud

Re: Can I do point in time recover using nodetool

2017-03-08 Thread benjamin roth
I remember a very similar question on the list some months ago. The short answer is that there is no short answer. I'd recommend you search the mailing list archive for "backup" or "recover". 2017-03-08 10:17 GMT+01:00 Bhardwaj, Rahul : > Hi All, > > > > Is there any possibility of restoring cass

Re: scylladb

2017-03-11 Thread benjamin roth
Thanks a lot for your detailed explanation! I am very curious about the future development of Scylladb! Especially about mvs and lwt! Am 11.03.2017 02:05 schrieb "Dor Laor" : > On Fri, Mar 10, 2017 at 4:45 PM, Kant Kodali wrote: > >> http://performanceterracotta.blogspot.com/2012/09/numa-java.ht

Re: scylladb

2017-03-11 Thread benjamin roth
Why? Am 12.03.2017 07:02 schrieb "Jeff Jirsa" : > > > On 2017-03-10 09:57 (-0800), Rakesh Kumar wrote: > > Cassanda vs Scylla is a valid comparison because they both are > compatible. Scylla is a drop-in replacement for Cassandra. > > No, they aren't, and no, it isn't > > > > >

Re: scylladb

2017-03-11 Thread benjamin roth
There is no reason to be angry. This is progress. This is the circle of live. It happens anywhere at any time. Am 12.03.2017 07:34 schrieb "Dor Laor" : > On Sat, Mar 11, 2017 at 10:02 PM, Jeff Jirsa wrote: > >> >> >> On 2017-03-10 09:57 (-0800), Rakesh Kumar wrote: >> > Cassanda vs Scylla is a

Re: scylladb

2017-03-13 Thread benjamin roth
@Dor,Jeff: I think Jeff pointed out an important fact: You cannot stop CS, swap binaries and start Scylla. To be honest that was AFAIR the only "Oooh :(" I had when reading the Scylla "marketing material". If that worked it would be very valuable from both Scylla's and a users' point of view. As

Re: repair performance

2017-03-17 Thread benjamin roth
It depends a lot ... - Repairs can be very slow, yes! (And unreliable, due to timeouts, outages, whatever) - You can use incremental repairs to speed things up for regular repairs - You can use "reaper" to schedule repairs and run them sliced, automated, failsafe The time repairs actually may var

Re: repair performance

2017-03-17 Thread benjamin roth
], dataCenters: [ProdDC2], hosts: [], # of > ranges: 1758) > > 3.0.7 is also the reason why we are not using reaper ... as far as i could > figure out it's not compatible with 3.0+ > > > > On Fri, 2017-03-17 at 22:13 +0100, benjamin roth wrote: > > It depends a l

Re: Running cassandra

2017-03-19 Thread benjamin roth
I once had the same problem. In my case it was the coloured output of grep that injected ansi codes into the CS startup command. Am 19.03.2017 18:07 schrieb "Long Quanzheng" : > Hi > It still doesn't work. > > The real problem is this error: > > Error: Could not find or load main class -ea > > Th

Re: Running cassandra

2017-03-19 Thread benjamin roth
You're welcome! 2017-03-19 18:41 GMT+01:00 Long Quanzheng : > You are RIGHT! > It's working after I remove the env variable GREP_OPTIONS. > > Thanks! > > 2017-03-19 10:08 GMT-07:00 benjamin roth : > >> I once had the same problem. In my case it was the col

Re: spikes in blocked native transport requests

2017-03-20 Thread benjamin roth
Did you check STW GCs? You can do that with 'nodetool gcstats', by looking at the gc.log or observing GC related JMX metrics. 2017-03-20 8:52 GMT+01:00 Roland Otta : > we have a datacenter which is currently used exlusively for spark batch > jobs. > > in case batch jobs are running against that e

Re: nodes are always out of sync

2017-04-01 Thread benjamin roth
I think your way to communicate needs work. No one forces you to answer on questions. Am 01.04.2017 21:09 schrieb "daemeon reiydelle" : > What you are doing is correctly going to result in this, IF there is > substantial backlog/network/disk or whatever pressure. > > What do you think will happen

Re: nodes are always out of sync

2017-04-01 Thread benjamin roth
Tl;Dr: there are race conditions in a repair and it is not trivial to fix them. So we rather stay with these race conditions. Actually they don't really hurt. The worst case is that ranges are repaired that don't really need a repair. Am 01.04.2017 21:14 schrieb "Chris Lohfink" : > Repairs do not

Re: nodes are always out of sync

2017-04-02 Thread benjamin roth
Btw.: I created an issue for that some months ago https://issues.apache.org/jira/browse/CASSANDRA-12991 2017-04-01 22:25 GMT+02:00 Roland Otta : > thank you both chris and benjamin for taking time to clarify that. > > > On Sat, 2017-04-01 at 21:17 +0200, benjamin roth wrote: > >

Re: Node always dieing

2017-04-06 Thread benjamin roth
Cassandra cannot write an SSTable to disk. Are you sure the disk/volume where SSTables reside (normally /var/lib/cassandra/data) is writeable for the CS user and has enough free space? The CDC warning also implies that. The other warnings indicate you are probably not running CS as root and you did

Re: Node always dieing

2017-04-06 Thread benjamin roth
= 1048575 net.ipv4.tcp_keepalive_time=600 On /etc/pam.d/su ... sessionrequired pam_limits.so ... Distro is the currently Ubuntu LTS. Thanks On 04/06/2017 10:39 AM, benjamin roth wrote: Cassandra cannot write an SSTable to disk. Are you sure the disk/volume where SSTables reside (normally /var/lib

Re: Node always dieing

2017-04-06 Thread benjamin roth
truncate_request_timeout_in_ms=600; > unlogged_batch_across_partitions_warn_threshold=10; > user_defined_function_fail_timeout=1500; > user_defined_function_warn_timeout=500; > user_function_timeout_policy=die; windows_timer_interval=1; > write_request_timeout_in_ms=60

Re: Multiple nodes decommission

2017-04-11 Thread benjamin roth
I did not test it but I'd bet that parallel decommision will lead to inconsistencies. Each decommission results in range movements and range reassignments which becomes effective after a successful decommission. If you start several decommissions at once, I guess the calculated reassignments are in

WriteTimeoutException with LWT after few milliseconds

2017-04-12 Thread benjamin roth
Hi folks, Can someone explain why that occurs? Write timeout after 0.006s Query: 'INSERT INTO log_moment_import ("source", "reference", "user_id", "moment_id", "date", "finished") VALUES (3, '1305821272790495', 65675537, 0, '2017-04-12 13:00:51', NULL) IF NOT EXISTS Primary key and parition key i

Re: WriteTimeoutException with LWT after few milliseconds

2017-04-12 Thread benjamin roth
its > a timeout)... but in that case you would get an instant error for inserts. > wouldn't you? > > br, > roland > > > > On Wed, 2017-04-12 at 15:09 +0200, benjamin roth wrote: > > Hi folks, > > Can someone explain why that occurs? > > Write timeou

Re: hanging validation compaction

2017-04-12 Thread benjamin roth
You should connect to the node with JConsole and see where the compaction thread is stuck 2017-04-13 8:34 GMT+02:00 Roland Otta : > hi, > > we have the following issue on our 3.10 development cluster. > > we are doing regular repairs with thelastpickle's fork of creaper. > sometimes the repair (i

Re: hanging validation compaction

2017-04-13 Thread benjamin roth
6) > java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1142) > java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:617) > org.apache.cassandra.concurrent.NamedThreadFactory. > lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79) &

Re: force processing of pending hinted handoffs

2017-04-13 Thread benjamin roth
There is a nodetool command to resume hints. Maybe that helps? Am 13.04.2017 09:42 schrieb "Roland Otta" : > oh ... the operation is deprecated according to the docs ... > > > On Thu, 2017-04-13 at 07:40 +, Roland Otta wrote: > > i figured out that there is an mbean > > org.apache.cassandra.d

Re: force processing of pending hinted handoffs

2017-04-13 Thread benjamin roth
; hase been paused with the pausehandoff before. > i have tested it (resuming .. pausing & resuming) but it has no effect on > those old hints > > On Thu, 2017-04-13 at 10:27 +0200, benjamin roth wrote: > > There is a nodetool command to resume hints. Maybe that helps? >

Re: hanging validation compaction

2017-04-13 Thread benjamin roth
e opening an issue > > br, > roland > > > On Thu, 2017-04-13 at 21:35 +1200, Nate McCall wrote: > > Not sure what is going on there either. Roland - can you open an issue > with the information above: > https://issues.apache.org/jira/browse/CASSANDRA > > On Th

Re: hanging validation compaction

2017-04-13 Thread benjamin roth
gain > > after that it worked ... but it also could be that it just worked by > accident after the last restart and is not related to my config changes > > On Thu, 2017-04-13 at 11:58 +0200, benjamin roth wrote: > > If you restart the server the same validation completes successfu

Re: hanging validation compaction

2017-04-13 Thread benjamin roth
ang.Thread.run(Thread.java:745) > > > br, > roland > > > On Thu, 2017-04-13 at 10:04 +, Roland Otta wrote: > > i did 2 restarts before which did not help > > after that i have set for testing purposes file_cache_size_in_mb: 0 and > buffer_pool_use_heap_if_exhauste

Re: WriteTimeoutException with LWT after few milliseconds

2017-04-13 Thread benjamin roth
gt;> what if one ore more nodes are marked down and proper consistency cant be >> achieved? >> of course the error message does not indicate that problem (as it says >> its a timeout)... but in that case you would get an instant error for >> inserts. wouldn't you? >&

Re: hanging validation compaction

2017-04-13 Thread benjamin roth
icular column_family > > is it possible to track down which sstable of that cf is affected or > should i upload all of them? > > > br, > roland > > > On Thu, 2017-04-13 at 13:57 +0200, benjamin roth wrote: > > I think thats a good reproduction case for the issue - you

Re: Counter performance

2017-04-17 Thread benjamin roth
Do you have a different compaction strategy on the counter tables? 2017-04-17 10:07 GMT+02:00 Eren Yilmaz : > We are using Cassandra (3.7) counter tables in our application, and there > are about 10 counter tables. The counter tables are in a separate keyspace > with RF=3 (total 10 nodes). The ta

Re: Counter performance

2017-04-17 Thread benjamin roth
tegy then. > > > > compaction = { 'class' : > 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy', > 'sstable_size_in_mb' : 512 } > > > > *From:* benjamin roth [mailto:brs...@gmail.com] > *Sent:* Monday, April 17, 2017 12:12 PM > *To:* us

RE: Counter performance

2017-04-17 Thread benjamin roth
sergrid-created data. *From:* benjamin roth [mailto:brs...@gmail.com] *Sent:* Monday, April 17, 2017 4:12 PM *To:* user@cassandra.apache.org *Subject:* Re: Counter performance Do you see difference when tracing the selects? 2017-04-17 13:36 GMT+02:00 Eren Yilmaz : Applica

RE: Counter performance

2017-04-17 Thread benjamin roth
s in this version? Or, what do we expect with this log output? > > > > *From:* benjamin roth [mailto:brs...@gmail.com] > *Sent:* Monday, April 17, 2017 5:44 PM > *To:* user@cassandra.apache.org > *Subject:* RE: Counter performance > > > > You could enable a slow que

Re: Counter performance

2017-04-17 Thread benjamin roth
message to / > cassandra-01 [MessagingService-Outgoing-/ cassandra-01 ] | 2017-04-17 > 18:31:49.627000 | cassandra-05 | 3453 | cassandra-01 > > REQUEST_RESPONSE message received from / > cassandra-05 [MessagingService-Incoming-/ cassandra-05 ] | 2

Re: WriteTimeoutException with LWT after few milliseconds

2017-04-19 Thread benjamin roth
Thanks, Jeff! As soon as I have some spare time I will try to reproduce and open a Jira for it. 2017-04-19 16:27 GMT+02:00 Jeff Jirsa : > > > On 2017-04-13 05:13 (-0700), benjamin roth wrote: > > I found out that if the WTEs occur, there was already another process > &g

Re: TRUNCATE on a disk almost full - possible?

2017-04-21 Thread benjamin roth
Truncate needs no space. It just creates a hard link of all affected SSTables under the corresponding -SNAPSHOT dir (at least with default settings) and then removes the SSTables. Also this operation should be rather fast as it is mostly a file-deletion process with some metadata updates. 2017-04-

Re: Cassandra Cluster issues

2017-05-08 Thread benjamin roth
Hm that question is like "My car does not start - whats the problem?". You have to monitor, monitor, monitor, monitor. I'd strongly advice to graph as many metrics as you can. Read them from the JMX interface and write them to a TSDB, visualize them e.g. with Grafana. Then read logs, trace your que

Re: Is it possible to repair a single partition.

2017-06-27 Thread benjamin roth
Then the partition is too big or has too many sstables that contain data for that partition so that the query times out. You can run a manual compaction on that table. That helped me several times. + I hope you are not trying to read that parition at once. Please use paging to query large partitio

Re: Corrupted commit log prevents Cassandra start

2017-07-07 Thread benjamin roth
Hi Hannu, I remember there have been discussions about this in the past. Most probably there is already a JIRA for this. I roughly remember a consense like that: - Default behaviour should remain - It should be configurable to the needs and preferences of the DBA - It should at least spit out erro

Re: Cassandra seems slow when having many read operations

2017-07-21 Thread benjamin roth
Apart from all that you can try to reduce the compression chunk size from the default 64kb to 16kb or even down to 4kb. This can help a lot if your read io on disk is very high and the page cache is not efficient. Am 21.07.2017 23:03 schrieb "Petrus Gomes" : > Thanks a lot to share the result. >

Re: Cassandra seems slow when having many read operations

2017-07-21 Thread benjamin roth
creased heap to 24GB. do you use cms or g1gc? > did you change any other parameters? > As for the chunk size, we found change 64kb to 16kb didn't make a > difference in low key cache rate environment > > > > On Fri, Jul 21, 2017 at 9:27 PM, benjamin roth wrote: > >

<    1   2