java.lang.IllegalArgumentException: Mutation of X bytes is too large for the maxiumum size of Y

2015-10-06 Thread George Sigletos
Hello, I have been frequently receiving those warnings: java.lang.IllegalArgumentException: Mutation of 35141120 bytes is too large for the maxiumum size of 33554432 at org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:221) ~[apache-cassandra-2.1.9.jar:2.1.9] at

Re: java.lang.IllegalArgumentException: Mutation of X bytes is too large for the maxiumum size of Y

2015-10-06 Thread Kiran mk
Do you see more dropped mutation messages in nodetool tpstats output. On Oct 6, 2015 7:51 PM, "George Sigletos" wrote: > Hello, > > I have been frequently receiving those warnings: > > java.lang.IllegalArgumentException: Mutation of 35141120 bytes is too > large for the

Re: java.lang.IllegalArgumentException: Mutation of X bytes is too large for the maxiumum size of Y

2015-10-06 Thread George Sigletos
I see no dropped mutation in any of the nodes: Message type Dropped RANGE_SLICE 0 READ_REPAIR 0 PAGED_RANGE 0 BINARY 0 READ 0 MUTATION 0 _TRACE 0

Re: Cassandra Configuration VS Static IPs.

2015-10-06 Thread Gene
Rather than using public IPs on the Cassandra nodes at all I would set up a VPN. 1. Make sure all Cassandra nodes are on the same private network with static IPs 2. Set up a new micro instance with a static public IP, make sure it has a static IP on the private network as well 3. Install OpenVPN

Maximum node decommission // bootstrap at once.

2015-10-06 Thread Kevin Burton
We're in the middle of migrating datacenters. We're migrating from 13 nodes to 30 nodes in the new datacenter. The plan was to bootstrap the 30 nodes first, wait until they have joined. then we're going to decommission the old ones. How many nodes can we bootstrap at once? How many can we

Re: Maximum node decommission // bootstrap at once.

2015-10-06 Thread Bryan Cheng
Honestly, we've had more luck bootstrapping in our old DC (defining topology properties as the new DC) and using rsync to migrate the data files to new machines in the new datacenter. We had 10gig within the datacenter but significantly less than this cross-DC, which lead to a lot of broken

Re: Maximum node decommission // bootstrap at once.

2015-10-06 Thread Robert Coli
On Tue, Oct 6, 2015 at 12:32 PM, Kevin Burton wrote: > How many nodes can we bootstrap at once? How many can we decommission? > short answer : 1 node can join or part at simultaneously longer answer : https://issues.apache.org/jira/browse/CASSANDRA-2434 /

Re: Maximum node decommission // bootstrap at once.

2015-10-06 Thread Kevin Burton
I'm not sure which is faster/easier. Just joining one box at a time and then decommissioning or using replace_address. this stuff is always something you do rarely and then more complex than it needs to be. This complicates long term migration too. Having to have gigabit is somewhat of a

Re: Maximum node decommission // bootstrap at once.

2015-10-06 Thread Kevin Burton
OH. interesting. Yeah. That's another strategy. We've already done a bunch of TCP tuning... we get about 1Gbit with large TCP windows. So I think we have that part done. It's sad that CS can't resume... Plan be we will just rsync the data.. Does it pretty much work just by putting the data in

Re: Maximum node decommission // bootstrap at once.

2015-10-06 Thread Robert Coli
On Tue, Oct 6, 2015 at 2:14 PM, Kevin Burton wrote: > Plan be we will just rsync the data.. Does it pretty much work just by > putting the data in a directory or do you have to do anything special? > http://www.pythian.com/blog/bulk-loading-options-for-cassandra/ Be

Re: How are writes handled while adding nodes to cluster?

2015-10-06 Thread Tyler Hobbs
When a node is joining, writes are sent to both the current replicas *and* the joining replica. However, the joining replica does not count towards the consistency level. So, for example, if you write at ConsistencyLevel.TWO, and only one existing replica and the joining replica respond, the

Re: How are writes handled while adding nodes to cluster?

2015-10-06 Thread Robert Coli
On Tue, Oct 6, 2015 at 11:10 AM, Tyler Hobbs wrote: > When a node is joining, writes are sent to both the current replicas *and* > the joining replica. However, the joining replica does not count towards > the consistency level. So, for example, if you write at >

Cassandra class Client.java usage from Eclipse console

2015-10-06 Thread Xin Ren
Cassandra version: 3.0 native protocol version: 4 Hi, I'm trying to start Cassandra server and client *within Eclipse*, and just wondering the usage of this class org.apache.cassandra.transport.Client in Cassandra source code. I git cloned the source code and opened the project in Eclipse Luna,

How are writes handled while adding nodes to cluster?

2015-10-06 Thread Erik Forsberg
Hi! How are writes handled while I'm adding a node to a cluster, i.e. while the new node is in JOINING state? Are they queued up as hinted handoffs, or are they being written to the joining node? In the former case I guess I have to make sure my max_hint_window_in_ms is long enough for the node

Re: Maximum node decommission // bootstrap at once.

2015-10-06 Thread Bryan Cheng
Robert, I might be misinterpreting you but I *think* your link is talking about bootstrapping a new node by bulk loading replica data from your existing cluster? I was referring to using Cassandra's bootstrap to get the node to join and run (as a member of DC2 but with physical residence in DC1),

Re: Maximum node decommission // bootstrap at once.

2015-10-06 Thread Robert Coli
On Tue, Oct 6, 2015 at 3:41 PM, Bryan Cheng wrote: > Robert, I might be misinterpreting you but I *think* your link is talking > about bootstrapping a new node by bulk loading replica data from your > existing cluster? > Sure... more generically it's about treating the

Re: Cassandra Configuration VS Static IPs.

2015-10-06 Thread Jonathan Haddad
Personally I wouldn't bother with a VPN. You can limit access to a group of servers easily via security groups and not have a single point of failure. I wrote a utility to manage this. If you're bringing up a new VPC + Cassandra servers every day, you can trivially apply a set of rules to your

Re: Repair corrupt SSTable from power outage?

2015-10-06 Thread John Anderson
I'm running 2.1.9 On Fri, Oct 2, 2015 at 8:40 AM, Tyler Hobbs wrote: > What version of Cassandra are you running? It sounds like the disk > failure policy is incorrectly being applied to scrub, which kind of defeats > the purpose of it. I'd recommend opening a ticket ( >