Only the last added node is visible in the cluster
While creating a multinode cluster, my nodes are unable to identify all the nodes in the cluster. Only the *last added' node is visible when I do: ./nodetool -h localhost ring I am trying to create a 4 nodes cluster. On starting the seed node, the above command shows just itself(ok.. good), then when I start the 2nd node the first one disappears there is only 2nd visible in the ring. On starting 3rd one, just the 3rd one remains. In cassandra.yaml of each node, I configured the listen_adress equal to ip address of that node for seeds I just put the ip address of 1st node everywhere. Can anyone point to me what may be causing this ?
MemtableThroughput test in ColumnFamily.apply
Hi,
while hunting down some memory consumption issues in 0.7.10 I realized
that MemtableThroughput condition is tested before writing the new data.
As this causes memtables to grow larger than expected I changed
Memtable apply(DecoratedKey key, ColumnFamily columnFamily)
{
long start = System.nanoTime();
boolean flushRequested = memtable.isThresholdViolated();
memtable.put(key, columnFamily);
ColumnFamily cachedRow = getRawCachedRow(key);
if (cachedRow != null)
cachedRow.addAll(columnFamily);
writeStats.addNano(System.nanoTime() - start);
return flushRequested ? memtable : null;
}
to
Memtable apply(DecoratedKey key, ColumnFamily columnFamily)
{
long start = System.nanoTime();
memtable.put(key, columnFamily);
ColumnFamily cachedRow = getRawCachedRow(key);
if (cachedRow != null)
cachedRow.addAll(columnFamily);
writeStats.addNano(System.nanoTime() - start);
boolean flushRequested = memtable.isThresholdViolated();
return flushRequested ? memtable : null;
}
Are there any objections to this change? So far it works for me.
Best,
Thomas
Re: MemtableThroughput test in ColumnFamily.apply
Makes sense to me, although I don't see it making a material
difference when there are 1000 mutations in a memtable vs 1001.
On Sat, Feb 25, 2012 at 11:23 AM, Thomas Richter t...@tricnet.de wrote:
Hi,
while hunting down some memory consumption issues in 0.7.10 I realized
that MemtableThroughput condition is tested before writing the new data.
As this causes memtables to grow larger than expected I changed
Memtable apply(DecoratedKey key, ColumnFamily columnFamily)
{
long start = System.nanoTime();
boolean flushRequested = memtable.isThresholdViolated();
memtable.put(key, columnFamily);
ColumnFamily cachedRow = getRawCachedRow(key);
if (cachedRow != null)
cachedRow.addAll(columnFamily);
writeStats.addNano(System.nanoTime() - start);
return flushRequested ? memtable : null;
}
to
Memtable apply(DecoratedKey key, ColumnFamily columnFamily)
{
long start = System.nanoTime();
memtable.put(key, columnFamily);
ColumnFamily cachedRow = getRawCachedRow(key);
if (cachedRow != null)
cachedRow.addAll(columnFamily);
writeStats.addNano(System.nanoTime() - start);
boolean flushRequested = memtable.isThresholdViolated();
return flushRequested ? memtable : null;
}
Are there any objections to this change? So far it works for me.
Best,
Thomas
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com
Re: MemtableThroughput test in ColumnFamily.apply
Hi,
I agree, but the problem in our case is that we have rather small
memtables (5MB) and during hinted handoff there are several dozen or
even hundreds of MBs inserted without flushing the tables. And in that
case it makes a difference.
Best,
Thomas
On 02/25/2012 07:06 PM, Jonathan Ellis wrote:
Makes sense to me, although I don't see it making a material
difference when there are 1000 mutations in a memtable vs 1001.
On Sat, Feb 25, 2012 at 11:23 AM, Thomas Richter t...@tricnet.de wrote:
Hi,
while hunting down some memory consumption issues in 0.7.10 I realized
that MemtableThroughput condition is tested before writing the new data.
As this causes memtables to grow larger than expected I changed
Memtable apply(DecoratedKey key, ColumnFamily columnFamily)
{
long start = System.nanoTime();
boolean flushRequested = memtable.isThresholdViolated();
memtable.put(key, columnFamily);
ColumnFamily cachedRow = getRawCachedRow(key);
if (cachedRow != null)
cachedRow.addAll(columnFamily);
writeStats.addNano(System.nanoTime() - start);
return flushRequested ? memtable : null;
}
to
Memtable apply(DecoratedKey key, ColumnFamily columnFamily)
{
long start = System.nanoTime();
memtable.put(key, columnFamily);
ColumnFamily cachedRow = getRawCachedRow(key);
if (cachedRow != null)
cachedRow.addAll(columnFamily);
writeStats.addNano(System.nanoTime() - start);
boolean flushRequested = memtable.isThresholdViolated();
return flushRequested ? memtable : null;
}
Are there any objections to this change? So far it works for me.
Best,
Thomas
Re: Only the last added node is visible in the cluster
My guess would be you're using the same token everywhere. -Brandon On Sat, Feb 25, 2012 at 9:48 AM, Aditya Gupta ady...@gmail.com wrote: While creating a multinode cluster, my nodes are unable to identify all the nodes in the cluster. Only the *last added' node is visible when I do: ./nodetool -h localhost ring I am trying to create a 4 nodes cluster. On starting the seed node, the above command shows just itself(ok.. good), then when I start the 2nd node the first one disappears there is only 2nd visible in the ring. On starting 3rd one, just the 3rd one remains. In cassandra.yaml of each node, I configured the listen_adress equal to ip address of that node for seeds I just put the ip address of 1st node everywhere. Can anyone point to me what may be causing this ?
Re: Only the last added node is visible in the cluster
Then my next guess is you cloned one system to make the others in a virtual env, and the token is recorded in the system keyspace. In any case, some nodetool ring output at each node addition will clarify this. -Brandon On Sat, Feb 25, 2012 at 1:20 PM, Aditya Gupta ady...@gmail.com wrote: Nope, I just re-verified :) I have split up the range into 4 parts for 4 nodes. I have specified that in the intial_token On Sun, Feb 26, 2012 at 12:33 AM, Brandon Williams dri...@gmail.com wrote: My guess would be you're using the same token everywhere. -Brandon On Sat, Feb 25, 2012 at 9:48 AM, Aditya Gupta ady...@gmail.com wrote: While creating a multinode cluster, my nodes are unable to identify all the nodes in the cluster. Only the *last added' node is visible when I do: ./nodetool -h localhost ring I am trying to create a 4 nodes cluster. On starting the seed node, the above command shows just itself(ok.. good), then when I start the 2nd node the first one disappears there is only 2nd visible in the ring. On starting 3rd one, just the 3rd one remains. In cassandra.yaml of each node, I configured the listen_adress equal to ip address of that node for seeds I just put the ip address of 1st node everywhere. Can anyone point to me what may be causing this ?
Re: Only the last added node is visible in the cluster
yes, exactly I did cloned a single VMware machine to make other instances.. so how do I correct this now ? On Sun, Feb 26, 2012 at 2:18 AM, Brandon Williams dri...@gmail.com wrote: Then my next guess is you cloned one system to make the others in a virtual env, and the token is recorded in the system keyspace. In any case, some nodetool ring output at each node addition will clarify this. -Brandon On Sat, Feb 25, 2012 at 1:20 PM, Aditya Gupta ady...@gmail.com wrote: Nope, I just re-verified :) I have split up the range into 4 parts for 4 nodes. I have specified that in the intial_token On Sun, Feb 26, 2012 at 12:33 AM, Brandon Williams dri...@gmail.com wrote: My guess would be you're using the same token everywhere. -Brandon On Sat, Feb 25, 2012 at 9:48 AM, Aditya Gupta ady...@gmail.com wrote: While creating a multinode cluster, my nodes are unable to identify all the nodes in the cluster. Only the *last added' node is visible when I do: ./nodetool -h localhost ring I am trying to create a 4 nodes cluster. On starting the seed node, the above command shows just itself(ok.. good), then when I start the 2nd node the first one disappears there is only 2nd visible in the ring. On starting 3rd one, just the 3rd one remains. In cassandra.yaml of each node, I configured the listen_adress equal to ip address of that node for seeds I just put the ip address of 1st node everywhere. Can anyone point to me what may be causing this ?
Re: Only the last added node is visible in the cluster
The output of nodetool ring after each addition of nodes, make just the last added node visible in the ring. When I retry to add the node(which are not visible) it says that it is already a part of the ring. Could you indicate how should I rectify this now, as you seem to have figured out the issue ? On Sun, Feb 26, 2012 at 2:49 AM, Aditya Gupta ady...@gmail.com wrote: yes, exactly I did cloned a single VMware machine to make other instances.. so how do I correct this now ? On Sun, Feb 26, 2012 at 2:18 AM, Brandon Williams dri...@gmail.comwrote: Then my next guess is you cloned one system to make the others in a virtual env, and the token is recorded in the system keyspace. In any case, some nodetool ring output at each node addition will clarify this. -Brandon On Sat, Feb 25, 2012 at 1:20 PM, Aditya Gupta ady...@gmail.com wrote: Nope, I just re-verified :) I have split up the range into 4 parts for 4 nodes. I have specified that in the intial_token On Sun, Feb 26, 2012 at 12:33 AM, Brandon Williams dri...@gmail.com wrote: My guess would be you're using the same token everywhere. -Brandon On Sat, Feb 25, 2012 at 9:48 AM, Aditya Gupta ady...@gmail.com wrote: While creating a multinode cluster, my nodes are unable to identify all the nodes in the cluster. Only the *last added' node is visible when I do: ./nodetool -h localhost ring I am trying to create a 4 nodes cluster. On starting the seed node, the above command shows just itself(ok.. good), then when I start the 2nd node the first one disappears there is only 2nd visible in the ring. On starting 3rd one, just the 3rd one remains. In cassandra.yaml of each node, I configured the listen_adress equal to ip address of that node for seeds I just put the ip address of 1st node everywhere. Can anyone point to me what may be causing this ?
Re: Only the last added node is visible in the cluster
On Sat, Feb 25, 2012 at 3:39 PM, Aditya Gupta ady...@gmail.com wrote: The output of nodetool ring after each addition of nodes, make just the last added node visible in the ring. When I retry to add the node(which are not visible) it says that it is already a part of the ring. Could you indicate how should I rectify this now, as you seem to have figured out the issue ? The simplest thing to do is rm -rf /var/lib/cassandra on all the nodes. -Brandon
Re: Only the last added node is visible in the cluster
Thanks removing the system keyspace worked! Thanks! On Sun, Feb 26, 2012 at 3:39 AM, Brandon Williams dri...@gmail.com wrote: On Sat, Feb 25, 2012 at 3:39 PM, Aditya Gupta ady...@gmail.com wrote: The output of nodetool ring after each addition of nodes, make just the last added node visible in the ring. When I retry to add the node(which are not visible) it says that it is already a part of the ring. Could you indicate how should I rectify this now, as you seem to have figured out the issue ? The simplest thing to do is rm -rf /var/lib/cassandra on all the nodes. -Brandon
