from:"laxmikanth sadula"

Re: disable reads from node till hints are fully synced

2017-09-11 Thread laxmikanth sadula

Hi,

If you are using datastax java driver, I think this might work.

http://docs.datastax.com/en/latest-java-driver-api/com/datastax/driver/core/policies/WhiteListPolicy.html

On Sep 11, 2017 2:28 AM, "Jeff Jirsa"  wrote:

> There's not - you can disable native/binary to make it less likely, but
> you can't stop reads entirely because you need gossip up in order to have
> hints deliver
>
> What you can do is use severity to make the dynamic snitch MUCH less
> likely to pick that node (and disable binary so it's not a coordinator).
> That often works for what you're trying to do, though it's imperfect.
> Brandon described this a bit here:
>
> https://www.datastax.com/dev/blog/dynamic-snitching-in-
> cassandra-past-present-and-future
>
>
>
> --
> Jeff Jirsa
>
>
> On Sep 10, 2017, at 1:28 PM, Aleksandr Ivanov  wrote:
>
> Hello,
>
> from time to time we have situations where node is down for longer period
> (but less than max_hint_window_in_ms). After node is up and hints are
> activly syncing to affected node, clients get inconsistent data (client
> uses LOCAL_ONE consistency due to performance reasons).
>
> Is any way exist to disable reads from such node till hints are fully
> synced?
>
> Regards,
> Aleksandr
>
>

Re: Secondary Index

2017-06-25 Thread laxmikanth sadula

Thanks for the reply vladimir but we can't upgrade c* binary that soon as
we have a quick requirement for the use case I said in my first mail.

So just wanted to know the difference between the 2 queries I asked in my
last mail


1. select * from ks1.cf1 where status=1;
2. select * from ks1.cf1 where id1=123456 and status=1;

where id1 is partition key and status is indexed column as I said above.

Could you please tell me the performance difference btwn above 2 queries.

Thanks in advance,

Techpyaasaa

On Jun 25, 2017 11:29 PM, "Vladimir Yudovin"  wrote:

> Hi,
>
> beyond scope of your question (as you use 2.1.17) but starting from v3.4 SASI
> is avaialble
> ,
> doc is about DSE, but is applicable for free version as well.
>
> Best regards, Vladimir Yudovin,
> *Winguzone  - Cloud Cassandra Hosting*
>
>
>  On Mon, 19 Jun 2017 14:00:40 -0400 *techpyaasa .
> >* wrote 
>
> Hi,
>
> I want to create Index on already existing table which has more than 3
> GB/node.
> We are using c*-2.1.17 with 2 DCs , each DC with 3 groups and each group
> has 7 nodes.(Total 42 nodes in cluster)
>
> So is it ok to create Index on this table now or will it have any problem?
> If its ok , how much time it would take for this process?
>
>
> Thanks in advance,
> TechPyaasa
>
>
>

Re: Disk full during new node bootstrap

2017-02-04 Thread laxmikanth sadula

Yes .. same number of tokens...
256

On Sat, Feb 4, 2017 at 11:56 AM, Jonathan Haddad  wrote:

> Are you using the same number of tokens on the new node as the old ones?
>
> On Fri, Feb 3, 2017 at 8:31 PM techpyaasa .  wrote:
>
>> Hi,
>>
>> We are using c* 2.0.17 , 2 DCs , RF=3.
>>
>> When I try to add new node to one group in a DC , I got disk full. Can
>> someone please tell what is the best way to resolve this?
>>
>> Run compaction for nodes in that group(to which I'm going to add new
>> node, as data streams to new nodes from nodes of group to which it is added)
>>
>> OR
>>
>> Boootstrap/add  2(multiple nodes) at a time?
>>
>>
>> Please suggest better way to fix this.
>>
>> Thanks in advance
>>
>> Techpyaasa
>>
>>


-- 
Regards,
Laxmikanth
99621 38051

Re: Cannot set TTL in COPY command

2016-10-26 Thread laxmikanth sadula

You mean to say instead of
*COPY keyspace1.columnFamily1 FROM 'dump_data.csv' WITH TTL = '7200';*use


*COPY keyspace1.columnFamily1 FROM 'dump_data.csv'
WITH DEFAULT_TIME_TO_LIVE = '7200';*
I tried this way too, but again exception thrown saying "*Unrecognized COPY
FROM options: default_time_to_live*" :( :(

On Wed, Oct 26, 2016 at 8:53 PM, Lahiru Gamathige 
wrote:

> You have to use with default_time_to_live = 7200.
>
> On Wed, Oct 26, 2016 at 8:07 AM, techpyaasa . 
> wrote:
>
>> Hi all,
>>
>> I'm getting following exception when I try to set TTL using COPY command,
>> where as it is working fine without TTL option. Followed doc at
>> https://docs.datastax.com/en/cql/3.1/cql/cql_reference/copy_r.html
>>
>> *"**Improper COPY command.**"*
>>
>> Command used as below
>>
>> *COPY keyspace1.columnFamily1 FROM 'dump_data.csv' WITH TTL = '7200';*
>>
>>
>> Version is as below
>>
>> *[cqlsh 4.1.1 | Cassandra 2.0.17 | CQL spec 3.1.1 | Thrift protocol
>> 19.39.0]*
>>
>>
>> Can some one please tell me how to set TTL using COPY command?
>>
>> Thanks
>> Techpyaasa
>>
>
>


-- 
Regards,
Laxmikanth
99621 38051

Re: Difference in token range count

2016-09-30 Thread laxmikanth sadula

Hi Eric,

Thanks for the reply...
RF=3 for all DCs...

On Fri, Sep 30, 2016 at 9:57 PM, Eric Stevens  wrote:

> What is your replication factor in this DC?
>
> On Fri, Sep 30, 2016 at 8:08 AM techpyaasa .  wrote:
>
>> Hi ,
>>
>> We have c*-2.0.17  with 3 data centers . Each data center has 9 nodes. 
>> vnodes enabled in all nodes.
>>
>> When I ran -local repair(./nodetool -local repair keyspace_name1 
>> columnfamily_1) on one of data center I saw following print
>>
>> "Starting repair command #3, repairing *2647 ranges* for keyspace 
>> keyspace_name1"
>>
>> The count of ranges , it is supposed to be *2304*(256*9) as we have 9 nodes 
>> in one data center right but why it is showing as 2647 ranges ??
>>
>> Can someone please clarify why this difference in token ranges count?
>>
>> Thanks
>> techpyaasa
>>
>>


-- 
Regards,
Laxmikanth
99621 38051

Re: New node block in autobootstrap

2016-09-27 Thread laxmikanth sadula

Ok... Thanks for the reply...
I'm going to retry nodetool rebuild with following changes as you said

net.ipv4.tcp_keepalive_time=60 net.ipv4.tcp_keepalive_probes=3
net.ipv4.tcp_keepalive_intvl=10

Hope this changes would be enough on the new node where I'm running
'nodetool rebuild' and hope NOT required on all existing nodes from which
data is going to get streamed..Am I right?

On Sep 28, 2016 1:04 AM, "Paulo Motta" <pauloricard...@gmail.com> wrote:

> Yeah this is likely to be caused by idle connections being shut down, so
> you may need to update your tcp_keepalive* and/or network/firewall settings.
>
> 2016-09-27 15:29 GMT-03:00 laxmikanth sadula <laxmikanth...@gmail.com>:
>
>> Hi paul,
>>
>> Thanks for the reply...
>>
>> I'm getting following streaming exceptions during nodetool rebuild in
>> c*-2.0.17
>>
>> *04:24:49,759 StreamSession.java (line 461) [Stream
>> #5e1b7f40-8496-11e6-8847-1b88665e430d] Streaming error occurred*
>> *java.io.IOException: Connection timed out*
>> *at sun.nio.ch.FileDispatcherImpl.write0(Native Method)*
>> *at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)*
>> *at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)*
>> *at sun.nio.ch.IOUtil.write(IOUtil.java:65)*
>> *at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:487)*
>> *at
>> org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:44)*
>> *at
>> org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:339)*
>> *at
>> org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:311)*
>> *at java.lang.Thread.run(Thread.java:745)*
>> *DEBUG [STREAM-OUT-/xxx.xxx.98.168] 2016-09-27 04:24:49,764
>> ConnectionHandler.java (line 104) [Stream
>> #5e1b7f40-8496-11e6-8847-1b88665e430d] Closing stream connection handler on
>> /xxx.xxx.98.168*
>> * INFO [STREAM-OUT-/xxx.xxx.98.168] 2016-09-27 04:24:49,764
>> StreamResultFuture.java (line 186) [Stream
>> #5e1b7f40-8496-11e6-8847-1b88665e430d] Session with /xxx.xxx.98.168 is
>> complete*
>> *ERROR [STREAM-OUT-/xxx.xxx.98.168] 2016-09-27 04:24:49,764
>> StreamSession.java (line 461) [Stream
>> #5e1b7f40-8496-11e6-8847-1b88665e430d] Streaming error occurred*
>> *java.io.IOException: Broken pipe*
>> *at sun.nio.ch.FileDispatcherImpl.write0(Native Method)*
>> *at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)*
>> *at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)*
>> *at sun.nio.ch.IOUtil.write(IOUtil.java:65)*
>> *at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:487)*
>> *at
>> org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:44)*
>> *at
>> org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:339)*
>> *at
>> org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:319)*
>> *at java.lang.Thread.run(Thread.java:745)*
>> *DEBUG [STREAM-IN-/xxx.xxx.98.168] 2016-09-27 04:24:49,909
>> ConnectionHandler.java (line 244) [Stream
>> #5e1b7f40-8496-11e6-8847-1b88665e430d] Received File (Header (cfId:
>> 68af9ee0-96f8-3b1d-a418-e5ae844f2cc2, #3, version: jb, estimated keys:
>> 4736, transfer size: 2306880, compressed?: true), file:
>> /home/cassandra/data_directories/data/keyspace_name1/archiving_metadata/keyspace_name1-archiving_metadata-tmp-jb-27-Data.db)*
>> *ERROR [STREAM-IN-/xxx.xxx.98.168] 2016-09-27 04:24:49,909
>> StreamSession.java (line 461) [Stream
>> #5e1b7f40-8496-11e6-8847-1b88665e430d] Streaming error occurred*
>> *java.lang.RuntimeException: Outgoing stream handler has been closed*
>> *at
>> org.apache.cassandra.streaming.ConnectionHandler.sendMessage(ConnectionHandler.java:126)*
>> *at
>> org.apache.cassandra.streaming.StreamSession.receive(StreamSession.java:524)*
>> *at
>> org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:413)*
>> *at
>> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:245)*
>> *at java.lang.Thread.run(Thread.java:745)*
>>
>> On Sep 27, 2016 11:48 PM, "Paulo Motta" <pauloricard...@gmail.com> wrote:
>>
>>> What type of streaming timeout are you getting? Do you have a stack
>>> trace? What version are you in?
>>>
>>> See more information about tuning tcp_keepalive* here:
>>> https://docs.da

Re: New node block in autobootstrap

2016-09-27 Thread laxmikanth sadula

Hi paul,

Thanks for the reply...

I'm getting following streaming exceptions during nodetool rebuild in
c*-2.0.17

*04:24:49,759 StreamSession.java (line 461) [Stream
#5e1b7f40-8496-11e6-8847-1b88665e430d] Streaming error occurred*
*java.io.IOException: Connection timed out*
*at sun.nio.ch.FileDispatcherImpl.write0(Native Method)*
*at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)*
*at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)*
*at sun.nio.ch.IOUtil.write(IOUtil.java:65)*
*at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:487)*
*at
org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:44)*
*at
org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:339)*
*at
org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:311)*
*at java.lang.Thread.run(Thread.java:745)*
*DEBUG [STREAM-OUT-/xxx.xxx.98.168] 2016-09-27 04:24:49,764
ConnectionHandler.java (line 104) [Stream
#5e1b7f40-8496-11e6-8847-1b88665e430d] Closing stream connection handler on
/xxx.xxx.98.168*
* INFO [STREAM-OUT-/xxx.xxx.98.168] 2016-09-27 04:24:49,764
StreamResultFuture.java (line 186) [Stream
#5e1b7f40-8496-11e6-8847-1b88665e430d] Session with /xxx.xxx.98.168 is
complete*
*ERROR [STREAM-OUT-/xxx.xxx.98.168] 2016-09-27 04:24:49,764
StreamSession.java (line 461) [Stream
#5e1b7f40-8496-11e6-8847-1b88665e430d] Streaming error occurred*
*java.io.IOException: Broken pipe*
*at sun.nio.ch.FileDispatcherImpl.write0(Native Method)*
*at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)*
*at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)*
*at sun.nio.ch.IOUtil.write(IOUtil.java:65)*
*at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:487)*
*at
org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:44)*
*at
org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:339)*
*at
org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:319)*
*at java.lang.Thread.run(Thread.java:745)*
*DEBUG [STREAM-IN-/xxx.xxx.98.168] 2016-09-27 04:24:49,909
ConnectionHandler.java (line 244) [Stream
#5e1b7f40-8496-11e6-8847-1b88665e430d] Received File (Header (cfId:
68af9ee0-96f8-3b1d-a418-e5ae844f2cc2, #3, version: jb, estimated keys:
4736, transfer size: 2306880, compressed?: true), file:
/home/cassandra/data_directories/data/keyspace_name1/archiving_metadata/keyspace_name1-archiving_metadata-tmp-jb-27-Data.db)*
*ERROR [STREAM-IN-/xxx.xxx.98.168] 2016-09-27 04:24:49,909
StreamSession.java (line 461) [Stream
#5e1b7f40-8496-11e6-8847-1b88665e430d] Streaming error occurred*
*java.lang.RuntimeException: Outgoing stream handler has been closed*
*at
org.apache.cassandra.streaming.ConnectionHandler.sendMessage(ConnectionHandler.java:126)*
*at
org.apache.cassandra.streaming.StreamSession.receive(StreamSession.java:524)*
*at
org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:413)*
*at
org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:245)*
*at java.lang.Thread.run(Thread.java:745)*

On Sep 27, 2016 11:48 PM, "Paulo Motta" <pauloricard...@gmail.com> wrote:

> What type of streaming timeout are you getting? Do you have a stack trace?
> What version are you in?
>
> See more information about tuning tcp_keepalive* here:
> https://docs.datastax.com/en/cassandra/2.0/cassandra/troubleshooting/
> trblshootIdleFirewall.html
>
> 2016-09-27 14:07 GMT-03:00 laxmikanth sadula <laxmikanth...@gmail.com>:
>
>> @Paulo Motta
>>
>> Even we are facing Streaming timeout exceptions during 'nodetool rebuild'
>> , I set streaming_socket_timeout_in_ms to 8640 (24 hours) as suggested
>> in datastax blog  - https://support.datastax.com/h
>> c/en-us/articles/206502913-FAQ-How-to-reduce-the-impact-of-s
>> treaming-errors-or-failures  , but still we are getting streaming
>> exceptions.
>>
>> And what is the suggestible settings/values for kernel tcp_keepalive
>> which would help streaming succeed ?
>>
>> Thank you
>>
>> On Tue, Aug 16, 2016 at 12:21 AM, Paulo Motta <pauloricard...@gmail.com>
>> wrote:
>>
>>> What version are you in? This seems like a typical case were there was a
>>> problem with streaming (hanging, etc), do you have access to the logs?
>>> Maybe look for streaming errors? Typically streaming errors are related to
>>> timeouts, so you should review your cassandra
>>> streaming_socket_timeout_in_ms and kernel tcp_keepalive settings.
>>>
>>> If you're on 2.2+ you can resume a failed bootstrap with nodetoo

Re: New node block in autobootstrap

2016-09-27 Thread laxmikanth sadula

@Paulo Motta

Even we are facing Streaming timeout exceptions during 'nodetool rebuild' ,
I set streaming_socket_timeout_in_ms to 8640 (24 hours) as suggested in
datastax blog  - https://support.datastax.com/hc/en-us/articles/206502913-
FAQ-How-to-reduce-the-impact-of-streaming-errors-or-failures  , but still
we are getting streaming exceptions.

And what is the suggestible settings/values for kernel tcp_keepalive which
would help streaming succeed ?

Thank you

On Tue, Aug 16, 2016 at 12:21 AM, Paulo Motta 
wrote:

> What version are you in? This seems like a typical case were there was a
> problem with streaming (hanging, etc), do you have access to the logs?
> Maybe look for streaming errors? Typically streaming errors are related to
> timeouts, so you should review your cassandra
> streaming_socket_timeout_in_ms and kernel tcp_keepalive settings.
>
> If you're on 2.2+ you can resume a failed bootstrap with nodetool
> bootstrap resume. There were also some streaming hanging problems fixed
> recently, so I'd advise you to upgrade to the latest version of your
> particular series for a more robust version.
>
> Is there any reason why you didn't use the replace procedure
> (-Dreplace_address) to replace the node with the same tokens? This would be
> a bit faster than remove + bootstrap procedure.
>
> 2016-08-15 15:37 GMT-03:00 Jérôme Mainaud :
>
>> Hello,
>>
>> A client of mime have problems when adding a node in the cluster.
>> After 4 days, the node is still in joining mode, it doesn't have the same
>> level of load than the other and there seems to be no streaming from and to
>> the new node.
>>
>> This node has a history.
>>
>>1. At the begin, it was in a seed in the cluster.
>>2. Ops detected that client had problems with it.
>>3. They tried to reset it but failed. In their process they launched
>>several repair and rebuild process on the node.
>>4. Then they asked me to help them.
>>5. We stopped the node,
>>6. removed it from the list of seeds (more precisely it was replaced
>>by another node),
>>7. removed it from the cluster (I choose not to use decommission
>>since node data was compromised)
>>8. deleted all files from data, commitlog and savedcache directories.
>>9. after the leaving process ended, it was started as a fresh new
>>node and began autobootstrap.
>>
>>
>> As I don’t have direct access to the cluster I don't have a lot of
>> information, but I will have tomorrow (logs and results of some commands).
>> And I can ask for people any required information.
>>
>> Does someone have any idea of what could have happened and what I should
>> investigate first ?
>> What would you do to unlock the situation ?
>>
>> Context: The cluster consists of two DC, each with 15 nodes. Average load
>> is around 3 TB per node. The joining node froze a little after 2 TB.
>>
>> Thank you for your help.
>> Cheers,
>>
>>
>> --
>> Jérôme Mainaud
>> jer...@mainaud.com
>>
>
>


-- 
Regards,
Laxmikanth
99621 38051

Re: How long/how many days 'nodetool gossipinfo' will have decommissioned nodes info

2016-09-26 Thread laxmikanth sadula

Thank you @Jaoquin and @DuyHai

On Mon, Sep 26, 2016 at 10:00 PM, DuyHai Doan  wrote:

> I've read from some  that the gossip info will stay
> around for 72h before being removed.
>
> On Mon, Sep 26, 2016 at 6:19 PM, Joaquin Casares <
> joaq...@thelastpickle.com> wrote:
>
>> Hello Techpyassa,
>>
>> Sometimes old gossip information tends to echo around for quite a bit
>> longer than intended. I'm unsure how long the LEFT messages are supposed to
>> be echoed for, but if you want to force the removal of a removed node from
>> gossip, you can use the Assassinate Endpoint JMX command. On larger
>> clusters, running this command synchronously across all machines may be
>> required. Instructions on Assassinate Endpoint can be found here:
>>
>> https://gist.github.com/justenwalker/8338334
>>
>> If you're planning on recommissioning the same node, upon bootstrapping
>> the gossiped message should change to a JOINING message overwriting the
>> LEFT message.
>>
>> I've personally never checked `nodetool gossipinfo` before
>> recommissioning a node and typically only ensure the node does not appear
>> in `nodetool status`.
>>
>> Hope that helps,
>>
>> Joaquin Casares
>> Consultant
>> Austin, TX
>>
>> Apache Cassandra Consulting
>> http://www.thelastpickle.com
>>
>> On Sun, Sep 25, 2016 at 2:17 PM, Laxmikanth S 
>> wrote:
>>
>>> Hi,
>>>
>>> Recently we have decommissioned nodes from Cassandra cluster , but even
>>> after nearly 48 hours 'nodetool gossipinfo' still shows the removed nodes(
>>> as LEFT).
>>>
>>> I just wanted to recommission the same node again. So just wanted to
>>> know , will it create a problem if I recommission the same node(same IP)
>>>  again while its state is maintained as LEFT in 'nodetool gossipnfo'.
>>>
>>>
>>> Thanks,
>>> Techpyaasa
>>>
>>
>>
>


-- 
Regards,
Laxmikanth
99621 38051

Nodetool rebuild exception on c*-2.0.17

2016-09-22 Thread laxmikanth sadula

Hi,

We have c* 2.0.17 cluster with 2 DCs - DC1, DC2.  We tried to add new data
center DC3 and ran "nodetool rebuild 'DC1'" and we faced below exception on
few nodes after some data got streamed to new nodes in new data center DC3.


*Exception in thread "main" java.lang.RuntimeException: Error while
rebuilding node: Stream failed*
*at
org.apache.cassandra.service.StorageService.rebuild(StorageService.java:936)*
*at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)*
*at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)*
*at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)*
*at java.lang.reflect.Method.invoke(Method.java:606)*
*at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:75)*
*at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)*
*at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)*
*at java.lang.reflect.Method.invoke(Method.java:606)*
*at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:279)*
*at
com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112)*
*at
com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46)*
*at
com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)*
*at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)*
*at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252)*
*at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)*
*at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)*
*at
javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1487)*
*at
javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:97)*
*at
javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1328)*
*at
javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1420)*
*at
javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:848)*
*at sun.reflect.GeneratedMethodAccessor29.invoke(Unknown Source)*
*at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)*
*at java.lang.reflect.Method.invoke(Method.java:606)*
*at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)*
*at sun.rmi.transport.Transport$2.run(Transport.java:202)*
*at sun.rmi.transport.Transport$2.run(Transport.java:199)*
*at java.security.AccessController.doPrivileged(Native Method)*
*at sun.rmi.transport.Transport.serviceCall(Transport.java:198)*
*at
sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:567)*
*at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:828)*
*at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.access$400(TCPTransport.java:619)*
*at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler$1.run(TCPTransport.java:684)*
*at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler$1.run(TCPTransport.java:681)*
*at java.security.AccessController.doPrivileged(Native Method)*
*at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:681)*
*at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)*
*at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)*
*at java.lang.Thread.run(Thread.java:745)*


We  have 4  user keyspaces , so we altered all keyspaces as below before
issuing rebuild.

*ALTER KEYSPACE keyspace_name WITH replication = {'class':
'NetworkTopologyStrategy', 'DC1': '3' , 'DC2' : '3' , 'DC3' : '3'};*


*Output of describecluster*

./nodetool describecluster
Cluster Information:
Name: Ss Cluster
Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch
Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
Schema versions:
3b688e54-47be-39e8-ae45-e71ba98d69e2: [xxx.xxx.198.75, xxx.xxx.198.132,
xxx.xxx.198.133, xxx.xxx.12.115, xxx.xxx.198.78, xxx.xxx.12.123,
xxx.xxx.98.205, xxx.xxx.98.219, xxx.xxx.98.220, xxx.xxx.198.167,
xxx.xxx.98.172, xxx.xxx.98.173, xxx.xxx.98.170, xxx.xxx.198.168,
xxx.xxx.98.171, xxx.xxx.198.169, xxx.xxx.12.146, xxx.xxx.98.168,
xxx.xxx.12.145, xxx.xxx.98.169, xxx.xxx.12.144, xxx.xxx.12.143,
xxx.xxx.12.140, xxx.xxx.12.139, xxx.xxx.198.126, xxx.xxx.12.136,
xxx.xxx.12.135, xxx.xxx.198.191, xxx.xxx.12.133, xxx.xxx.12.131]



*nodetool status output:*

./nodetool status
Note: Ownership information does not include topology; for complete
information, specify a keyspace
Datacenter: DC1
===
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address  Load   Tokens  Owns   Host ID
  Rack
UN  xxx.xxx.198.75   639.41 GB  256 3.2%
fdd32c67-3cea-4174-b59b-c1ea14e1a334  GRP1
UN  xxx.xxx.198.132  581.94 GB  256 3.4%
6a465101-29e7-4792-8269-851200a70023  GRP2
UN  xxx.xxx.198.133  618.22 GB  256 3.6%
751ce15a-10f1-44cf-9357-04da7e21b511  GRP2
UN

Nodetool rebuild exception

2016-09-22 Thread laxmikanth sadula

Hi,

We have c* 2.0.17 cluster with 2 DCs - DC1, DC2.  We tried to add new data
center DC3 and ran "nodetool rebuild 'DC1'" and we faced below exception on
few nodes after some data got streamed to new nodes in new data center DC3.


*Exception in thread "main" java.lang.RuntimeException: Error while
rebuilding node: Stream failed*
*at
org.apache.cassandra.service.StorageService.rebuild(StorageService.java:936)*
*at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)*
*at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)*
*at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)*
*at java.lang.reflect.Method.invoke(Method.java:606)*
*at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:75)*
*at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)*
*at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)*
*at java.lang.reflect.Method.invoke(Method.java:606)*
*at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:279)*
*at
com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112)*
*at
com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46)*
*at
com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)*
*at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)*
*at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252)*
*at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)*
*at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)*
*at
javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1487)*
*at
javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:97)*
*at
javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1328)*
*at
javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1420)*
*at
javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:848)*
*at sun.reflect.GeneratedMethodAccessor29.invoke(Unknown Source)*
*at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)*
*at java.lang.reflect.Method.invoke(Method.java:606)*
*at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)*
*at sun.rmi.transport.Transport$2.run(Transport.java:202)*
*at sun.rmi.transport.Transport$2.run(Transport.java:199)*
*at java.security.AccessController.doPrivileged(Native Method)*
*at sun.rmi.transport.Transport.serviceCall(Transport.java:198)*
*at
sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:567)*
*at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:828)*
*at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.access$400(TCPTransport.java:619)*
*at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler$1.run(TCPTransport.java:684)*
*at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler$1.run(TCPTransport.java:681)*
*at java.security.AccessController.doPrivileged(Native Method)*
*at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:681)*
*at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)*
*at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)*

*at java.lang.Thread.run(Thread.java:745)*

We  have 4  user keyspaces , so we altered all keyspaces as below before
issuing rebuild.

ALTER KEYSPACE keyspace_name WITH replication = {'class':
'NetworkTopologyStrategy', 'IDC': '3' , 'DR' : '3' , 'US3' : '3'};


*Output of describecluster*

*./nodetool describecluster*
*Cluster Information:*
* Name: Ss Cluster*
* Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch*
* Partitioner: org.apache.cassandra.dht.Murmur3Partitioner*
* Schema versions:*
* 3b688e54-47be-39e8-ae45-e71ba98d69e2: [xxx.xxx.198.75, xxx.xxx.198.132,
xxx.xxx.198.133, xxx.xxx.12.115, xxx.xxx.198.78, xxx.xxx.12.123,
xxx.xxx.98.205, xxx.xxx.98.219, xxx.xxx.98.220, xxx.xxx.198.167,
xxx.xxx.98.172, xxx.xxx.98.173, xxx.xxx.98.170, xxx.xxx.198.168,
xxx.xxx.98.171, xxx.xxx.198.169, xxx.xxx.12.146, xxx.xxx.98.168,
xxx.xxx.12.145, xxx.xxx.98.169, xxx.xxx.12.144, xxx.xxx.12.143,
xxx.xxx.12.140, xxx.xxx.12.139, xxx.xxx.198.126, xxx.xxx.12.136,
xxx.xxx.12.135, xxx.xxx.198.191, xxx.xxx.12.133, xxx.xxx.12.131]*



*nodetool status output:*

*./nodetool status*
*Note: Ownership information does not include topology; for complete
information, specify a keyspace*
*Datacenter: IDC*
*===*
*Status=Up/Down*
*|/ State=Normal/Leaving/Joining/Moving*
*--  Address  Load   Tokens  Owns   Host ID
  Rack*
*UN  xxx.xxx.198.75   639.41 GB  256 3.2%
fdd32c67-3cea-4174-b59b-c1ea14e1a334  GRP1*
*UN  xxx.xxx.198.132  581.94 GB  256 3.4%
6a465101-29e7-4792-8269-851200a70023  GRP2*
*UN  xxx.xxx.198.133  618.22 GB  256 3.6%

Re: Throughout of hints delivery

2016-09-17 Thread laxmikanth sadula

Hi Matija,

All nodes are UP & running and even GC patterns are all well. But I see lot
of "Timed out replaying hints" in HintedHandOff Manger, I suspect this
might be the reason why GBs of hints getting piled up instead of proper
delivery.
So this clearly indicates some network related issues , so just wanted to
know the way to monitor hints delivery throughput and also tcp
throughput(packets sent, received ,dropped etc., on an interface).

If anyone monitoring such stats, please let me know.

On Sat, Sep 17, 2016 at 11:26 PM, Matija Gobec <matija0...@gmail.com> wrote:

> Hi,
>
> You should first figure out why you have so many hints and then think
> about throughput of hints delivery.
> Hints are generated for dead nodes and in a healthy cluster are not
> present.
> Are all your nodes alive and running? What is the issue of inter DC
> connectivity?
>
> Matija
>
> --
>
> *Matija Gobec*
> *Co-Founder & Senior Consultant*
> www.smartcat.io
>
>
>
> *Data <https://www.smartcat.io/expertise/> --> Knowledge
> <https://www.smartcat.io/blog/> --> Power <https://www.smartcat.io/news/> *
>
> On Sat, Sep 17, 2016 at 3:16 PM, laxmikanth sadula <
> laxmikanth...@gmail.com> wrote:
>
>> Hi,
>>
>> Is there any way to monitor hints delivery throughout/performance/issue
>> delivering hints?
>>
>> We have 2 DC c* cluster with 2.0.17 with RF=3 setup. Due to inter DC
>> connectivity issues/some other issues hints shoot upto GBs/node.
>>
>> So I would like to monitor hints throughput/pin point the reason for
>> hints growth on nodes.
>>
>> Kindly let us know if any one of you have such thing to monitor hints.
>>
>>
>> Thanks
>> Laxmikanth
>>
>>
>> --
>> Regards,
>> Laxmikanth
>> 99621 38051
>>
>>
>>
>

-- 
Regards,
Laxmikanth
99621 38051

Throughout of hints delivery

2016-09-17 Thread laxmikanth sadula

Hi,

Is there any way to monitor hints delivery throughout/performance/issue
delivering hints?

We have 2 DC c* cluster with 2.0.17 with RF=3 setup. Due to inter DC
connectivity issues/some other issues hints shoot upto GBs/node.

So I would like to monitor hints throughput/pin point the reason for hints
growth on nodes.

Kindly let us know if any one of you have such thing to monitor hints.


Thanks
Laxmikanth


-- 
Regards,
Laxmikanth
99621 38051

Is to ok restart DECOMMISION

2016-09-15 Thread laxmikanth sadula

I started decommssioned a node in our cassandra cluster.
But its taking too long time (more than 12 hrs) , so I would like to
restart(stop/kill the node & restart 'node decommission' again)..

Does killing node/stopping decommission and restarting decommission will
cause any issues to cluster?

Using c*-2.0.17 , 2 Data centers, each DC with 3 groups each , each group
with 3 nodes with RF-3

-- 
Thanks...!

Re: New data center to an existing cassandra cluster

2016-08-27 Thread laxmikanth sadula

Yes  , RF=3 in existing datacenters DC1 & DC2 and going to be same RF in
new datacenter DC3 which I'm going to add.


On Sat, Aug 27, 2016 at 11:15 PM, Alexander DEJANOVSKI <
adejanov...@gmail.com> wrote:

> Reads at quorum in dc3 will involve dc1 and dc2 as they will require a
> response from more than half the replicas throughout the Cluster.
>
> If you're using RF=3 in each DC, each read will need at least 5 responses,
> which DC3 cannot provide on its own.
>
> You can have troubles if DC3 has more than half then replicas, but I
> guess/hope it is not the case, otherwise you're fine.
>
> You would be in trouble though if you were using local_quorum on DC3 or
> ONE on any DC.
>
>
>
> Le sam. 27 août 2016 19:11, Surbhi Gupta <surbhi.gupt...@gmail.com> a
> écrit :
>
>> Yes, it will have issue during the time new nodes are building
>> So it is always advised to use LOCAL_QUORUM instead of QUORUM and
>> LOCAL_ONE instead of ONE
>>
>> On 27 August 2016 at 09:45, laxmikanth sadula <laxmikanth...@gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> I'm going to add a new data center DC3 to an existing cassandra cluster
>>> which has already 2 data centers DC1 , DC2.
>>>
>>> The thing I'm worried of is about tables in one keyspace which has
>>> QUORUM reads and NOT LOCAL_QUORUM.
>>> So while adding a new data centers with auto_bootstrap:false and
>>> 'nodetool rebuild' , will the queries to tables in this keyspace will have
>>> any issue ?
>>>
>>> Thanks in advance.
>>>
>>> --
>>> Regards,
>>> Laxmikanth
>>>
>>
>>


-- 
Regards,
Laxmikanth
99621 38051

New data center to an existing cassandra cluster

2016-08-27 Thread laxmikanth sadula

Hi,

I'm going to add a new data center DC3 to an existing cassandra cluster
which has already 2 data centers DC1 , DC2.

The thing I'm worried of is about tables in one keyspace which has QUORUM
reads and NOT LOCAL_QUORUM.
So while adding a new data centers with auto_bootstrap:false and 'nodetool
rebuild' , will the queries to tables in this keyspace will have any issue ?

Thanks in advance.

-- 
Regards,
Laxmikanth

Re: disable reads from node till hints are fully synced

Re: Secondary Index

Re: Disk full during new node bootstrap

Re: Cannot set TTL in COPY command

Re: Difference in token range count

Re: New node block in autobootstrap

Re: New node block in autobootstrap

Re: New node block in autobootstrap

Re: How long/how many days 'nodetool gossipinfo' will have decommissioned nodes info

Nodetool rebuild exception on c*-2.0.17

Nodetool rebuild exception

Re: Throughout of hints delivery

Throughout of hints delivery

Is to ok restart DECOMMISION

Re: New data center to an existing cassandra cluster

New data center to an existing cassandra cluster

16 matches

Site Navigation

Mail list logo

Footer information