Have you run Cassandra 3.11.x on Alma9 or Alma8

[Surbhi Gupta]

Hi,

Has any of you run Cassandra 3.11.x on Alma9 or Alma8?

Any issues or concerns?

We are going to upgrade from cent 7 to Alma8 or 9 , and wanted to
understand if there is/are any known issue?

Thanks
Surbhi

Re: Repair errors

[Surbhi Gupta]

357,5338449508113440752],
>> (4959134492108085445,4965331080956982133],
>> (5938148666505886222,5945280202710590417],
>> (8428867157147807368,8431880058869458408],
>> (5338449508113440752,5343449756371539147]]] Validation failed in /
>> 172.16.20.16:7000. Check the logs on the repair participants for further
>> details
>> at
>> org.apache.cassandra.tools.RepairRunner.progress(RepairRunner.java:137)
>> at
>> org.apache.cassandra.utils.progress.jmx.JMXNotificationProgressListener.handleNotification(JMXNotificationProgressListener.java:77)
>> at
>> java.management/com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.dispatchNotification(ClientNotifForwarder.java:633)
>> at
>> java.management/com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.doRun(ClientNotifForwarder.java:555)
>> at
>> java.management/com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.run(ClientNotifForwarder.java:474)
>> at
>> java.management/com.sun.jmx.remote.internal.ClientNotifForwarder$LinearExecutor.lambda$execute$0(ClientNotifForwarder.java:108)
>> at java.base/java.lang.Thread.run(Thread.java:829)
>>
>> I'm not sure what to do next?
>>
>> -Joe
>> On 8/6/2023 8:58 AM, Josh McKenzie wrote:
>>
>> Quick drive-by observation:
>>
>> Did not get replies from all endpoints.. Check the
>> logs on the repair participants for further details
>>
>>
>> dropping message of type HINT_REQ due to error
>> org.apache.cassandra.net.AsyncChannelOutputPlus$FlushException: The
>> channel this output stream was writing to has been closed
>>
>>
>> Caused by: io.netty.channel.unix.Errors$NativeIoException:
>> writeAddress(..) failed: Connection timed out
>>
>>
>> java.lang.RuntimeException: Did not get replies from all endpoints.
>>
>> These all point to the same shaped problem: for whatever reason, the
>> coordinator of this repair didn't receive replies from the replicas
>> executing it. Could be that they're dead, could be they took too long,
>> could be they never got the start message, etc. Distributed operations are
>> tricky like that.
>>
>> Logs on the replicas doing the actual repairs should give you more
>> insight; this is a pretty low level generic set of errors that basically
>> amounts to "we didn't hear back from the other participants in time so we
>> timed out."
>>
>> On Fri, Aug 4, 2023, at 12:02 PM, Surbhi Gupta wrote:
>>
>> Can you please try to do nodetool describecluster from every node of the
>> cluster?
>>
>> One time I noticed issue when nodetool status shows all nodes UN but
>> describecluster was not.
>>
>> Thanks
>> Surbhi
>>
>> On Fri, Aug 4, 2023 at 8:59 AM Joe Obernberger <
>> joseph.obernber...@gmail.com> wrote:
>>
>> Hi All - been using reaper to do repairs, but it has hung.  I tried to
>> run:
>> nodetool repair -pr
>> on each of the nodes, but they all fail with some form of this error:
>>
>> error: Repair job has failed with the error message: Repair command #521
>> failed with error Did not get replies from all endpoints.. Check the
>> logs on the repair participants for further details
>> -- StackTrace --
>> java.lang.RuntimeException: Repair job has failed with the error
>> message: Repair command #521 failed with error Did not get replies from
>> all endpoints.. Check the logs on the repair participants for further
>> details
>>  at
>> org.apache.cassandra.tools.RepairRunner.progress(RepairRunner.java:137)
>>  at
>>
>> org.apache.cassandra.utils.progress.jmx.JMXNotificationProgressListener.handleNotification(JMXNotificationProgressListener.java:77)
>>  at
>>
>> java.management/com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.dispatchNotification(ClientNotifForwarder.java:633)
>>  at
>>
>> java.management/com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.doRun(ClientNotifForwarder.java:555)
>>  at
>>
>> java.management/com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.run(ClientNotifForwarder.java:474)
>>  at
>>
>> java.management/com.sun.jmx.remote.internal.ClientNotifForwarder$LinearExecutor.lambda$execute$0(ClientNotifForwarder.java:108)
>>  at java.base/java.lang.Thread.run(Thread.java:829)
>>
>> Using version 4.1.2-1
>> nodetool status
>> Datacenter: datacenter1
>> ===
>> Status=Up/Down
>&

Re: Upgrade from 3.11.5 to 4.1.x

[Surbhi Gupta]

Thanks,

I am looking to to upgrade to 4.1.x .
Please advise.

Thanks
Surbhi

On Thu, Aug 10, 2023 at 5:39 PM MyWorld  wrote:

> Though it's recommended to upgrade to latest version of 3.11.x and then to
> ver 4  but even upgrading directly won't be a problem. Just check the
> release notes.
>
> However for production, I would recommend to go for 4.0.x latest stable
> version.
>
> Regards
> Ashish
>
> On Sat, 8 Jul, 2023, 05:44 Surbhi Gupta,  wrote:
>
>> Hi,
>>
>> We have to upgrade from 3.11.5 to 4.1.x .
>> Can we do it in one go ?
>> Or do we have to go to an intermediate version first?
>>
>> Thanks
>> Surbhi
>>
>

Re: Materialized View inconsistency issue

[Surbhi Gupta]

Thanks everyone.


On Wed, 9 Aug 2023 at 01:00, Regis Le Bretonnic
 wrote:
>
> Hi Surbhi
>
> We do use cassandra materialized views even if not recommended.
> There are known issues you have to make with. Despite of them, we still use 
> VM.
> What we observe is :
> * there are  inconsistency issues but few. Most of them are rows that should 
> not exist in the MV...
> * we made a spark script downloading the master table and the MV, and 
> comparing them and fixing data (as said previously we have very few errors 
> and we run it maybe once a year)
>
> * Things go very very very bad when you add or remove a node ! Limit this 
> operation if possible and do it knowing what can happen (we isolate the 
> ring/datacenter and fix data before putting it back to production. We did 
> this only once in the last 4 years).
>
> PS : all proposals avoiding MV failed for our project. Basically managing a 
> table like a MV (by deleting and inserting rows from code) is worse and more 
> corrupted than what MV does...
> The worse issue is adding and removing nodes. Maybe cassandra 4 improves this 
> point (not tested yet).
>
> Have fun...
>
> Le mar. 8 août 2023 à 22:36, Surbhi Gupta  a écrit :
>>
>> Hi,
>>
>> We get complaints about Materialized View inconsistency issues.
>> We are on 3.11.5 and on 3.11.5 Materialized Views were not production ready.
>> We are ok to upgrade.
>>
>> On which version of cassandra MVs doesnt have inconsistency issues?
>>
>> Thanks
>> Surbhi

Materialized View inconsistency issue

[Surbhi Gupta]

Hi,

We get complaints about Materialized View inconsistency issues.
We are on 3.11.5 and on 3.11.5 Materialized Views were not production ready.
We are ok to upgrade.

On which version of cassandra MVs doesnt have inconsistency issues?

Thanks
Surbhi

Re: Repair errors

[Surbhi Gupta]

Can you please try to do nodetool describecluster from every node of the
cluster?

One time I noticed issue when nodetool status shows all nodes UN but
describecluster was not.

Thanks
Surbhi

On Fri, Aug 4, 2023 at 8:59 AM Joe Obernberger 
wrote:

> Hi All - been using reaper to do repairs, but it has hung.  I tried to run:
> nodetool repair -pr
> on each of the nodes, but they all fail with some form of this error:
>
> error: Repair job has failed with the error message: Repair command #521
> failed with error Did not get replies from all endpoints.. Check the
> logs on the repair participants for further details
> -- StackTrace --
> java.lang.RuntimeException: Repair job has failed with the error
> message: Repair command #521 failed with error Did not get replies from
> all endpoints.. Check the logs on the repair participants for further
> details
>  at
> org.apache.cassandra.tools.RepairRunner.progress(RepairRunner.java:137)
>  at
>
> org.apache.cassandra.utils.progress.jmx.JMXNotificationProgressListener.handleNotification(JMXNotificationProgressListener.java:77)
>  at
>
> java.management/com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.dispatchNotification(ClientNotifForwarder.java:633)
>  at
>
> java.management/com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.doRun(ClientNotifForwarder.java:555)
>  at
>
> java.management/com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.run(ClientNotifForwarder.java:474)
>  at
>
> java.management/com.sun.jmx.remote.internal.ClientNotifForwarder$LinearExecutor.lambda$execute$0(ClientNotifForwarder.java:108)
>  at java.base/java.lang.Thread.run(Thread.java:829)
>
> Using version 4.1.2-1
> nodetool status
> Datacenter: datacenter1
> ===
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> --  Address LoadTokens  Owns  Host
> ID   Rack
> UN  172.16.100.45   505.66 GiB  250 ?
> 07bccfce-45f1-41a3-a5c4-ee748a7a9b98  rack1
> UN  172.16.100.251  380.75 GiB  200 ?
> 274a6e8d-de37-4e0b-b000-02d221d858a5  rack1
> UN  172.16.100.35   479.2 GiB   200 ?
> 59150c47-274a-46fb-9d5e-bed468d36797  rack1
> UN  172.16.100.252  248.69 GiB  200 ?
> 8f0d392f-0750-44e2-91a5-b30708ade8e4  rack1
> UN  172.16.100.249  411.53 GiB  200 ?
> 49e4f571-7d1c-4e1e-aca7-5bbe076596f7  rack1
> UN  172.16.100.38   333.26 GiB  200 ?
> 0d9509cc-2f23-4117-a883-469a1be54baf  rack1
> UN  172.16.100.36   405.33 GiB  200 ?
> d9702f96-256e-45ae-8e12-69a42712be50  rack1
> UN  172.16.100.39   437.74 GiB  200 ?
> 93f9cb0f-ea71-4e3d-b62a-f0ea0e888c47  rack1
> UN  172.16.100.248  344.4 GiB   200 ?
> 4bbbe57c-6219-41e5-bbac-de92a9594d53  rack1
> UN  172.16.100.44   409.36 GiB  200 ?
> b2e5366e-8386-40ec-a641-27944a5a7cfa  rack1
> UN  172.16.100.37   236.08 GiB  120 ?
> 08a19658-40be-4e55-8709-812b3d4ac750  rack1
> UN  172.16.20.16975 GiB 500 ?
> 1ccd2cc5-3ee5-43c5-a8c3-7065bdc24297  rack1
> UN  172.16.100.34   340.77 GiB  200 ?
> 352fd049-32f8-4be8-9275-68b145ac2832  rack1
> UN  172.16.100.42   974.86 GiB  500 ?
> b088a8e6-42f3-4331-a583-47ef5149598f  rack1
>
> Note: Non-system keyspaces don't have the same replication settings,
> effective ownership information is meaningless
>
> Debug log has:
>
>
> DEBUG [ScheduledTasks:1] 2023-08-04 11:56:04,955
> MigrationCoordinator.java:264 - Pulling unreceived schema versions...
> INFO  [HintsDispatcher:11344] 2023-08-04 11:56:21,369
> HintsDispatchExecutor.java:318 - Finished hinted handoff of file
> 1ccd2cc5-3ee5-43c5-a8c3-7065bdc24297-1690426370160-2.hints to endpoint
> /172.16.20.16:7000: 1ccd2cc5-3ee5-43c5-a8c3-7065bdc24297, partially
> WARN
> [Messaging-OUT-/172.16.100.34:7000->/172.16.20.16:7000-LARGE_MESSAGES]
> 2023-08-04 11:56:21,916 OutboundConnection.java:491 -
> /172.16.100.34:7000->/172.16.20.16:7000-LARGE_MESSAGES-[no-channel]
> dropping message of type HINT_REQ due to error
> org.apache.cassandra.net.AsyncChannelOutputPlus$FlushException: The
> channel this output stream was writing to has been closed
>  at
> org.apache.cassandra.net
> .AsyncChannelOutputPlus.propagateFailedFlush(AsyncChannelOutputPlus.java:200)
>  at
> org.apache.cassandra.net
> .AsyncChannelOutputPlus.waitUntilFlushed(AsyncChannelOutputPlus.java:158)
>  at
> org.apache.cassandra.net
> .AsyncChannelOutputPlus.waitForSpace(AsyncChannelOutputPlus.java:140)
>  at
> org.apache.cassandra.net
> .AsyncChannelOutputPlus.beginFlush(AsyncChannelOutputPlus.java:97)
>  at
> org.apache.cassandra.net
> .AsyncMessageOutputPlus.doFlush(AsyncMessageOutputPlus.java:100)
>  at
>
> org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:122)
>  at
>
> org.apache.cassandra.hints.HintMessage$Serializer.serialize(HintMessage.java:139)
>  at
>
> 

Upgrade from 3.11.5 to 4.1.x

[Surbhi Gupta]

Hi,

We have to upgrade from 3.11.5 to 4.1.x .
Can we do it in one go ?
Or do we have to go to an intermediate version first?

Thanks
Surbhi

Is there a way to find out if a server is part of application connection string?

[Surbhi Gupta]

Hi,

We have a cluster with many applications connecting to it.
We need to decommission few of the servers from the cluster .
Without asking the application team, is there any way to know the ips
of the application connection string?
Does cassandra logs (system or debug) this information somewhere?

Application team might have different ips than seed nodes.

Thanks in advance.
Surbhi

Re: Bootstrapping new node throwing error - Mutation too large

[Surbhi Gupta]

Thanks Scott

On Wed, Mar 1, 2023 at 4:00 PM C. Scott Andreas 
wrote:

> The performance implications would primarily be due to the challenge of
> handling mutations this large themselves rather than the commitlog segment
> size. These would occupy large, contiguous areas of heap and increase
> memory pressure in the process.
>
> Increasing commit_log_segment_size_in_mb is likely the best / only
> approach, along with addressing mutation size for future writes from the
> application.
>
> I'd also strongly recommend upgrading from 3.11.5 to 3.11.15. 3.11.5 was
> released about 3.5 years ago, with a large number of bugfixes available in
> 3.11.15. That release is also drop-in, so you can upgrade simply by rev'ing
> the version and performing a rolling restart of the instances.
>
>
> – Scott
>
> On Mar 1, 2023, at 2:43 PM, Surbhi Gupta  wrote:
>
>
> Hi Cassandra Community,
>
> We have to expand our cluster and I tried to add the first node to the
> cluster and when the new node was bootstrapping , I noticed the error like
> below in the system.log, but the bootstrap process was successful .
>
> We are on 3.11.5 .
>
> ERROR [MutationStage-7] 2023-03-01 07:01:40,026
> AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread
> Thread[MutationStage-7,5,main]
>
> java.lang.IllegalArgumentException: Mutation of 24.510MiB is too large for
> the maximum size of 16.000MiB
>
> I tried to add another node after 5 days of adding 1st node and the 2nd
> node also gave an error but this time the mutation size was bigger .
>
> ERROR [BatchlogTasks:1] 2023-03-01 01:39:59,357 CassandraDaemon.java:228
> - Exception in thread Thread[BatchlogTasks:1,5,main]
>
> java.lang.IllegalArgumentException: Mutation of 40.717MiB is too large for
> the maximum size of 16.000MiB
>
> And on the first node( which was added 5 days ago), we saw messages like
> below till the 2nd node bootstrap was completed.
>  "Mutation of 24.510MiB is too large for the maximum size of 16.000MiB"
> but after the 2nd node bootstarp process was completed, we started seeing
> the message about hint byte is too large but there are no hint files in any
> of the nodes on hints directory . And the message about mutation too large
> has stopped popping in system.log on the first added node. Now we are
> seeing hints error as below on the first newly added node.
>
> ERROR [BatchlogTasks:1] 2023-03-01 07:48:32,091 CassandraDaemon.java:228
> - Exception in thread Thread[BatchlogTasks:1,5,main]
>
> java.lang.IllegalArgumentException: Hint of 25700336 bytes is too large -
> the maximum size is 16777216
>
> As per the application team , nothing changed .
>
> As per the workaround commitlog_segment_size_in_mb can be increased to
> accommodate the increased size but that doesnt seem to be a concrete
> solution and it can have performance impact ,  because by design intent
> the maximum allowed segment size is 50% of the configured
> commit_log_segment_size_in_mb. This is so Cassandra avoids writing segments
> with large amounts of empty space.
>
> Looks like i am hitting
> https://issues.apache.org/jira/browse/CASSANDRA-15152
> <https://issues.apache.org/jira/browse/CASSANDRA-15152>
>
> Anyone have any suggestions?
>
> Thanks
> Surbhi
>
>
>
>
>
>
>
>
>

Bootstrapping new node throwing error - Mutation too large

[Surbhi Gupta]

Hi Cassandra Community,

We have to expand our cluster and I tried to add the first node to the
cluster and when the new node was bootstrapping , I noticed the error like
below in the system.log, but the bootstrap process was successful .

We are on 3.11.5 .

ERROR [MutationStage-7] 2023-03-01 07:01:40,026
AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread
Thread[MutationStage-7,5,main]

java.lang.IllegalArgumentException: Mutation of 24.510MiB is too large for
the maximum size of 16.000MiB

I tried to add another node after 5 days of adding 1st node and the 2nd
node also gave an error but this time the mutation size was bigger .

ERROR [BatchlogTasks:1] 2023-03-01 01:39:59,357 CassandraDaemon.java:228 -
Exception in thread Thread[BatchlogTasks:1,5,main]

java.lang.IllegalArgumentException: Mutation of 40.717MiB is too large for
the maximum size of 16.000MiB

And on the first node( which was added 5 days ago), we saw messages like
below till the 2nd node bootstrap was completed.
 "Mutation of 24.510MiB is too large for the maximum size of 16.000MiB"
but after the 2nd node bootstarp process was completed, we started seeing
the message about hint byte is too large but there are no hint files in any
of the nodes on hints directory . And the message about mutation too large
has stopped popping in system.log on the first added node. Now we are
seeing hints error as below on the first newly added node.

ERROR [BatchlogTasks:1] 2023-03-01 07:48:32,091 CassandraDaemon.java:228 -
Exception in thread Thread[BatchlogTasks:1,5,main]

java.lang.IllegalArgumentException: Hint of 25700336 bytes is too large -
the maximum size is 16777216

As per the application team , nothing changed .

As per the workaround commitlog_segment_size_in_mb can be increased to
accommodate the increased size but that doesnt seem to be a concrete
solution and it can have performance impact ,  because by design intent the
maximum allowed segment size is 50% of the configured
commit_log_segment_size_in_mb. This is so Cassandra avoids writing segments
with large amounts of empty space.

Looks like i am hitting
https://issues.apache.org/jira/browse/CASSANDRA-15152


Anyone have any suggestions?

Thanks
Surbhi

Re: Anyone connecting the Cassandra on a server

[Surbhi Gupta]

You can use tcpdump

On Fri, 19 Nov 2021 at 10:34, Soumya Jena 
wrote:

> You can just do a netstat on port 9042 to see if anything connected .
>
> Something like
> netstat -anp | grep 9042 .
>
> Or you can also check for read/write client requests metrics . You can
> check if specific tables are taking read or writes .
> There is also a metrics to see number of connected clients .
> org.apache.cassandra.metrics.Client.connectedNativeClients.
>
> Hope this helps . Thanks !!
>
> On Fri, Nov 19, 2021 at 11:43 PM Saha, Sushanta K <
> sushanta.s...@verizonwireless.com> wrote:
>
>> I need to shutdown an old Apache Cassandra server for good. Running
>> 3.0.x. Any way I can determine if anyone is still connecting to the
>> Cassandra instance running on this server?
>>
>> Thanks
>>  Sushanta
>>
>>

Re: Streaming failure Issue

[Surbhi Gupta]

Hi ,

Try to adjust phi_convict_threshold and see if that helps.
When we did migration from on prim to AWS, this was one of the factor to
consider.

Thanks


On Tue, Oct 5, 2021 at 4:00 AM MyWorld  wrote:

> Hi all,
>
> Need urgent help.
> We have one Physical Data Center of 5 nodes with 1 TB data on each
> (Location: Dallas). Currently we are using Cassandra ver 3.0.9. Now we are
> Adding one more Data Center of 5 nodes(Location GCP-US) and have joined it
> to the existing one.
>
> While running nodetool rebuild command, we are getting following error :
> On GCP node (where we ran rebuild command) :
>
>> ERROR [STREAM-IN-/192.x.x.x] 2021-10-05 15:56:52,246
>> StreamSession.java:639 - [Stream #66646d30-25a2-11ec-903b-774f88efe725]
>> Remote peer 192.x.x.x failed stream session.
>> INFO  [STREAM-IN-/192.x.x.x] 2021-10-05 15:56:52,266
>> StreamResultFuture.java:183 - [Stream
>> #66646d30-25a2-11ec-903b-774f88efe725] Session with /192.x.x.x is complete
>
>
> On DL source node :
>
>> INFO  [STREAM-IN-/34.x.x.x] 2021-10-05 15:55:53,785
>> StreamResultFuture.java:183 - [Stream
>> #66646d30-25a2-11ec-903b-774f88efe725] Session with /34.x.x.x is complete
>> ERROR [STREAM-OUT-/34.x.x.x] 2021-10-05 15:55:53,785
>> StreamSession.java:534 - [Stream #66646d30-25a2-11ec-903b-774f88efe725]
>> Streaming error occurred
>> java.lang.RuntimeException: Transfer of file
>> /var/lib/cassandra/data/clickstream/glusr_usr_paid_url_mv-3c49c392b35511e9bd0a8f42dfb09617/mc-45676-big-Data.db
>> already completed or aborted (perhaps session failed?).
>> at
>> org.apache.cassandra.streaming.messages.OutgoingFileMessage.startTransfer(OutgoingFileMessage.java:120)
>> ~[apache-cassandra-3.0.9.jar:3.0.9]
>> at
>> org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:50)
>> ~[apache-cassandra-3.0.9.jar:3.0.9]
>> at
>> org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:42)
>> ~[apache-cassandra-3.0.9.jar:3.0.9]
>> at
>> org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:48)
>> ~[apache-cassandra-3.0.9.jar:3.0.9]
>> at
>> org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:387)
>> ~[apache-cassandra-3.0.9.jar:3.0.9]
>> at
>> org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:367)
>> ~[apache-cassandra-3.0.9.jar:3.0.9]
>> at java.lang.Thread.run(Thread.java:748) [na:1.8.0_192]
>> WARN  [STREAM-IN-/34.x.x.x] 2021-10-05 15:55:53,786
>> StreamResultFuture.java:210 - [Stream
>> #66646d30-25a2-11ec-903b-774f88efe725] Stream failed
>
>
> Before starting this rebuild, we have made the following changes:
> 1. Set setstreamthroughput to 600 Mb/sec
> 2. Set setinterdcstreamthroughput to 600 Mb/sec
> 3. streaming_socket_timeout_in_ms is 24 hrs
> 4. Disabled autocompaction on GCP node as this was heavily utilising CPU
> resource
>
> FYI, GCP rebuild process starts with data streaming from 3 nodes, and all
> fails one by one after streaming for a few hours.
> Please help out how to correct this issue.
> Is there any other way to rebuild such big data.
> We have a few tables with 200 - 400GB of data and some smaller tables.
> Also, we have Mviews in our environment
>
> Regards,
> Ashish Gupta
>
>

Re: Hints are not getting partially processed

[Surbhi Gupta]

I tried

nodetool sethintedhandoffthrottlekb 0;nodetool pausehandoff;sleep
15;nodetool resumehandoff;

But still the same

On Sun, 26 Sept 2021 at 08:25, Surbhi Gupta 
wrote:

> All nodes are up and running .
> Checked system.log and debug.log but nothing useful i got it.
>
>
> On Sun, 26 Sept 2021 at 08:03, Surbhi Gupta 
> wrote:
>
>> I tried truncatehints and some of the hints file did not delete but some
>> hints file got deleted.
>>
>>
>> On Sun, 26 Sept 2021 at 04:00, Surbhi Gupta 
>> wrote:
>>
>>> Hi,
>>>
>>> We are on 3.11.5 and we have 2 DC cassandra cluster.
>>> Suddenly we started seeing hints issue .
>>> system.logs says that hints are getting partially replayed  and we are
>>> seeing hints dropped in target nodes  after hinted_handoff_period is over.
>>> We checked debug.log as well but nothing we found there.
>>>
>>> INFO  [HintsDispatcher:680] 2021-09-26 03:54:26,399
>>> HintsDispatchExecutor.java:289 - Finished hinted handoff of file
>>> aa-1632638838462-1.hints to endpoint /10.xx.yyy.zz:
>>> aa, partially
>>>
>>> This is happening only on one node. I stopped the cassandra and deleted
>>> the hints files and started cassandra but it started happening again after
>>> few hours.
>>>
>>> Any idea ?
>>>
>>> Thanks
>>> Surbhi
>>>
>>>
>>>

Re: Hints are not getting partially processed

[Surbhi Gupta]

All nodes are up and running .
Checked system.log and debug.log but nothing useful i got it.


On Sun, 26 Sept 2021 at 08:03, Surbhi Gupta 
wrote:

> I tried truncatehints and some of the hints file did not delete but some
> hints file got deleted.
>
>
> On Sun, 26 Sept 2021 at 04:00, Surbhi Gupta 
> wrote:
>
>> Hi,
>>
>> We are on 3.11.5 and we have 2 DC cassandra cluster.
>> Suddenly we started seeing hints issue .
>> system.logs says that hints are getting partially replayed  and we are
>> seeing hints dropped in target nodes  after hinted_handoff_period is over.
>> We checked debug.log as well but nothing we found there.
>>
>> INFO  [HintsDispatcher:680] 2021-09-26 03:54:26,399
>> HintsDispatchExecutor.java:289 - Finished hinted handoff of file
>> aa-1632638838462-1.hints to endpoint /10.xx.yyy.zz:
>> aa, partially
>>
>> This is happening only on one node. I stopped the cassandra and deleted
>> the hints files and started cassandra but it started happening again after
>> few hours.
>>
>> Any idea ?
>>
>> Thanks
>> Surbhi
>>
>>
>>

Re: Hints are not getting partially processed

[Surbhi Gupta]

I tried truncatehints and some of the hints file did not delete but some
hints file got deleted.


On Sun, 26 Sept 2021 at 04:00, Surbhi Gupta 
wrote:

> Hi,
>
> We are on 3.11.5 and we have 2 DC cassandra cluster.
> Suddenly we started seeing hints issue .
> system.logs says that hints are getting partially replayed  and we are
> seeing hints dropped in target nodes  after hinted_handoff_period is over.
> We checked debug.log as well but nothing we found there.
>
> INFO  [HintsDispatcher:680] 2021-09-26 03:54:26,399
> HintsDispatchExecutor.java:289 - Finished hinted handoff of file
> aa-1632638838462-1.hints to endpoint /10.xx.yyy.zz:
> aa, partially
>
> This is happening only on one node. I stopped the cassandra and deleted
> the hints files and started cassandra but it started happening again after
> few hours.
>
> Any idea ?
>
> Thanks
> Surbhi
>
>
>

Hints are not getting partially processed

[Surbhi Gupta]

Hi,

We are on 3.11.5 and we have 2 DC cassandra cluster.
Suddenly we started seeing hints issue .
system.logs says that hints are getting partially replayed  and we are
seeing hints dropped in target nodes  after hinted_handoff_period is over.
We checked debug.log as well but nothing we found there.

INFO  [HintsDispatcher:680] 2021-09-26 03:54:26,399
HintsDispatchExecutor.java:289 - Finished hinted handoff of file
aa-1632638838462-1.hints to endpoint /10.xx.yyy.zz:
aa, partially

This is happening only on one node. I stopped the cassandra and deleted the
hints files and started cassandra but it started happening again after few
hours.

Any idea ?

Thanks
Surbhi

Which open source or free tool do you use to monitor cassandra clusters?

[Surbhi Gupta]

Hi,

Which open source or free tool do you use to monitor cassandra clusters
which have similar features like Opscenter?

Thanks
Surbhi

Re: Memory requirements for Cassandra reaper

[Surbhi Gupta]

Thanks a lot.

On Tue, 4 May 2021 at 19:51, Erick Ramirez 
wrote:

> 2GB is allocated to the Reaper JVM on startup (see
> https://github.com/thelastpickle/cassandra-reaper/blob/2.2.4/src/packaging/bin/cassandra-reaper#L90-L91
> ).
>
> If you just want to test it out on a machine with only 8GB, you can update
> the cassandra-reaper script to only use 1GB by setting -Xms1G and -Xmx1G
> but you won't be able to do much with it. It might also be necessary to
> reduce the heap allocated to Cassandra down to 2GB so there's enough RAM
> left for the operating system.
>
> For test and production environments, I recommend deploying Reaper on a
> dedicated machine so it doesn't affect the performance of whatever cluster
> it is connecting to. Reaper needs a minimum of 2 vCPUs + 2GB of RAM and
> this works in most cases.
>
> As a side note, if you just want to play around with the likes of Reaper
> and Medusa (backups) then I'd recommend having a look at deploying
> https://k8ssandra.io/ -- it's a production-ready platform for running
> Apache Cassandra on Kubernetes with all the tools bundled in:
>
>- Reaper  for automated repairs
>- Medusa  for
>backups and restores
>- Metrics Collector
> for
>monitoring with Prometheus + Grafana
>- *Stargate.io * for accessing your data using
>REST, GraphQL and JSON Doc APIs
>- Traefik templates for k8s cluster ingress
>
> Cheers!
>
>>

Memory requirements for Cassandra reaper

[Surbhi Gupta]

Hi,

What are the memory requirements for Cassandra reaper?
I was trying to setup cassandra reaper on a 8GB box where cassandra is
taking 3GB heap size , but i got error "Cannot allocate memory"

Hence wanted to understand the memory requirements for cassandra reaper .
What should be the size of the memory on the box where we need to run
reaper ?

Thanks
Surbhi

Dont want to split sstables for repaired and non repaired while repairing with -pr option

[Surbhi Gupta]

Hi,

I dont want to split sstables ,repaired and non repaired , while repairing
with -pr option.

nodetool repair -pr splits the sstable into repaired and non repaired and
disk size increases.

I dont want to increase the disk size.
What are my options ?

Thanks
Surbhi

Re: Best strategy to run repair

[Surbhi Gupta]

Does describering not give the correct sub ranges for each node ?

On Mon, 22 Mar 2021 at 20:28, manish khandelwal <
manishkhandelwa...@gmail.com> wrote:

> Also try to use Cassandra reaper (as Kane also mentioned) for subrange
> repair. Doing subrange repair yourself may lead to a lot of trouble as
> calculating correct subranges is not an easy task.
>
> On Tue, Mar 23, 2021 at 3:38 AM Kane Wilson  wrote:
>
>> -pr on all nodes takes much longer as you'll do at least triple the
>> amount of merkle calculations I believe (with RF 3) and tends to be quite
>> problematic.
>>
>> Subrange is the way to go, which is what cassandra-reaper will do for you
>> if you have it set up.
>>
>> raft.so - Cassandra consulting, support, and managed services
>>
>>
>> On Tue, Mar 23, 2021 at 7:33 AM Surbhi Gupta 
>> wrote:
>>
>>> Hi,
>>>
>>> We are on open source 3.11.5 .
>>> We need to repair a production cluster .
>>> We are using num_token as 256 .
>>> What will be a better option to run repair ?
>>> 1. nodetool -pr  (Primary range repair on all nodes, one node at a time)
>>> OR
>>> 2. nodetool -st -et (Subrange repair , taking the ranges for each node
>>> from nodetool describering) and run 256 repairs on each node .
>>>
>>> Thanks
>>> Surbhi
>>>
>>>

Best strategy to run repair

[Surbhi Gupta]

Hi,

We are on open source 3.11.5 .
We need to repair a production cluster .
We are using num_token as 256 .
What will be a better option to run repair ?
1. nodetool -pr  (Primary range repair on all nodes, one node at a time)
OR
2. nodetool -st -et (Subrange repair , taking the ranges for each node from
nodetool describering) and run 256 repairs on each node .

Thanks
Surbhi

Re: Rollback Cassandra after 1 node upgrade

[Surbhi Gupta]

Hi Manish,

Please provide both versions.

Thanks
Surbhi

On Fri, Sep 4, 2020 at 8:55 PM manish khandelwal <
manishkhandelwa...@gmail.com> wrote:

> Hi
>
> We have been forced into rolling back our Cassandra after 1 node upgrade.
> The node was upgraded 10 days ago. We have the backup of the old data.
>
> Strategy one which we are thinking :
> 1. Rollback to old binaries and configuration.
> 2. Restore the old data from backup.
> 3. Run Repair.
>
> Another Strategy is to bootstrap the node as new after changing the
> binaries.
>
> Which of the strategies is best?
>
> Regards
> Manish
>
>
>

Re: Cassandra upgrade from 3.11.3 -> 3.11.6

[Surbhi Gupta]

In case of any issue, it gets very difficult to debug when we have multiple
versions.

On Tue, 23 Jun 2020 at 22:23, Jürgen Albersdorfer 
wrote:

> Hi, I would say „It depends“ - as it always does. I have had a 21 Node
> Cluster running in Production in one DC with versions ranging from 3.11.1
> to 3.11.6 without having had any single issue for over a year. I just
> upgraded all nodes to 3.11.6 for the sake of consistency.
>
> Von meinem iPhone gesendet
>
> Am 24.06.2020 um 02:56 schrieb Surbhi Gupta :
>
> 
>
> Hi ,
>
> We have recently upgraded from 3.11.0 to 3.11.5 . There is a sstable
> format change from 3.11.4 .
> We also had to expand the cluster and we also discussed about expansion
> first and than upgrade. But finally we upgraded and than expanded.
> As per our experience what I could tell you is, it is not advisable to add
> new nodes on higher version.
> There are many bugs which got fixed from 3.11.3 to 3.11.6.
>
> Thanks
> Surbhi
>
> On Tue, Jun 23, 2020 at 5:04 PM Jai Bheemsen Rao Dhanwada <
> jaibheem...@gmail.com> wrote:
>
>> Hello,
>>
>> I am trying to upgrade from 3.11.3 to 3.11.6.
>> Can I add new nodes with the 3.11.6  version to the cluster running with
>> 3.11.3?
>> Also, I see the SSTable format changed from mc-* to md-*, does this cause
>> any issues?
>>
>>

Re: Cassandra upgrade from 3.11.3 -> 3.11.6

[Surbhi Gupta]

Hi ,

We have recently upgraded from 3.11.0 to 3.11.5 . There is a sstable format
change from 3.11.4 .
We also had to expand the cluster and we also discussed about expansion
first and than upgrade. But finally we upgraded and than expanded.
As per our experience what I could tell you is, it is not advisable to add
new nodes on higher version.
There are many bugs which got fixed from 3.11.3 to 3.11.6.

Thanks
Surbhi

On Tue, Jun 23, 2020 at 5:04 PM Jai Bheemsen Rao Dhanwada <
jaibheem...@gmail.com> wrote:

> Hello,
>
> I am trying to upgrade from 3.11.3 to 3.11.6.
> Can I add new nodes with the 3.11.6  version to the cluster running with
> 3.11.3?
> Also, I see the SSTable format changed from mc-* to md-*, does this cause
> any issues?
>
>

Bootstrap resume , streamed all data again and 2nd bootstrap id in netstats

[Surbhi Gupta]

Hi,

We are on 3.11.5 .
We are trying to add a node in a DC and after all the streaming is done, no
streaming is active in nodetool netstats output , the node was just waiting
for 1 hour doing nothing.
So we thought it might be hung, so we tried
nodetool bootstrap resume

But bootstrap resume , started streaming all the data again and after all
streaming is done it again showed same behavior as the normal bootstrap
like doing nothing and just stuck  , now disk is used twice as it should be
and it has 500s of pending compaction.

When bootstrap resume created new session id and now in nodetool netstats
the status is showing  Normal (Which came from first bootstrap , which
eventually finished after waiting for a long time when bootstrap resume was
going on ) .

Now the condition is, node is in UN state as seen from all the nodes and
started accepting the traffic .

However bootstrap resume is still going on . What happens in this scenario ?

[root@abcdef ~]# nta netstats |grep -v "100%"

Mode: NORMAL

Bootstrap b940a710-a6a8-11ea-b467-3d5e11ea164a

/10.abc

Receiving 1414 files, 74428129981 bytes total. Already
received 46 files, 3425181474 bytes total

/10.def

Receiving 1392 files, 61042286685 bytes total. Already
received 44 files, 4198620698 bytes total

/10.ijk

Receiving 1449 files, 70624858458 bytes total. Already
received 45 files, 6266730847 bytes total

/10.lmn

Receiving 1399 files, 59352202847 bytes total. Already
received 45 files, 4518550733 bytes total

/10.xyz

Receiving 1463 files, 74140648517 bytes total. Already
received 45 files, 3231112921 bytes total

Read Repair Statistics:

Attempted: 31108

Mismatch (Blocking): 0

Mismatch (Background): 67

Pool NameActive   Pending  Completed   Dropped

Large messages  n/a 02068193 0

Small messages  n/a30  501343037 0

Gossip messages n/a 0 101098 0

Thanks

Surbhi

Re: Truncate Materialized View

[Surbhi Gupta]

Thanks James, there were no hints pending at the time of dropping the
materialized view. But the when we dropped the materialized view it somehow
corrupted the hints.

On Fri, May 15, 2020 at 4:15 PM James Shaw  wrote:

> Surbhi:
>   I don't think you may truncate the materialized view.
> What exact error got ?  If you think it is same as the bug, then you may
> try to avoid the bug triggered condition. It says pending hints. So you may
> let all hints applied, then try drop the view.
>
> Thanks,
>
> James
>
> On Fri, May 15, 2020 at 1:35 PM Surbhi Gupta 
> wrote:
>
>> Anyone has truncated materialized views ?
>>
>> On Thu, 14 May 2020 at 11:59, Surbhi Gupta 
>> wrote:
>>
>>> Hi,
>>>
>>> We are on 3.11.0 .
>>> We have 11 Materialized view on a table.
>>> After discussion with application team , we found out that they are
>>> using only 4 out of 11 .
>>> We tried to drop the materialized view and got hit by the bug
>>> https://issues.apache.org/jira/browse/CASSANDRA-13696 which made our
>>> whole cluster unstable and multiple nodes were down at the same time .
>>>
>>> We will upgrade this cluster but my question is , can we truncate the
>>> Materialized view rather than dropping?
>>>
>>> How will it impact ?
>>> Does update into base table , updates all materialized views (This is my
>>> understanding) ?
>>> If yes, then truncate the data from the materialized view will create
>>> too many read repairs later, correct?
>>> Please correct me if I am wrong .
>>>
>>> Thanks
>>> Surbhi
>>>
>>>

Re: Truncate Materialized View

[Surbhi Gupta]

Anyone has truncated materialized views ?

On Thu, 14 May 2020 at 11:59, Surbhi Gupta  wrote:

> Hi,
>
> We are on 3.11.0 .
> We have 11 Materialized view on a table.
> After discussion with application team , we found out that they are using
> only 4 out of 11 .
> We tried to drop the materialized view and got hit by the bug
> https://issues.apache.org/jira/browse/CASSANDRA-13696 which made our
> whole cluster unstable and multiple nodes were down at the same time .
>
> We will upgrade this cluster but my question is , can we truncate the
> Materialized view rather than dropping?
>
> How will it impact ?
> Does update into base table , updates all materialized views (This is my
> understanding) ?
> If yes, then truncate the data from the materialized view will create too
> many read repairs later, correct?
> Please correct me if I am wrong .
>
> Thanks
> Surbhi
>
>

Truncate Materialized View

[Surbhi Gupta]

Hi,

We are on 3.11.0 .
We have 11 Materialized view on a table.
After discussion with application team , we found out that they are using
only 4 out of 11 .
We tried to drop the materialized view and got hit by the bug
https://issues.apache.org/jira/browse/CASSANDRA-13696 which made our whole
cluster unstable and multiple nodes were down at the same time .

We will upgrade this cluster but my question is , can we truncate the
Materialized view rather than dropping?

How will it impact ?
Does update into base table , updates all materialized views (This is my
understanding) ?
If yes, then truncate the data from the materialized view will create too
many read repairs later, correct?
Please correct me if I am wrong .

Thanks
Surbhi

Add a new node of 3.11.5 in a 3.11.0 Cassandra Cluster

[Surbhi Gupta]

Hi,

We are facing some issue in bootstrapping new node in 3.11.0 and
bootstrapping is failing.
We have two tasks here :
1. Expand the cluster (Due to disk concern and dropped mutation)
2. Upgrade the cluster from 3.11.0 to 3.11.5 because of various bugs we are
hitting in 3.11.0 .

So my question here is :
1. Can we add new node with 3.11.5 rpm on a 3.11.0 cluster ?
2. Because there is a sstable format change from mc to md , is there any
issue in bootstrapping a node with 3.11.5 version ?

We are not able to upgrade the cluster first and add nodes because of the
disk concerns . upgradesstable will need space because of sstable format
change.

Please advise.

Thanks
Surbhi

Re: Bootstraping is failing

[Surbhi Gupta]

I tried to change the heap size from 31GB to 62GB on the bootstrapping node
because , I noticed that , when it reached the mid way of bootstrapping ,
heap reached to around 90% or more and node just freeze .
But still it is the same behavior , it again reached midway and heap again
reached 90% or more and node just freeze and none of the node tool command
returns the output, other node also removed this node from the joining as
they were not able to gossip.
We are on 3.11.0 .

I tried to take heap dump when the node had 90% + heap utilization of 62GB
heap size and opened the leak report and found 3 leak suspect and out of
three 2 were as below:

1. The thread *io.netty.util.concurrent.FastThreadLocalThread @
0x7fbe9533bf98 StreamReceiveTask:26* keeps local variables with total
size *16,898,023,552
(31.10%)*bytes.
The memory is accumulated in one instance of
*"io.netty.util.Recycler$DefaultHandle[]"* loaded by
*"sun.misc.Launcher$AppClassLoader
@ 0x7fb917c76dc8"*.

2. The thread *io.netty.util.concurrent.FastThreadLocalThread @
0x7fbb846fb800 StreamReceiveTask:29* keeps local variables with total
size *11,696,214,424
(21.53%)*bytes.
The memory is accumulated in one instance of
*"io.netty.util.Recycler$DefaultHandle[]"* loaded by
*"sun.misc.Launcher$AppClassLoader
@ 0x7fb917c76dc8"*.

Am I getting hit by https://issues.apache.org/jira/browse/CASSANDRA-13929

I haven't changed the tcp settings . My tcp settings are more than
recommended, what I wanted to understand , how tcp settings can effect the
bootstrapping process ?

Thanks
Surbhi

On Thu, 7 May 2020 at 17:01, Surbhi Gupta  wrote:

> When we are starting the node, it is starting bootstrap automatically and
> restreaming the whole data again.  It is not resuming .
>
> On Thu, May 7, 2020 at 4:47 PM Adam Scott  wrote:
>
>> I think you want to run `nodetool bootstrap resume` (
>> https://cassandra.apache.org/doc/latest/tools/nodetool/bootstrap.html)
>> to pick up where it last left off. Sorry for the late reply.
>>
>>
>> On Thu, May 7, 2020 at 2:22 PM Surbhi Gupta 
>> wrote:
>>
>>> So after failed bootstrapped , if we start cassandra again on the new
>>> node , will it resume bootstrap or will it start over?
>>>
>>> On Thu, 7 May 2020 at 13:32, Adam Scott  wrote:
>>>
>>>> I recommend it on all nodes.  This will eliminate that as a source of
>>>> trouble further on down the road.
>>>>
>>>>
>>>> On Thu, May 7, 2020 at 1:30 PM Surbhi Gupta 
>>>> wrote:
>>>>
>>>>> streaming_socket_timeout_in_ms is 24 hour.
>>>>>   So tcp settings should be changed on the new bootstrap node or on
>>>>> all nodes ?
>>>>>
>>>>>
>>>>> On Thu, 7 May 2020 at 13:23, Adam Scott 
>>>>> wrote:
>>>>>
>>>>>>
>>>>>> *edit
>>>>>> /etc/sysctl.confnet.ipv4.tcp_keepalive_time=60 
>>>>>> net.ipv4.tcp_keepalive_probes=3net.ipv4.tcp_keepalive_intvl=10*
>>>>>> then run sysctl -p to cause the kernel to reload the settings
>>>>>>
>>>>>> 5 minutes (300) seconds is probably too long.
>>>>>>
>>>>>> On Thu, May 7, 2020 at 1:09 PM Surbhi Gupta 
>>>>>> wrote:
>>>>>>
>>>>>>> [root@abc cassandra]# cat /proc/sys/net/ipv4/tcp_keepalive_time
>>>>>>>
>>>>>>> 300
>>>>>>>
>>>>>>> [root@abc cassandra]# cat /proc/sys/net/ipv4/tcp_keepalive_intvl
>>>>>>>
>>>>>>> 30
>>>>>>>
>>>>>>> [root@abc cassandra]# cat /proc/sys/net/ipv4/tcp_keepalive_probes
>>>>>>>
>>>>>>> 9
>>>>>>>
>>>>>>> On Thu, 7 May 2020 at 12:32, Adam Scott 
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Maybe a firewall killing a connection?
>>>>>>>>
>>>>>>>> What does the following show?
>>>>>>>> cat /proc/sys/net/ipv4/tcp_keepalive_time
>>>>>>>> cat /proc/sys/net/ipv4/tcp_keepalive_intvl
>>>>>>>> cat /proc/sys/net/ipv4/tcp_keepalive_probes
>>>>>>>>
>>>>>>>> On Thu, May 7, 2020 at 10:31 AM Surbhi Gupta <
>>>>>>>> surbhi.gupt...@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> We 

Re: Bootstraping is failing

[Surbhi Gupta]

When we are starting the node, it is starting bootstrap automatically and
restreaming the whole data again.  It is not resuming .

On Thu, May 7, 2020 at 4:47 PM Adam Scott  wrote:

> I think you want to run `nodetool bootstrap resume` (
> https://cassandra.apache.org/doc/latest/tools/nodetool/bootstrap.html)
> to pick up where it last left off. Sorry for the late reply.
>
>
> On Thu, May 7, 2020 at 2:22 PM Surbhi Gupta 
> wrote:
>
>> So after failed bootstrapped , if we start cassandra again on the new
>> node , will it resume bootstrap or will it start over?
>>
>> On Thu, 7 May 2020 at 13:32, Adam Scott  wrote:
>>
>>> I recommend it on all nodes.  This will eliminate that as a source of
>>> trouble further on down the road.
>>>
>>>
>>> On Thu, May 7, 2020 at 1:30 PM Surbhi Gupta 
>>> wrote:
>>>
>>>> streaming_socket_timeout_in_ms is 24 hour.
>>>>   So tcp settings should be changed on the new bootstrap node or on all
>>>> nodes ?
>>>>
>>>>
>>>> On Thu, 7 May 2020 at 13:23, Adam Scott  wrote:
>>>>
>>>>>
>>>>> *edit
>>>>> /etc/sysctl.confnet.ipv4.tcp_keepalive_time=60 
>>>>> net.ipv4.tcp_keepalive_probes=3net.ipv4.tcp_keepalive_intvl=10*
>>>>> then run sysctl -p to cause the kernel to reload the settings
>>>>>
>>>>> 5 minutes (300) seconds is probably too long.
>>>>>
>>>>> On Thu, May 7, 2020 at 1:09 PM Surbhi Gupta 
>>>>> wrote:
>>>>>
>>>>>> [root@abc cassandra]# cat /proc/sys/net/ipv4/tcp_keepalive_time
>>>>>>
>>>>>> 300
>>>>>>
>>>>>> [root@abc cassandra]# cat /proc/sys/net/ipv4/tcp_keepalive_intvl
>>>>>>
>>>>>> 30
>>>>>>
>>>>>> [root@abc cassandra]# cat /proc/sys/net/ipv4/tcp_keepalive_probes
>>>>>>
>>>>>> 9
>>>>>>
>>>>>> On Thu, 7 May 2020 at 12:32, Adam Scott 
>>>>>> wrote:
>>>>>>
>>>>>>> Maybe a firewall killing a connection?
>>>>>>>
>>>>>>> What does the following show?
>>>>>>> cat /proc/sys/net/ipv4/tcp_keepalive_time
>>>>>>> cat /proc/sys/net/ipv4/tcp_keepalive_intvl
>>>>>>> cat /proc/sys/net/ipv4/tcp_keepalive_probes
>>>>>>>
>>>>>>> On Thu, May 7, 2020 at 10:31 AM Surbhi Gupta <
>>>>>>> surbhi.gupt...@gmail.com> wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> We are trying to expand a datacenter and trying to add nodes but
>>>>>>>> when node is bootstrapping , it goes half way through and then fail 
>>>>>>>> with
>>>>>>>> below error, We have increased stremthroughput from 200 to 400 when we 
>>>>>>>> were
>>>>>>>> trying for the 2nd time but still it failed. We are on 3.11.0 , using 
>>>>>>>> G1GC
>>>>>>>> with 31GB heap.
>>>>>>>>
>>>>>>>> ERROR [MessagingService-Incoming-/10.X.X.X] 2020-05-07 09:42:38,933
>>>>>>>> CassandraDaemon.java:228 - Exception in thread
>>>>>>>> Thread[MessagingService-Incoming-/10.X.X.X,main]
>>>>>>>>
>>>>>>>> java.io.IOError: java.io.EOFException: Stream ended prematurely
>>>>>>>>
>>>>>>>> at
>>>>>>>> org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer$1.computeNext(UnfilteredRowIteratorSerializer.java:227)
>>>>>>>> ~[apache-cassandra-3.11.0.jar:3.11.0]
>>>>>>>>
>>>>>>>> at
>>>>>>>> org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer$1.computeNext(UnfilteredRowIteratorSerializer.java:215)
>>>>>>>> ~[apache-cassandra-3.11.0.jar:3.11.0]
>>>>>>>>
>>>>>>>> at
>>>>>>>> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
>>>>>>>> ~[apache-cassandra-3.11.0.jar:3.11.0]
>>>>>>>>
>>>>>>>> at
>>>>>>>> org.apache.cassand

Re: Bootstraping is failing

[Surbhi Gupta]

So after failed bootstrapped , if we start cassandra again on the new node
, will it resume bootstrap or will it start over?

On Thu, 7 May 2020 at 13:32, Adam Scott  wrote:

> I recommend it on all nodes.  This will eliminate that as a source of
> trouble further on down the road.
>
>
> On Thu, May 7, 2020 at 1:30 PM Surbhi Gupta 
> wrote:
>
>> streaming_socket_timeout_in_ms is 24 hour.
>>   So tcp settings should be changed on the new bootstrap node or on all
>> nodes ?
>>
>>
>> On Thu, 7 May 2020 at 13:23, Adam Scott  wrote:
>>
>>>
>>> *edit
>>> /etc/sysctl.confnet.ipv4.tcp_keepalive_time=60 
>>> net.ipv4.tcp_keepalive_probes=3net.ipv4.tcp_keepalive_intvl=10*
>>> then run sysctl -p to cause the kernel to reload the settings
>>>
>>> 5 minutes (300) seconds is probably too long.
>>>
>>> On Thu, May 7, 2020 at 1:09 PM Surbhi Gupta 
>>> wrote:
>>>
>>>> [root@abc cassandra]# cat /proc/sys/net/ipv4/tcp_keepalive_time
>>>>
>>>> 300
>>>>
>>>> [root@abc cassandra]# cat /proc/sys/net/ipv4/tcp_keepalive_intvl
>>>>
>>>> 30
>>>>
>>>> [root@abc cassandra]# cat /proc/sys/net/ipv4/tcp_keepalive_probes
>>>>
>>>> 9
>>>>
>>>> On Thu, 7 May 2020 at 12:32, Adam Scott  wrote:
>>>>
>>>>> Maybe a firewall killing a connection?
>>>>>
>>>>> What does the following show?
>>>>> cat /proc/sys/net/ipv4/tcp_keepalive_time
>>>>> cat /proc/sys/net/ipv4/tcp_keepalive_intvl
>>>>> cat /proc/sys/net/ipv4/tcp_keepalive_probes
>>>>>
>>>>> On Thu, May 7, 2020 at 10:31 AM Surbhi Gupta 
>>>>> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> We are trying to expand a datacenter and trying to add nodes but when
>>>>>> node is bootstrapping , it goes half way through and then fail with below
>>>>>> error, We have increased stremthroughput from 200 to 400 when we were
>>>>>> trying for the 2nd time but still it failed. We are on 3.11.0 , using 
>>>>>> G1GC
>>>>>> with 31GB heap.
>>>>>>
>>>>>> ERROR [MessagingService-Incoming-/10.X.X.X] 2020-05-07 09:42:38,933
>>>>>> CassandraDaemon.java:228 - Exception in thread
>>>>>> Thread[MessagingService-Incoming-/10.X.X.X,main]
>>>>>>
>>>>>> java.io.IOError: java.io.EOFException: Stream ended prematurely
>>>>>>
>>>>>> at
>>>>>> org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer$1.computeNext(UnfilteredRowIteratorSerializer.java:227)
>>>>>> ~[apache-cassandra-3.11.0.jar:3.11.0]
>>>>>>
>>>>>> at
>>>>>> org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer$1.computeNext(UnfilteredRowIteratorSerializer.java:215)
>>>>>> ~[apache-cassandra-3.11.0.jar:3.11.0]
>>>>>>
>>>>>> at
>>>>>> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
>>>>>> ~[apache-cassandra-3.11.0.jar:3.11.0]
>>>>>>
>>>>>> at
>>>>>> org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.deserialize30(PartitionUpdate.java:839)
>>>>>> ~[apache-cassandra-3.11.0.jar:3.11.0]
>>>>>>
>>>>>> at
>>>>>> org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.deserialize(PartitionUpdate.java:814)
>>>>>> ~[apache-cassandra-3.11.0.jar:3.11.0]
>>>>>>
>>>>>> at
>>>>>> org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:425)
>>>>>> ~[apache-cassandra-3.11.0.jar:3.11.0]
>>>>>>
>>>>>> at
>>>>>> org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:434)
>>>>>> ~[apache-cassandra-3.11.0.jar:3.11.0]
>>>>>>
>>>>>> at
>>>>>> org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:371)
>>>>>> ~[apache-cassandra-3.11.0.jar:3.11.0]
>>>>>>
>>>>>> at
>>>>>> org.apache.cassandra.net.MessageIn.read(MessageIn.java:123)
>&

Re: Bootstraping is failing

[Surbhi Gupta]

streaming_socket_timeout_in_ms is 24 hour.
  So tcp settings should be changed on the new bootstrap node or on all
nodes ?


On Thu, 7 May 2020 at 13:23, Adam Scott  wrote:

>
> *edit
> /etc/sysctl.confnet.ipv4.tcp_keepalive_time=60 
> net.ipv4.tcp_keepalive_probes=3net.ipv4.tcp_keepalive_intvl=10*
> then run sysctl -p to cause the kernel to reload the settings
>
> 5 minutes (300) seconds is probably too long.
>
> On Thu, May 7, 2020 at 1:09 PM Surbhi Gupta 
> wrote:
>
>> [root@abc cassandra]# cat /proc/sys/net/ipv4/tcp_keepalive_time
>>
>> 300
>>
>> [root@abc cassandra]# cat /proc/sys/net/ipv4/tcp_keepalive_intvl
>>
>> 30
>>
>> [root@abc cassandra]# cat /proc/sys/net/ipv4/tcp_keepalive_probes
>>
>> 9
>>
>> On Thu, 7 May 2020 at 12:32, Adam Scott  wrote:
>>
>>> Maybe a firewall killing a connection?
>>>
>>> What does the following show?
>>> cat /proc/sys/net/ipv4/tcp_keepalive_time
>>> cat /proc/sys/net/ipv4/tcp_keepalive_intvl
>>> cat /proc/sys/net/ipv4/tcp_keepalive_probes
>>>
>>> On Thu, May 7, 2020 at 10:31 AM Surbhi Gupta 
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> We are trying to expand a datacenter and trying to add nodes but when
>>>> node is bootstrapping , it goes half way through and then fail with below
>>>> error, We have increased stremthroughput from 200 to 400 when we were
>>>> trying for the 2nd time but still it failed. We are on 3.11.0 , using G1GC
>>>> with 31GB heap.
>>>>
>>>> ERROR [MessagingService-Incoming-/10.X.X.X] 2020-05-07 09:42:38,933
>>>> CassandraDaemon.java:228 - Exception in thread
>>>> Thread[MessagingService-Incoming-/10.X.X.X,main]
>>>>
>>>> java.io.IOError: java.io.EOFException: Stream ended prematurely
>>>>
>>>> at
>>>> org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer$1.computeNext(UnfilteredRowIteratorSerializer.java:227)
>>>> ~[apache-cassandra-3.11.0.jar:3.11.0]
>>>>
>>>> at
>>>> org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer$1.computeNext(UnfilteredRowIteratorSerializer.java:215)
>>>> ~[apache-cassandra-3.11.0.jar:3.11.0]
>>>>
>>>> at
>>>> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
>>>> ~[apache-cassandra-3.11.0.jar:3.11.0]
>>>>
>>>> at
>>>> org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.deserialize30(PartitionUpdate.java:839)
>>>> ~[apache-cassandra-3.11.0.jar:3.11.0]
>>>>
>>>> at
>>>> org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.deserialize(PartitionUpdate.java:814)
>>>> ~[apache-cassandra-3.11.0.jar:3.11.0]
>>>>
>>>> at
>>>> org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:425)
>>>> ~[apache-cassandra-3.11.0.jar:3.11.0]
>>>>
>>>> at
>>>> org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:434)
>>>> ~[apache-cassandra-3.11.0.jar:3.11.0]
>>>>
>>>> at
>>>> org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:371)
>>>> ~[apache-cassandra-3.11.0.jar:3.11.0]
>>>>
>>>> at org.apache.cassandra.net.MessageIn.read(MessageIn.java:123)
>>>> ~[apache-cassandra-3.11.0.jar:3.11.0]
>>>>
>>>> at
>>>> org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:192)
>>>> ~[apache-cassandra-3.11.0.jar:3.11.0]
>>>>
>>>> at
>>>> org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:180)
>>>> ~[apache-cassandra-3.11.0.jar:3.11.0]
>>>>
>>>> at
>>>> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:94)
>>>> ~[apache-cassandra-3.11.0.jar:3.11.0]
>>>>
>>>> Caused by: java.io.EOFException: Stream ended prematurely
>>>>
>>>> at
>>>> net.jpountz.lz4.LZ4BlockInputStream.readFully(LZ4BlockInputStream.java:218)
>>>> ~[lz4-1.3.0.jar:na]
>>>>
>>>> at
>>>> net.jpountz.lz4.LZ4BlockInputStream.refill(LZ4BlockInputStream.java:150)
>>>> ~[lz4-1.3.0.jar:na]
>>>>
>>

Re: Bootstraping is failing

[Surbhi Gupta]

[root@abc cassandra]# cat /proc/sys/net/ipv4/tcp_keepalive_time

300

[root@abc cassandra]# cat /proc/sys/net/ipv4/tcp_keepalive_intvl

30

[root@abc cassandra]# cat /proc/sys/net/ipv4/tcp_keepalive_probes

9

On Thu, 7 May 2020 at 12:32, Adam Scott  wrote:

> Maybe a firewall killing a connection?
>
> What does the following show?
> cat /proc/sys/net/ipv4/tcp_keepalive_time
> cat /proc/sys/net/ipv4/tcp_keepalive_intvl
> cat /proc/sys/net/ipv4/tcp_keepalive_probes
>
> On Thu, May 7, 2020 at 10:31 AM Surbhi Gupta 
> wrote:
>
>> Hi,
>>
>> We are trying to expand a datacenter and trying to add nodes but when
>> node is bootstrapping , it goes half way through and then fail with below
>> error, We have increased stremthroughput from 200 to 400 when we were
>> trying for the 2nd time but still it failed. We are on 3.11.0 , using G1GC
>> with 31GB heap.
>>
>> ERROR [MessagingService-Incoming-/10.X.X.X] 2020-05-07 09:42:38,933
>> CassandraDaemon.java:228 - Exception in thread
>> Thread[MessagingService-Incoming-/10.X.X.X,main]
>>
>> java.io.IOError: java.io.EOFException: Stream ended prematurely
>>
>> at
>> org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer$1.computeNext(UnfilteredRowIteratorSerializer.java:227)
>> ~[apache-cassandra-3.11.0.jar:3.11.0]
>>
>> at
>> org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer$1.computeNext(UnfilteredRowIteratorSerializer.java:215)
>> ~[apache-cassandra-3.11.0.jar:3.11.0]
>>
>> at
>> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
>> ~[apache-cassandra-3.11.0.jar:3.11.0]
>>
>> at
>> org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.deserialize30(PartitionUpdate.java:839)
>> ~[apache-cassandra-3.11.0.jar:3.11.0]
>>
>> at
>> org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.deserialize(PartitionUpdate.java:814)
>> ~[apache-cassandra-3.11.0.jar:3.11.0]
>>
>> at
>> org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:425)
>> ~[apache-cassandra-3.11.0.jar:3.11.0]
>>
>> at
>> org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:434)
>> ~[apache-cassandra-3.11.0.jar:3.11.0]
>>
>> at
>> org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:371)
>> ~[apache-cassandra-3.11.0.jar:3.11.0]
>>
>> at org.apache.cassandra.net.MessageIn.read(MessageIn.java:123)
>> ~[apache-cassandra-3.11.0.jar:3.11.0]
>>
>> at
>> org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:192)
>> ~[apache-cassandra-3.11.0.jar:3.11.0]
>>
>> at
>> org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:180)
>> ~[apache-cassandra-3.11.0.jar:3.11.0]
>>
>> at
>> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:94)
>> ~[apache-cassandra-3.11.0.jar:3.11.0]
>>
>> Caused by: java.io.EOFException: Stream ended prematurely
>>
>> at
>> net.jpountz.lz4.LZ4BlockInputStream.readFully(LZ4BlockInputStream.java:218)
>> ~[lz4-1.3.0.jar:na]
>>
>> at
>> net.jpountz.lz4.LZ4BlockInputStream.refill(LZ4BlockInputStream.java:150)
>> ~[lz4-1.3.0.jar:na]
>>
>> at
>> net.jpountz.lz4.LZ4BlockInputStream.read(LZ4BlockInputStream.java:117)
>> ~[lz4-1.3.0.jar:na]
>>
>> at java.io.DataInputStream.readFully(DataInputStream.java:195)
>> ~[na:1.8.0_242]
>>
>> at java.io.DataInputStream.readFully(DataInputStream.java:169)
>> ~[na:1.8.0_242]
>>
>> at
>> org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:402)
>> ~[apache-cassandra-3.11.0.jar:3.11.0]
>>
>> at
>> org.apache.cassandra.db.marshal.AbstractType.readValue(AbstractType.java:437)
>> ~[apache-cassandra-3.11.0.jar:3.11.0]
>>
>> at
>> org.apache.cassandra.db.rows.Cell$Serializer.deserialize(Cell.java:245)
>> ~[apache-cassandra-3.11.0.jar:3.11.0]
>>
>> at
>> org.apache.cassandra.db.rows.UnfilteredSerializer.readComplexColumn(UnfilteredSerializer.java:665)
>> ~[apache-cassandra-3.11.0.jar:3.11.0]
>>
>> at
>> org.apache.cassandra.db.rows.UnfilteredSerializer.lambda$deserializeRowBody$1(UnfilteredSerializer.java:606)
>> ~[apache-cassandra-3.11.0.jar:3.11.0]
>>
>> at
>> org.apache.ca

Bootstraping is failing

[Surbhi Gupta]

Hi,

We are trying to expand a datacenter and trying to add nodes but when node
is bootstrapping , it goes half way through and then fail with below error,
We have increased stremthroughput from 200 to 400 when we were trying for
the 2nd time but still it failed. We are on 3.11.0 , using G1GC with 31GB
heap.

ERROR [MessagingService-Incoming-/10.X.X.X] 2020-05-07 09:42:38,933
CassandraDaemon.java:228 - Exception in thread
Thread[MessagingService-Incoming-/10.X.X.X,main]

java.io.IOError: java.io.EOFException: Stream ended prematurely

at
org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer$1.computeNext(UnfilteredRowIteratorSerializer.java:227)
~[apache-cassandra-3.11.0.jar:3.11.0]

at
org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer$1.computeNext(UnfilteredRowIteratorSerializer.java:215)
~[apache-cassandra-3.11.0.jar:3.11.0]

at
org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
~[apache-cassandra-3.11.0.jar:3.11.0]

at
org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.deserialize30(PartitionUpdate.java:839)
~[apache-cassandra-3.11.0.jar:3.11.0]

at
org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.deserialize(PartitionUpdate.java:814)
~[apache-cassandra-3.11.0.jar:3.11.0]

at
org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:425)
~[apache-cassandra-3.11.0.jar:3.11.0]

at
org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:434)
~[apache-cassandra-3.11.0.jar:3.11.0]

at
org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:371)
~[apache-cassandra-3.11.0.jar:3.11.0]

at org.apache.cassandra.net.MessageIn.read(MessageIn.java:123)
~[apache-cassandra-3.11.0.jar:3.11.0]

at
org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:192)
~[apache-cassandra-3.11.0.jar:3.11.0]

at
org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:180)
~[apache-cassandra-3.11.0.jar:3.11.0]

at
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:94)
~[apache-cassandra-3.11.0.jar:3.11.0]

Caused by: java.io.EOFException: Stream ended prematurely

at
net.jpountz.lz4.LZ4BlockInputStream.readFully(LZ4BlockInputStream.java:218)
~[lz4-1.3.0.jar:na]

at
net.jpountz.lz4.LZ4BlockInputStream.refill(LZ4BlockInputStream.java:150)
~[lz4-1.3.0.jar:na]

at
net.jpountz.lz4.LZ4BlockInputStream.read(LZ4BlockInputStream.java:117)
~[lz4-1.3.0.jar:na]

at java.io.DataInputStream.readFully(DataInputStream.java:195)
~[na:1.8.0_242]

at java.io.DataInputStream.readFully(DataInputStream.java:169)
~[na:1.8.0_242]

at
org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:402)
~[apache-cassandra-3.11.0.jar:3.11.0]

at
org.apache.cassandra.db.marshal.AbstractType.readValue(AbstractType.java:437)
~[apache-cassandra-3.11.0.jar:3.11.0]

at
org.apache.cassandra.db.rows.Cell$Serializer.deserialize(Cell.java:245)
~[apache-cassandra-3.11.0.jar:3.11.0]

at
org.apache.cassandra.db.rows.UnfilteredSerializer.readComplexColumn(UnfilteredSerializer.java:665)
~[apache-cassandra-3.11.0.jar:3.11.0]

at
org.apache.cassandra.db.rows.UnfilteredSerializer.lambda$deserializeRowBody$1(UnfilteredSerializer.java:606)
~[apache-cassandra-3.11.0.jar:3.11.0]

at
org.apache.cassandra.utils.btree.BTree.applyForwards(BTree.java:1242)
~[apache-cassandra-3.11.0.jar:3.11.0]

at org.apache.cassandra.utils.btree.BTree.apply(BTree.java:1197)
~[apache-cassandra-3.11.0.jar:3.11.0]

at org.apache.cassandra.db.Columns.apply(Columns.java:377)
~[apache-cassandra-3.11.0.jar:3.11.0]

at
org.apache.cassandra.db.rows.UnfilteredSerializer.deserializeRowBody(UnfilteredSerializer.java:600)
~[apache-cassandra-3.11.0.jar:3.11.0]

at
org.apache.cassandra.db.rows.UnfilteredSerializer.deserializeOne(UnfilteredSerializer.java:475)
~[apache-cassandra-3.11.0.jar:3.11.0]

at
org.apache.cassandra.db.rows.UnfilteredSerializer.deserialize(UnfilteredSerializer.java:431)
~[apache-cassandra-3.11.0.jar:3.11.0]

at
org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer$1.computeNext(UnfilteredRowIteratorSerializer.java:222)
~[apache-cassandra-3.11.0.jar:3.11.0]

... 11 common frames omitted

Thanks
Surbhi

Re: OOM only on one datacenter nodes

[Surbhi Gupta]

We are using JRE and not JDK , hence not able to take heap dump .

On Sun, 5 Apr 2020 at 19:21, Jeff Jirsa  wrote:

>
> Set the jvm flags to heap dump on oom
>
> Open up the result in a heap inspector of your preference (like yourkit or
> similar)
>
> Find a view that counts objects by total retained size. Take a screenshot.
> Send that.
>
>
>
> On Apr 5, 2020, at 6:51 PM, Surbhi Gupta  wrote:
>
> 
> I just checked, we have setup the Heapsize to be 31GB not 32GB in DC2.
>
> I checked the CPU and RAM both are same on all the nodes in DC1 and DC2.
> What specific parameter I should check on OS ?
> We are using CentOS release 6.10.
>
> Currently disk_access_modeis not set hence it is auto in our env. Should
> setting disk_access_mode  to mmap_index_only  will help ?
>
> Thanks
> Surbhi
>
> On Sun, 5 Apr 2020 at 01:31, Alex Ott  wrote:
>
>> Have you set -Xmx32g ? In this case you may get significantly less
>> available memory because of switch to 64-bit references.  See
>> http://java-performance.info/over-32g-heap-java/ for details, and set
>> slightly less than 32Gb
>>
>> Reid Pinchback  at "Sun, 5 Apr 2020 00:50:43 +" wrote:
>>  RP> Surbi:
>>
>>  RP> If you aren’t seeing connection activity in DC2, I’d check to see if
>> the operations hitting DC1 are quorum ops instead of local quorum.  That
>>  RP> still wouldn’t explain DC2 nodes going down, but would at least
>> explain them doing more work than might be on your radar right now.
>>
>>  RP> The hint replay being slow to me sounds like you could be fighting
>> GC.
>>
>>  RP> You mentioned bumping the DC2 nodes to 32gb.  You might have already
>> been doing this, but if not, be sure to be under 32gb, like 31gb.
>>  RP> Otherwise you’re using larger object pointers and could actually
>> have less effective ability to allocate memory.
>>
>>  RP> As the problem is only happening in DC2, then there has to be a
>> thing that is true in DC2 that isn’t true in DC1.  A difference in
>> hardware, a
>>  RP> difference in O/S version, a difference in networking config or
>> physical infrastructure, a difference in client-triggered activity, or a
>>  RP> difference in how repairs are handled. Somewhere, there is a
>> difference.  I’d start with focusing on that.
>>
>>  RP> From: Erick Ramirez 
>>  RP> Reply-To: "user@cassandra.apache.org" 
>>  RP> Date: Saturday, April 4, 2020 at 8:28 PM
>>  RP> To: "user@cassandra.apache.org" 
>>  RP> Subject: Re: OOM only on one datacenter nodes
>>
>>  RP> Message from External Sender
>>
>>  RP> With a lack of heapdump for you to analyse, my hypothesis is that
>> your DC2 nodes are taking on traffic (from some client somewhere) but you're
>>  RP> just not aware of it. The hints replay is just a side-effect of the
>> nodes getting overloaded.
>>
>>  RP> To rule out my hypothesis in the first instance, my recommendation
>> is to monitor the incoming connections to the nodes in DC2. If you don't
>>  RP> have monitoring in place, you could simply run netstat at regular
>> intervals and go from there. Cheers!
>>
>>  RP> GOT QUESTIONS? Apache Cassandra experts from the community and
>> DataStax have answers! Share your expertise on
>> https://community.datastax.com/.
>>
>>
>>
>> --
>> With best wishes,Alex Ott
>> Principal Architect, DataStax
>> http://datastax.com/
>>
>> -
>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
>> For additional commands, e-mail: user-h...@cassandra.apache.org
>>
>>

Re: OOM only on one datacenter nodes

[Surbhi Gupta]

I just checked, we have setup the Heapsize to be 31GB not 32GB in DC2.

I checked the CPU and RAM both are same on all the nodes in DC1 and DC2.
What specific parameter I should check on OS ?
We are using CentOS release 6.10.

Currently disk_access_modeis not set hence it is auto in our env. Should
setting disk_access_mode  to mmap_index_only  will help ?

Thanks
Surbhi

On Sun, 5 Apr 2020 at 01:31, Alex Ott  wrote:

> Have you set -Xmx32g ? In this case you may get significantly less
> available memory because of switch to 64-bit references.  See
> http://java-performance.info/over-32g-heap-java/ for details, and set
> slightly less than 32Gb
>
> Reid Pinchback  at "Sun, 5 Apr 2020 00:50:43 +" wrote:
>  RP> Surbi:
>
>  RP> If you aren’t seeing connection activity in DC2, I’d check to see if
> the operations hitting DC1 are quorum ops instead of local quorum.  That
>  RP> still wouldn’t explain DC2 nodes going down, but would at least
> explain them doing more work than might be on your radar right now.
>
>  RP> The hint replay being slow to me sounds like you could be fighting GC.
>
>  RP> You mentioned bumping the DC2 nodes to 32gb.  You might have already
> been doing this, but if not, be sure to be under 32gb, like 31gb.
>  RP> Otherwise you’re using larger object pointers and could actually have
> less effective ability to allocate memory.
>
>  RP> As the problem is only happening in DC2, then there has to be a thing
> that is true in DC2 that isn’t true in DC1.  A difference in hardware, a
>  RP> difference in O/S version, a difference in networking config or
> physical infrastructure, a difference in client-triggered activity, or a
>  RP> difference in how repairs are handled. Somewhere, there is a
> difference.  I’d start with focusing on that.
>
>  RP> From: Erick Ramirez 
>  RP> Reply-To: "user@cassandra.apache.org" 
>  RP> Date: Saturday, April 4, 2020 at 8:28 PM
>  RP> To: "user@cassandra.apache.org" 
>  RP> Subject: Re: OOM only on one datacenter nodes
>
>  RP> Message from External Sender
>
>  RP> With a lack of heapdump for you to analyse, my hypothesis is that
> your DC2 nodes are taking on traffic (from some client somewhere) but you're
>  RP> just not aware of it. The hints replay is just a side-effect of the
> nodes getting overloaded.
>
>  RP> To rule out my hypothesis in the first instance, my recommendation is
> to monitor the incoming connections to the nodes in DC2. If you don't
>  RP> have monitoring in place, you could simply run netstat at regular
> intervals and go from there. Cheers!
>
>  RP> GOT QUESTIONS? Apache Cassandra experts from the community and
> DataStax have answers! Share your expertise on
> https://community.datastax.com/.
>
>
>
> --
> With best wishes,Alex Ott
> Principal Architect, DataStax
> http://datastax.com/
>
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
>
>

OOM only on one datacenter nodes

[Surbhi Gupta]

Hi,

We have two datacenter with 5 nodes each and have replication factor of 3.
We have traffic on DC1 and DC2 is just for disaster recovery and there is
no direct traffic.
We are using 24cpu with 128GB RAM machines .
For DC1 where we have live traffic , we don't see any issue, however for
DC2 where we don't have live traffic we see lots OOM(Out of Memory)and node
goes down(only on DC2 nodes).

We were using 16GB heap with G1GC in DC1 and DC2 both .
As DC2 nodes were OOM so we increased 16GB to 24GB and then to 32GB but
still DC2 nodes goes down with OOM , but obviously not as frequently as it
used to go down when heap was 16GB .
DC1 nodes are still on 16GB heap and none of the nodes goes down .

We are on open source 3.11.0 .
We are having Materialized views.
We see lots of hints pending on DC2 nodes and hints replay is very very
slow on DC2 nodes compare to DC1 nodes.

Other than heap sizes mentioned above , all the configs are same in
all nodes in the clusters.
We are using JRE and can't collect the heap dump.

Any idea, what can be the cause ?

Currently disk_access_modeis not set hence it is auto in our env. Should
setting disk_access_mode  to mmap_index_only  will help ?

My question is "*Why DC2 nodes OOM and DC1 nodes doesn't?*"

Thanks
Surbhi

Re: Upgradesstables - PerSSTableIndexWriter.java:211 - Rejecting value

[Surbhi Gupta]

Just to add further , I could get the code reference as below:
https://github.com/xedin/sasi/blob/master/src/java/org/apache/cassandra/db/index/sasi/disk/PerSSTableIndexWriter.java

while (analyzer.hasNext())
{
ByteBuffer token = analyzer.next();
int size = token.remaining();
if (token.remaining() >= OnDiskIndexBuilder.MAX_TERM_SIZE)
{
logger.info("Rejecting value (size {}, maximum {} bytes) for column {}
(analyzed {}) at {} SSTable.",
term.remaining(),
OnDiskIndexBuilder.MAX_TERM_SIZE,
columnIndex.getColumnName(),
columnIndex.getMode().isAnalyzed,
descriptor);
continue;
}

On Mon, 9 Mar 2020 at 09:36, Surbhi Gupta  wrote:

>
> https://javadoc.io/static/org.apache.cassandra/cassandra-all/3.11.4/constant-values.html#org.apache.cassandra.index.sasi.disk.OnDiskIndexBuilder.MAX_TERM_SIZE
> The MAX_TERM_SIZE value is 1024
>
> Can we change it ?
>
>
> The messages below is INFO but because it is saying "Rejecting value" , I
> would like to understand the impact of this rejection.
>
> On Mon, 9 Mar 2020 at 08:47, Surbhi Gupta 
> wrote:
>
>> We have SASI index .
>> Any solution ?
>>
>> On Thu, 5 Mar 2020 at 15:20, Surbhi Gupta 
>> wrote:
>>
>>> Hi,
>>>
>>> We are in process of upgrading from 3.11.0 to 3.115 .
>>> While upgrading SSTables we are noticing messages like below in
>>> system.log.
>>> What are the significance of these messages?
>>>
>>> INFO  [CompactionExecutor:3] 2020-03-05 16:12:41,393
>>> PerSSTableIndexWriter.java:211 - Rejecting value (size 1.022KiB, maximum
>>> 1.000KiB) for column body (analyzed true) at
>>> /mnt/cdata/golf/enus-76e27eb0f4c111e7b07c8dd03431d478/md-196-big SSTable.
>>>
>>> INFO  [CompactionExecutor:3] 2020-03-05 16:12:41,396
>>> PerSSTableIndexWriter.java:211 - Rejecting value (size 1.178KiB, maximum
>>> 1.000KiB) for column body (analyzed true) at
>>> /mnt/cdata/golf/enus-76e27eb0f4c111e7b07c8dd03431d478/md-196-big SSTable.
>>>
>>> INFO  [CompactionExecutor:3] 2020-03-05 16:12:41,400
>>> PerSSTableIndexWriter.java:211 - Rejecting value (size 2.808KiB, maximum
>>> 1.000KiB) for column body (analyzed true) at
>>> /mnt/cdata/golf/enus-76e27eb0f4c111e7b07c8dd03431d478/md-196-big SSTable.
>>>
>>> INFO  [CompactionExecutor:3] 2020-03-05 16:12:41,401
>>> PerSSTableIndexWriter.java:211 - Rejecting value (size 1.562KiB, maximum
>>> 1.000KiB) for column body (analyzed true) at
>>> /mnt/cdata/golf/enus-76e27eb0f4c111e7b07c8dd03431d478/md-196-big SSTable.
>>>
>>> INFO  [CompactionExecutor:3] 2020-03-05 16:12:41,401
>>> PerSSTableIndexWriter.java:211 - Rejecting value (size 1.182KiB, maximum
>>> 1.000KiB) for column body (analyzed true) at
>>> /mnt/cdata/golf/enus-76e27eb0f4c111e7b07c8dd03431d478/md-196-big SSTable.
>>>
>>> INFO  [CompactionExecutor:3] 2020-03-05 16:12:41,404
>>> PerSSTableIndexWriter.java:211 - Rejecting value (size 1.848KiB, maximum
>>> 1.000KiB) for column body (analyzed true) at
>>> /mnt/cdata/golf/enus-76e27eb0f4c111e7b07c8dd03431d478/md-196-big SSTable.
>>>
>>> Thanks
>>> Surbhi
>>>
>>

Re: Upgradesstables - PerSSTableIndexWriter.java:211 - Rejecting value

[Surbhi Gupta]

https://javadoc.io/static/org.apache.cassandra/cassandra-all/3.11.4/constant-values.html#org.apache.cassandra.index.sasi.disk.OnDiskIndexBuilder.MAX_TERM_SIZE
The MAX_TERM_SIZE value is 1024

Can we change it ?


The messages below is INFO but because it is saying "Rejecting value" , I
would like to understand the impact of this rejection.

On Mon, 9 Mar 2020 at 08:47, Surbhi Gupta  wrote:

> We have SASI index .
> Any solution ?
>
> On Thu, 5 Mar 2020 at 15:20, Surbhi Gupta 
> wrote:
>
>> Hi,
>>
>> We are in process of upgrading from 3.11.0 to 3.115 .
>> While upgrading SSTables we are noticing messages like below in
>> system.log.
>> What are the significance of these messages?
>>
>> INFO  [CompactionExecutor:3] 2020-03-05 16:12:41,393
>> PerSSTableIndexWriter.java:211 - Rejecting value (size 1.022KiB, maximum
>> 1.000KiB) for column body (analyzed true) at
>> /mnt/cdata/golf/enus-76e27eb0f4c111e7b07c8dd03431d478/md-196-big SSTable.
>>
>> INFO  [CompactionExecutor:3] 2020-03-05 16:12:41,396
>> PerSSTableIndexWriter.java:211 - Rejecting value (size 1.178KiB, maximum
>> 1.000KiB) for column body (analyzed true) at
>> /mnt/cdata/golf/enus-76e27eb0f4c111e7b07c8dd03431d478/md-196-big SSTable.
>>
>> INFO  [CompactionExecutor:3] 2020-03-05 16:12:41,400
>> PerSSTableIndexWriter.java:211 - Rejecting value (size 2.808KiB, maximum
>> 1.000KiB) for column body (analyzed true) at
>> /mnt/cdata/golf/enus-76e27eb0f4c111e7b07c8dd03431d478/md-196-big SSTable.
>>
>> INFO  [CompactionExecutor:3] 2020-03-05 16:12:41,401
>> PerSSTableIndexWriter.java:211 - Rejecting value (size 1.562KiB, maximum
>> 1.000KiB) for column body (analyzed true) at
>> /mnt/cdata/golf/enus-76e27eb0f4c111e7b07c8dd03431d478/md-196-big SSTable.
>>
>> INFO  [CompactionExecutor:3] 2020-03-05 16:12:41,401
>> PerSSTableIndexWriter.java:211 - Rejecting value (size 1.182KiB, maximum
>> 1.000KiB) for column body (analyzed true) at
>> /mnt/cdata/golf/enus-76e27eb0f4c111e7b07c8dd03431d478/md-196-big SSTable.
>>
>> INFO  [CompactionExecutor:3] 2020-03-05 16:12:41,404
>> PerSSTableIndexWriter.java:211 - Rejecting value (size 1.848KiB, maximum
>> 1.000KiB) for column body (analyzed true) at
>> /mnt/cdata/golf/enus-76e27eb0f4c111e7b07c8dd03431d478/md-196-big SSTable.
>>
>> Thanks
>> Surbhi
>>
>

Re: Upgradesstables - PerSSTableIndexWriter.java:211 - Rejecting value

[Surbhi Gupta]

We have SASI index .
Any solution ?

On Thu, 5 Mar 2020 at 15:20, Surbhi Gupta  wrote:

> Hi,
>
> We are in process of upgrading from 3.11.0 to 3.115 .
> While upgrading SSTables we are noticing messages like below in system.log.
> What are the significance of these messages?
>
> INFO  [CompactionExecutor:3] 2020-03-05 16:12:41,393
> PerSSTableIndexWriter.java:211 - Rejecting value (size 1.022KiB, maximum
> 1.000KiB) for column body (analyzed true) at
> /mnt/cdata/golf/enus-76e27eb0f4c111e7b07c8dd03431d478/md-196-big SSTable.
>
> INFO  [CompactionExecutor:3] 2020-03-05 16:12:41,396
> PerSSTableIndexWriter.java:211 - Rejecting value (size 1.178KiB, maximum
> 1.000KiB) for column body (analyzed true) at
> /mnt/cdata/golf/enus-76e27eb0f4c111e7b07c8dd03431d478/md-196-big SSTable.
>
> INFO  [CompactionExecutor:3] 2020-03-05 16:12:41,400
> PerSSTableIndexWriter.java:211 - Rejecting value (size 2.808KiB, maximum
> 1.000KiB) for column body (analyzed true) at
> /mnt/cdata/golf/enus-76e27eb0f4c111e7b07c8dd03431d478/md-196-big SSTable.
>
> INFO  [CompactionExecutor:3] 2020-03-05 16:12:41,401
> PerSSTableIndexWriter.java:211 - Rejecting value (size 1.562KiB, maximum
> 1.000KiB) for column body (analyzed true) at
> /mnt/cdata/golf/enus-76e27eb0f4c111e7b07c8dd03431d478/md-196-big SSTable.
>
> INFO  [CompactionExecutor:3] 2020-03-05 16:12:41,401
> PerSSTableIndexWriter.java:211 - Rejecting value (size 1.182KiB, maximum
> 1.000KiB) for column body (analyzed true) at
> /mnt/cdata/golf/enus-76e27eb0f4c111e7b07c8dd03431d478/md-196-big SSTable.
>
> INFO  [CompactionExecutor:3] 2020-03-05 16:12:41,404
> PerSSTableIndexWriter.java:211 - Rejecting value (size 1.848KiB, maximum
> 1.000KiB) for column body (analyzed true) at
> /mnt/cdata/golf/enus-76e27eb0f4c111e7b07c8dd03431d478/md-196-big SSTable.
>
> Thanks
> Surbhi
>

Upgradesstables - PerSSTableIndexWriter.java:211 - Rejecting value

[Surbhi Gupta]

Hi,

We are in process of upgrading from 3.11.0 to 3.115 .
While upgrading SSTables we are noticing messages like below in system.log.
What are the significance of these messages?

INFO  [CompactionExecutor:3] 2020-03-05 16:12:41,393
PerSSTableIndexWriter.java:211 - Rejecting value (size 1.022KiB, maximum
1.000KiB) for column body (analyzed true) at
/mnt/cdata/golf/enus-76e27eb0f4c111e7b07c8dd03431d478/md-196-big SSTable.

INFO  [CompactionExecutor:3] 2020-03-05 16:12:41,396
PerSSTableIndexWriter.java:211 - Rejecting value (size 1.178KiB, maximum
1.000KiB) for column body (analyzed true) at
/mnt/cdata/golf/enus-76e27eb0f4c111e7b07c8dd03431d478/md-196-big SSTable.

INFO  [CompactionExecutor:3] 2020-03-05 16:12:41,400
PerSSTableIndexWriter.java:211 - Rejecting value (size 2.808KiB, maximum
1.000KiB) for column body (analyzed true) at
/mnt/cdata/golf/enus-76e27eb0f4c111e7b07c8dd03431d478/md-196-big SSTable.

INFO  [CompactionExecutor:3] 2020-03-05 16:12:41,401
PerSSTableIndexWriter.java:211 - Rejecting value (size 1.562KiB, maximum
1.000KiB) for column body (analyzed true) at
/mnt/cdata/golf/enus-76e27eb0f4c111e7b07c8dd03431d478/md-196-big SSTable.

INFO  [CompactionExecutor:3] 2020-03-05 16:12:41,401
PerSSTableIndexWriter.java:211 - Rejecting value (size 1.182KiB, maximum
1.000KiB) for column body (analyzed true) at
/mnt/cdata/golf/enus-76e27eb0f4c111e7b07c8dd03431d478/md-196-big SSTable.

INFO  [CompactionExecutor:3] 2020-03-05 16:12:41,404
PerSSTableIndexWriter.java:211 - Rejecting value (size 1.848KiB, maximum
1.000KiB) for column body (analyzed true) at
/mnt/cdata/golf/enus-76e27eb0f4c111e7b07c8dd03431d478/md-196-big SSTable.

Thanks
Surbhi

Downgrading from 3.11.5 to 3.11.0

[Surbhi Gupta]

Hi,

As the SSTable file formats have changed from 3.11.4 to "md "
https://docs.datastax.com/en/landing_page/doc/landing_page/compatibility.html

We are going to take snapshots but still wanted to understand .
After we do upgrades stable when we upgrade from 3.11.0 to 3.11.5 , and
later in future if we found any issue , then can we downgrade to 3.11.0
with SSTable format "md" (we may not be able to restore from the snapshot
if time lapse is between upgrades is more) .

Any thoughts?

Thanks
Surbhi

Re: Consequences of dropping Materialized views

[Surbhi Gupta]

Application team confirmed that they are * not* referencing the dropped MVs
anywhere for reading or writing


On Tue, 18 Feb 2020 at 22:25, Surbhi Gupta  wrote:

> So should upgrading to 3.11.1 will solve this issue?
>
> On Tue, 18 Feb 2020 at 22:18, Surbhi Gupta 
> wrote:
>
>> Thanks Eric...
>>
>> On Tue, 18 Feb 2020 at 22:06, Erick Ramirez 
>> wrote:
>>
>>> Just to add to my above point because here we are dropping MV not a
>>>> regular table.
>>>> And MV does read before write , Is this the reason we are seeing the
>>>> below message? Trying to understand
>>>>
>>>> WARN  [HintsDispatcher:6737] 2020-02-18 14:22:24,932 HintsReader.java:237 
>>>> - Failed to read a hint for /10.X.X.X: 
>>>> f75e58e8-c255-4318-a553-06487b6bbe8c - table with id 
>>>> 7bb4b2d2-8622-11e8-a44b-a92f2dd93ff1 is unknown in file 
>>>> f75e58e8-c255-4318-a553-06487b6bbe8c-1582060925656-1.hints
>>>>
>>>>
>>> That warning message means that the hint for host /10.X.X.X (the IP you
>>> obfuscated) could not be read from the hint file (file
>>> f75e58e8-c255-4318-a553-06487b6bbe8c-1582060925656-1.hints) because the
>>> table (or MV) it belongs to with CF ID
>>> 7bb4b2d2-8622-11e8-a44b-a92f2dd93ff1 doesn't exist anymore.
>>>
>>>

Re: Consequences of dropping Materialized views

[Surbhi Gupta]

So should upgrading to 3.11.1 will solve this issue?

On Tue, 18 Feb 2020 at 22:18, Surbhi Gupta  wrote:

> Thanks Eric...
>
> On Tue, 18 Feb 2020 at 22:06, Erick Ramirez 
> wrote:
>
>> Just to add to my above point because here we are dropping MV not a
>>> regular table.
>>> And MV does read before write , Is this the reason we are seeing the
>>> below message? Trying to understand
>>>
>>> WARN  [HintsDispatcher:6737] 2020-02-18 14:22:24,932 HintsReader.java:237 - 
>>> Failed to read a hint for /10.X.X.X: f75e58e8-c255-4318-a553-06487b6bbe8c - 
>>> table with id 7bb4b2d2-8622-11e8-a44b-a92f2dd93ff1 is unknown in file 
>>> f75e58e8-c255-4318-a553-06487b6bbe8c-1582060925656-1.hints
>>>
>>>
>> That warning message means that the hint for host /10.X.X.X (the IP you
>> obfuscated) could not be read from the hint file (file
>> f75e58e8-c255-4318-a553-06487b6bbe8c-1582060925656-1.hints) because the
>> table (or MV) it belongs to with CF ID
>> 7bb4b2d2-8622-11e8-a44b-a92f2dd93ff1 doesn't exist anymore.
>>
>>

Re: Consequences of dropping Materialized views

[Surbhi Gupta]

Thanks Eric...

On Tue, 18 Feb 2020 at 22:06, Erick Ramirez 
wrote:

> Just to add to my above point because here we are dropping MV not a
>> regular table.
>> And MV does read before write , Is this the reason we are seeing the
>> below message? Trying to understand
>>
>> WARN  [HintsDispatcher:6737] 2020-02-18 14:22:24,932 HintsReader.java:237 - 
>> Failed to read a hint for /10.X.X.X: f75e58e8-c255-4318-a553-06487b6bbe8c - 
>> table with id 7bb4b2d2-8622-11e8-a44b-a92f2dd93ff1 is unknown in file 
>> f75e58e8-c255-4318-a553-06487b6bbe8c-1582060925656-1.hints
>>
>>
> That warning message means that the hint for host /10.X.X.X (the IP you
> obfuscated) could not be read from the hint file (file
> f75e58e8-c255-4318-a553-06487b6bbe8c-1582060925656-1.hints) because the
> table (or MV) it belongs to with CF ID
> 7bb4b2d2-8622-11e8-a44b-a92f2dd93ff1 doesn't exist anymore.
>
>

Re: Consequences of dropping Materialized views

[Surbhi Gupta]

Just to add to my above point because here we are dropping MV not a regular
table.
And MV does read before write , Is this the reason we are seeing the below
message? Trying to understand

WARN  [HintsDispatcher:6737] 2020-02-18 14:22:24,932
HintsReader.java:237 - Failed to read a hint for /10.X.X.X:
f75e58e8-c255-4318-a553-06487b6bbe8c - table with id
7bb4b2d2-8622-11e8-a44b-a92f2dd93ff1 is unknown in file
f75e58e8-c255-4318-a553-06487b6bbe8c-1582060925656-1.hints


On Tue, 18 Feb 2020 at 21:41, Erick Ramirez 
wrote:

> I'm not sure I understand your last response. Was there a question in
> there somewhere? Cheers!
>
>>

Re: Consequences of dropping Materialized views

[Surbhi Gupta]

Hi Eric,
As per  https://issues.apache.org/jira/browse/CASSANDRA-13696 , this issue
happens even with write traffic

"I did more investigation today. Seems it's more serious than I thought.
Even there's no down node, "drop table" + write traffic, will trigger the
problem."

Thanks
Surbhi

On Tue, 18 Feb 2020 at 20:04, Erick Ramirez 
wrote:

> We are Cassandra 3.11.0 unfortunately :(
>>
>
> Oh, right. That's why the hint read failure is causing the node to
> shutdown. We've at least identified that. I was worried that there was
> another bug we didn't know about. Cheers!
>
>>

Re: Consequences of dropping Materialized views

[Surbhi Gupta]

We tested this is dev and test before dropping in production but did not
see this issue in dev/test .
I am yet to hear from application team.
However if it was application read was happening from the dropped MV then
we would have caught this error in dev/test itself.
And we are running Cassandra 3.11.0 in dev/test and prod .

Just a thought

On Tue, 18 Feb 2020 at 19:49, Surbhi Gupta  wrote:

> We are Cassandra 3.11.0 unfortunately :(
>
> On Tue, 18 Feb 2020 at 19:41, Erick Ramirez 
> wrote:
>
>> Clearly the hint error invoked the fs error handler - probably
>>> incorrectly - which shut down the db. That’s not ok and deserves a JIRA.
>>>
>>
>> It's supposed to have been fixed by CASSANDRA-13696 in 3.0.15/3.11.1 but
>> I'm waiting for Surbhi to confirm the exact C* version. Cheers!
>>
>

Re: Consequences of dropping Materialized views

[Surbhi Gupta]

We are Cassandra 3.11.0 unfortunately :(

On Tue, 18 Feb 2020 at 19:41, Erick Ramirez 
wrote:

> Clearly the hint error invoked the fs error handler - probably incorrectly
>> - which shut down the db. That’s not ok and deserves a JIRA.
>>
>
> It's supposed to have been fixed by CASSANDRA-13696 in 3.0.15/3.11.1 but
> I'm waiting for Surbhi to confirm the exact C* version. Cheers!
>

Re: Consequences of dropping Materialized views

[Surbhi Gupta]

Thanks Eric, Let me go back to the app team

On Tue, Feb 18, 2020 at 6:49 PM Erick Ramirez 
wrote:

> We are on cassandra 3.11 , we are using G1GC and using 16GB of heap.
>>
>
> Which exact version of C* is it again?
>
>> WARN [MessagingService-Incoming-/10.X.X.X] 2020-02-18 14:21:47,115
>> IncomingTcpConnection.java:103 - UnknownColumnFamilyException reading from
>> socket; closing
>>
>
> This is expected because your app is trying to read the MV you just
> dropped.
>
> ERROR [MutationStage-9] 2020-02-18 14:21:47,267 Keyspace.java:593 - 
> Attempting to mutate non-existant table 7bb4b2d2-8622-11e8-a44b-a92f2dd93ff1
>>
>>
> Any view update transactions which are already in-flight will fail. Again,
> this is expected since the MV updates are not synchronous with the base
> table updates.
>
> WARN  [BatchlogTasks:1] 2020-02-18 14:21:53,786 BatchlogManager.java:252 - 
> Skipped batch replay of a19c6480-5294-11ea-9e09-3948c59ad0f5 due to {}
>>
>>
> As above, this is expected since the MV updates which are already in the
> queue will fail to apply because the MV got dropped.
>
> WARN  [HintsDispatcher:6737] 2020-02-18 14:22:24,932 HintsReader.java:237 - 
> Failed to read a hint for /10.X.X.X: f75e58e8-c255-4318-a553-06487b6bbe8c - 
> table with id 7bb4b2d2-8622-11e8-a44b-a92f2dd93ff1 is unknown in file 
> f75e58e8-c255-4318-a553-06487b6bbe8c-1582060925656-1.hints
>>
>>
> This is also expected. It won't be able to read the hint for a
> non-existent MV.
>
> So where to from here? The symptoms you described indicate that you
> haven't updated your app *before* you dropped the MVs. That is key here
> -- your app shouldn't be issuing reads for the MV since your intention is
> to drop it. What I think happened is that the nodes were backed up with
> read requests for the MV which no longer exists.
>
> Can you confirm that you have changed your application so it is not
> supposed to issue reads to the MV? You shouldn't drop MVs if you're still
> going to issue read requests from them just like you shouldn't drop tables
> your app is still expected to access. I hope that makes sense. Cheers!
>

Re: Consequences of dropping Materialized views

[Surbhi Gupta]

Thanks Jonathan, it still helps...

Anyone knows the solution?

On Tue, 18 Feb 2020 at 18:08, Jonathan Koppenhofer 
wrote:

> Forensics are gone at this point, so I can't verify exact errors, but
> wanted to mention we had seen something similar to corroborate your
> experience and warn others.
>
> The version would have been 3.0.15 or 3.11.3 as that is what we were
> deploying on our clusters at the time. I think it was more likely 3.0.15.
>
> So sorry for the "vagueness" :(
>
> On Tue, Feb 18, 2020, 8:54 PM Surbhi Gupta 
> wrote:
>
>> Jonathan,
>> As per https://issues.apache.org/jira/browse/CASSANDRA-13696 the issue,
>> Digest mismatch Exception if hints file has UnknownColumnFamily,  is fixed
>> for 3.0.15 , did you still faced this issue on 3.0.15 ?
>>
>> Thanks
>> Surbhi
>>
>> On Tue, 18 Feb 2020 at 17:40, Jonathan Koppenhofer 
>> wrote:
>>
>>> I believe we had something similar happen on 3.0.15 a while back. We had
>>> a cluster that created mass havoc by creating MVs on a large existing
>>> dataset. We thought we had stabilized the cluster, but saw similar issues
>>> as you when we dropped the MVs.
>>>
>>> We interpreted our errors to mean that we should not attempt to write to
>>> base tables while also dropping downstream materialized views. We
>>> essentially had the app stop their app, then drop the views 1 by 1 with
>>> some pause between. That then seemed to work fine, but yes, be careful with
>>> everything MVs.
>>>
>>> We now disallow the use of MVs globally.
>>>
>>> On Tue, Feb 18, 2020, 8:27 PM Surbhi Gupta 
>>> wrote:
>>>
>>>> We are on cassandra 3.11 , we are using G1GC and using 16GB of heap.
>>>>
>>>> So we had to drop 7 MVs in production, as soon as we dropped the first
>>>> Materialized View, our cluster became unstable and app started giving 100%
>>>> error, what we noticed:
>>>> 1. As soon as MV was dropped , cluster became unstable and nodes were
>>>> showing down from each other.
>>>> 2. We saw below warnings in system.log which is understood,
>>>>  WARN [MessagingService-Incoming-/10.X.X.X] 2020-02-18 14:21:47,115
>>>> IncomingTcpConnection.java:103 - UnknownColumnFamilyException reading from
>>>> socket; closing
>>>>
>>>> org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find table 
>>>> for cfId 7bb4b2d2-8622-11e8-a44b-a92f2dd93ff1. If a table was just 
>>>> created, this is likely due to the schema not being fully propagated.  
>>>> Please wait for schema agreement on table creation.
>>>>
>>>> 3. We noticed below errors as well:
>>>>
>>>> ERROR [MutationStage-9] 2020-02-18 14:21:47,267 Keyspace.java:593 - 
>>>> Attempting to mutate non-existant table 
>>>> 7bb4b2d2-8622-11e8-a44b-a92f2dd93ff1
>>>>
>>>> 4. We noticed messages like below:
>>>>
>>>> WARN  [BatchlogTasks:1] 2020-02-18 14:21:53,786 BatchlogManager.java:252 - 
>>>> Skipped batch replay of a19c6480-5294-11ea-9e09-3948c59ad0f5 due to {}
>>>>
>>>> 5. Hints file corrupted:
>>>>
>>>> WARN  [HintsDispatcher:6737] 2020-02-18 14:22:24,932 HintsReader.java:237 
>>>> - Failed to read a hint for /10.X.X.X: 
>>>> f75e58e8-c255-4318-a553-06487b6bbe8c - table with id 
>>>> 7bb4b2d2-8622-11e8-a44b-a92f2dd93ff1 is unknown in file 
>>>> f75e58e8-c255-4318-a553-06487b6bbe8c-1582060925656-1.hints
>>>> ERROR [HintsDispatcher:6737] 2020-02-18 14:22:24,933 
>>>> HintsDispatchExecutor.java:243 - Failed to dispatch hints file 
>>>> f75e58e8-c255-4318-a553-06487b6bbe8c-1582060925656-1.hints: file is 
>>>> corrupted ({})
>>>> org.apache.cassandra.io.FSReadError: java.io.IOException: Digest mismatch 
>>>> exception
>>>>
>>>>
>>>> 5. And then Cassandra shut down:
>>>>
>>>> ERROR [HintsDispatcher:6737] 2020-02-18 14:22:24,937 
>>>> StorageService.java:430 - Stopping gossiper
>>>> WARN  [HintsDispatcher:6737] 2020-02-18 14:22:24,937 
>>>> StorageService.java:321 - Stopping gossip by operator request
>>>> INFO  [HintsDispatcher:6737] 2020-02-18 14:22:24,937 Gossiper.java:1530 - 
>>>> Announcing shutdown
>>>>
>>>>
>>>> Any views? Below are the issues I
>>>>
>>>> https://support.datastax.com/hc/en-us/articles/36368126

Re: Consequences of dropping Materialized views

[Surbhi Gupta]

Jonathan,
As per https://issues.apache.org/jira/browse/CASSANDRA-13696 the issue,
Digest mismatch Exception if hints file has UnknownColumnFamily,  is fixed
for 3.0.15 , did you still faced this issue on 3.0.15 ?

Thanks
Surbhi

On Tue, 18 Feb 2020 at 17:40, Jonathan Koppenhofer 
wrote:

> I believe we had something similar happen on 3.0.15 a while back. We had a
> cluster that created mass havoc by creating MVs on a large existing
> dataset. We thought we had stabilized the cluster, but saw similar issues
> as you when we dropped the MVs.
>
> We interpreted our errors to mean that we should not attempt to write to
> base tables while also dropping downstream materialized views. We
> essentially had the app stop their app, then drop the views 1 by 1 with
> some pause between. That then seemed to work fine, but yes, be careful with
> everything MVs.
>
> We now disallow the use of MVs globally.
>
> On Tue, Feb 18, 2020, 8:27 PM Surbhi Gupta 
> wrote:
>
>> We are on cassandra 3.11 , we are using G1GC and using 16GB of heap.
>>
>> So we had to drop 7 MVs in production, as soon as we dropped the first
>> Materialized View, our cluster became unstable and app started giving 100%
>> error, what we noticed:
>> 1. As soon as MV was dropped , cluster became unstable and nodes were
>> showing down from each other.
>> 2. We saw below warnings in system.log which is understood,
>>  WARN [MessagingService-Incoming-/10.X.X.X] 2020-02-18 14:21:47,115
>> IncomingTcpConnection.java:103 - UnknownColumnFamilyException reading from
>> socket; closing
>>
>> org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find table 
>> for cfId 7bb4b2d2-8622-11e8-a44b-a92f2dd93ff1. If a table was just created, 
>> this is likely due to the schema not being fully propagated.  Please wait 
>> for schema agreement on table creation.
>>
>> 3. We noticed below errors as well:
>>
>> ERROR [MutationStage-9] 2020-02-18 14:21:47,267 Keyspace.java:593 - 
>> Attempting to mutate non-existant table 7bb4b2d2-8622-11e8-a44b-a92f2dd93ff1
>>
>> 4. We noticed messages like below:
>>
>> WARN  [BatchlogTasks:1] 2020-02-18 14:21:53,786 BatchlogManager.java:252 - 
>> Skipped batch replay of a19c6480-5294-11ea-9e09-3948c59ad0f5 due to {}
>>
>> 5. Hints file corrupted:
>>
>> WARN  [HintsDispatcher:6737] 2020-02-18 14:22:24,932 HintsReader.java:237 - 
>> Failed to read a hint for /10.X.X.X: f75e58e8-c255-4318-a553-06487b6bbe8c - 
>> table with id 7bb4b2d2-8622-11e8-a44b-a92f2dd93ff1 is unknown in file 
>> f75e58e8-c255-4318-a553-06487b6bbe8c-1582060925656-1.hints
>> ERROR [HintsDispatcher:6737] 2020-02-18 14:22:24,933 
>> HintsDispatchExecutor.java:243 - Failed to dispatch hints file 
>> f75e58e8-c255-4318-a553-06487b6bbe8c-1582060925656-1.hints: file is 
>> corrupted ({})
>> org.apache.cassandra.io.FSReadError: java.io.IOException: Digest mismatch 
>> exception
>>
>>
>> 5. And then Cassandra shut down:
>>
>> ERROR [HintsDispatcher:6737] 2020-02-18 14:22:24,937 StorageService.java:430 
>> - Stopping gossiper
>> WARN  [HintsDispatcher:6737] 2020-02-18 14:22:24,937 StorageService.java:321 
>> - Stopping gossip by operator request
>> INFO  [HintsDispatcher:6737] 2020-02-18 14:22:24,937 Gossiper.java:1530 - 
>> Announcing shutdown
>>
>>
>> Any views? Below are the issues I
>>
>> https://support.datastax.com/hc/en-us/articles/36368126-Hints-file-with-unknown-CFID-can-cause-nodes-to-fail
>>
>> https://issues.apache.org/jira/browse/CASSANDRA-13696
>>
>> https://issues.apache.org/jira/browse/CASSANDRA-6822
>>
>> https://support.datastax.com/hc/en-us/articles/36368126-Hints-file-with-unknown-CFID-can-cause-nodes-to-fail
>>
>>
>>
>>
>>
>> On Wed, 12 Feb 2020 at 19:10, Surbhi Gupta 
>> wrote:
>>
>>> Thanks Eric ...
>>> This is helpful...
>>>
>>>
>>> On Wed, 12 Feb 2020 at 17:46, Erick Ramirez 
>>> wrote:
>>>
>>>> There shouldn't be any negative impact from dropping MVs and there's
>>>> certainly no risk to the base table if that is your concern. All it will do
>>>> is remove all the data in the respective views plus drop any pending view
>>>> mutations from the batch log. If anything, you should see some performance
>>>> gain since updates to the base table will only trigger 4 view updates
>>>> instead of the previous 11. Cheers!
>>>>
>>>> Erick Ramirez  |  Developer Relations
>>>>
>>>> erick.rami...@datastax.com | datastax.co

Re: Consequences of dropping Materialized views

[Surbhi Gupta]

We are on cassandra 3.11 , we are using G1GC and using 16GB of heap.

So we had to drop 7 MVs in production, as soon as we dropped the first
Materialized View, our cluster became unstable and app started giving 100%
error, what we noticed:
1. As soon as MV was dropped , cluster became unstable and nodes were
showing down from each other.
2. We saw below warnings in system.log which is understood,
 WARN [MessagingService-Incoming-/10.X.X.X] 2020-02-18 14:21:47,115
IncomingTcpConnection.java:103 - UnknownColumnFamilyException reading from
socket; closing

org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find
table for cfId 7bb4b2d2-8622-11e8-a44b-a92f2dd93ff1. If a table was
just created, this is likely due to the schema not being fully
propagated.  Please wait for schema agreement on table creation.

3. We noticed below errors as well:

ERROR [MutationStage-9] 2020-02-18 14:21:47,267 Keyspace.java:593 -
Attempting to mutate non-existant table
7bb4b2d2-8622-11e8-a44b-a92f2dd93ff1

4. We noticed messages like below:

WARN  [BatchlogTasks:1] 2020-02-18 14:21:53,786
BatchlogManager.java:252 - Skipped batch replay of
a19c6480-5294-11ea-9e09-3948c59ad0f5 due to {}

5. Hints file corrupted:

WARN  [HintsDispatcher:6737] 2020-02-18 14:22:24,932
HintsReader.java:237 - Failed to read a hint for /10.X.X.X:
f75e58e8-c255-4318-a553-06487b6bbe8c - table with id
7bb4b2d2-8622-11e8-a44b-a92f2dd93ff1 is unknown in file
f75e58e8-c255-4318-a553-06487b6bbe8c-1582060925656-1.hints
ERROR [HintsDispatcher:6737] 2020-02-18 14:22:24,933
HintsDispatchExecutor.java:243 - Failed to dispatch hints file
f75e58e8-c255-4318-a553-06487b6bbe8c-1582060925656-1.hints: file is
corrupted ({})
org.apache.cassandra.io.FSReadError: java.io.IOException: Digest
mismatch exception


5. And then Cassandra shut down:

ERROR [HintsDispatcher:6737] 2020-02-18 14:22:24,937
StorageService.java:430 - Stopping gossiper
WARN  [HintsDispatcher:6737] 2020-02-18 14:22:24,937
StorageService.java:321 - Stopping gossip by operator request
INFO  [HintsDispatcher:6737] 2020-02-18 14:22:24,937
Gossiper.java:1530 - Announcing shutdown


Any views? Below are the issues I

https://support.datastax.com/hc/en-us/articles/36368126-Hints-file-with-unknown-CFID-can-cause-nodes-to-fail

https://issues.apache.org/jira/browse/CASSANDRA-13696

https://issues.apache.org/jira/browse/CASSANDRA-6822

https://support.datastax.com/hc/en-us/articles/36368126-Hints-file-with-unknown-CFID-can-cause-nodes-to-fail





On Wed, 12 Feb 2020 at 19:10, Surbhi Gupta  wrote:

> Thanks Eric ...
> This is helpful...
>
>
> On Wed, 12 Feb 2020 at 17:46, Erick Ramirez 
> wrote:
>
>> There shouldn't be any negative impact from dropping MVs and there's
>> certainly no risk to the base table if that is your concern. All it will do
>> is remove all the data in the respective views plus drop any pending view
>> mutations from the batch log. If anything, you should see some performance
>> gain since updates to the base table will only trigger 4 view updates
>> instead of the previous 11. Cheers!
>>
>> Erick Ramirez  |  Developer Relations
>>
>> erick.rami...@datastax.com | datastax.com <http://www.datastax.com>
>> <https://www.linkedin.com/company/datastax>
>> <https://www.facebook.com/datastax> <https://twitter.com/datastax>
>> <http://feeds.feedburner.com/datastax> <https://github.com/datastax/>
>>
>> <https://www.datastax.com/accelerate>
>>
>>
>>
>> On Thu, 13 Feb 2020 at 04:26, Surbhi Gupta 
>> wrote:
>>
>>> Hi,
>>>
>>> So application team created 11 materialized views on a base table in
>>> production and we need to drop 7 Materialized views as they are not in use.
>>> Wanted to understand the impact of dropping the materialized views.
>>> We are on Cassandra 3.11.1 , multi datacenter with replication factor of
>>> 3 in each datacenter.
>>> We are using LOCAL_QUORUM for write consistency and LOCAL_ONE for read
>>> consistency.
>>>
>>> Any thoughts or suggestion to keep in mind before dropping the
>>> Materialized views.
>>>
>>> Thanks
>>> Surbhi
>>>
>>>
>>>
>>>

Re: Consequences of dropping Materialized views

[Surbhi Gupta]

Thanks Eric ...
This is helpful...


On Wed, 12 Feb 2020 at 17:46, Erick Ramirez 
wrote:

> There shouldn't be any negative impact from dropping MVs and there's
> certainly no risk to the base table if that is your concern. All it will do
> is remove all the data in the respective views plus drop any pending view
> mutations from the batch log. If anything, you should see some performance
> gain since updates to the base table will only trigger 4 view updates
> instead of the previous 11. Cheers!
>
> Erick Ramirez  |  Developer Relations
>
> erick.rami...@datastax.com | datastax.com <http://www.datastax.com>
> <https://www.linkedin.com/company/datastax>
> <https://www.facebook.com/datastax> <https://twitter.com/datastax>
> <http://feeds.feedburner.com/datastax> <https://github.com/datastax/>
>
> <https://www.datastax.com/accelerate>
>
>
>
> On Thu, 13 Feb 2020 at 04:26, Surbhi Gupta 
> wrote:
>
>> Hi,
>>
>> So application team created 11 materialized views on a base table in
>> production and we need to drop 7 Materialized views as they are not in use.
>> Wanted to understand the impact of dropping the materialized views.
>> We are on Cassandra 3.11.1 , multi datacenter with replication factor of
>> 3 in each datacenter.
>> We are using LOCAL_QUORUM for write consistency and LOCAL_ONE for read
>> consistency.
>>
>> Any thoughts or suggestion to keep in mind before dropping the
>> Materialized views.
>>
>> Thanks
>> Surbhi
>>
>>
>>
>>

Consequences of dropping Materialized views

[Surbhi Gupta]

Hi,

So application team created 11 materialized views on a base table in
production and we need to drop 7 Materialized views as they are not in use.
Wanted to understand the impact of dropping the materialized views.
We are on Cassandra 3.11.1 , multi datacenter with replication factor of 3
in each datacenter.
We are using LOCAL_QUORUM for write consistency and LOCAL_ONE for read
consistency.

Any thoughts or suggestion to keep in mind before dropping the Materialized
views.

Thanks
Surbhi

Re: Overload because of hint pressure + MVs

[Surbhi Gupta]

We are using G1 ...

On Tue, 11 Feb 2020 at 08:51, Reid Pinchback 
wrote:

> A caveat to the 31GB recommendation for G1GC.  If you have tight latency
> SLAs instead of throughput SLAs then this doesn’t necessary pan out to be
> beneficial.
>
>
>
> Yes the GCs are less frequent, but they can hurt more when they do happen.
> The win is if your usage pattern is such that the added time helps you
> squeak past deciding copying into old gen when a smaller heap/more frequent
> GC cycle would have decided it had to do promotions.  C* tends to have a
> lot of medium-lifetime objects on the heap so it can really come down to
> the specifics of what your clients are typically doing.
>
>
>
> Also, reallocation of RAM from O/S buffer cache to Java heap will also
> change the dynamics of dirty page flushes from your writes, which again
> directly surfaces in C* read latency numbers during I/O stalls from the
> write spikes in the background.  So really bumping up heap is an alteration
> that can be a double-whammy for the latency sensitive.  Those only caring
> about throughput won’t care and it’s probably unconditionally a win to go
> to 31GB.
>
>
>
> R
>
>
>
> *From: *Erick Ramirez 
> *Reply-To: *"user@cassandra.apache.org" 
> *Date: *Monday, February 10, 2020 at 3:55 PM
> *To: *"user@cassandra.apache.org" 
> *Subject: *Re: Overload because of hint pressure + MVs
>
>
>
> *Message from External Sender*
>
> Currently the value of phi_convict_threshold is not set which makes it to
> 8 (default) .
> Can this also cause hints buildup even when we can see that all nodes are
> UP ?
>
>
>
> You can bump it up to 12 to reduce the sensitivity but it's likely GC
> pauses causing it. Phi convict is the side-effect, not the cause.
>
>
>
> Just to add , we are using 24GB heap size.
>
>
>
> Are you using CMS? If using G1, I'd recommend bumping it up to 31GB if the
> servers have 40+ GB of RAM. Cheers!
>

Re: Overload because of hint pressure + MVs

[Surbhi Gupta]

Just to add , we are using 24GB heap size.

On Mon, 10 Feb 2020 at 09:08, Surbhi Gupta  wrote:

> Hi Jon,
>
> We are on multi datacenter(On Prim) setup.
> We also noticed too many messages like below:
>
> DEBUG [GossipStage:1] 2020-02-10 09:38:52,953 FailureDetector.java:457 -
> Ignoring interval time of 3258125997 for /10.x.x.x
>
> DEBUG [GossipStage:1] 2020-02-10 09:38:52,954 FailureDetector.java:457 -
> Ignoring interval time of 2045630029 for /10.y.y.y
>
> DEBUG [GossipStage:1] 2020-02-10 09:38:52,954 FailureDetector.java:457 -
> Ignoring interval time of 2045416737 for /10.z.z.z
>
>
>
> Currently the value of phi_convict_threshold is not set which makes it to
> 8 (default) .
> Can this also cause hints buildup even when we can see that all nodes are
> UP ?
> Recommended value of phi_convict_threshold  is 12 in AWS multi datacenter
> environment.
>
> Thanks
> Surbhi
>
> On Sun, 9 Feb 2020 at 21:42, Surbhi Gupta 
> wrote:
>
>> Thanks a lot Jon..
>> Will try the recommendations and let you know the results
>>
>> On Fri, Feb 7, 2020 at 10:52 AM Jon Haddad  wrote:
>>
>>> There's a few things you can do here that might help.
>>>
>>> First off, if you're using the default heap settings, that's a serious
>>> problem.  If you've got the head room, my recommendation is to use 16GB
>>> heap with 12 GB new gen and pin your memtable heap space to 2GB.  Set your
>>> max tenuring threshold to 6 and your survivor ratio to 6.  You don't need a
>>> lot of old gen space with cassandra, almost everything that will show up
>>> there is memtable related, and we allocate a *lot* whenever we read data
>>> off disk.
>>>
>>> Most folks use the default disk read ahead setting of 128KB.  You can
>>> check this setting using blockdev --report, under the RA column.  You'll
>>> see 256 there, that's in 512 byte sectors.  MVs rely on a read before a
>>> write, so for every read off disk you do, you'll pull additional 128KB into
>>> your page cache.  This is usually a waste and puts WAY too much pressure on
>>> your disk.  On SSD, I always change this to 4KB.
>>>
>>> Next, be sure you're setting your compression rate accordingly.  I wrote
>>> a long post on the topic here:
>>> https://thelastpickle.com/blog/2018/08/08/compression_performance.html.
>>> Our default compression is very unfriendly for read heavy workloads if
>>> you're reading small rows.  If your records are small, 4KB compression
>>> chunk length is your friend.
>>>
>>> I have some slides showing pretty good performance improvements from the
>>> above 2 changes.  Specifically, I went from 16K reads a second at 180ms p99
>>> latency up to 63K reads / second at 21ms p99.  Disk usage dropped by a
>>> factor of 10.  Throw in those JVM changes I recommended and things should
>>> improve even further.
>>>
>>> Generally speaking, I recommend avoiding MVs, as they can be a giant
>>> mine if you aren't careful.  They're not doing any magic behind the scenes
>>> that makes scaling easier, and in a lot of cases they're a hinderance.  You
>>> still need to understand the underlying data and how it's laid out to use
>>> them properly, which is 99% of the work.
>>>
>>> Jon
>>>
>>> On Fri, Feb 7, 2020 at 10:32 AM Michael Shuler 
>>> wrote:
>>>
>>>> That JIRA still says Open, so no, it has not been fixed (unless there's
>>>> a fixed duplicate in JIRA somewhere).
>>>>
>>>> For clarification, you could update that ticket with a comment
>>>> including
>>>> your environmental details, usage of MV, etc. I'll bump the priority up
>>>> and include some possible branchX fixvers.
>>>>
>>>> Michael
>>>>
>>>> On 2/7/20 10:53 AM, Surbhi Gupta wrote:
>>>> > Hi,
>>>> >
>>>> > We are getting hit by the below bug.
>>>> > Other than lowering hinted_handoff_throttle_in_kb to 100 any other
>>>> work
>>>> > around ?
>>>> >
>>>> > https://issues.apache.org/jira/browse/CASSANDRA-13810
>>>> >
>>>> > Any idea if it got fixed in later version.
>>>> > We are on Open source Cassandra 3.11.1  .
>>>> >
>>>> > Thanks
>>>> > Surbhi
>>>> >
>>>> >
>>>>
>>>> -
>>>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
>>>> For additional commands, e-mail: user-h...@cassandra.apache.org
>>>>
>>>>

Re: Overload because of hint pressure + MVs

[Surbhi Gupta]

Hi Jon,

We are on multi datacenter(On Prim) setup.
We also noticed too many messages like below:

DEBUG [GossipStage:1] 2020-02-10 09:38:52,953 FailureDetector.java:457 -
Ignoring interval time of 3258125997 for /10.x.x.x

DEBUG [GossipStage:1] 2020-02-10 09:38:52,954 FailureDetector.java:457 -
Ignoring interval time of 2045630029 for /10.y.y.y

DEBUG [GossipStage:1] 2020-02-10 09:38:52,954 FailureDetector.java:457 -
Ignoring interval time of 2045416737 for /10.z.z.z



Currently the value of phi_convict_threshold is not set which makes it to 8
(default) .
Can this also cause hints buildup even when we can see that all nodes are
UP ?
Recommended value of phi_convict_threshold  is 12 in AWS multi datacenter
environment.

Thanks
Surbhi

On Sun, 9 Feb 2020 at 21:42, Surbhi Gupta  wrote:

> Thanks a lot Jon..
> Will try the recommendations and let you know the results
>
> On Fri, Feb 7, 2020 at 10:52 AM Jon Haddad  wrote:
>
>> There's a few things you can do here that might help.
>>
>> First off, if you're using the default heap settings, that's a serious
>> problem.  If you've got the head room, my recommendation is to use 16GB
>> heap with 12 GB new gen and pin your memtable heap space to 2GB.  Set your
>> max tenuring threshold to 6 and your survivor ratio to 6.  You don't need a
>> lot of old gen space with cassandra, almost everything that will show up
>> there is memtable related, and we allocate a *lot* whenever we read data
>> off disk.
>>
>> Most folks use the default disk read ahead setting of 128KB.  You can
>> check this setting using blockdev --report, under the RA column.  You'll
>> see 256 there, that's in 512 byte sectors.  MVs rely on a read before a
>> write, so for every read off disk you do, you'll pull additional 128KB into
>> your page cache.  This is usually a waste and puts WAY too much pressure on
>> your disk.  On SSD, I always change this to 4KB.
>>
>> Next, be sure you're setting your compression rate accordingly.  I wrote
>> a long post on the topic here:
>> https://thelastpickle.com/blog/2018/08/08/compression_performance.html.
>> Our default compression is very unfriendly for read heavy workloads if
>> you're reading small rows.  If your records are small, 4KB compression
>> chunk length is your friend.
>>
>> I have some slides showing pretty good performance improvements from the
>> above 2 changes.  Specifically, I went from 16K reads a second at 180ms p99
>> latency up to 63K reads / second at 21ms p99.  Disk usage dropped by a
>> factor of 10.  Throw in those JVM changes I recommended and things should
>> improve even further.
>>
>> Generally speaking, I recommend avoiding MVs, as they can be a giant mine
>> if you aren't careful.  They're not doing any magic behind the scenes that
>> makes scaling easier, and in a lot of cases they're a hinderance.  You
>> still need to understand the underlying data and how it's laid out to use
>> them properly, which is 99% of the work.
>>
>> Jon
>>
>> On Fri, Feb 7, 2020 at 10:32 AM Michael Shuler 
>> wrote:
>>
>>> That JIRA still says Open, so no, it has not been fixed (unless there's
>>> a fixed duplicate in JIRA somewhere).
>>>
>>> For clarification, you could update that ticket with a comment including
>>> your environmental details, usage of MV, etc. I'll bump the priority up
>>> and include some possible branchX fixvers.
>>>
>>> Michael
>>>
>>> On 2/7/20 10:53 AM, Surbhi Gupta wrote:
>>> > Hi,
>>> >
>>> > We are getting hit by the below bug.
>>> > Other than lowering hinted_handoff_throttle_in_kb to 100 any other
>>> work
>>> > around ?
>>> >
>>> > https://issues.apache.org/jira/browse/CASSANDRA-13810
>>> >
>>> > Any idea if it got fixed in later version.
>>> > We are on Open source Cassandra 3.11.1  .
>>> >
>>> > Thanks
>>> > Surbhi
>>> >
>>> >
>>>
>>> -
>>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
>>> For additional commands, e-mail: user-h...@cassandra.apache.org
>>>
>>>

Re: Overload because of hint pressure + MVs

[Surbhi Gupta]

Thanks a lot Jon..
Will try the recommendations and let you know the results

On Fri, Feb 7, 2020 at 10:52 AM Jon Haddad  wrote:

> There's a few things you can do here that might help.
>
> First off, if you're using the default heap settings, that's a serious
> problem.  If you've got the head room, my recommendation is to use 16GB
> heap with 12 GB new gen and pin your memtable heap space to 2GB.  Set your
> max tenuring threshold to 6 and your survivor ratio to 6.  You don't need a
> lot of old gen space with cassandra, almost everything that will show up
> there is memtable related, and we allocate a *lot* whenever we read data
> off disk.
>
> Most folks use the default disk read ahead setting of 128KB.  You can
> check this setting using blockdev --report, under the RA column.  You'll
> see 256 there, that's in 512 byte sectors.  MVs rely on a read before a
> write, so for every read off disk you do, you'll pull additional 128KB into
> your page cache.  This is usually a waste and puts WAY too much pressure on
> your disk.  On SSD, I always change this to 4KB.
>
> Next, be sure you're setting your compression rate accordingly.  I wrote a
> long post on the topic here:
> https://thelastpickle.com/blog/2018/08/08/compression_performance.html.
> Our default compression is very unfriendly for read heavy workloads if
> you're reading small rows.  If your records are small, 4KB compression
> chunk length is your friend.
>
> I have some slides showing pretty good performance improvements from the
> above 2 changes.  Specifically, I went from 16K reads a second at 180ms p99
> latency up to 63K reads / second at 21ms p99.  Disk usage dropped by a
> factor of 10.  Throw in those JVM changes I recommended and things should
> improve even further.
>
> Generally speaking, I recommend avoiding MVs, as they can be a giant mine
> if you aren't careful.  They're not doing any magic behind the scenes that
> makes scaling easier, and in a lot of cases they're a hinderance.  You
> still need to understand the underlying data and how it's laid out to use
> them properly, which is 99% of the work.
>
> Jon
>
> On Fri, Feb 7, 2020 at 10:32 AM Michael Shuler 
> wrote:
>
>> That JIRA still says Open, so no, it has not been fixed (unless there's
>> a fixed duplicate in JIRA somewhere).
>>
>> For clarification, you could update that ticket with a comment including
>> your environmental details, usage of MV, etc. I'll bump the priority up
>> and include some possible branchX fixvers.
>>
>> Michael
>>
>> On 2/7/20 10:53 AM, Surbhi Gupta wrote:
>> > Hi,
>> >
>> > We are getting hit by the below bug.
>> > Other than lowering hinted_handoff_throttle_in_kb to 100 any other work
>> > around ?
>> >
>> > https://issues.apache.org/jira/browse/CASSANDRA-13810
>> >
>> > Any idea if it got fixed in later version.
>> > We are on Open source Cassandra 3.11.1  .
>> >
>> > Thanks
>> > Surbhi
>> >
>> >
>>
>> -
>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
>> For additional commands, e-mail: user-h...@cassandra.apache.org
>>
>>

Overload because of hint pressure + MVs

[Surbhi Gupta]

Hi,

We are getting hit by the below bug.
Other than lowering hinted_handoff_throttle_in_kb to 100 any other work
around ?

https://issues.apache.org/jira/browse/CASSANDRA-13810

Any idea if it got fixed in later version.
We are on Open source Cassandra 3.11.1  .

Thanks
Surbhi

Re: Nodes becoming unresponsive

[Surbhi Gupta]

I have limited options to use JDK based tools because in our environment we
are running JRE .

I tried to debug more and could see using top that Command is MutationStage
in top output , Any clue we get from this ?

top - 16:30:47 up 94 days,  5:33,  1 user,  load average: 134.83, 142.48,
144.75
Tasks: 564 total,  58 running, 506 sleeping,   0 stopped,   0 zombie
Cpu(s): 95.2%us,  2.5%sy,  0.3%ni,  1.7%id,  0.0%wa,  0.0%hi,  0.3%si,
 0.0%st
Mem:  132236016k total, 131378384k used,   857632k free,   189208k buffers
Swap:0k total,0k used,0k free, 94530140k cached

   PID USER  PR  NI  VIRT  RES  SHR S %CPU %MEMTIME+  WCHAN
COMMAND

 11798 cassandr  20   0  261g  42g  14g R 14.4 33.3  76:47.38 -
MutationStage-1

 11762 cassandr  20   0  261g  42g  14g S 14.1 33.3  82:19.84 -
MutationStage-9

 11530 cassandr  20   0  261g  42g  14g R 13.8 33.3 100:00.22 -
MutationStage-3

 11501 cassandr  20   0  261g  42g  14g S 13.4 33.3   2598:38 -
MutationStage-5

 11688 cassandr  20   0  261g  42g  14g S 13.1 33.3  90:42.47 -
MutationStage-5

 11512 cassandr  20   0  261g  42g  14g R 12.8 33.3 153:13.59 -
MutationStage-1

 11534 cassandr  20   0  261g  42g  14g R 12.8 33.3 104:48.21 -
MutationStage-2

 11708 cassandr  20   0  261g  42g  14g S 12.5 33.3  87:17.64 -
MutationStage-6

 11783 cassandr  20   0  261g  42g  14g S 12.5 33.3  76:01.10 futex_wai
MutationStage-1

 11792 cassandr  20   0  261g  42g  14g S 12.5 33.3  76:19.90 futex_wai
MutationStage-1

 11504 cassandr  20   0  261g  42g  14g S 12.2 33.3 859:10.54 futex_wai
MutationStage-8

 11517 cassandr  20   0  261g  42g  14g R 12.2 33.3 116:18.38 -
MutationStage-2

 11535 cassandr  20   0  261g  42g  14g R 12.2 33.3  96:11.11 -
MutationStage-3

 11710 cassandr  20   0  261g  42g  14g R 12.2 33.3  86:50.77 -
MutationStage-7

 11730 cassandr  20   0  261g  42g  14g S 12.2 33.3  78:36.04 -
MutationStage-1

 11743 cassandr  20   0  261g  42g  14g R 12.2 33.3  80:27.18 -
MutationStage-1

 11773 cassandr  20   0  261g  42g  14g R 12.2 33.3  79:29.48 -
MutationStage-1

 11800 cassandr  20   0  261g  42g  14g S 12.2 33.3  77:01.39 futex_wai
MutationStage-1

 11830 cassandr  20   0  261g  42g  14g R 12.2 33.3  70:47.18 -
MutationStage-1

 11495 cassandr  20   0  261g  42g  14g R 11.8 33.3   7693:04 -
MutationStage-3

 11675 cassandr  20   0  261g  42g  14g R 11.8 33.3  94:13.22 -
MutationStage-4

 11683 cassandr  20   0  261g  42g  14g S 11.8 33.3  91:42.91 futex_wai
MutationStage-4

 11701 cassandr  20   0  261g  42g  14g S 11.8 33.3  85:16.00 -
MutationStage-7

 11703 cassandr  20   0  261g  42g  14g R 11.8 33.3  88:33.81 -
MutationStage-6

 11725 cassandr  20   0  261g  42g  14g R 11.8 33.3  78:12.70 -
MutationStage-1

 11752 cassandr  20   0  261g  42g  14g S 11.8 33.3  83:25.14 futex_wai
MutationStage-9

 11755 cassandr  20   0  261g  42g  14g R 11.8 33.3  82:38.87 -
MutationStage-9

 11776 cassandr  20   0  261g  42g  14g S 11.8 33.3  79:31.49 futex_wai
MutationStage-1

 11781 cassandr  20   0  261g  42g  14g R 11.8 33.3  75:01.54 -
MutationStage-1

 11796 cassandr  20   0  261g  42g  14g S 11.8 33.3  77:03.78 -
MutationStage-1

 11804 cassandr  20   0  261g  42g  14g R 11.8 33.3  81:38.46 -
MutationStage-1

 11818 cassandr  20   0  261g  42g  14g S 11.8 33.3  76:51.42 -
MutationStage-1

 11823 cassandr  20   0  261g  42g  14g R 11.8 33.3  75:56.69 -
MutationStage-1

 11506 cassandr  20   0  261g  42g  14g R 11.5 33.3 502:50.67 -
MutationStage-1

 11513 cassandr  20   0  261g  42g  14g R 11.5 33.3 140:00.60 -
MutationStage-1

 11515 cassandr  20   0  261g  42g  14g S 11.5 33.3 123:31.16 futex_wai
MutationStage-1

 11676 cassandr  20   0  261g  42g  14g S 11.5 33.3  93:44.36 futex_wai
MutationStage-4

 11680 cassandr  20   0  261g  42g  14g S 11.5 33.3  93:28.55 futex_wai
MutationStage-4

 11706 cassandr  20   0  261g  42g  14g R 11.5 33.3  89:17.10 -
MutationStage-6

 11729 cassandr  20   0  261g  42g  14g R 11.5 33.3  78:42.33 -
MutationStage-1


On Thu, 6 Feb 2020 at 10:17, Elliott Sims  wrote:

> Async-profiler (https://github.com/jvm-profiling-tools/async-profiler )
> flamegraphs can also be a really good tool to figure out the exact
> callgraph that's leading to the futex_wait, both in and out of the JVM.
>

Re: Nodes becoming unresponsive

[Surbhi Gupta]

Sure Eric...

I tried strace as well ...

Nodes becoming unresponsive

[Surbhi Gupta]

Hi,

We have noticed in a Cassandra Cluster , one of the node has 100% cpu
utilization, using top we can see that cassandra process is showing
futex_wait .

We are on CentOS release 6.10 (Final)  .As per below document the futex bug
was on Centos 6.6 .
https://support.datastax.com/hc/en-us/articles/206259833-Nodes-appear-unresponsive-due-to-a-Linux-futex-wait-kernel-bug

Below are the installed patches.

sudo rpm -q --changelog kernel-`uname -r` | grep futex | grep ref

- [kernel] futex: Mention key referencing differences between shared and
private futexes (Larry Woodman) [1167405]

- [kernel] futex: Ensure get_futex_key_refs() always implies a barrier
(Larry Woodman) [1167405]

- [kernel] futex: Fix errors in nested key ref-counting (Denys Vlasenko)
[1094458] {CVE-2014-0205}

- [kernel] futex_lock_pi() key refcnt fix (Danny Feng) [566347]
{CVE-2010-0623}

top - 21:23:34 up 93 days, 10:43,  1 user,  load average: 137.35, 147.74,
148.52

Tasks: 658 total,   1 running, 657 sleeping,   0 stopped,   0 zombie

Cpu(s): 93.9%us,  1.9%sy,  2.0%ni,  2.0%id,  0.0%wa,  0.0%hi,  0.2%si,
0.0%st

Mem:  132236016k total, 129681568k used,  2554448k free,   215888k buffers

Swap:0k total,0k used,0k free, 93679880k cached


   PID USER  PR  NI  VIRT  RES  SHR S %CPU %MEMTIME+  WCHAN
COMMAND


  7725 cassandr  20   0  258g  40g  13g S 2302.0 32.4 305169:26 futex_wai
java


 69075 logstash  39  19 10.5g 1.5g  14m S 42.1  1.2   6763:00 futex_wai
java


 30008 root  20   0  465m  55m  11m S 11.5  0.0   0:02.78 poll_sche
TaniumClient


 31785 cassandr  20   0 34.9g  31m  10m S  4.9  0.0   0:00.15 futex_wai
java


  5154 root  20   0 1523m 6260 1300 S  3.0  0.0   1073:05 hrtimer_n
collectd


  1129 root  20   0 000 S  1.3  0.0 294:55.87 kjournald
jbd2/dm-0-8


 64173 root  20   0 1512m  71m  13m S  1.3  0.1   0:55.69 futex_wai
TaniumClient

Any idea , what else can be looked for high CPU issue?

Thanks
Surbhi

Re: How to read content of hints file and apply them manually?

[Surbhi Gupta]

So this problem we face is , every time a node goes down or a node is under
high load or CPU. We see lots of hints piles up and doesn’t apply on the
other nodes. Last time when this happened we noticed, high pending
mutations but when I have gone back and checked the history of events , not
every time we see high pending mutations. So basically high load and cpu
caused high pending mutations however I feel it was not the vice versa.

Using top command it was very clear that Cassandra is the cause of the high
cpu.

Other than too, iostat, iotop what tools use you use to dig into high load
and high cpu issue ?

On Tue, Jan 28, 2020 at 1:12 PM Patrick McFadin  wrote:

> I would definitely check the IO stats then, If you see latency going over
> 20ms, you need to solve that problem.
>
> Patrick
>
> On Tue, Jan 28, 2020 at 12:01 PM Surbhi Gupta 
> wrote:
>
>> We have also noticed a lot of MutationStage pending .
>>
>>
>> On Tue, 28 Jan 2020 at 11:06, Richard Andersen 
>> wrote:
>>
>>> I am in agreement with Patrick, this is a typical symptom of saturated
>>> IO. Are there a high of drops and/or pending compactions?
>>>
>>> Get Outlook for Android <https://aka.ms/ghei36>
>>> --
>>> *From:* Patrick McFadin 
>>> *Sent:* Tuesday, January 28, 2020 11:25:49 AM
>>> *To:* user@cassandra.apache.org 
>>> *Subject:* Re: How to read content of hints file and apply them
>>> manually?
>>>
>>> Just to add in here. Any time I see any hints on a cluster, that's like
>>> seeing smoke. If you can't explain it, you have a fire somewhere and it's
>>> not going to get any better.
>>>
>>> By the few messages I've seen, I would start by looking at your IO
>>> subsystem on your nodes. Do you have enough throughput to write and read at
>>> the same time? These are exactly the symptoms I see when running Cassandra
>>> on a SAN or NAS.
>>>
>>> Patrick
>>>
>>> On Mon, Jan 27, 2020 at 8:17 PM Surbhi Gupta 
>>> wrote:
>>>
>>> We tried to tune sethintedhandoffthrottlekb to 100 , 1024 , 10240 but
>>> nothing helped .
>>> Our hints related parameters are as below, if you don't find any
>>> parameter below then it is not set in our environment and should be of the
>>> default value.
>>>
>>> max_hint_window_in_ms: 1080 # 3 hours
>>>
>>> hinted_handoff_enabled: true
>>>
>>> hinted_handoff_throttle_in_kb: 100
>>>
>>> max_hints_delivery_threads: 8
>>>
>>> hints_directory: /var/lib/cassandra/hints
>>>
>>> hints_flush_period_in_ms: 1
>>>
>>> max_hints_file_size_in_mb: 128
>>>
>>> On Mon, 27 Jan 2020 at 18:34, Jeff Jirsa  wrote:
>>>
>>>
>>> The high cpu is probably the hints getting replayed slamming the write
>>> path
>>>
>>> Slowing it down with the hint throttle may help
>>>
>>> It’s not instant.
>>>
>>> On Jan 27, 2020, at 6:05 PM, Erick Ramirez  wrote:
>>>
>>> 
>>>
>>> Increase the max_hint_window_in_ms setting in cassandra.yaml to more
>>> than 3 hours, perhaps 6 hours. If the issue still persists networking may
>>> need to be tested for bandwidth issues.
>>>
>>>
>>> Just a note of warning about bumping up the hint window without
>>> understanding the pros and cons. Be aware that doubling it means:
>>>
>>>- you'll end up doubling the size of stored hints in
>>>the hints_directory
>>>- there'll be twice as much hints to replay when node(s) come back
>>>online
>>>
>>> There's always 2 sides to fiddling with the knobs in C*. Cheers!
>>>
>>>

Re: How to read content of hints file and apply them manually?

[Surbhi Gupta]

We have also noticed a lot of MutationStage pending .


On Tue, 28 Jan 2020 at 11:06, Richard Andersen 
wrote:

> I am in agreement with Patrick, this is a typical symptom of saturated IO.
> Are there a high of drops and/or pending compactions?
>
> Get Outlook for Android <https://aka.ms/ghei36>
> --
> *From:* Patrick McFadin 
> *Sent:* Tuesday, January 28, 2020 11:25:49 AM
> *To:* user@cassandra.apache.org 
> *Subject:* Re: How to read content of hints file and apply them manually?
>
> Just to add in here. Any time I see any hints on a cluster, that's like
> seeing smoke. If you can't explain it, you have a fire somewhere and it's
> not going to get any better.
>
> By the few messages I've seen, I would start by looking at your IO
> subsystem on your nodes. Do you have enough throughput to write and read at
> the same time? These are exactly the symptoms I see when running Cassandra
> on a SAN or NAS.
>
> Patrick
>
> On Mon, Jan 27, 2020 at 8:17 PM Surbhi Gupta 
> wrote:
>
> We tried to tune sethintedhandoffthrottlekb to 100 , 1024 , 10240 but
> nothing helped .
> Our hints related parameters are as below, if you don't find any parameter
> below then it is not set in our environment and should be of the default
> value.
>
> max_hint_window_in_ms: 1080 # 3 hours
>
> hinted_handoff_enabled: true
>
> hinted_handoff_throttle_in_kb: 100
>
> max_hints_delivery_threads: 8
>
> hints_directory: /var/lib/cassandra/hints
>
> hints_flush_period_in_ms: 1
>
> max_hints_file_size_in_mb: 128
>
> On Mon, 27 Jan 2020 at 18:34, Jeff Jirsa  wrote:
>
>
> The high cpu is probably the hints getting replayed slamming the write
> path
>
> Slowing it down with the hint throttle may help
>
> It’s not instant.
>
> On Jan 27, 2020, at 6:05 PM, Erick Ramirez  wrote:
>
> 
>
> Increase the max_hint_window_in_ms setting in cassandra.yaml to more than
> 3 hours, perhaps 6 hours. If the issue still persists networking may need
> to be tested for bandwidth issues.
>
>
> Just a note of warning about bumping up the hint window without
> understanding the pros and cons. Be aware that doubling it means:
>
>- you'll end up doubling the size of stored hints in
>the hints_directory
>- there'll be twice as much hints to replay when node(s) come back
>online
>
> There's always 2 sides to fiddling with the knobs in C*. Cheers!
>
>

Re: How to read content of hints file and apply them manually?

[Surbhi Gupta]

We tried to tune sethintedhandoffthrottlekb to 100 , 1024 , 10240 but
nothing helped .
Our hints related parameters are as below, if you don't find any parameter
below then it is not set in our environment and should be of the default
value.

max_hint_window_in_ms: 1080 # 3 hours

hinted_handoff_enabled: true

hinted_handoff_throttle_in_kb: 100

max_hints_delivery_threads: 8

hints_directory: /var/lib/cassandra/hints

hints_flush_period_in_ms: 1

max_hints_file_size_in_mb: 128

On Mon, 27 Jan 2020 at 18:34, Jeff Jirsa  wrote:

>
> The high cpu is probably the hints getting replayed slamming the write path
>
> Slowing it down with the hint throttle may help
>
> It’s not instant.
>
> On Jan 27, 2020, at 6:05 PM, Erick Ramirez  wrote:
>
> 
>
>> Increase the max_hint_window_in_ms setting in cassandra.yaml to more than
>> 3 hours, perhaps 6 hours. If the issue still persists networking may need
>> to be tested for bandwidth issues.
>>
>
> Just a note of warning about bumping up the hint window without
> understanding the pros and cons. Be aware that doubling it means:
>
>- you'll end up doubling the size of stored hints in
>the hints_directory
>- there'll be twice as much hints to replay when node(s) come back
>online
>
> There's always 2 sides to fiddling with the knobs in C*. Cheers!
>
>

Re: How to read content of hints file and apply them manually?

[Surbhi Gupta]

Why we think it might be related to hints is , because if we truncate the
hints then load goes normal on the nodes.
FYI , We had to run repair after truncating hints.
Any thoughts ?


On Mon, 27 Jan 2020 at 15:27, Deepak Vohra 
wrote:

>
> Hints are a stopgap measure and not a fix to the underlying issue. Run a
> full repair.
> On Monday, January 27, 2020, 10:17:01 p.m. UTC, Surbhi Gupta <
> surbhi.gupt...@gmail.com> wrote:
>
>
> Hi,
>
> We are on Open source 3.11 .
> We have a issue in one of the cluster where lots of hints gets piled up
> and they don't get applied within hinted handoff period ( 3 hour in our
> case) .
> And load and CPU of the server goes very high.
> We see lot of messages   in system.log and debug.log . Our read repair
> chance and dc_local_repair chance is 0.1 . Any pointers are welcome .
>
> ERROR [ReadRepairStage:83] 2020-01-27 13:08:43,695
> CassandraDaemon.java:228 - Exception in thread
> Thread[ReadRepairStage:83,5,main]
>
> org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out
> - received only 0 responses.
>
> DEBUG [ReadRepairStage:111] 2020-01-27 13:10:06,663 ReadCallback.java:242
> - Digest mismatch:
>
> org.apache.cassandra.service.DigestMismatchException: Mismatch for key
> DecoratedKey(4759131696153881383, 9a21276d0af64de28d5d3023b69e)
> (142a55e1e28de7daa2ddc34a361
>
> 474a0 vs fcba30f022ef25f456914c341022963d)
>

How to read content of hints file and apply them manually?

[Surbhi Gupta]

Hi,

We are on Open source 3.11 .
We have a issue in one of the cluster where lots of hints gets piled up and
they don't get applied within hinted handoff period ( 3 hour in our case) .
And load and CPU of the server goes very high.
We see lot of messages   in system.log and debug.log . Our read repair
chance and dc_local_repair chance is 0.1 . Any pointers are welcome .

ERROR [ReadRepairStage:83] 2020-01-27 13:08:43,695 CassandraDaemon.java:228
- Exception in thread Thread[ReadRepairStage:83,5,main]

org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out -
received only 0 responses.

DEBUG [ReadRepairStage:111] 2020-01-27 13:10:06,663 ReadCallback.java:242 -
Digest mismatch:

org.apache.cassandra.service.DigestMismatchException: Mismatch for key
DecoratedKey(4759131696153881383, 9a21276d0af64de28d5d3023b69e)
(142a55e1e28de7daa2ddc34a361

474a0 vs fcba30f022ef25f456914c341022963d)

Re: Cassandra is not showing a node up hours after restart

[Surbhi Gupta]

Before Cassandra shutdown, nodetool drain should be executed first. As soon
as you do nodetool drain, others node will see this node down and no new
traffic will come to this node.
I generally gives 10 seconds gap between nodetool drain and Cassandra stop.

On Sun, Nov 24, 2019 at 9:52 AM Paul Mena  wrote:

> Thank you for the replies. I had made no changes to the config before the
> rolling restart.
>
>
> I can try another restart but was wondering if I should do it differently.
> I had simply done "service cassandra stop" followed by "service cassandra
> start".  Since then I've seen some suggestions to proceed the shutdown with
> "nodetool disablegossip" and/or "nodetool drain". Are these commands
> advisable? Are any other commands recommended either before the shutdown or
> after the startup?
>
>
> Thanks again!
>
>
> Paul
> --
> *From:* Naman Gupta 
> *Sent:* Sunday, November 24, 2019 11:18:14 AM
> *To:* user@cassandra.apache.org
> *Subject:* Re: Cassandra is not showing a node up hours after restart
>
> Did you change the name of datacenter or any other config changes before
> the rolling restart?
>
> On Sun, Nov 24, 2019 at 8:49 PM Paul Mena  wrote:
>
>> I am in the process of doing a rolling restart on a 4-node cluster
>> running Cassandra 2.1.9. I stopped and started Cassandra on node 1 via
>> "service cassandra stop/start", and noted nothing unusual in either
>> system.log or cassandra.log. Doing a "nodetool status" from node 1 shows
>> all four nodes up:
>>
>> user@node001=> nodetool status
>> Datacenter: datacenter1
>>
>> ===
>> Status=Up/Down
>> |/ State=Normal/Leaving/Joining/Moving
>> --  Address  Load   Tokens  OwnsHost ID  
>>  Rack
>> UN  192.168.187.121  538.95 GB  256 ?   
>> c99cf581-f4ae-4aa9-ab37-1a114ab2429b  rack1
>> UN  192.168.187.122  630.72 GB  256 ?   
>> bfa07f47-7e37-42b4-9c0b-024b3c02e93f  rack1
>> UN  192.168.187.123  572.73 GB  256 ?   
>> 273df9f3-e496-4c65-a1f2-325ed288a992  rack1
>> UN  192.168.187.124  625.05 GB  256 ?   
>> b8639cf1-5413-4ece-b882-2161bbb8a9c3  rack1
>>
>> But doing the same command from any other of the 3 nodes shows node 1
>> still down.
>>
>> user@node002=> nodetool status
>> Datacenter: datacenter1
>> ===
>> Status=Up/Down
>> |/ State=Normal/Leaving/Joining/Moving
>> --  Address  Load   Tokens  OwnsHost ID  
>>  Rack
>> DN  192.168.187.121  538.94 GB  256 ?   
>> c99cf581-f4ae-4aa9-ab37-1a114ab2429b  rack1
>> UN  192.168.187.122  630.72 GB  256 ?   
>> bfa07f47-7e37-42b4-9c0b-024b3c02e93f  rack1
>> UN  192.168.187.123  572.73 GB  256 ?   
>> 273df9f3-e496-4c65-a1f2-325ed288a992  rack1
>> UN  192.168.187.124  625.04 GB  256 ?   
>> b8639cf1-5413-4ece-b882-2161bbb8a9c3  rack1
>>
>> Is there something I can do to remedy this current situation - so that I
>> can continue with the rolling restart?
>>
>>

Re: Cassandra is not showing a node up hours after restart

[Surbhi Gupta]

It sounds silly but sometimes restarting again the node which is showing
down from other nodes fix the issue. This looks like a gossip issue.

On Sun, Nov 24, 2019 at 7:19 AM Paul Mena  wrote:

> I am in the process of doing a rolling restart on a 4-node cluster running
> Cassandra 2.1.9. I stopped and started Cassandra on node 1 via "service
> cassandra stop/start", and noted nothing unusual in either system.log or
> cassandra.log. Doing a "nodetool status" from node 1 shows all four nodes
> up:
>
> user@node001=> nodetool status
> Datacenter: datacenter1
>
> ===
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> --  Address  Load   Tokens  OwnsHost ID   
> Rack
> UN  192.168.187.121  538.95 GB  256 ?   
> c99cf581-f4ae-4aa9-ab37-1a114ab2429b  rack1
> UN  192.168.187.122  630.72 GB  256 ?   
> bfa07f47-7e37-42b4-9c0b-024b3c02e93f  rack1
> UN  192.168.187.123  572.73 GB  256 ?   
> 273df9f3-e496-4c65-a1f2-325ed288a992  rack1
> UN  192.168.187.124  625.05 GB  256 ?   
> b8639cf1-5413-4ece-b882-2161bbb8a9c3  rack1
>
> But doing the same command from any other of the 3 nodes shows node 1
> still down.
>
> user@node002=> nodetool status
> Datacenter: datacenter1
> ===
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> --  Address  Load   Tokens  OwnsHost ID   
> Rack
> DN  192.168.187.121  538.94 GB  256 ?   
> c99cf581-f4ae-4aa9-ab37-1a114ab2429b  rack1
> UN  192.168.187.122  630.72 GB  256 ?   
> bfa07f47-7e37-42b4-9c0b-024b3c02e93f  rack1
> UN  192.168.187.123  572.73 GB  256 ?   
> 273df9f3-e496-4c65-a1f2-325ed288a992  rack1
> UN  192.168.187.124  625.04 GB  256 ?   
> b8639cf1-5413-4ece-b882-2161bbb8a9c3  rack1
>
> Is there something I can do to remedy this current situation - so that I
> can continue with the rolling restart?
>
>

Re: Cassandra-stress testing

[Surbhi Gupta]

Have you tried ycsa?
It is a tool from yahoo for stress testing nosql databases.

On Tue, Aug 20, 2019 at 3:34 AM  wrote:

> Hi Everyone,
>
>
>
> Anyone before who have bused Cassandra-stress. I want to test if it’s
> possible to load 600 milllions records per hour in Cassandra or
>
> Find a better way to optimize Cassandra for this case.
>
> Any help will be highly appreciated.
>
>
>
> Sent from Mail  for Window
>

Re: can i...

[Surbhi Gupta]

Send the details

On Thu, Mar 7, 2019 at 8:45 AM Nick Hatfield 
wrote:

> Use this email to get some insight on how to fix database issues in our
> cluster?
>

Re: Cassandra | Cross Data Centre Replication Status

[Surbhi Gupta]

Repair will take way more time then rebuild.

On Wed, Oct 31, 2018 at 6:45 AM Kiran mk  wrote:

> Run the repair with -pr option on each node which will repair only the
>
> parition range.
>
>
>
> nodetool repair -pr
>
> On Wed, Oct 31, 2018 at 7:04 PM Surbhi Gupta 
> wrote:
>
> >
>
> > Nodetool repair will take way more time than nodetool rebuild.
>
> > How much data u have in your original data center?
>
> > Repair should be run to make the data consistent in case of node down
> more than hintedhandoff period and dropped mutations.
>
> > But as a thumb rule ,generally we run repair using opscenter (if using
> Datastax) most of the times.
>
> >
>
> > So in your case run “nodetool rebuild ” on all the
> nodes in new data center.
>
> > For making the rebuild process fast, increase three parameters,
> compaction throughput , stream throughput and interdcstream  throughput.
>
> >
>
> > Thanks
>
> > Surbhi
>
> > On Tue, Oct 30, 2018 at 11:29 PM Akshay Bhardwaj <
> akshay.bhardwaj1...@gmail.com> wrote:
>
> >>
>
> >> Hi Jonathan,
>
> >>
>
> >> That makes sense. Thank you for the explanation.
>
> >>
>
> >> Another quick question, as the cluster is still operative and the data
> for the past 2 weeks (since updating replication factor) is present in both
> the data centres, should I run "nodetool rebuild" or "nodetool repair"?
>
> >>
>
> >> I read that nodetool rebuild is faster and is useful till the new data
> centre is empty and no partition keys are present. So when is the good time
> to use either of the commands and what impact can it have on the data
> centre operations?
>
> >>
>
> >> Thanks and Regards
>
> >>
>
> >> Akshay Bhardwaj
>
> >> +91-97111-33849
>
> >>
>
> >>
>
> >> On Wed, Oct 31, 2018 at 2:34 AM Jonathan Haddad 
> wrote:
>
> >>>
>
> >>> You need to run "nodetool rebuild -- " on each node
> in the new DC to get the old data to replicate.  It doesn't do it
> automatically because Cassandra has no way of knowing if you're done adding
> nodes and if it were to migrate automatically, it could cause a lot of
> problems. Imagine streaming 100 nodes data to 3 nodes in the new DC, not
> fun.
>
> >>>
>
> >>> On Tue, Oct 30, 2018 at 1:59 PM Akshay Bhardwaj <
> akshay.bhardwaj1...@gmail.com> wrote:
>
> >>>>
>
> >>>> Hi Experts,
>
> >>>>
>
> >>>> I previously had 1 Cassandra data centre in AWS Singapore region with
> 5 nodes, with my keyspace's replication factor as 3 in Network topology.
>
> >>>>
>
> >>>> After this cluster has been running smoothly for 4 months (500 GB of
> data on each node's disk), I added 2nd data centre in AWS Mumbai region
> with yet again 5 nodes in Network topology.
>
> >>>>
>
> >>>> After updating my keyspace's replication factor to
> {"AWS_Sgp":3,"AWS_Mum":3}, my expectation was that the data present in Sgp
> region will immediately start replicating on the Mum region's nodes.
> However even after 2 weeks I do not see historical data to be replicated,
> but new data being written on Sgp region is present in Mum region as well.
>
> >>>>
>
> >>>> Any help or suggestions to debug this issue will be highly
> appreciated.
>
> >>>>
>
> >>>> Regards
>
> >>>> Akshay Bhardwaj
>
> >>>> +91-97111-33849
>
> >>>>
>
> >>>>
>
> >>>
>
> >>>
>
> >>> --
>
> >>> Jon Haddad
>
> >>> http://www.rustyrazorblade.com
>
> >>> twitter: rustyrazorblade
>
> >>>
>
> >>>
>
> >>
>
> >>
>
>
>
>
>
> --
>
> Best Regards,
>
> Kiran.M.K.
>
>
>
> -
>
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
>
> For additional commands, e-mail: user-h...@cassandra.apache.org
>
>
>
>

Re: Cassandra | Cross Data Centre Replication Status

[Surbhi Gupta]

Nodetool repair will take way more time than nodetool rebuild.
How much data u have in your original data center?
Repair should be run to make the data consistent in case of node down more
than hintedhandoff period and dropped mutations.
But as a thumb rule ,generally we run repair using opscenter (if using
Datastax) most of the times.

So in your case run “nodetool rebuild ” on all the
nodes in new data center.
For making the rebuild process fast, increase three parameters, compaction
throughput , stream throughput and interdcstream  throughput.

Thanks
Surbhi
On Tue, Oct 30, 2018 at 11:29 PM Akshay Bhardwaj <
akshay.bhardwaj1...@gmail.com> wrote:

> Hi Jonathan,
>
> That makes sense. Thank you for the explanation.
>
> Another quick question, as the cluster is still operative and the data for
> the past 2 weeks (since updating replication factor) is present in both the
> data centres, should I run "nodetool rebuild" or "nodetool repair"?
>
> I read that nodetool rebuild is faster and is useful till the new data
> centre is empty and no partition keys are present. So when is the good time
> to use either of the commands and what impact can it have on the data
> centre operations?
>
> Thanks and Regards
>
> Akshay Bhardwaj
> +91-97111-33849
>
>
> On Wed, Oct 31, 2018 at 2:34 AM Jonathan Haddad  wrote:
>
>> You need to run "nodetool rebuild -- " on each node in
>> the new DC to get the old data to replicate.  It doesn't do it
>> automatically because Cassandra has no way of knowing if you're done adding
>> nodes and if it were to migrate automatically, it could cause a lot of
>> problems. Imagine streaming 100 nodes data to 3 nodes in the new DC, not
>> fun.
>>
>> On Tue, Oct 30, 2018 at 1:59 PM Akshay Bhardwaj <
>> akshay.bhardwaj1...@gmail.com> wrote:
>>
>>> Hi Experts,
>>>
>>> I previously had 1 Cassandra data centre in AWS Singapore region with 5
>>> nodes, with my keyspace's replication factor as 3 in Network topology.
>>>
>>> After this cluster has been running smoothly for 4 months (500 GB of
>>> data on each node's disk), I added 2nd data centre in AWS Mumbai region
>>> with yet again 5 nodes in Network topology.
>>>
>>> After updating my keyspace's replication factor to
>>> {"AWS_Sgp":3,"AWS_Mum":3}, my expectation was that the data present in Sgp
>>> region will immediately start replicating on the Mum region's nodes.
>>> However even after 2 weeks I do not see historical data to be replicated,
>>> but new data being written on Sgp region is present in Mum region as well.
>>>
>>> Any help or suggestions to debug this issue will be highly appreciated.
>>>
>>> Regards
>>> Akshay Bhardwaj
>>> +91-97111-33849
>>>
>>>
>>>
>>
>> --
>> Jon Haddad
>> http://www.rustyrazorblade.com
>> twitter: rustyrazorblade
>>
>>
>>
>
>

Re: Read timeouts when performing rolling restart

[Surbhi Gupta]

Another thing to notice is :

system_auth WITH replication = {'class': 'SimpleStrategy',
'replication_factor': '1'}

system_auth has a replication factor of 1 and even if one node is down it
may impact the system because of the replication factor.



On Wed, 12 Sep 2018 at 09:46, Steinmaurer, Thomas <
thomas.steinmau...@dynatrace.com> wrote:

> Hi,
>
>
>
> I remember something that a client using the native protocol gets notified
> too early by Cassandra being ready due to the following issue:
>
> https://issues.apache.org/jira/browse/CASSANDRA-8236
>
>
>
> which looks similar, but above was marked as fixed in 2.2.
>
>
>
> Thomas
>
>
>
> *From:* Riccardo Ferrari 
> *Sent:* Mittwoch, 12. September 2018 18:25
> *To:* user@cassandra.apache.org
> *Subject:* Re: Read timeouts when performing rolling restart
>
>
>
> Hi Alain,
>
>
>
> Thank you for chiming in!
>
>
>
> I was thinking to perform the 'start_native_transport=false' test as well
> and indeed the issue is not showing up. Starting the/a node with native
> transport disabled and letting it cool down lead to no timeout exceptions
> no dropped messages, simply a crystal clean startup. Agreed it is a
> workaround
>
>
>
> # About upgrading:
>
> Yes, I desperately want to upgrade despite is a long and slow task. Just
> reviewing all the changes from 3.0.6 to 3.0.17
> is going to be a huge pain, top of your head, any breaking change I should
> absolutely take care of reviewing ?
>
>
>
> # describecluster output: YES they agree on the same schema version
>
>
>
> # keyspaces:
>
> system WITH replication = {'class': 'LocalStrategy'}
>
> system_schema WITH replication = {'class': 'LocalStrategy'}
>
> system_auth WITH replication = {'class': 'SimpleStrategy',
> 'replication_factor': '1'}
>
> system_distributed WITH replication = {'class': 'SimpleStrategy',
> 'replication_factor': '3'}
>
> system_traces WITH replication = {'class': 'SimpleStrategy',
> 'replication_factor': '2'}
>
>
>
>  WITH replication = {'class': 'SimpleStrategy',
> 'replication_factor': '3'}
>
>   WITH replication = {'class': 'SimpleStrategy',
> 'replication_factor': '3'}
>
>
>
> # Snitch
>
> Ec2Snitch
>
>
>
> ## About Snitch and replication:
>
> - We have the default DC and all nodes are in the same RACK
>
> - We are planning to move to GossipingPropertyFileSnitch configuring the
> cassandra-rackdc accortingly.
>
> -- This should be a transparent change, correct?
>
>
>
> - Once switched to GPFS, we plan to move to 'NetworkTopologyStrategy' with
> 'us-' DC and replica counts as before
>
> - Then adding a new DC inside the VPC, but this is another story...
>
>
>
> Any concerns here ?
>
>
>
> # nodetool status 
>
> --  Address Load   Tokens   Owns (effective)  Host
> ID   Rack
> UN  10.x.x.a  177 GB 256  50.3%
> d8bfe4ad-8138-41fe-89a4-ee9a043095b5  rr
> UN  10.x.x.b152.46 GB  256  51.8%
> 7888c077-346b-4e09-96b0-9f6376b8594f  rr
> UN  10.x.x.c   159.59 GB  256  49.0%
> 329b288e-c5b5-4b55-b75e-fbe9243e75fa  rr
> UN  10.x.x.d  162.44 GB  256  49.3%
> 07038c11-d200-46a0-9f6a-6e2465580fb1  rr
> UN  10.x.x.e174.9 GB   256  50.5%
> c35b5d51-2d14-4334-9ffc-726f9dd8a214  rr
> UN  10.x.x.f  194.71 GB  256  49.2%
> f20f7a87-d5d2-4f38-a963-21e24167b8ac  rr
>
>
>
> # gossipinfo
>
> /10.x.x.a
>   STATUS:827:NORMAL,-1350078789194251746
>   LOAD:289986:1.90078037902E11
>   SCHEMA:281088:af4461c3-d269-39bc-9d03-3566031c1e0a
>   DC:6:
>   RACK:8:rr
>   RELEASE_VERSION:4:3.0.6
>   SEVERITY:290040:0.5934718251228333
>   NET_VERSION:1:10
>   HOST_ID:2:d8bfe4ad-8138-41fe-89a4-ee9a043095b5
>   RPC_READY:868:true
>   TOKENS:826:
> /10.x.x.b
>   STATUS:16:NORMAL,-1023229528754013265
>   LOAD:7113:1.63730480619E11
>   SCHEMA:10:af4461c3-d269-39bc-9d03-3566031c1e0a
>   DC:6:
>   RACK:8:rr
>   RELEASE_VERSION:4:3.0.6
>   SEVERITY:7274:0.5988024473190308
>   NET_VERSION:1:10
>   HOST_ID:2:7888c077-346b-4e09-96b0-9f6376b8594f
>   TOKENS:15:
> /10.x.x.c
>   STATUS:732:NORMAL,-111717275923547
>   LOAD:245839:1.71409806942E11
>   SCHEMA:237168:af4461c3-d269-39bc-9d03-3566031c1e0a
>   DC:6:
>   RACK:8:rr
>   RELEASE_VERSION:4:3.0.6
>   SEVERITY:245989:0.0
>   NET_VERSION:1:10
>   HOST_ID:2:329b288e-c5b5-4b55-b75e-fbe9243e75fa
>   RPC_READY:763:true
>   TOKENS:731:
> /10.x.x.d
>   STATUS:14:NORMAL,-1004942496246544417
>   LOAD:313125:1.74447964917E11
>   SCHEMA:304268:af4461c3-d269-39bc-9d03-3566031c1e0a
>   DC:6:
>   RACK:8:rr
>   RELEASE_VERSION:4:3.0.6
>   SEVERITY:313215:0.25641027092933655
>   NET_VERSION:1:10
>   HOST_ID:2:07038c11-d200-46a0-9f6a-6e2465580fb1
>   RPC_READY:56:true
>   TOKENS:13:
> /10.x.x.e
>   STATUS:520:NORMAL,-1058809960483771749
>   LOAD:276118:1.87831573032E11
>   SCHEMA:267327:af4461c3-d269-39bc-9d03-3566031c1e0a
>   DC:6:
>   RACK:8:rr
>   RELEASE_VERSION:4:3.0.6
>   SEVERITY:276217:0.32786884903907776
>   NET_VERSION:1:10
>   HOST_ID:2:c35b5d51-2d14-4334-9ffc-726f9dd8a214
>   

Re: nodetool rebuild

[Surbhi Gupta]

Increase 3 throughput
Compaction throughput
Stream throughput
Interdcstream throughput (if rebuilding from another DC)

Make all of the above to 0 and see if there is any improvement and later
set the value if u can’t leave these values to 0.

On Wed, Sep 12, 2018 at 5:42 AM Vitali Dyachuk  wrote:

> Hi,
> I'm currently streaming data with nodetool rebuild on 2 nodes, each node
> is streaming from different location. The problem is that it takes ~7 days
> to stream 4Tb of data to 1 node, the speed on each side is ~150Mbit/s  so
> it should take around
> ~2,5 days . Although there are resources on the destnodes and in the
> source regions.
> I've increased stream throughput, but its only affects outbound
> connections.
> Tested with iperf the bandwidth is 600Mibt/s from both sides. Last week
> i've changed the CS from ST to LC because of huge sstables and compaction
> of them is still ongoing.
> How does rebuild command works ? Does it calculate the range then request
> the needed sstables from that node and start streaming ? How is it possible
> to speed up the streaming ?
>
> Vitali.
>

Re: Log application Queries

[Surbhi Gupta]

nodeool setlogginglevel is only valid for below :


   - org.apache.cassandra
   - org.apache.cassandra.db
   - org.apache.cassandra.service.StorageProxy


On 25 May 2018 at 09:01, Nitan Kainth <nitankai...@gmail.com> wrote:

> Thanks Surbhi. I found another way. I used nodetool settraceprobability 1
> and it is logging in system_traces.
>
> How is it different from nodeool setlogginglevel?
>
>
> Regards,
> Nitan K.
> Cassandra and Oracle Architect/SME
> Datastax Certified Cassandra expert
> Oracle 10g Certified
>
> On Fri, May 25, 2018 at 11:41 AM, Surbhi Gupta <surbhi.gupt...@gmail.com>
> wrote:
>
>> If using dse then u can enable in dse.yaml.
>>
>> # CQL slow log settings
>> cql_slow_log_options:
>> enabled: true
>> threshold_ms: 0
>> ttl_seconds: 259200
>>
>> As far as my understanding says setlogginglevel  is used for changing
>> the logging level as below but not for slow query .
>>
>>- ALL
>>- TRACE
>>- DEBUG
>>- INFO
>>- WARN
>>- ERROR
>>- OFF
>>
>>
>>
>> On 25 May 2018 at 08:24, Nitan Kainth <nitankai...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I would like to log all C* queries hitting cluster. Could someone please
>>> tell me how can I do it at cluster level?
>>> Will nodetool setlogginglevel work? If so, please share example with
>>> library name.
>>>
>>> C* version 3.11
>>>
>>
>>
>

Re: Log application Queries

[Surbhi Gupta]

If using dse then u can enable in dse.yaml.

# CQL slow log settings
cql_slow_log_options:
enabled: true
threshold_ms: 0
ttl_seconds: 259200

As far as my understanding says setlogginglevel  is used for changing the
logging level as below but not for slow query .

   - ALL
   - TRACE
   - DEBUG
   - INFO
   - WARN
   - ERROR
   - OFF



On 25 May 2018 at 08:24, Nitan Kainth  wrote:

> Hi,
>
> I would like to log all C* queries hitting cluster. Could someone please
> tell me how can I do it at cluster level?
> Will nodetool setlogginglevel work? If so, please share example with
> library name.
>
> C* version 3.11
>

Re: Question About Reaper

[Surbhi Gupta]

Do we have to setup the reaper on one of the node where Cassandra cluster
is running?
We are using a separate node where we have the connectivity to the
Cassandra cluster .

We have tried with the certificate settings in
/usr/local/bin/cassandra-reaper

We have put below in /usr/local/bin/cassandra-reaper

JVM_OPTS="$JVM_OPTS -Dssl.enable=true
-Djavax.net.ssl.keyStore=/etc/dse/cassandra/keystores/server-keystore.jks
-Djavax.net.ssl.keyStorePassword=xx
-Djavax.net.ssl.trustStore=/etc/dse/cassandra/keystores/server-truststore.jks
-Djavax.net.ssl.trustStorePassword=xxx"

But still getting below error:

Exception in thread "main" java.lang.IllegalStateException: Cannot
initialize SSL Context

at
com.datastax.driver.core.JdkSSLOptions.makeDefaultContext(JdkSSLOptions.java:81)

at com.datastax.driver.core.JdkSSLOptions.(JdkSSLOptions.java:49)

at
com.datastax.driver.core.JdkSSLOptions$Builder.build(JdkSSLOptions.java:128)

at
systems.composable.dropwizard.cassandra.ssl.JDKSSLOptionsFactory.build(JDKSSLOptionsFactory.java:15)

at java.util.Optional.map(Optional.java:215)

at
systems.composable.dropwizard.cassandra.CassandraFactory.build(CassandraFactory.java:477)

at
systems.composable.dropwizard.cassandra.CassandraFactory.build(CassandraFactory.java:447)

at
io.cassandrareaper.storage.CassandraStorage.(CassandraStorage.java:140)

at
io.cassandrareaper.ReaperApplication.initializeStorage(ReaperApplication.java:235)

at io.cassandrareaper.ReaperApplication.run(ReaperApplication.java:140)

at io.cassandrareaper.ReaperApplication.run(ReaperApplication.java:67)

at io.dropwizard.cli.EnvironmentCommand.run(EnvironmentCommand.java:43)

at io.dropwizard.cli.ConfiguredCommand.run(ConfiguredCommand.java:85)

at io.dropwizard.cli.Cli.run(Cli.java:75)

at io.dropwizard.Application.run(Application.java:79)

at io.cassandrareaper.ReaperApplication.main(ReaperApplication.java:87)

Caused by: java.security.NoSuchAlgorithmException: Error constructing
implementation (algorithm: Default, provider: SunJSSE, class:
sun.security.ssl.SSLContextImpl$DefaultSSLContext)

at java.security.Provider$Service.newInstance(Provider.java:1617)

at sun.security.jca.GetInstance.getInstance(GetInstance.java:236)

at sun.security.jca.GetInstance.getInstance(GetInstance.java:164)

at javax.net.ssl.SSLContext.getInstance(SSLContext.java:156)

at javax.net.ssl.SSLContext.getDefault(SSLContext.java:96)

at
com.datastax.driver.core.JdkSSLOptions.makeDefaultContext(JdkSSLOptions.java:79)

... 15 more

Caused by: java.security.PrivilegedActionException:
java.io.FileNotFoundException:
/etc/dse/cassandra/keystores/server-keystore.jks (No such file or directory)

at java.security.AccessController.doPrivileged(Native Method)

at
sun.security.ssl.SSLContextImpl$DefaultManagersHolder.getKeyManagers(SSLContextImpl.java:822)

at
sun.security.ssl.SSLContextImpl$DefaultManagersHolder.(SSLContextImpl.java:758)

at
sun.security.ssl.SSLContextImpl$DefaultSSLContext.(SSLContextImpl.java:913)

at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)

at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

at java.lang.reflect.Constructor.newInstance(Constructor.java:423)

at java.security.Provider$Service.newInstance(Provider.java:1595)

... 20 more

Caused by: java.io.FileNotFoundException:
/etc/dse/cassandra/keystores/server-keystore.jks (No such file or directory)

at java.io.FileInputStream.open0(Native Method)

at java.io.FileInputStream.open(FileInputStream.java:195)

at java.io.FileInputStream.(FileInputStream.java:138)

at java.io.FileInputStream.(FileInputStream.java:93)

at
sun.security.ssl.SSLContextImpl$DefaultManagersHolder$2.run(SSLContextImpl.java:826)

at
sun.security.ssl.SSLContextImpl$DefaultManagersHolder$2.run(SSLContextImpl.java:823)

On 24 May 2018 at 14:12, Dennis Lovely <d...@aegisco.com> wrote:

> looks like you're connecting to a service listening on SSL but you don't
> have the CA used in your truststore
>
> On Thu, May 24, 2018 at 1:58 PM, Surbhi Gupta <surbhi.gupt...@gmail.com>
> wrote:
>
>> Getting below error:
>>
>> Caused by: sun.security.validator.ValidatorException: PKIX path building
>> failed: sun.security.provider.certpath.SunCertPathBuilderException:
>> unable to find valid certification path to requested target
>>
>> at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:397)
>>
>> at sun.security.validator.PKIXValidator.engineValidate(PKIXVali
>> dator.java:302)
>>
>> at sun.security.validator.Validator.validate(Validator.java:260)
>>
>> at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustMana
>> gerImpl.java:324)
>>
>> at sun.security.ssl.X509TrustManager

Re: Question About Reaper

[Surbhi Gupta]

Getting below error:

Caused by: sun.security.validator.ValidatorException: PKIX path building
failed: sun.security.provider.certpath.SunCertPathBuilderException: unable
to find valid certification path to requested target

at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:397)

at
sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302)

at sun.security.validator.Validator.validate(Validator.java:260)

at
sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324)

at
sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:281)

at
sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136)

at
sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1501)

... 20 common frames omitted

Any thought?

On 24 May 2018 at 10:35, Surbhi Gupta <surbhi.gupt...@gmail.com> wrote:

> Another question, We use 9142 cqlsh port in one of the datacenter and on
> other datacenter we use 9042 port.
> How should we configure this ?
>
> On 24 May 2018 at 10:22, Surbhi Gupta <surbhi.gupt...@gmail.com> wrote:
>
>> What is the impact of
>> PARALLEL - all replicas at the same time ?
>> Will it make repair faster,?
>> Do we expect more CPU , Load and memory usage in case if we use Parallel
>> , compare to other settings ?
>>
>>
>>
>> On 21 May 2018 at 22:55, Alexander Dejanovski <a...@thelastpickle.com>
>> wrote:
>>
>>> You won't be able to have less segments than vnodes, so just use 256
>>> segments per node, use parallel as repair parallelism, and set intensity to
>>> 1.
>>>
>>> You apparently have more than 3TB per node, and that kind of density is
>>> always challenging when it comes to run "fast" repairs.
>>>
>>> Cheers,
>>>
>>> Le mar. 22 mai 2018 à 07:28, Surbhi Gupta <surbhi.gupt...@gmail.com> a
>>> écrit :
>>>
>>>> We are on Dse 4.8.15 and it is cassandra 2.1.
>>>> What are the best configuration to use for reaper for 144 nodes with
>>>> 256 vnodes and it shows around 532TB data when we start opscenter repairs.
>>>>
>>>> We need to finish repair soon.
>>>>
>>>> On Mon, May 21, 2018 at 10:53 AM Alexander Dejanovski <
>>>> a...@thelastpickle.com> wrote:
>>>>
>>>>> Hi Subri,
>>>>>
>>>>> Reaper might indeed be your best chance to reduce the overhead of
>>>>> vnodes there.
>>>>> The latest betas include a new feature that will group vnodes sharing
>>>>> the same replicas in the same segment. This will allow to have less
>>>>> segments than vnodes, and is available with Cassandra 2.2 and onwards (the
>>>>> improvement is especially beneficial with Cassandra 3.0+ as such token
>>>>> ranges will be repaired in a single session).
>>>>>
>>>>> We have a gitter that you can join if you want to ask questions.
>>>>>
>>>>> Cheers,
>>>>>
>>>>> Le lun. 21 mai 2018 à 15:29, Surbhi Gupta <surbhi.gupt...@gmail.com>
>>>>> a écrit :
>>>>>
>>>>>> Thanks Abdul
>>>>>>
>>>>>> On Mon, May 21, 2018 at 6:28 AM Abdul Patel <abd786...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> We have a paramater in reaper yaml file called
>>>>>>> repairManagerSchrdulingIntervalSeconds default is 10 seconds   , i
>>>>>>> tested with 8,6,5 seconds and found 5 seconds optimal for my environment
>>>>>>> ..you go down further but it will have cascading effects in cpu and 
>>>>>>> memory
>>>>>>> consumption.
>>>>>>> So test well.
>>>>>>>
>>>>>>>
>>>>>>> On Monday, May 21, 2018, Surbhi Gupta <surbhi.gupt...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Thanks a lot for your inputs,
>>>>>>>> Abdul, how did u tune reaper?
>>>>>>>>
>>>>>>>> On Sun, May 20, 2018 at 10:10 AM Jonathan Haddad <j...@jonhaddad.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> FWIW the largest deployment I know about is a single reaper
>>>>>>>>> instance managing 50 clusters and over 2000 nodes.
>>>>>>>>>
>>>>>&

Re: Question About Reaper

[Surbhi Gupta]

Another question, We use 9142 cqlsh port in one of the datacenter and on
other datacenter we use 9042 port.
How should we configure this ?

On 24 May 2018 at 10:22, Surbhi Gupta <surbhi.gupt...@gmail.com> wrote:

> What is the impact of
> PARALLEL - all replicas at the same time ?
> Will it make repair faster,?
> Do we expect more CPU , Load and memory usage in case if we use Parallel ,
> compare to other settings ?
>
>
>
> On 21 May 2018 at 22:55, Alexander Dejanovski <a...@thelastpickle.com>
> wrote:
>
>> You won't be able to have less segments than vnodes, so just use 256
>> segments per node, use parallel as repair parallelism, and set intensity to
>> 1.
>>
>> You apparently have more than 3TB per node, and that kind of density is
>> always challenging when it comes to run "fast" repairs.
>>
>> Cheers,
>>
>> Le mar. 22 mai 2018 à 07:28, Surbhi Gupta <surbhi.gupt...@gmail.com> a
>> écrit :
>>
>>> We are on Dse 4.8.15 and it is cassandra 2.1.
>>> What are the best configuration to use for reaper for 144 nodes with 256
>>> vnodes and it shows around 532TB data when we start opscenter repairs.
>>>
>>> We need to finish repair soon.
>>>
>>> On Mon, May 21, 2018 at 10:53 AM Alexander Dejanovski <
>>> a...@thelastpickle.com> wrote:
>>>
>>>> Hi Subri,
>>>>
>>>> Reaper might indeed be your best chance to reduce the overhead of
>>>> vnodes there.
>>>> The latest betas include a new feature that will group vnodes sharing
>>>> the same replicas in the same segment. This will allow to have less
>>>> segments than vnodes, and is available with Cassandra 2.2 and onwards (the
>>>> improvement is especially beneficial with Cassandra 3.0+ as such token
>>>> ranges will be repaired in a single session).
>>>>
>>>> We have a gitter that you can join if you want to ask questions.
>>>>
>>>> Cheers,
>>>>
>>>> Le lun. 21 mai 2018 à 15:29, Surbhi Gupta <surbhi.gupt...@gmail.com> a
>>>> écrit :
>>>>
>>>>> Thanks Abdul
>>>>>
>>>>> On Mon, May 21, 2018 at 6:28 AM Abdul Patel <abd786...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> We have a paramater in reaper yaml file called
>>>>>> repairManagerSchrdulingIntervalSeconds default is 10 seconds   , i
>>>>>> tested with 8,6,5 seconds and found 5 seconds optimal for my environment
>>>>>> ..you go down further but it will have cascading effects in cpu and 
>>>>>> memory
>>>>>> consumption.
>>>>>> So test well.
>>>>>>
>>>>>>
>>>>>> On Monday, May 21, 2018, Surbhi Gupta <surbhi.gupt...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Thanks a lot for your inputs,
>>>>>>> Abdul, how did u tune reaper?
>>>>>>>
>>>>>>> On Sun, May 20, 2018 at 10:10 AM Jonathan Haddad <j...@jonhaddad.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> FWIW the largest deployment I know about is a single reaper
>>>>>>>> instance managing 50 clusters and over 2000 nodes.
>>>>>>>>
>>>>>>>> There might be bigger, but I either don’t know about it or can’t
>>>>>>>> remember.
>>>>>>>>
>>>>>>>> On Sun, May 20, 2018 at 10:04 AM Abdul Patel <abd786...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> I recently tested reaper and it actually helped us alot. Even with
>>>>>>>>> our small footprint 18 node reaper takes close to 6 hrs.>>>>>>>> took 13
>>>>>>>>> hrs ,i was able to tune it 50%>. But it really depends on number 
>>>>>>>>> nodes. For
>>>>>>>>> example if you have 4 nodes then it runs on 4*256 =1024 
>>>>>>>>> segements ,
>>>>>>>>> so for your env. Ut will be 256*144 close to 36k segements.
>>>>>>>>> Better test on poc box how much time it takes and then proceed
>>>>>>>>> further ..i have tested so far in 1 dc only , we can actually have 
>>>>>>>>> seperate
>>>>>>>>> reaper instance handling seperate dc but havent tested it yet.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Sunday, May 20, 2018, Surbhi Gupta <surbhi.gupt...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> We have a cluster with 144 nodes( 3 datacenter) with 256 Vnodes .
>>>>>>>>>> When we tried to start repairs from opscenter then it showed
>>>>>>>>>> 1.9Million ranges to repair .
>>>>>>>>>> And even after doing compaction and strekamthroughput to 0 ,
>>>>>>>>>> opscenter is not able to help us much to finish repair in 9 days 
>>>>>>>>>> timeframe .
>>>>>>>>>>
>>>>>>>>>> What is your thought on Reaper ?
>>>>>>>>>> Do you think , Reaper might be able to help us in this scenario ?
>>>>>>>>>>
>>>>>>>>>> Thanks
>>>>>>>>>> Surbhi
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>> Jon Haddad
>>>>>>>> http://www.rustyrazorblade.com
>>>>>>>> twitter: rustyrazorblade
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>
>>>>> --
>>>> -
>>>> Alexander Dejanovski
>>>> France
>>>> @alexanderdeja
>>>>
>>>> Consultant
>>>> Apache Cassandra Consulting
>>>> http://www.thelastpickle.com
>>>>
>>>>
>>>> --
>> -
>> Alexander Dejanovski
>> France
>> @alexanderdeja
>>
>> Consultant
>> Apache Cassandra Consulting
>> http://www.thelastpickle.com
>>
>
>

Re: Question About Reaper

[Surbhi Gupta]

What is the impact of
PARALLEL - all replicas at the same time ?
Will it make repair faster,?
Do we expect more CPU , Load and memory usage in case if we use Parallel ,
compare to other settings ?



On 21 May 2018 at 22:55, Alexander Dejanovski <a...@thelastpickle.com>
wrote:

> You won't be able to have less segments than vnodes, so just use 256
> segments per node, use parallel as repair parallelism, and set intensity to
> 1.
>
> You apparently have more than 3TB per node, and that kind of density is
> always challenging when it comes to run "fast" repairs.
>
> Cheers,
>
> Le mar. 22 mai 2018 à 07:28, Surbhi Gupta <surbhi.gupt...@gmail.com> a
> écrit :
>
>> We are on Dse 4.8.15 and it is cassandra 2.1.
>> What are the best configuration to use for reaper for 144 nodes with 256
>> vnodes and it shows around 532TB data when we start opscenter repairs.
>>
>> We need to finish repair soon.
>>
>> On Mon, May 21, 2018 at 10:53 AM Alexander Dejanovski <
>> a...@thelastpickle.com> wrote:
>>
>>> Hi Subri,
>>>
>>> Reaper might indeed be your best chance to reduce the overhead of vnodes
>>> there.
>>> The latest betas include a new feature that will group vnodes sharing
>>> the same replicas in the same segment. This will allow to have less
>>> segments than vnodes, and is available with Cassandra 2.2 and onwards (the
>>> improvement is especially beneficial with Cassandra 3.0+ as such token
>>> ranges will be repaired in a single session).
>>>
>>> We have a gitter that you can join if you want to ask questions.
>>>
>>> Cheers,
>>>
>>> Le lun. 21 mai 2018 à 15:29, Surbhi Gupta <surbhi.gupt...@gmail.com> a
>>> écrit :
>>>
>>>> Thanks Abdul
>>>>
>>>> On Mon, May 21, 2018 at 6:28 AM Abdul Patel <abd786...@gmail.com>
>>>> wrote:
>>>>
>>>>> We have a paramater in reaper yaml file called
>>>>> repairManagerSchrdulingIntervalSeconds default is 10 seconds   , i
>>>>> tested with 8,6,5 seconds and found 5 seconds optimal for my environment
>>>>> ..you go down further but it will have cascading effects in cpu and memory
>>>>> consumption.
>>>>> So test well.
>>>>>
>>>>>
>>>>> On Monday, May 21, 2018, Surbhi Gupta <surbhi.gupt...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Thanks a lot for your inputs,
>>>>>> Abdul, how did u tune reaper?
>>>>>>
>>>>>> On Sun, May 20, 2018 at 10:10 AM Jonathan Haddad <j...@jonhaddad.com>
>>>>>> wrote:
>>>>>>
>>>>>>> FWIW the largest deployment I know about is a single reaper instance
>>>>>>> managing 50 clusters and over 2000 nodes.
>>>>>>>
>>>>>>> There might be bigger, but I either don’t know about it or can’t
>>>>>>> remember.
>>>>>>>
>>>>>>> On Sun, May 20, 2018 at 10:04 AM Abdul Patel <abd786...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I recently tested reaper and it actually helped us alot. Even with
>>>>>>>> our small footprint 18 node reaper takes close to 6 hrs.>>>>>>> 13
>>>>>>>> hrs ,i was able to tune it 50%>. But it really depends on number 
>>>>>>>> nodes. For
>>>>>>>> example if you have 4 nodes then it runs on 4*256 =1024 
>>>>>>>> segements ,
>>>>>>>> so for your env. Ut will be 256*144 close to 36k segements.
>>>>>>>> Better test on poc box how much time it takes and then proceed
>>>>>>>> further ..i have tested so far in 1 dc only , we can actually have 
>>>>>>>> seperate
>>>>>>>> reaper instance handling seperate dc but havent tested it yet.
>>>>>>>>
>>>>>>>>
>>>>>>>> On Sunday, May 20, 2018, Surbhi Gupta <surbhi.gupt...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> We have a cluster with 144 nodes( 3 datacenter) with 256 Vnodes .
>>>>>>>>> When we tried to start repairs from opscenter then it showed
>>>>>>>>> 1.9Million ranges to repair .
>>>>>>>>> And even after doing compaction and strekamthroughput to 0 ,
>>>>>>>>> opscenter is not able to help us much to finish repair in 9 days 
>>>>>>>>> timeframe .
>>>>>>>>>
>>>>>>>>> What is your thought on Reaper ?
>>>>>>>>> Do you think , Reaper might be able to help us in this scenario ?
>>>>>>>>>
>>>>>>>>> Thanks
>>>>>>>>> Surbhi
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>> Jon Haddad
>>>>>>> http://www.rustyrazorblade.com
>>>>>>> twitter: rustyrazorblade
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>
>>>> --
>>> -
>>> Alexander Dejanovski
>>> France
>>> @alexanderdeja
>>>
>>> Consultant
>>> Apache Cassandra Consulting
>>> http://www.thelastpickle.com
>>>
>>>
>>> --
> -
> Alexander Dejanovski
> France
> @alexanderdeja
>
> Consultant
> Apache Cassandra Consulting
> http://www.thelastpickle.com
>

Re: Question About Reaper

[Surbhi Gupta]

We are on Dse 4.8.15 and it is cassandra 2.1.
What are the best configuration to use for reaper for 144 nodes with 256
vnodes and it shows around 532TB data when we start opscenter repairs.

We need to finish repair soon.

On Mon, May 21, 2018 at 10:53 AM Alexander Dejanovski <
a...@thelastpickle.com> wrote:

> Hi Subri,
>
> Reaper might indeed be your best chance to reduce the overhead of vnodes
> there.
> The latest betas include a new feature that will group vnodes sharing the
> same replicas in the same segment. This will allow to have less segments
> than vnodes, and is available with Cassandra 2.2 and onwards (the
> improvement is especially beneficial with Cassandra 3.0+ as such token
> ranges will be repaired in a single session).
>
> We have a gitter that you can join if you want to ask questions.
>
> Cheers,
>
> Le lun. 21 mai 2018 à 15:29, Surbhi Gupta <surbhi.gupt...@gmail.com> a
> écrit :
>
>> Thanks Abdul
>>
>> On Mon, May 21, 2018 at 6:28 AM Abdul Patel <abd786...@gmail.com> wrote:
>>
>>> We have a paramater in reaper yaml file called
>>> repairManagerSchrdulingIntervalSeconds default is 10 seconds   , i tested
>>> with 8,6,5 seconds and found 5 seconds optimal for my environment ..you go
>>> down further but it will have cascading effects in cpu and memory
>>> consumption.
>>> So test well.
>>>
>>>
>>> On Monday, May 21, 2018, Surbhi Gupta <surbhi.gupt...@gmail.com> wrote:
>>>
>>>> Thanks a lot for your inputs,
>>>> Abdul, how did u tune reaper?
>>>>
>>>> On Sun, May 20, 2018 at 10:10 AM Jonathan Haddad <j...@jonhaddad.com>
>>>> wrote:
>>>>
>>>>> FWIW the largest deployment I know about is a single reaper instance
>>>>> managing 50 clusters and over 2000 nodes.
>>>>>
>>>>> There might be bigger, but I either don’t know about it or can’t
>>>>> remember.
>>>>>
>>>>> On Sun, May 20, 2018 at 10:04 AM Abdul Patel <abd786...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I recently tested reaper and it actually helped us alot. Even with
>>>>>> our small footprint 18 node reaper takes close to 6 hrs.>>>>> hrs ,i was able to tune it 50%>. But it really depends on number nodes. 
>>>>>> For
>>>>>> example if you have 4 nodes then it runs on 4*256 =1024 
>>>>>> segements ,
>>>>>> so for your env. Ut will be 256*144 close to 36k segements.
>>>>>> Better test on poc box how much time it takes and then proceed
>>>>>> further ..i have tested so far in 1 dc only , we can actually have 
>>>>>> seperate
>>>>>> reaper instance handling seperate dc but havent tested it yet.
>>>>>>
>>>>>>
>>>>>> On Sunday, May 20, 2018, Surbhi Gupta <surbhi.gupt...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> We have a cluster with 144 nodes( 3 datacenter) with 256 Vnodes .
>>>>>>> When we tried to start repairs from opscenter then it showed
>>>>>>> 1.9Million ranges to repair .
>>>>>>> And even after doing compaction and strekamthroughput to 0 ,
>>>>>>> opscenter is not able to help us much to finish repair in 9 days 
>>>>>>> timeframe .
>>>>>>>
>>>>>>> What is your thought on Reaper ?
>>>>>>> Do you think , Reaper might be able to help us in this scenario ?
>>>>>>>
>>>>>>> Thanks
>>>>>>> Surbhi
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>> Jon Haddad
>>>>> http://www.rustyrazorblade.com
>>>>> twitter: rustyrazorblade
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>
>> --
> -
> Alexander Dejanovski
> France
> @alexanderdeja
>
> Consultant
> Apache Cassandra Consulting
> http://www.thelastpickle.com
>
>
>

Re: Question About Reaper

[Surbhi Gupta]

Thanks Abdul

On Mon, May 21, 2018 at 6:28 AM Abdul Patel <abd786...@gmail.com> wrote:

> We have a paramater in reaper yaml file called
> repairManagerSchrdulingIntervalSeconds default is 10 seconds   , i tested
> with 8,6,5 seconds and found 5 seconds optimal for my environment ..you go
> down further but it will have cascading effects in cpu and memory
> consumption.
> So test well.
>
>
> On Monday, May 21, 2018, Surbhi Gupta <surbhi.gupt...@gmail.com> wrote:
>
>> Thanks a lot for your inputs,
>> Abdul, how did u tune reaper?
>>
>> On Sun, May 20, 2018 at 10:10 AM Jonathan Haddad <j...@jonhaddad.com>
>> wrote:
>>
>>> FWIW the largest deployment I know about is a single reaper instance
>>> managing 50 clusters and over 2000 nodes.
>>>
>>> There might be bigger, but I either don’t know about it or can’t
>>> remember.
>>>
>>> On Sun, May 20, 2018 at 10:04 AM Abdul Patel <abd786...@gmail.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> I recently tested reaper and it actually helped us alot. Even with our
>>>> small footprint 18 node reaper takes close to 6 hrs.>>> ,i was able to tune it 50%>. But it really depends on number nodes. For
>>>> example if you have 4 nodes then it runs on 4*256 =1024 segements ,
>>>> so for your env. Ut will be 256*144 close to 36k segements.
>>>> Better test on poc box how much time it takes and then proceed further
>>>> ..i have tested so far in 1 dc only , we can actually have seperate reaper
>>>> instance handling seperate dc but havent tested it yet.
>>>>
>>>>
>>>> On Sunday, May 20, 2018, Surbhi Gupta <surbhi.gupt...@gmail.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> We have a cluster with 144 nodes( 3 datacenter) with 256 Vnodes .
>>>>> When we tried to start repairs from opscenter then it showed
>>>>> 1.9Million ranges to repair .
>>>>> And even after doing compaction and strekamthroughput to 0 , opscenter
>>>>> is not able to help us much to finish repair in 9 days timeframe .
>>>>>
>>>>> What is your thought on Reaper ?
>>>>> Do you think , Reaper might be able to help us in this scenario ?
>>>>>
>>>>> Thanks
>>>>> Surbhi
>>>>>
>>>>>
>>>>> --
>>> Jon Haddad
>>> http://www.rustyrazorblade.com
>>> twitter: rustyrazorblade
>>>
>>>
>>>
>>
>>

Re: Question About Reaper

[Surbhi Gupta]

Thanks a lot for your inputs,
Abdul, how did u tune reaper?

On Sun, May 20, 2018 at 10:10 AM Jonathan Haddad <j...@jonhaddad.com> wrote:

> FWIW the largest deployment I know about is a single reaper instance
> managing 50 clusters and over 2000 nodes.
>
> There might be bigger, but I either don’t know about it or can’t remember.
>
> On Sun, May 20, 2018 at 10:04 AM Abdul Patel <abd786...@gmail.com> wrote:
>
>> Hi,
>>
>> I recently tested reaper and it actually helped us alot. Even with our
>> small footprint 18 node reaper takes close to 6 hrs.> ,i was able to tune it 50%>. But it really depends on number nodes. For
>> example if you have 4 nodes then it runs on 4*256 =1024 segements ,
>> so for your env. Ut will be 256*144 close to 36k segements.
>> Better test on poc box how much time it takes and then proceed further
>> ..i have tested so far in 1 dc only , we can actually have seperate reaper
>> instance handling seperate dc but havent tested it yet.
>>
>>
>> On Sunday, May 20, 2018, Surbhi Gupta <surbhi.gupt...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> We have a cluster with 144 nodes( 3 datacenter) with 256 Vnodes .
>>> When we tried to start repairs from opscenter then it showed 1.9Million
>>> ranges to repair .
>>> And even after doing compaction and strekamthroughput to 0 , opscenter
>>> is not able to help us much to finish repair in 9 days timeframe .
>>>
>>> What is your thought on Reaper ?
>>> Do you think , Reaper might be able to help us in this scenario ?
>>>
>>> Thanks
>>> Surbhi
>>>
>>>
>>> --
> Jon Haddad
> http://www.rustyrazorblade.com
> twitter: rustyrazorblade
>
>
>

Question About Reaper

[Surbhi Gupta]

Hi,

We have a cluster with 144 nodes( 3 datacenter) with 256 Vnodes .
When we tried to start repairs from opscenter then it showed 1.9Million
ranges to repair .
And even after doing compaction and strekamthroughput to 0 , opscenter is
not able to help us much to finish repair in 9 days timeframe .

What is your thought on Reaper ?
Do you think , Reaper might be able to help us in this scenario ?

Thanks
Surbhi

Re: Cassandra crashes....

[Surbhi Gupta]

16GB heap is too small for G1GC . Try at least 32GB of heap size
On Tue, Aug 22, 2017 at 7:58 AM Fay Hou [Storage Service] ­ <
fay...@coupang.com> wrote:

> What errors do you see?
> 16gb of 256 GB . Heap is too small. I would give heap at least 160gb.
>
>
> On Aug 22, 2017 7:42 AM, "Thakrar, Jayesh" 
> wrote:
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> Hi All,
>
>
>
>
>
> We are somewhat new users to Cassandra 3.10 on Linux and wanted to ping
> the user group for their experiences.
>
>
>
>
>
> Our usage profile is  batch jobs that load millions of rows to Cassandra
> every hour.
>
>
> And there are similar period batch jobs that read millions of rows and do
> some processing, outputting the result to HDFS (no issues with HDFS).
>
>
>
>
>
> We often seen Cassandra daemons crash.
>
>
> Key points of our environment are:
>
>
> *Pretty good servers:* 54 cores (with hyperthreading), 256 GB RAM, 3.2 TB
> SSD drive
>
>
> *Compaction:* TWCS compaction with 7 day windows as the data retention
> period is limited - about 120 days.
>
>
> *JDK: *Java 1.8.0.71 and G1 GC
>
>
>
> *Heap Size:* 16 GB
>
>
> *Large SSTables:* 50 GB to 300+ GB
>
>
>
>
>
>
>
> We see the daemons crash after some back-to-back long GCs (1.5 to 3.5
> seconds).
>
>
> Note that we had set the target for GC pauses to be 200 ms
>
>
>
>
>
> We have been somewhat able to tame the crashes by updating the TWCS
> compaction properties
>
>
>
> to have min/max compaction sstables = 4 and by drastically reducing the
> size of the New/Eden space (to 5% of heap space = 800 MB).
>
>
> Its been about 12 hours and our stop-the-world gc pauses are under 90 ms.
>
>
> Since the servers have more than sufficient resources, we are not seeing
> any noticeable performance impact.
>
>
>
>
>
> Is this kind of tuning normal/expected?
>
>
>
>
>
> Thanks,
>
>
> Jayesh
>
>
>
>
>
>
>
>
>
>
>
>
>

Re: Tool to manage cassandra

[Surbhi Gupta]

If u are using dse then u can use opscenter

On Fri, Jun 16, 2017 at 6:01 AM Ram Bhatia  wrote:

> Hi
>
>
>
>
>
>
>
>
>
> May I know, if there a tool similar to Oracle Enterprise Manager for
> managing Cassandra ?
>
>
>
>
>
>
>
>
>
> Thank you in advance for your help,
>
>
>
>
> Ram Bhatia
>
>
>
>
> -
>
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
>
> For additional commands, e-mail: user-h...@cassandra.apache.org
>
>
>
>

Re: How do you do automatic restacking of AWS instance for cassandra?

[Surbhi Gupta]

We get the new AMI release with the new OS updates and we are not allowed
to use the old AMI .


On Sat, May 27, 2017 at 7:11 PM Jeff Jirsa <jji...@apache.org> wrote:

>
>
>
>
> On 2017-05-27 18:04 (-0700), Surbhi Gupta <surbhi.gupt...@gmail.com>
> wrote:
>
> > Thanks a lot for all of your reply.
>
> > Our requirement is :
>
> > Our company releases AMI almost every month where they have some or the
>
> > other security packages.
>
> > So as per our security team we need to move our cassandra cluster to the
>
> > new AMI .
>
> > As this process happens every month, we would like to automate the
> process .
>
> > Few points to consider here:
>
> >
>
> > 1. We are using ephemeral drives to store cassandra data
>
> > 2. We are on dse 4.8.x
>
> >
>
> > So currently to do the process, we pinup a new nodes with new DC name and
>
> > join that DC, alter the keyspace, do rebuild  and later alter the
> keyspace
>
> > again to remove the old DC .
>
> >
>
> > But all of this process is manually done as of now.
>
> >
>
> > So i wanted to understand , on AWS, how do you do above kind of task
>
> > automatically ?
>
>
>
>
>
> At a previous employer, they used M4 class instances with data on a
> dedicated EBS volumes, so we could swap AMIs / stop / start / adjust
> instances without having to deal with this. This worked reasonably well for
> their scale (which was petabytes of data).
>
>
>
> Other companies using ephemeral tend to be more willing to just terminate
> instances and replace them (-Dcassandra.replace_address). If you stop
> cassandra, then boot a replacement with 'replace_address' set, it'll take
> over for the stopped instance, including re-streaming all data (as best it
> can, subject to consistency level and repair status). This may be easier
> for you to script than switching your fleet to EBS, but it's not without
> risk.
>
>
>
>
>
>
>
> -
>
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
>
> For additional commands, e-mail: user-h...@cassandra.apache.org
>
>
>
>

Re: How do you do automatic restacking of AWS instance for cassandra?

[Surbhi Gupta]

Thanks a lot for all of your reply.
Our requirement is :
Our company releases AMI almost every month where they have some or the
other security packages.
So as per our security team we need to move our cassandra cluster to the
new AMI .
As this process happens every month, we would like to automate the process .
Few points to consider here:

1. We are using ephemeral drives to store cassandra data
2. We are on dse 4.8.x

So currently to do the process, we pinup a new nodes with new DC name and
join that DC, alter the keyspace, do rebuild  and later alter the keyspace
again to remove the old DC .

But all of this process is manually done as of now.

So i wanted to understand , on AWS, how do you do above kind of task
automatically ?

Thanks
Surbhi


On 27 May 2017 at 16:11, Marc Selwan <marc.sel...@datastax.com> wrote:

> Hi Surbhi,
>
> The only time I've heard of restacking, it was a specific term a financial
> services company used internally to describe a security related procedure
> specific to them.
>
> If this sounds like you/the company you work for, send me a PM because I
> don't believe I can share those details in a public mailing list outside of
> that organization.
>
> Best,
> Marc
>
> On Thu, May 25, 2017, 11:22 AM daemeon reiydelle <daeme...@gmail.com>
> wrote:
>
>> What is restacking?
>>
>>
>>
>>
>>
>> *Daemeon C.M. ReiydelleUSA (+1) 415.501.0198 <(415)%20501-0198>London
>> (+44) (0) 20 8144 9872 <+44%2020%208144%209872>*
>>
>>
>> *“All men dream, but not equally. Those who dream by night in the dusty
>> recesses of their minds wake up in the day to find it was vanity, but the
>> dreamers of the day are dangerous men, for they may act their dreams with
>> open eyes, to make it possible.” — T.E. Lawrence*
>>
>>
>> On Thu, May 25, 2017 at 10:24 AM, Surbhi Gupta <surbhi.gupt...@gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> Wanted to understand, how do you do automatic restacking of cassandra
>>> nodes on AWS?
>>>
>>> Thanks
>>> Surbhi
>>>
>>
>> --
> Marc Selwan | DataStax | Solutions Engineer | (925) 413-7079
>
>
>

How do you do automatic restacking of AWS instance for cassandra?

[Surbhi Gupta]

Hi,

Wanted to understand, how do you do automatic restacking of cassandra nodes
on AWS?

Thanks
Surbhi

Re: Unsuccessful back-up and restore with differing counts

[Surbhi Gupta]

Below link has the method u r looking for
http://datascale.io/cloning-cassandra-clusters-fast-way/

On Sat, May 13, 2017 at 9:49 AM srinivasarao daruna 
wrote:

> I am using vnodes. Is there a documenation that you can suggest to
> understand how to assign same tokens in new cluster.? I will try it again.
>
>
> On May 13, 2017 12:32 PM, "Nitan Kainth"  wrote:
>
> As Jonathan mentioned, if you are using vnodes , you should back up
> nodetool output and assign same token to nodes and then copy corresponding
> sstables.
>
> If using, initial token, then assign same value for subsequent node.
>
> Sstable loader should work independently, not sure why you are getting
> wrong counts for that
>
> Sent from my iPhone
>
> On May 13, 2017, at 9:34 AM, Jonathan Haddad  wrote:
>
> Did you create the nodes with the same tokens?
>
> On Sat, May 13, 2017 at 8:44 AM srinivasarao daruna <
> sree.srin...@gmail.com> wrote:
>
>> Hi,
>>
>> We have a cassandra cluster built on Apache Cassandra 3.9 with 6 nodes
>> and RF = 3. As part of re-building the cluster, we are testing the backup
>> and restore strategy.
>>
>> We took the snapshot and uploaded the files to S3 and data has been saved
>> the data with folder names (backup_folder1 - 6 for nodes 1 - 6).
>> Created a new cluster with the same number of nodes, and copied the data
>> from S3 and created the schema.
>>
>> *Strategy 1: (using nodetool refresh)*
>> 1) Copied back the data from S3 into one machine each based on the
>> folders created (backup_folder1  - 6 to 6 nodes)
>> 2) and performed nodetool refresh on the cluster.
>>
>> Ran the count:
>>
>> Count on previous cluster: 12125800
>> Count on new cluster: 10504780
>>
>> *Strategy 2: using sstableloader*
>>
>> 1) Copied back the data from S3 into one machine each based on the
>> folders created (backup_folder1  - 6 to 6 nodes)
>> 2) and performed sstableloader on each node.
>>
>> Ran the count:
>>
>> Count on previous cluster: 12125800
>> Count on new cluster: 11705084
>>
>>
>> Looking at the results, i have bit disappointed that neither of the
>> approach resulted 100% restore for me.
>> If there is an error in taking the backup, it should have not given
>> different counts.
>>
>> Any ideas on successful back-up and restore strategies.? and what could
>> ve gone wrong in my process.?
>>
>> Thank You,
>> Regards,
>> Srini
>>
>>
>>
>>
>>
>>
>>
>>
>
>
>

antlr-runtime-3.2.jar is turning into 0 bytes and dse is going down

[Surbhi Gupta]

Hi,

We have single node instance where we have cassandra , mysql and
application running at the same node for developers.

We are at dse 4.8.9 and dse is going down after sometime .
What we have noticed is that few of the jar at /usr/share/dse/common are
turning into 0 bytes.
Jars are as follows:

antlr-runtime-3.2.jar
antlr-2.7.7.jar
antlr-3.2.jar


When i install the dse all jars are of non zero bytes but not sure what is
happening and some of the jars are turning into 0 bytes and dse is shutting
down.

We were earlier at dse 4.6.10 and we did not see this issue.

Anybody faced this?
Any suggestion?

Thanks
Surbhi

Re: How to find total data size of a keyspace.

[Surbhi Gupta]

Nodetool status key space_name .
On Tue, Feb 28, 2017 at 4:53 AM anuja jain  wrote:

> Hi,
> Using nodetool cfstats gives me data size of each  table/column family and
> nodetool ring gives me load of all keyspace in cluster but I need total
> data size of one keyspace in the cluster. How can I get that?
>
>
>

Re: Cassandra Node Restart Stuck in STARTING?

[Surbhi Gupta]

Attaching the system.log can give more details ...

On 16 November 2016 at 11:05, Daniel Subak  wrote:

> Hey everyone,
>
> Ran into an issue running a node restart where "nodetool netstats"
> reported the node as "STARTING" with no streams when run locally. "nodetool
> status" run on other nodes reported that node as "DN". Both of those were
> expected. However, tailing the logs, there didn't seem to be anything
> noteworthy happening (below are the last few log lines in system.log.) Has
> anyone seen this behavior before? We'd love to be able to better monitor
> what is happening during a restart if anyone has some information on what
> happens during this phase! Happy to provide more info if needed, but even a
> high level general explanation would provide some clarity
>
> Thanks,
> Dan
>
> INFO  [main] 2016-11-16 18:07:55,907 ColumnFamilyStore.java:405 -
> Initializing system_schema.keyspaces
> INFO  [main] 2016-11-16 18:07:55,942 ColumnFamilyStore.java:405 -
> Initializing system_schema.tables
> INFO  [main] 2016-11-16 18:07:55,971 ColumnFamilyStore.java:405 -
> Initializing system_schema.columns
> INFO  [main] 2016-11-16 18:07:55,992 ColumnFamilyStore.java:405 -
> Initializing system_schema.triggers
> INFO  [main] 2016-11-16 18:07:56,010 ColumnFamilyStore.java:405 -
> Initializing system_schema.dropped_columns
> INFO  [main] 2016-11-16 18:07:56,026 ColumnFamilyStore.java:405 -
> Initializing system_schema.views
> INFO  [main] 2016-11-16 18:07:56,047 ColumnFamilyStore.java:405 -
> Initializing system_schema.types
> INFO  [main] 2016-11-16 18:07:56,066 ColumnFamilyStore.java:405 -
> Initializing system_schema.functions
> INFO  [main] 2016-11-16 18:07:56,081 ColumnFamilyStore.java:405 -
> Initializing system_schema.aggregates
> INFO  [main] 2016-11-16 18:07:56,093 ColumnFamilyStore.java:405 -
> Initializing system_schema.indexes
> INFO  [main] 2016-11-16 18:07:56,102 ViewManager.java:139 - Not submitting
> build tasks for views in keyspace system_schema as storage service is not
> initialized
>

Re: Priority for cassandra nodes in cluster

[Surbhi Gupta]

If u ask conceptually, it is possible but not recommended.
If u really want to do it use the initial token setting and provide the
broad range to the nodes where u want more data. But u need to understand
about the replication factor consideration, if u keep rf as 3 on a 3 node
cluster that means all nodes has all the data.

On Saturday, November 12, 2016, sat  wrote:

> Hi,
>
> Thanks all for your valuable suggestion.
>
> Thanks and Regards
> A.SathishKumar
>
> On Sat, Nov 12, 2016 at 2:59 PM, Ben Bromhead  > wrote:
>
>> +1 w/ Benjamin.
>>
>> However if you wish to make use of spare hardware capacity, look to
>> something like mesos DC/OS or kubernetes. You can run multiple services
>> across a fleet of hardware, but provision equal resources to Cassandra and
>> have somewhat reliable hardware sharing mechanisms.
>>
>> On Sat, 12 Nov 2016 at 14:12 Jon Haddad > > wrote:
>>
>>> Agreed w/ Benjamin.  Trying to diagnose issues in prod will be a
>>> nightmare.  Keep your DB servers homogeneous.
>>>
>>> On Nov 12, 2016, at 1:52 PM, Benjamin Roth >> > wrote:
>>>
>>> 1. From a 15 year experience of running distributed Services: dont Mix
>>> Services on machines if you don't have to. Dedicate each server to a single
>>> task if you can afford it. It is easier to manage and reduces risks in case
>>> of overload or failure
>>> 2. You can assign a different number of tokens for each node by setting
>>> this in Cassandra.yaml before you bootstrap that node
>>>
>>> Am 12.11.2016 22:48 schrieb "sat" >> >:
>>>
 Hi,

 We are planning to install 3 node cluster in production environment. Is
 it possible to provide weightage or priority to the nodes in cluster.

 Eg., We want more more records to be written to first 2 nodes and less
 to the 3rd node. We are thinking of this approach because we want to
 install other IO intensive messaging server in the 3rd node, in order to
 reduce the load we are requesting for this approach.


 Thanks and Regards
 A.SathishKumar


>>> --
>> Ben Bromhead
>> CTO | Instaclustr 
>> +1 650 284 9692
>> Managed Cassandra / Spark on AWS, Azure and Softlayer
>>
>
>
>
> --
> A.SathishKumar
> 044-24735023
>

Re: Keyspace/CF creation Timeouts

[Surbhi Gupta]

What is the error getting logged in system.log ?

On 25 October 2016 at 15:31, Jai Bheemsen Rao Dhanwada <
jaibheem...@gmail.com> wrote:

> No, I am not trying to create many column families.
> I already have 500 CF in my cluster and now I am trying to create one KS
> and 3 CF.
>
> write_request_timeout_in_ms = 1 -> 10 seconds
>
> On Tue, Oct 25, 2016 at 3:00 PM, Surbhi Gupta <surbhi.gupt...@gmail.com>
> wrote:
>
>> As you have many keyspaces and column family to be created that might be
>> the reason that within the stipulated time response is not coming back and
>> u r seeing time out.
>> Can you pls check write_request_timeout_in_ms also?
>>
>> On 25 October 2016 at 14:55, Edward Capriolo <edlinuxg...@gmail.com>
>> wrote:
>>
>>> I do not believe the ConsistencyLevel matters for schema changes. In
>>> recent versions request_timeout_in_ms has been replaced by N variables
>>> which allow different timeout values for different types of operations.
>>>
>>> You seem to have both a lot of keyspaces and column families. It seems
>>> likely that you have a large cluster since you have mentioned multiple data
>>> centers. Many people have similar problems namely: the operation that
>>> changes the schema will timeout at the client level, but the cluster will
>>> eventually carry out the change.
>>>
>>> Many people seem to be writing their schema changing code to issue the
>>> request, ignore the response (in the case of timeout) and then use a
>>> command like "describe schema|cluster" to confirm the change propagation.
>>> Generally it is viewed as a annoying problem that people deal with.
>>>
>>> On Tue, Oct 25, 2016 at 4:41 PM, Jai Bheemsen Rao Dhanwada <
>>> jaibheem...@gmail.com> wrote:
>>>
>>>> 1. Yes, all nodes are up and running,
>>>> 2. We are using the Local_QUORUM.
>>>>
>>>> On Tue, Oct 25, 2016 at 1:28 PM, Surbhi Gupta <surbhi.gupt...@gmail.com
>>>> > wrote:
>>>>
>>>>> 1. Make sure all nodes are up and running while you are trying to
>>>>> create the Keyspaces and Column Family.
>>>>> 2. What is the write consistency level u r using?
>>>>>
>>>>>
>>>>> On 25 October 2016 at 13:18, Jai Bheemsen Rao Dhanwada <
>>>>> jaibheem...@gmail.com> wrote:
>>>>>
>>>>>> Hello All,
>>>>>>
>>>>>> I have recently started noticing timeouts while creating KS/CF. this
>>>>>> is happening with increase in no.of keyspaces.
>>>>>>
>>>>>> Does anyone have an idea what to look for? as I don't see any error
>>>>>> or exception in the logs.
>>>>>> or is there some kind of parameter change required?
>>>>>>
>>>>>> C* Version: 2.1.16
>>>>>> Total KS: 170
>>>>>> Total CF: 500
>>>>>> Total DC : 8
>>>>>>
>>>>>> request_timeout_in_ms: 1
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>

Re: Keyspace/CF creation Timeouts

[Surbhi Gupta]

As you have many keyspaces and column family to be created that might be
the reason that within the stipulated time response is not coming back and
u r seeing time out.
Can you pls check write_request_timeout_in_ms also?

On 25 October 2016 at 14:55, Edward Capriolo <edlinuxg...@gmail.com> wrote:

> I do not believe the ConsistencyLevel matters for schema changes. In
> recent versions request_timeout_in_ms has been replaced by N variables
> which allow different timeout values for different types of operations.
>
> You seem to have both a lot of keyspaces and column families. It seems
> likely that you have a large cluster since you have mentioned multiple data
> centers. Many people have similar problems namely: the operation that
> changes the schema will timeout at the client level, but the cluster will
> eventually carry out the change.
>
> Many people seem to be writing their schema changing code to issue the
> request, ignore the response (in the case of timeout) and then use a
> command like "describe schema|cluster" to confirm the change propagation.
> Generally it is viewed as a annoying problem that people deal with.
>
> On Tue, Oct 25, 2016 at 4:41 PM, Jai Bheemsen Rao Dhanwada <
> jaibheem...@gmail.com> wrote:
>
>> 1. Yes, all nodes are up and running,
>> 2. We are using the Local_QUORUM.
>>
>> On Tue, Oct 25, 2016 at 1:28 PM, Surbhi Gupta <surbhi.gupt...@gmail.com>
>> wrote:
>>
>>> 1. Make sure all nodes are up and running while you are trying to create
>>> the Keyspaces and Column Family.
>>> 2. What is the write consistency level u r using?
>>>
>>>
>>> On 25 October 2016 at 13:18, Jai Bheemsen Rao Dhanwada <
>>> jaibheem...@gmail.com> wrote:
>>>
>>>> Hello All,
>>>>
>>>> I have recently started noticing timeouts while creating KS/CF. this is
>>>> happening with increase in no.of keyspaces.
>>>>
>>>> Does anyone have an idea what to look for? as I don't see any error or
>>>> exception in the logs.
>>>> or is there some kind of parameter change required?
>>>>
>>>> C* Version: 2.1.16
>>>> Total KS: 170
>>>> Total CF: 500
>>>> Total DC : 8
>>>>
>>>> request_timeout_in_ms: 1
>>>>
>>>
>>>
>>
>