Re: Connection reset by peer error

[cass savy]

The app and Cassandra are connected via firewall. For some reason,
connections are still remaining on Cassandra side even after stopping
services on app server.


On Mon, Jun 30, 2014 at 3:29 PM, Jacob Rhoden jacob.rho...@me.com wrote:

 How are the two machines connected? Direct cable? Via a hub, router,
 firewall, wan?

 __
 Sent from iPhone

 On 1 Jul 2014, at 6:01 am, cass savy casss...@gmail.com wrote:

 We use Datastax Java driver version 1.0.6. Application is running into
 issues connecting to the 3 node cluster. What is cause for it? Application
 is not able to establish a connection at all. I see this error
 intermittently few time every other day.


 Is the issue related to read/write timeout?Do I need to increase *timeout*
 values in yaml ?
 *APP logs*

 2014-06-27 17:33:47

 Full thread dump Java HotSpot(TM) 64-Bit Server VM (23.5-b02 mixed mode):



 RMI TCP Connection(105)-10.198.49.16 - Thread t@247

java.lang.Thread.State: RUNNABLE

 at java.net.SocketInputStream.socketRead0(Native Method)

 at
 java.net.SocketInputStream.read(SocketInputStream.java:150)

 at
 java.net.SocketInputStream.read(SocketInputStream.java:121)

 at
 java.io.BufferedInputStream.fill(BufferedInputStream.java:235)

 at
 java.io.BufferedInputStream.read(BufferedInputStream.java:254)

 - locked 68d37818 (a java.io.BufferedInputStream)

 at
 java.io.FilterInputStream.read(FilterInputStream.java:83)

 at
 sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)

 at
 sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)

 at
 sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)

 at
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)

 at
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)

 at java.lang.Thread.run(Thread.java:722)



 *All I see in the Cassandra logs:*
 ERROR [Native-Transport-Requests:2704] 2014-06-27 16:33:23,339
 ErrorMessage.java (line 210) Unexpected exception during request
 java.io.IOException: Connection reset by peer
 at sun.nio.ch.FileDispatcher.read0(Native Method)
 at sun.nio.ch.SocketDispatcher.read(Unknown Source)
 at sun.nio.ch.IOUtil.readIntoNativeBuffer(Unknown Source)
 at sun.nio.ch.IOUtil.read(Unknown Source)
 at sun.nio.ch.SocketChannelImpl.read(Unknown Source)
 at
 org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:59)
 at
 org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:472)
 at
 org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:333)
 at
 org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:35)
 at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown
 Source)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
 Source)
 at java.lang.Thread.run(Unknown Source)

Upgrade to DSE 4.5

[cass savy]

We run on DSE 3.1.3 and only use the Cassandra in prod cluster.

What is the release that I need to be on right away. Because if I need to
upgrade to DSE 4.5.c* 2.0.7. I need to take 3 paths to get there. I see lot
of improvements for solr/Hadoop features in DSE 4.0 and above.

Can I upgrade to  DSE 3.2.7 for now , be on it for sometime and then later
upgrade to DSE 4.5?

I do not want to upgrade to multiple versions on the same day and its hard
to know which one is the culprit is we run into issues?

Has any of you have DSE 4.5 in prod or DSE 3.2.7 and have found any issues?

Re: Better option to load data to cassandra

[cass savy]

Sstableloader works well for large tables if you want to move data from
Cassandra to Cassandra. This works if both C* are on the same version.

Sstable2json and json2sstable is another alternative.
On Nov 11, 2014 4:53 AM, srinivas rao pinnakasrin...@gmail.com wrote:

 Hi Team,

 Please suggest me the better options to load data from NoSql to Cassndra.



 Thanks
 Srini

Re: Cassandra install on JRE vs JDK

[cass savy]

Thanks Mark  for quick response. What version of Cassandra and JDK are you
using in Prod.


On Wed, Feb 18, 2015 at 11:58 AM, Mark Reddy mark.l.re...@gmail.com wrote:

 Yes you can use Oracle JDK if your prefer, I've been using the JDK with
 Cassandra in production for years without issue.

 Regards,
 Mark

 On 18 February 2015 at 19:49, cass savy casss...@gmail.com wrote:

 Can we install Oracle JDK instead of JRE  in Cassandra servers? We have
 few clusters running JDK when we upgraded to C*2.0.

 Is there any known issue or impact with using  JDK vs JRE?
 What is the reason to not use Oracle JDK in C* servers?
 Is there any performance impact ?

 Please advice.

Re: Cassandra install on JRE vs JDK

[cass savy]

Thanks Robert  for quick response. I use Oracle JDK and not OpenJDK.


On Wed, Feb 18, 2015 at 11:54 AM, Robert Stupp sn...@snazy.de wrote:

 The ”natural” dependency of Cassandra is the JRE (not the JDK) - e.g. in
 the Debian package.
 You should be safe using JRE instead of JDK.

 If you’re asking whether to use a non-Oracle JVM - the answer would be:
 use the Oracle JVM.
 OpenJDK might work, but I’d not recommend it.


 Am 18.02.2015 um 20:49 schrieb cass savy casss...@gmail.com:

 Can we install Oracle JDK instead of JRE  in Cassandra servers? We have
 few clusters running JDK when we upgraded to C*2.0.

 Is there any known issue or impact with using  JDK vs JRE?
 What is the reason to not use Oracle JDK in C* servers?
 Is there any performance impact ?

 Please advice.



 —
 Robert Stupp
 @snazy

Data tiered compaction and data model question

[cass savy]

We want to track events in log  Cf/table and should be able to query for
events that occurred in range of mins or hours for given day. Multiple
events can occur in a given minute.  Listed 2 table designs and leaning
towards table 1 to avoid large wide row.  Please advice on

*Table 1*: not very widerow, still be able to query for range of minutes
for given day
and/or given day and range of hours

Create table *log_Event*

(

 event_day text,

 event_hr int,

 event_time timeuuid,

 data text,

PRIMARY KEY ( *(event_day,event_hr),*event_time)

)
*Table 2: This will be very wide row*

Create table *log_Event*

( event_day text,

 event_time timeuuid,

 data text,

PRIMARY KEY ( *event_day,*event_time)

)


*Datatiered compaction: recommended for time series data as per below doc.
Our data will be kept only for 30 days. Hence thought of using this
compaction strategy.*

http://www.datastax.com/dev/blog/datetieredcompactionstrategy

Create table 1 listed above with this compaction strategy. Added some rows
and did manual flush.  I do not see any sstables created yet. Is that
expected?

 compaction={'max_sstable_age_days': '1', 'class':
'DateTieredCompactionStrategy'}

C*1.2 on JVM 1.7

[cass savy]

Has anybody run DSE 3.2.6(C*1.2.16) on JRE 1.7. I know its recommended that
we have to get to JVM version 7 to get to C*2.0 and higher.

We are experiencing latency issues during rolling upgrade from 1.2 to 2.0.
Hence cannot get o C*1.2, but plan to just upgrade JRE from 1.6 to 1.7. I
wanted to know if anyof you have C*1.2 on JRE 1.7 in PROD and have run into
issues.

Leap sec

[cass savy]

Just curious to know on how you are preparing Prod C* clusters for leap sec.

What are the workaorund other than upgrading kernel to 3.4+?
Are you upgrading clusters to Java 7 or higher on client and C* servers?

Re: Leap sec

[cass savy]

Are you suggesting the JDK 7 for client/driver and c* side as well. We use
Java driver 2.1.4 and plan to goto JDK7u80 on application end. In addition,
do I also have to upgrade to Java 7u60+ on C* servers as well.

On Fri, May 15, 2015 at 1:06 PM, Jim Witschey jim.witsc...@datastax.com
wrote:

  This post has some good advice for preparing for the leap second:
  http://www.datastax.com/dev/blog/preparing-for-the-leap-second

 Post author here. Let me know if you have any questions about it.

  What are the workaorund other than upgrading kernel to 3.4+?

 A couple NTP-based workarounds are described here, in the Preventing
 Issues and Workarounds section:

 https://access.redhat.com/articles/199563

 The goal of those workarounds is to prevent the leap second from being
 applied all at once. I make no guarantees about them, just pointing
 them out in case you wanted to investigate them yourself.

 As noted in my post and the Red Hat article, it's probably best to
 just upgrade your kernel/OS if you can.

  Are you upgrading clusters to Java 7 or higher on client and C* servers?

 Just to be clear, JDK 7 had its own timer problems until 7u60, so you
 should upgrade to or past that version.

 Jim Witschey

 Software Engineer in Test | jim.witsc...@datastax.com

Re: Nodetool rebuild on vnodes enabled

[cass savy]

1. Cassandra version is 2.0.14
2. Followed doc on adding new DC .  Added new DC with 3 nodes with vnodes
instead of manual tokens.

http://docs.datastax.com/en/cassandra/2.0/cassandra/operations/ops_add_dc_to_cluster_t.html

3. As per above doc and in past as well, rebuild is what i used when I add
new DC and had to rebuild data

4. I have 2 DC, one with only cassandra, other new DC with cassandra and
spark and with vnodes


On Tue, Nov 17, 2015 at 4:29 PM, Robert Coli <rc...@eventbrite.com> wrote:

> On Tue, Nov 17, 2015 at 3:24 PM, cass savy <casss...@gmail.com> wrote:
>
>> I am exploring vnodes on DSE spark enabled DC. I  added new nodes with 64
>> vnodes, stream thruput 100mb instead of default 200mb, sokcet_timeout set
>> to 1hr.
>>
>
> 1) what version of Cassandra (please the version of Apache Cassandra, not
> DSE)?
>
> 2) that's not really the way you're supposed to add vnodes?
>
> 3) why are you using "rebuild" instead of just bootstrapping?
>
> 4) how many total DCs do you have?
>
> Maybe I'm misunderstanding, but "rebuild" is for the case where you have
> more than one DC. Don't use it if you have fewer than two.
>
> =Rob
>
>

Switching to Vnodes

[cass savy]

We want to move our clusters to use Vnodes. I know the docs online say we
have to create new DC with vnodes and move to new dC and decommission old
one. We use DSE for our c* clusters.C* version is 2.0.14

1. Is there any other way to migrate existing nodes to vnodes?
2. What are the known issues with that approach?
3. We have few secondary indexes in the keyspace, will that cause any
issues with moving to vnodes?

4. What are the issues encountered after moving to vnodes in PROD
5. anybody recommend Vnodes for Spark nodes.

*Approach : Moving to new DC with vnodes enabled*:
When I tested it for  a  keyspace which has secondary indexes, rebuilds on
Vnode enabled Datacenter takes days and don't know when it completes or
even if it will complete. I tried with 256,32,64 tokens per node but no
luck.

Please advise.

Re: Switching to Vnodes

[cass savy]

Victor,
We have 21 nodes in 3 DC, spark DC has 3 nodes. Primary datacenter nodes
has 300gb of data.

What the num_tokens you have in prod cluster? are u using default 256?

On Wed, Dec 9, 2015 at 2:19 PM, Victor Chen <victor.h.c...@gmail.com> wrote:

> I have a 12 node cluster in prod using vnodes and C* version 2.18. I have
> never used rebuild, and instead prefer bootstrapping new nodes, even if it
> means there is additional shuffling of data and cleanup needed on the
> initial nodes in each DC, mostly b/c you can tell when bootstrapping is
> finished. w/ rebuild, like you have observed, there's really no way to be
> sure, apart from comparing load. I have no experience with vnodes and spark
> though, so I can't really comment on that. We are using secondary indexes
> though, and aren't seeing many issues. How much data do you have per node
> and in total and how many nodes?
>
> On Wed, Dec 9, 2015 at 4:26 PM, cass savy <casss...@gmail.com> wrote:
>
>> We want to move our clusters to use Vnodes. I know the docs online say we
>> have to create new DC with vnodes and move to new dC and decommission old
>> one. We use DSE for our c* clusters.C* version is 2.0.14
>>
>> 1. Is there any other way to migrate existing nodes to vnodes?
>> 2. What are the known issues with that approach?
>> 3. We have few secondary indexes in the keyspace, will that cause any
>> issues with moving to vnodes?
>>
>> 4. What are the issues encountered after moving to vnodes in PROD
>> 5. anybody recommend Vnodes for Spark nodes.
>>
>> *Approach : Moving to new DC with vnodes enabled*:
>> When I tested it for  a  keyspace which has secondary indexes, rebuilds
>> on Vnode enabled Datacenter takes days and don't know when it completes or
>> even if it will complete. I tried with 256,32,64 tokens per node but no
>> luck.
>>
>> Please advise.
>>
>>
>>
>
>

Write/read heavy usecase in one cluster

[cass savy]

How do you determine if we can share cluster in prod for 2 different
applications

 1. Has anybody shared cluster in prod a write heavy use case that captures
user login info (few 100 rpm)  and hardly performs few reads per day
and
 Use case that is read heavy use case that is 92% read with 10k requests
per min,higher consistency level of quorum


2. Use of  in-memory tables for lookup tables  that will be referred to for
every request prior to writing to transactional tables. Has anyone ised it
in prod and what were the issues encountered. what will be tuning/reco to
follow for prod

3. Use of multiple data directories for different applications like having
different data partitions for write/read heavy and one separate for
commitlog/caches

4. plan to use C* 2.1 with vnodes/murmur for above usecases. Need feedback
of if people have tried tuning heap size, off-heap parameters in c* 2.0 and
above. in prod

5. Java 8 with c* 2.0 and higher pros/cons especially with G1GC garbage
collection

Replication lag between data center

[cass savy]

How can we determine/measure the replication lag or latency between on
premise data centers or cross region/Availability zones?

Re: Switching to Vnodes

[cass savy]

Thanks Jeff for detailed clarifications.

We tried rebuild data in Spark DC nodes one node at a time in May again but
ran into issues. prod has 3 DC, DC1(9 nodes)  and DC2 (9 nodes) are only C*
and DC3 has spark with 3 nodes and vnodes enabled with numtokens=32

We also dropped few unused indexes. Then tried rebuild data in spark node 1
using  DC2 which takes less traffic so that DC1 will not be impacted. When
we tried this approach in May, we got  latency hit on DC1. We stopped
rebuild job and still saw the latency impact. Latency issue stopped once we
remove  replication to Spark nodes at keyspace level.

we need to rebuild data in new DC which has spark enabled. I am thinking of
this new approach to see if this will avoid latency impact.

1. Enable replication for keyspace with rep factor =1 for Analytics DC
2. keep the vnodes to be 32 and
3. do the nodetool rebuild in first spark DC node  using DC2

OR

1. disable vnodes and use manual tokens for all 3 nodes in Spark DC
2. cleanup all old data/logs
3. enable replication factor =1 for keyspace for Analytics DC
4. rebuild data using DC2

I want to ensure that  new approach does not send traffic to DC1 at all.
 let me know if there is any other options or if we need to change any of
the above approach.

Appreciate feedback.


On Wed, Dec 9, 2015 at 2:37 PM, Jeff Jirsa <jeff.ji...@crowdstrike.com>
wrote:

> Streaming with vnodes is not always pleasant – rebuild uses streaming (as
> does bootstrap, repair, and decommission). The rebuild delay you see may or
> may not be related to that. It could also be that the streams timed out,
> and you don’t have a stream timeout set. Are you seeing data move? Are the
> new nodes busy compacting? Secondary indexes themselves may not cause
> problems, but there are cases where very large indexes (due to very large
> partitions or unusual cardinalities) may case problems.
>
>
>1. The other way is to backup your data, make a new vnode cluster, and
>load your data in with sstableloader
>2. Known issues are that streaming with vnodes creates a lot of small
>tables and does a lot more work than streaming without vnodes
>    3. Not necessarily
>4. See #2
>
>
> From: cass savy
> Reply-To: "user@cassandra.apache.org"
> Date: Wednesday, December 9, 2015 at 1:26 PM
> To: "user@cassandra.apache.org"
> Subject: Switching to Vnodes
>
> We want to move our clusters to use Vnodes. I know the docs online say we
> have to create new DC with vnodes and move to new dC and decommission old
> one. We use DSE for our c* clusters.C* version is 2.0.14
>
> 1. Is there any other way to migrate existing nodes to vnodes?
> 2. What are the known issues with that approach?
> 3. We have few secondary indexes in the keyspace, will that cause any
> issues with moving to vnodes?
>
> 4. What are the issues encountered after moving to vnodes in PROD
> 5. anybody recommend Vnodes for Spark nodes.
>
> *Approach : Moving to new DC with vnodes enabled*:
> When I tested it for  a  keyspace which has secondary indexes, rebuilds on
> Vnode enabled Datacenter takes days and don't know when it completes or
> even if it will complete. I tried with 256,32,64 tokens per node but no
> luck.
>
> Please advise.
>
>
>

Replication time across regions in AWS

[cass savy]

Has anybody tested time taken to replicate data across regions between
us-east and west or AZ in AWS? Few docs/blogs says its in few 10s of
millisecs.

Please can you provide your insights. Also is there a way to measure the
time lag.

Cassandra on Cloud platforms experience

[cass savy]

I would like to hear from the community on their experiences or lesson
learnt on hosting Cassandra in cloud platforms like

1. Google Cloud Platform
2. AWS
3. Azure

1.  Which cloud hosting is better and Why?
2.  What differences of C* over vendor provided NoSQL DB like (Bigtable,
Dynamo,Azure Document DB)
3. AWS is more mature in his offerings and Azure is getting there or its
there already based on what I have been investigating so far?

4. What is drive to pick one vs another -Is it cost, infrastructure,
hardware SKU, availability, scalability, performance,ease of deployment and
maintenance,..etc?

Please let me know your thoughts and suggestions if somebody has done a
deep dive into these 3 cloud platforms for C*.


We use datastax cassandra and exploring new usecases in AWS and also
evaluating  or POC it in Azure/GCP

DSE 5.0 Upgrade

[cass savy]

Team,

1. What is the stable version for DSE 5.0 to upgrade from DSE 4.8.x?

2. Is anybody switched to using DSE Unified auth model which enfores to use
one auth policy as primary and other secondary?

3. Do I need to use multi-auth/DSE Unified auth for me upgrade to DSE 5.0
or higher?
Our old clusters are using internal authentication and we learnt that
we have to go with DSE Unifiedauth model as part of upgrade?

4. Does anybody migrate from Java driver to Datastax enterprise Java driver
1.1 or 1.2 recently which will be only driver that will support DSE 5.0 and
above and new auth policies added by DSE?

Or

Has anybody made Java driver version 3.2 work with DSE 5.0 or 5.1?

4. For AWS, what is prod recommended AMI with CentOS and for DSE 5.x
versions?

AWS Cassandra backup/Restore tools

[cass savy]

AWS Backup/Restore process/tools for C*/DSE C*:

Has anyone used Opscenter 6.1 backup tool to backup/restore data for larger
datasets online ?

If yes, did you run into issues using that tool to backup/restore data in
PROD that caused any performance or any other impact to the cluster?

If no, what are other tools that people have used or recommended for backup
and restore of Cassandra keyspaces?

Please advice.

Elastic IP for Cassandra in AWS

[cass savy]

We are working on automation framework to build new C* ring in AWS and
looking for suggestions to determine ways to automatically allocate
EIP/public IP during new cluster build and scale up/replace node situations.

We are using  elastic IP/public ip for broadcast address and private IP for
listen address for AWS Cassandra clusters.

 Has anybody done this for their PROD cluster? If yes, can you suggest
means to dynamically/automatically allocate/associate   EIP/public IP to C*
node using any automation framework/services or tools?

Re: Elastic IP for Cassandra in AWS

[cass savy]

How can we solve EIP allocation using APIs for  multiple nodes that are in
one ASG? What are the config management tools  that you are referring to?

On Mon, Oct 16, 2017 at 9:23 PM, kurt greaves  wrote:

> AWS API's provide the functionality to allocate and associate elastic IPs
> to instances. Generally the API's aren't pretty but they work. What issues
> are you having? If it's a configuration problem there are a variety of
> config management tools that you can use to populate the yaml/env files
> with the correct IP addresses.​
>