Re: Cassandra client drivers

[Abdul Patel]

Thanks Andy

On Monday, May 7, 2018, Andy Tolbert  wrote:

> Hi Abdul,
>
> If you are already at C* 3.1.0 and the driver you are using works, it's
> almost certain to also work with 3.11.2 as well as there are no protocol
> changes between these versions.  I would advise testing your application
> out against a 3.11.2 test environment first though, just to be safe ;).
>
> Thanks,
> Andy
>
> On Mon, May 7, 2018 at 5:47 PM, Abdul Patel  wrote:
>
>> Hi
>>
>> I am.planning for upgrade from 3.1.0 to 3.11.2 , just wanted to confirm
>> if cleint drivers need to upgraded? Or it will work with 3.1.0 drivers?
>>
>>
>>
>

Re: Cassandra client drivers

[Andy Tolbert]

Hi Abdul,

If you are already at C* 3.1.0 and the driver you are using works, it's
almost certain to also work with 3.11.2 as well as there are no protocol
changes between these versions.  I would advise testing your application
out against a 3.11.2 test environment first though, just to be safe ;).

Thanks,
Andy

On Mon, May 7, 2018 at 5:47 PM, Abdul Patel  wrote:

> Hi
>
> I am.planning for upgrade from 3.1.0 to 3.11.2 , just wanted to confirm if
> cleint drivers need to upgraded? Or it will work with 3.1.0 drivers?
>
>
>

Re: compaction: huge number of random reads

[kurt greaves]

If you've got small partitions/small reads you should test lowering your
compression chunk size on the table and disabling read ahead. This sounds
like it might just be a case of read amplification.

On Tue., 8 May 2018, 05:43 Kyrylo Lebediev, 
wrote:

> Dear Experts,
>
>
> I'm observing strange behavior on a cluster 2.1.20 during compactions.
>
>
> My setup is:
>
> 12 nodes  m4.2xlarge (8 vCPU, 32G RAM) Ubuntu 16.04, 2T EBS gp2.
>
> Filesystem: XFS, blocksize 4k, device read-ahead - 4k
>
> /sys/block/vxdb/queue/nomerges = 0
>
> SizeTieredCompactionStrategy
>
>
> After data loads when effectively nothing else is talking to the cluster
> and compactions is the only activity, I see something like this:
> $ iostat -dkx 1
> ...
>
> Device: rrqm/s   wrqm/s r/s w/srkB/swkB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> xvda  0.00 0.000.000.00 0.00 0.00
> 0.00 0.000.000.000.00   0.00   0.00
> xvdb  0.00 0.00 4769.00  213.00 19076.00 26820.00
> 18.42 7.951.171.063.76   0.20 100.00
>
> Device: rrqm/s   wrqm/s r/s w/srkB/swkB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> xvda  0.00 0.000.000.00 0.00 0.00
> 0.00 0.000.000.000.00   0.00   0.00
> xvdb  0.00 0.00 6098.00  177.00 24392.00 22076.00
> 14.81 6.461.360.96   15.16   0.16 100.00
>
> Writes are fine: 177 writes/sec <-> ~22Mbytes/sec,
>
> But for some reason compactions generate a huge number of small reads:
> 6098 reads/s <-> ~24Mbytes/sec.  ===>   Read size is 4k
>
>
> Why instead much smaller amount of large reads I'm getting huge number of
> 4k reads instead?
>
> What could be the reason?
>
> Thanks,
>
> Kyrill
>
>
>

Cassandra client drivers

[Abdul Patel]

Hi

I am.planning for upgrade from 3.1.0 to 3.11.2 , just wanted to confirm if
cleint drivers need to upgraded? Or it will work with 3.1.0 drivers?

compaction: huge number of random reads

[Kyrylo Lebediev]

Dear Experts,


I'm observing strange behavior on a cluster 2.1.20 during compactions.


My setup is:

12 nodes  m4.2xlarge (8 vCPU, 32G RAM) Ubuntu 16.04, 2T EBS gp2.

Filesystem: XFS, blocksize 4k, device read-ahead - 4k

/sys/block/vxdb/queue/nomerges = 0

SizeTieredCompactionStrategy


After data loads when effectively nothing else is talking to the cluster and 
compactions is the only activity, I see something like this:
$ iostat -dkx 1
...


Device: rrqm/s   wrqm/s r/s w/srkB/swkB/s avgrq-sz 
avgqu-sz   await r_await w_await  svctm  %util
xvda  0.00 0.000.000.00 0.00 0.00 0.00 
0.000.000.000.00   0.00   0.00
xvdb  0.00 0.00 4769.00  213.00 19076.00 26820.0018.42 
7.951.171.063.76   0.20 100.00

Device: rrqm/s   wrqm/s r/s w/srkB/swkB/s avgrq-sz 
avgqu-sz   await r_await w_await  svctm  %util
xvda  0.00 0.000.000.00 0.00 0.00 0.00 
0.000.000.000.00   0.00   0.00
xvdb  0.00 0.00 6098.00  177.00 24392.00 22076.0014.81 
6.461.360.96   15.16   0.16 100.00

Writes are fine: 177 writes/sec <-> ~22Mbytes/sec,

But for some reason compactions generate a huge number of small reads:
6098 reads/s <-> ~24Mbytes/sec.  ===>   Read size is 4k


Why instead much smaller amount of large reads I'm getting huge number of 4k 
reads instead?

What could be the reason?


Thanks,

Kyrill

Auto Compactions not running on Cassandra 3.10

[Anshul Rathore]

Hi

We are using 4 node cassandra 3.10 cluster.
For some reason autocompaction is not running on one of the table which
uses DTCS. TTL on this table is 3 months. Table has high write load , and
medium read load.

We have 4 disks per node , each disk growed to around 5k-6k sstables going
back to around 6-7 months.
I tried running major compaction which takes 1-2 days on each node, which
results in 20-30 sstables on all disks except 1.
But also after running major compaction , we now see sstables have started
growing again. So minor/auto compaction is not running on this table.

*Few Observations*
*  - *There are 2-3 partition keys i see while doing  major compaction
throwing large partition warning(100mb threshold) , their sizes around
100-200MB
  - All other tables use STCS , their auto compaction  is working fine.

Any pointers would be helpful

Thanks
Anshul

Re: 3.11.2 io.netty.channel.unix.FileDescriptor.readAddress(...)(Unknown Source)

[Kyrylo Lebediev]

Hi!

You are talking about  messages like below, right?


INFO  [epollEventLoopGroup-2-8] 2018-05-05 16:57:45,537 Message.java:623 - 
Unexpected exception during request; channel = [id: 0xa4879fdd, 
L:/10.175.20.112:9042 - R:/10.175.20.73:2508]

io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)() failed: 
Connection reset by peer

at io.netty.channel.unix.FileDescriptor.readAddress(...)(Unknown 
Source) ~[netty-all-4.0.44.Final.jar:4.0.44.Final]

As far as I understand, these messages mean that some C* client (10.175.20.73 
in your case) closed connection abnormally and the issue is on client side. So, 
this should be investigated on client side. I'd say if your app works fine it's 
up to you to investigate the issue or not ("due diligence").


At the same time there are some messages in the log related to hinted-handoff, 
which could mean your cluster suffers from write timeouts (unless these 
messages are caused by temporary unavailability of  10.175.20.114 because of 
node reboot etc). This definitely worth investigation. It can be caused by: 
non-optimal GC setting (btw messages like "ParNew GC in 309ms"  may imply this, 
delay of 309 ms is too long), hw performance issues, networking issues etc.


Regards,

Kyrill



From: Xiangfei Ni 
Sent: Monday, May 7, 2018 5:44:21 AM
To: user@cassandra.apache.org
Subject: 3.11.2 io.netty.channel.unix.FileDescriptor.readAddress(...)(Unknown 
Source)


Hi Expert,

  I have this error in the Cassandra logs,the nodes are ok and behave as normal,

INFO  [epollEventLoopGroup-2-8] 2018-05-05 16:57:45,537 Message.java:623 - 
Unexpected exception during request; channel = [id: 0xa4879fdd, 
L:/10.175.20.112:9042 - R:/10.175.20.73:2508]

io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)() failed: 
Connection reset by peer

at io.netty.channel.unix.FileDescriptor.readAddress(...)(Unknown 
Source) ~[netty-all-4.0.44.Final.jar:4.0.44.Final]

INFO  [epollEventLoopGroup-2-9] 2018-05-05 16:57:45,537 Message.java:623 - 
Unexpected exception during request; channel = [id: 0x86e1a3d9, 
L:/10.175.20.112:9042 - R:/10.175.20.37:44006]

io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)() failed: 
Connection reset by peer

at io.netty.channel.unix.FileDescriptor.readAddress(...)(Unknown 
Source) ~[netty-all-4.0.44.Final.jar:4.0.44.Final]

INFO  [HintsDispatcher:43] 2018-05-05 16:57:49,991 HintsStore.java:126 - 
Deleted hint file 5696cbef-edd1-42a6-b4f1-3c2f0881b8cb-1525510648347-1.hints

INFO  [HintsDispatcher:43] 2018-05-05 16:57:49,991 
HintsDispatchExecutor.java:282 - Finished hinted handoff of file 
5696cbef-edd1-42a6-b4f1-3c2f0881b8cb-1525510648347-1.hints to endpoint 
/10.175.20.114: 5696cbef-edd1-42a6-b4f1-3c2f0881b8cb

INFO  [Service Thread] 2018-05-05 16:58:01,159 GCInspector.java:284 - ParNew GC 
in 309ms.  CMS Old Gen: 1258291920 -> 1258482424; Par Eden Space: 503316480 -> 
0; Par Survivor Space: 700080 -> 782256

WARN  [GossipTasks:1] 2018-05-05 16:58:23,273 Gossiper.java:791 - Gossip stage 
has 2 pending tasks; skipping status check (no nodes will be marked down)

INFO  [HintsDispatcher:44] 2018-05-05 16:58:33,913 HintsStore.java:126 - 
Deleted hint file 21033fe2-dca9-42a5-8e68-496dd3517f53-1525510690633-1.hints

INFO  [HintsDispatcher:44] 2018-05-05 16:58:33,913 
HintsDispatchExecutor.java:282 - Finished hinted handoff of file 
21033fe2-dca9-42a5-8e68-496dd3517f53-1525510690633-1.hints to endpoint 
/10.175.20.113: 21033fe2-dca9-42a5-8e68-496dd3517f53

INFO  [epollEventLoopGroup-2-10] 2018-05-05 16:58:41,750 Message.java:623 - 
Unexpected exception during request; channel = [id: 0xc640c7a2, 
L:/10.175.20.112:9042 - R:/10.175.20.37:56784]

io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)() failed: 
Connection reset by peer

at io.netty.channel.unix.FileDescriptor.readAddress(...)(Unknown 
Source) ~[netty-all-4.0.44.Final.jar:4.0.44.Final]

the limit setting in CentOS 6.8 is setting as below :

* hard memlock unlimited

* hard nofile 10

* soft nofile 10

* soft memlock unlimited

* hard nproc 32768

* hard as unlimited

* soft nproc 32768

* soft as unlimited

root soft nofile 32768

root hard nofile 32768



  can I ignore this error or do you have any suggestions about this error?



Best Regards,



倪项菲/ David Ni

中移德电网络科技有限公司

Virtue Intelligent Network Ltd, co.

Add: 2003,20F No.35 Luojia creative city,Luoyu Road,Wuhan,HuBei

Mob: +86 13797007811|Tel: + 86 27 5024 2516