CPU high load

[Aoi Kadoya]

Hi,

I am running 6 nodes vnode cluster with DSE 4.8.1, and since few weeks
ago, all of the cluster nodes are hitting avg. 15-20 cpu load.
These nodes are running on VMs(VMware vSphere) that have 8vcpu
(1core/socket)-16 vRAM.(JVM options : -Xms8G -Xmx8G -Xmn800M)

At first I thought this is because of CPU iowait, however, iowait is
constantly low(in fact it's 0 almost all time time), CPU steal time is
also 0%.

When I took a thread dump, I found some of "SharedPool-Worker" threads
are consuming CPU and those threads seem to be waiting for something
so I assume this is the cause of cpu load.

"SharedPool-Worker-1" #240 daemon prio=5 os_prio=0
tid=0x7fabf459e000 nid=0x39b3 waiting on condition
[0x7faad7f02000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:85)
at java.lang.Thread.run(Thread.java:745)

Thread dump looks like this, but I am not sure what is this
sharedpool-worker waiting for.
Would you please help me with the further trouble shooting?
I am also reading the thread posted by Yuan as the situation is very
similar to mine but I didn't get any blocked, dropped or pending count
in my tpstat result.

Thanks,
Aoi

Re: Is my cluster normal?

[Yuan Fang]

Hi Jonathan,

Here is the result:

ubuntu@ip-172-31-44-250:~$ iostat -dmx 2 10
Linux 3.13.0-74-generic (ip-172-31-44-250) 07/12/2016 _x86_64_ (4 CPU)

Device: rrqm/s   wrqm/s r/s w/srMB/swMB/s avgrq-sz
avgqu-sz   await r_await w_await  svctm  %util
xvda  0.01 2.130.741.55 0.01 0.0227.77
0.000.740.890.66   0.43   0.10
xvdf  0.01 0.58  237.41   52.5012.90 6.21   135.02
2.328.013.65   27.72   0.57  16.63

Device: rrqm/s   wrqm/s r/s w/srMB/swMB/s avgrq-sz
avgqu-sz   await r_await w_await  svctm  %util
xvda  0.00 7.500.002.50 0.00 0.0432.00
0.001.600.001.60   1.60   0.40
xvdf  0.00 0.00  353.500.0024.12 0.00   139.75
0.491.371.370.00   0.58  20.60

Device: rrqm/s   wrqm/s r/s w/srMB/swMB/s avgrq-sz
avgqu-sz   await r_await w_await  svctm  %util
xvda  0.00 0.000.001.00 0.00 0.00 8.00
0.000.000.000.00   0.00   0.00
xvdf  0.00 2.00  463.50   35.0030.69 2.86   137.84
0.881.771.298.17   0.60  30.00

Device: rrqm/s   wrqm/s r/s w/srMB/swMB/s avgrq-sz
avgqu-sz   await r_await w_await  svctm  %util
xvda  0.00 0.000.001.00 0.00 0.00 8.00
0.000.000.000.00   0.00   0.00
xvdf  0.00 0.00   99.50   36.00 8.54 4.40   195.62
1.553.881.45   10.61   1.06  14.40

Device: rrqm/s   wrqm/s r/s w/srMB/swMB/s avgrq-sz
avgqu-sz   await r_await w_await  svctm  %util
xvda  0.00 5.000.001.50 0.00 0.0334.67
0.001.330.001.33   1.33   0.20
xvdf  0.00 1.50  703.00  195.0048.8323.76   165.57
6.498.361.66   32.51   0.55  49.80

Device: rrqm/s   wrqm/s r/s w/srMB/swMB/s avgrq-sz
avgqu-sz   await r_await w_await  svctm  %util
xvda  0.00 0.000.001.00 0.00 0.0472.00
0.000.000.000.00   0.00   0.00
xvdf  0.00 2.50  149.50   69.5010.12 6.68   157.14
0.743.421.188.23   0.51  11.20

Device: rrqm/s   wrqm/s r/s w/srMB/swMB/s avgrq-sz
avgqu-sz   await r_await w_await  svctm  %util
xvda  0.00 5.000.002.50 0.00 0.0324.00
0.000.000.000.00   0.00   0.00
xvdf  0.00 0.00   61.50   22.50 5.36 2.75   197.64
0.333.931.50   10.58   0.88   7.40

Device: rrqm/s   wrqm/s r/s w/srMB/swMB/s avgrq-sz
avgqu-sz   await r_await w_await  svctm  %util
xvda  0.00 0.000.000.50 0.00 0.00 8.00
0.000.000.000.00   0.00   0.00
xvdf  0.00 0.00  375.000.0024.84 0.00   135.64
0.451.201.200.00   0.57  21.20

Device: rrqm/s   wrqm/s r/s w/srMB/swMB/s avgrq-sz
avgqu-sz   await r_await w_await  svctm  %util
xvda  0.00 1.000.006.00 0.00 0.03 9.33
0.000.000.000.00   0.00   0.00
xvdf  0.00 0.00  542.50   23.5035.08 2.83   137.16
0.801.411.157.23   0.49  28.00

Device: rrqm/s   wrqm/s r/s w/srMB/swMB/s avgrq-sz
avgqu-sz   await r_await w_await  svctm  %util
xvda  0.00 3.500.501.50 0.00 0.0224.00
0.000.000.000.00   0.00   0.00
xvdf  0.00 1.50  272.00  153.5016.1818.67   167.73
   14.32   33.661.39   90.84   0.81  34.60



On Tue, Jul 12, 2016 at 12:34 PM, Jonathan Haddad  wrote:

> When you have high system load it means your CPU is waiting for
> *something*, and in my experience it's usually slow disk.  A disk connected
> over network has been a culprit for me many times.
>
> On Tue, Jul 12, 2016 at 12:33 PM Jonathan Haddad 
> wrote:
>
>> Can do you do:
>>
>> iostat -dmx 2 10
>>
>>
>>
>> On Tue, Jul 12, 2016 at 11:20 AM Yuan Fang  wrote:
>>
>>> Hi Jeff,
>>>
>>> The read being low is because we do not have much read operations right
>>> now.
>>>
>>> The heap is only 4GB.
>>>
>>> MAX_HEAP_SIZE=4GB
>>>
>>> On Thu, Jul 7, 2016 at 7:17 PM, Jeff Jirsa 
>>> wrote:
>>>
 EBS iops scale with volume size.



 A 600G EBS volume only guarantees 1800 iops – if you’re exhausting
 those on writes, you’re going to suffer on reads.



 You have a 16G server, and probably a good chunk of that allocated to
 heap. Consequently, you have almost no page cache, so your reads are going
 to hit the disk. Your reads being very low is not uncommon if you have no

Re: Is my cluster normal?

[Riccardo Ferrari]

While I'm surprised you don't have any dropped message I have to point the
finger against the following tpstats line:

Native-Transport-Requests   128   128  347508531 2
 15891862

where the the first '128' are the active reuests and the second '128' are
the pending ones. Might not be strictly related, however this might be of
interest:

https://issues.apache.org/jira/browse/CASSANDRA-11363

there's a chance that tuning the 'native_transport_*' related options can
mitigate/solve the issue.

Best,

On Tue, Jul 12, 2016 at 9:34 PM, Jonathan Haddad  wrote:

> When you have high system load it means your CPU is waiting for
> *something*, and in my experience it's usually slow disk.  A disk connected
> over network has been a culprit for me many times.
>
> On Tue, Jul 12, 2016 at 12:33 PM Jonathan Haddad 
> wrote:
>
>> Can do you do:
>>
>> iostat -dmx 2 10
>>
>>
>>
>> On Tue, Jul 12, 2016 at 11:20 AM Yuan Fang  wrote:
>>
>>> Hi Jeff,
>>>
>>> The read being low is because we do not have much read operations right
>>> now.
>>>
>>> The heap is only 4GB.
>>>
>>> MAX_HEAP_SIZE=4GB
>>>
>>> On Thu, Jul 7, 2016 at 7:17 PM, Jeff Jirsa 
>>> wrote:
>>>
 EBS iops scale with volume size.



 A 600G EBS volume only guarantees 1800 iops – if you’re exhausting
 those on writes, you’re going to suffer on reads.



 You have a 16G server, and probably a good chunk of that allocated to
 heap. Consequently, you have almost no page cache, so your reads are going
 to hit the disk. Your reads being very low is not uncommon if you have no
 page cache – the default settings for Cassandra (64k compression chunks)
 are really inefficient for small reads served off of disk. If you drop the
 compression chunk size (4k, for example), you’ll probably see your read
 throughput increase significantly, which will give you more iops for
 commitlog, so write throughput likely goes up, too.







 *From: *Jonathan Haddad 
 *Reply-To: *"user@cassandra.apache.org" 
 *Date: *Thursday, July 7, 2016 at 6:54 PM
 *To: *"user@cassandra.apache.org" 
 *Subject: *Re: Is my cluster normal?



 What's your CPU looking like? If it's low, check your IO with iostat or
 dstat. I know some people have used Ebs and say it's fine but ive been
 burned too many times.

 On Thu, Jul 7, 2016 at 6:12 PM Yuan Fang  wrote:

 Hi Riccardo,



 Very low IO-wait. About 0.3%.

 No stolen CPU. It is a casssandra only instance. I did not see any
 dropped messages.





 ubuntu@cassandra1:/mnt/data$ nodetool tpstats

 Pool NameActive   Pending  Completed   Blocked
  All time blocked

 MutationStage 1 1  929509244 0
 0

 ViewMutationStage 0 0  0 0
 0

 ReadStage 4 04021570 0
 0

 RequestResponseStage  0 0  731477999 0
 0

 ReadRepairStage   0 0 165603 0
 0

 CounterMutationStage  0 0  0 0
 0

 MiscStage 0 0  0 0
 0

 CompactionExecutor255  92022 0
 0

 MemtableReclaimMemory 0 0   1736 0
 0

 PendingRangeCalculator0 0  6 0
 0

 GossipStage   0 0 345474 0
 0

 SecondaryIndexManagement  0 0  0 0
 0

 HintsDispatcher   0 0  4 0
 0

 MigrationStage0 0 35 0
 0

 MemtablePostFlush 0 0   1973 0
 0

 ValidationExecutor0 0  0 0
 0

 Sampler   0 0  0 0
 0

 MemtableFlushWriter   0 0   1736 0
 0

 InternalResponseStage 0 0   5311 0
 0

 

Re: NoHostAvailableException coming up on our server

[Abhinav Solan]

I am using 3.0.0 version over apache-cassandra-3.3

On Tue, Jul 12, 2016 at 2:37 PM Riccardo Ferrari  wrote:

> What driver version are you using?
>
> You can look at the LoggingRetryPolicy to have more meaningful messages in
> your logs.
>
> best,
>
> On Tue, Jul 12, 2016 at 9:02 PM, Abhinav Solan 
> wrote:
>
>> Thanks, Johnny
>> Actually, they were running .. it went through a series of read and
>> writes .. and recovered after the error.
>> Is there any settings I can specify in preparing the Session at java
>> client driver level, here are my current settings -
>>
>> PoolingOptions poolingOptions = new PoolingOptions()
>>  .setConnectionsPerHost(HostDistance.LOCAL, 20, 20)
>>  .setMaxRequestsPerConnection(HostDistance.LOCAL, 128)
>>  .setNewConnectionThreshold(HostDistance.LOCAL, 100);
>>
>>  Cluster.Builder builder = Cluster.builder()
>>  .addContactPoints(cp)
>>  .withPoolingOptions(poolingOptions)
>>  .withProtocolVersion(ProtocolVersion.NEWEST_SUPPORTED)
>>  .withPort(port);
>>
>>
>>
>> On Tue, Jul 12, 2016 at 11:47 AM Johnny Miller 
>> wrote:
>>
>>> Abhinav - your getting that as the driver isn’t finding any hosts up for
>>> your query. You probably need to check if all the nodes in your cluster are
>>> running.
>>>
>>> See:
>>> http://docs.datastax.com/en/drivers/java/3.0/com/datastax/driver/core/exceptions/NoHostAvailableException.html
>>>
>>>
>>> Johnny
>>>
>>> On 12 Jul 2016, at 18:46, Abhinav Solan  wrote:
>>>
>>> Hi Everyone,
>>>
>>> I am getting this error on our server, it comes and goes seems the
>>> connection drops a comes back after a while -
>>>
>>> Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: 
>>> All host(s) tried for query failed (tried: :9042 
>>> (com.datastax.driver.core.exceptions.ConnectionException: 
>>> [] Pool is CLOSING))
>>> at 
>>> com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:218)
>>> at 
>>> com.datastax.driver.core.RequestHandler.access$1000(RequestHandler.java:43)
>>> at 
>>> com.datastax.driver.core.RequestHandler$SpeculativeExecution.sendRequest(RequestHandler.java:284)
>>> at 
>>> com.datastax.driver.core.RequestHandler.startNewExecution(RequestHandler.java:115)
>>> at 
>>> com.datastax.driver.core.RequestHandler.sendRequest(RequestHandler.java:91)
>>> at 
>>> com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:129)
>>>
>>> Can anyone suggest me what can be done to handle this error ?
>>>
>>>
>>> Thanks,
>>>
>>> Abhinav
>>>
>>>
>>>
>

Re: NoHostAvailableException coming up on our server

[Riccardo Ferrari]

What driver version are you using?

You can look at the LoggingRetryPolicy to have more meaningful messages in
your logs.

best,

On Tue, Jul 12, 2016 at 9:02 PM, Abhinav Solan 
wrote:

> Thanks, Johnny
> Actually, they were running .. it went through a series of read and writes
> .. and recovered after the error.
> Is there any settings I can specify in preparing the Session at java
> client driver level, here are my current settings -
>
> PoolingOptions poolingOptions = new PoolingOptions()
>  .setConnectionsPerHost(HostDistance.LOCAL, 20, 20)
>  .setMaxRequestsPerConnection(HostDistance.LOCAL, 128)
>  .setNewConnectionThreshold(HostDistance.LOCAL, 100);
>
>  Cluster.Builder builder = Cluster.builder()
>  .addContactPoints(cp)
>  .withPoolingOptions(poolingOptions)
>  .withProtocolVersion(ProtocolVersion.NEWEST_SUPPORTED)
>  .withPort(port);
>
>
>
> On Tue, Jul 12, 2016 at 11:47 AM Johnny Miller 
> wrote:
>
>> Abhinav - your getting that as the driver isn’t finding any hosts up for
>> your query. You probably need to check if all the nodes in your cluster are
>> running.
>>
>> See:
>> http://docs.datastax.com/en/drivers/java/3.0/com/datastax/driver/core/exceptions/NoHostAvailableException.html
>>
>>
>> Johnny
>>
>> On 12 Jul 2016, at 18:46, Abhinav Solan  wrote:
>>
>> Hi Everyone,
>>
>> I am getting this error on our server, it comes and goes seems the
>> connection drops a comes back after a while -
>>
>> Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All 
>> host(s) tried for query failed (tried: :9042 
>> (com.datastax.driver.core.exceptions.ConnectionException: [] 
>> Pool is CLOSING))
>>  at 
>> com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:218)
>>  at 
>> com.datastax.driver.core.RequestHandler.access$1000(RequestHandler.java:43)
>>  at 
>> com.datastax.driver.core.RequestHandler$SpeculativeExecution.sendRequest(RequestHandler.java:284)
>>  at 
>> com.datastax.driver.core.RequestHandler.startNewExecution(RequestHandler.java:115)
>>  at 
>> com.datastax.driver.core.RequestHandler.sendRequest(RequestHandler.java:91)
>>  at 
>> com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:129)
>>
>> Can anyone suggest me what can be done to handle this error ?
>>
>>
>> Thanks,
>>
>> Abhinav
>>
>>
>>

Re: Questions on LCS behaviour after big BulkLoad cluster bootstrap

[Johnny Miller]

Garo,

When your loading data in using LCS in a bulk fashion like this, there are a 
few things you should do.

You can disable STCS in L0 
(https://issues.apache.org/jira/browse/CASSANDRA-6621 
) with the JVM flag 
"-Dcassandra.disable_stcs_in_l0=true” which should stop you getting huge 
sstables in L0 while LCS is catching up. Once the load is complete you can then 
shutdown the node and perform an sstableofflinerelevel 
(https://docs.datastax.com/en/cassandra/2.2/cassandra/tools/toolsSSTableOfflineRelevel.html
 
)
 . This should help LCS catch up with things and reduce the pending compactions 
etc. however, it may just take a while to catchup still

Hope this helps.

Johnny

> On 6 Jul 2016, at 07:56, Juho Mäkinen  wrote:
> 
> Hello. I'm in the process of migrating my old 60 node cluster into a new 72 
> node cluster running 2.2.6. I fired BulkLoader on the old cluster to stream 
> all data from every node in the old cluster to my new cluster, and I'm now 
> watching as my new cluster is doing compactions. What I like is to understand 
> the LeveledCompactionStrategy behaviour in more detail.
> 
> I'm taking one node as an example, but all other nodes have quite same 
> situation.
> 
> There are 53 live SSTables in a big table. This can be seen both by looking 
> la-*Data.db files and also with nodetool cfstats: "SSTables in each level: 
> [31/4, 10, 12, 0, 0, 0, 0, 0, 0]"
> 
> If I look on the SSTable files in the disk I see some huge SSTables, like a 
> 37 GiB, 57 GiB, 74 GiB, which are all on Level 0 (used sstablemetadata to see 
> this). The size of all live sstables are about 920 GiB.
> 
> Then there are tmp-la-*Data.db and tmplink-la-*Data.db files (the tmplink 
> files are hardlinks to the tmp file due to CASSANDRA-6916). I guess that 
> these come from the single active compaction. The total size of these files 
> are around ~65 GiB.
> 
> On the compaction side compactionstats shows that there's just one compaction 
> running, which is heavily CPU bound: (I've reformatted the output here)
> pending tasks: 5390
> bytes done: 673623792733 (673 GiB)
> bytes left: 3325656896682 (3325 GiB)
> Active compaction remaining time :   2h44m39s
> 
> Why is the bytes done and especially bytes left such big? I don't have that 
> much data in my node.
> 
> Also how does Cassandra calculate the pending tasks with LCS?
> 
> Why are there a few such big SSTables in the active sstable list? Is it 
> because LCS falls back to STCS if L0 is too full? Should I use the 
> stcs_im_l0:false option? What will happen to these big sstables in the future?
> 
> I'm currently just waiting for the compactions to eventually finish, but I'm 
> hoping to learn in more detail what the system does and possibly to help 
> similar migration in the future.
> 
> Thanks,
> 
>  - Garo
> 
>  
> 
> 

Re: (C)* stable version after 3.5

[Johnny Miller]

Hey Varun - yeah, I would stick to 3.0.x for stability reasons unless there is 
a feature in the later releases you want. You should be able to upgrade 
directly to 3.0 from 2.1 without going via 2.2

> On 12 Jul 2016, at 15:01, Alain RODRIGUEZ  wrote:
> 
> Hi,
> 
> The only "fix" release after 3.5 is 3.7. Yet hard to say if it is more 
> stable, we can hope so.
> 
> For Tic-Toc releases (on 3.X)
> 
> Odd numbers are fix releases.
> Even numbers are feature releases.
> 
> Not sure why you want something above 3.5, but take care, those versions are 
> really recent, and less tested so maybe not that "stable". If you want 
> something more stable, I believe you can go with 3.0.8.
> 
> Yet I am not telling you not to do that, some people need to start testing 
> new things right... So if you choose 3.7 because you want some feature from 
> there, it is perfectly ok, just move carefully, maybe read some opened 
> tickets and previous experiences from the community and test the upgrade 
> process first on a dev cluster.
> 
> Also I am not sure if the 2.2 major version is something you can skip while 
> upgrading through a rolling restart. I believe you can, but it is not what is 
> recommended. Testing will let you know anyway.
> 
> Good luck and tell us how it went :-).
> 
> C*heers,
> ---
> Alain Rodriguez - al...@thelastpickle.com 
> France
> 
> The Last Pickle - Apache Cassandra Consulting
> http://www.thelastpickle.com 
> 
> 2016-07-12 11:05 GMT+02:00 Varun Barala  >:
> Hi all users,
> 
> 
> Currently we are using cassandra-2.1.13 but we want to upgrade 2.1.13 to 3.x 
> in production.
> 
> Could anyone please tell me which is the most stable cassandra version after 
> 3.5.
> 
> Thanking You!!
> 
> 
> Regards,
> Varun Barala
> 

Re: Is my cluster normal?

[Jonathan Haddad]

When you have high system load it means your CPU is waiting for
*something*, and in my experience it's usually slow disk.  A disk connected
over network has been a culprit for me many times.

On Tue, Jul 12, 2016 at 12:33 PM Jonathan Haddad  wrote:

> Can do you do:
>
> iostat -dmx 2 10
>
>
>
> On Tue, Jul 12, 2016 at 11:20 AM Yuan Fang  wrote:
>
>> Hi Jeff,
>>
>> The read being low is because we do not have much read operations right
>> now.
>>
>> The heap is only 4GB.
>>
>> MAX_HEAP_SIZE=4GB
>>
>> On Thu, Jul 7, 2016 at 7:17 PM, Jeff Jirsa 
>> wrote:
>>
>>> EBS iops scale with volume size.
>>>
>>>
>>>
>>> A 600G EBS volume only guarantees 1800 iops – if you’re exhausting those
>>> on writes, you’re going to suffer on reads.
>>>
>>>
>>>
>>> You have a 16G server, and probably a good chunk of that allocated to
>>> heap. Consequently, you have almost no page cache, so your reads are going
>>> to hit the disk. Your reads being very low is not uncommon if you have no
>>> page cache – the default settings for Cassandra (64k compression chunks)
>>> are really inefficient for small reads served off of disk. If you drop the
>>> compression chunk size (4k, for example), you’ll probably see your read
>>> throughput increase significantly, which will give you more iops for
>>> commitlog, so write throughput likely goes up, too.
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> *From: *Jonathan Haddad 
>>> *Reply-To: *"user@cassandra.apache.org" 
>>> *Date: *Thursday, July 7, 2016 at 6:54 PM
>>> *To: *"user@cassandra.apache.org" 
>>> *Subject: *Re: Is my cluster normal?
>>>
>>>
>>>
>>> What's your CPU looking like? If it's low, check your IO with iostat or
>>> dstat. I know some people have used Ebs and say it's fine but ive been
>>> burned too many times.
>>>
>>> On Thu, Jul 7, 2016 at 6:12 PM Yuan Fang  wrote:
>>>
>>> Hi Riccardo,
>>>
>>>
>>>
>>> Very low IO-wait. About 0.3%.
>>>
>>> No stolen CPU. It is a casssandra only instance. I did not see any
>>> dropped messages.
>>>
>>>
>>>
>>>
>>>
>>> ubuntu@cassandra1:/mnt/data$ nodetool tpstats
>>>
>>> Pool NameActive   Pending  Completed   Blocked
>>>  All time blocked
>>>
>>> MutationStage 1 1  929509244 0
>>>   0
>>>
>>> ViewMutationStage 0 0  0 0
>>>   0
>>>
>>> ReadStage 4 04021570 0
>>>   0
>>>
>>> RequestResponseStage  0 0  731477999 0
>>>   0
>>>
>>> ReadRepairStage   0 0 165603 0
>>>   0
>>>
>>> CounterMutationStage  0 0  0 0
>>>   0
>>>
>>> MiscStage 0 0  0 0
>>>   0
>>>
>>> CompactionExecutor255  92022 0
>>>   0
>>>
>>> MemtableReclaimMemory 0 0   1736 0
>>>   0
>>>
>>> PendingRangeCalculator0 0  6 0
>>>   0
>>>
>>> GossipStage   0 0 345474 0
>>>   0
>>>
>>> SecondaryIndexManagement  0 0  0 0
>>>   0
>>>
>>> HintsDispatcher   0 0  4 0
>>>   0
>>>
>>> MigrationStage0 0 35 0
>>>   0
>>>
>>> MemtablePostFlush 0 0   1973 0
>>>   0
>>>
>>> ValidationExecutor0 0  0 0
>>>   0
>>>
>>> Sampler   0 0  0 0
>>>   0
>>>
>>> MemtableFlushWriter   0 0   1736 0
>>>   0
>>>
>>> InternalResponseStage 0 0   5311 0
>>>   0
>>>
>>> AntiEntropyStage  0 0  0 0
>>>   0
>>>
>>> CacheCleanupExecutor  0 0  0 0
>>>   0
>>>
>>> Native-Transport-Requests   128   128  347508531 2
>>>15891862
>>>
>>>
>>>
>>> Message type   Dropped
>>>
>>> READ 0
>>>
>>> RANGE_SLICE  0
>>>
>>> _TRACE   0
>>>
>>> HINT 0
>>>
>>> MUTATION 0
>>>
>>> COUNTER_MUTATION 0
>>>
>>> BATCH_STORE  0
>>>
>>> BATCH_REMOVE 0
>>>
>>> REQUEST_RESPONSE 0
>>>
>>> PAGED_RANGE  0
>>>
>>> READ_REPAIR  0
>>>
>>>
>>>
>>>
>>>
>>>

Re: Is my cluster normal?

[Jonathan Haddad]

Can do you do:

iostat -dmx 2 10



On Tue, Jul 12, 2016 at 11:20 AM Yuan Fang  wrote:

> Hi Jeff,
>
> The read being low is because we do not have much read operations right
> now.
>
> The heap is only 4GB.
>
> MAX_HEAP_SIZE=4GB
>
> On Thu, Jul 7, 2016 at 7:17 PM, Jeff Jirsa 
> wrote:
>
>> EBS iops scale with volume size.
>>
>>
>>
>> A 600G EBS volume only guarantees 1800 iops – if you’re exhausting those
>> on writes, you’re going to suffer on reads.
>>
>>
>>
>> You have a 16G server, and probably a good chunk of that allocated to
>> heap. Consequently, you have almost no page cache, so your reads are going
>> to hit the disk. Your reads being very low is not uncommon if you have no
>> page cache – the default settings for Cassandra (64k compression chunks)
>> are really inefficient for small reads served off of disk. If you drop the
>> compression chunk size (4k, for example), you’ll probably see your read
>> throughput increase significantly, which will give you more iops for
>> commitlog, so write throughput likely goes up, too.
>>
>>
>>
>>
>>
>>
>>
>> *From: *Jonathan Haddad 
>> *Reply-To: *"user@cassandra.apache.org" 
>> *Date: *Thursday, July 7, 2016 at 6:54 PM
>> *To: *"user@cassandra.apache.org" 
>> *Subject: *Re: Is my cluster normal?
>>
>>
>>
>> What's your CPU looking like? If it's low, check your IO with iostat or
>> dstat. I know some people have used Ebs and say it's fine but ive been
>> burned too many times.
>>
>> On Thu, Jul 7, 2016 at 6:12 PM Yuan Fang  wrote:
>>
>> Hi Riccardo,
>>
>>
>>
>> Very low IO-wait. About 0.3%.
>>
>> No stolen CPU. It is a casssandra only instance. I did not see any
>> dropped messages.
>>
>>
>>
>>
>>
>> ubuntu@cassandra1:/mnt/data$ nodetool tpstats
>>
>> Pool NameActive   Pending  Completed   Blocked
>>  All time blocked
>>
>> MutationStage 1 1  929509244 0
>>   0
>>
>> ViewMutationStage 0 0  0 0
>>   0
>>
>> ReadStage 4 04021570 0
>>   0
>>
>> RequestResponseStage  0 0  731477999 0
>>   0
>>
>> ReadRepairStage   0 0 165603 0
>>   0
>>
>> CounterMutationStage  0 0  0 0
>>   0
>>
>> MiscStage 0 0  0 0
>>   0
>>
>> CompactionExecutor255  92022 0
>>   0
>>
>> MemtableReclaimMemory 0 0   1736 0
>>   0
>>
>> PendingRangeCalculator0 0  6 0
>>   0
>>
>> GossipStage   0 0 345474 0
>>   0
>>
>> SecondaryIndexManagement  0 0  0 0
>>   0
>>
>> HintsDispatcher   0 0  4 0
>>   0
>>
>> MigrationStage0 0 35 0
>>   0
>>
>> MemtablePostFlush 0 0   1973 0
>>   0
>>
>> ValidationExecutor0 0  0 0
>>   0
>>
>> Sampler   0 0  0 0
>>   0
>>
>> MemtableFlushWriter   0 0   1736 0
>>   0
>>
>> InternalResponseStage 0 0   5311 0
>>   0
>>
>> AntiEntropyStage  0 0  0 0
>>   0
>>
>> CacheCleanupExecutor  0 0  0 0
>>   0
>>
>> Native-Transport-Requests   128   128  347508531 2
>>15891862
>>
>>
>>
>> Message type   Dropped
>>
>> READ 0
>>
>> RANGE_SLICE  0
>>
>> _TRACE   0
>>
>> HINT 0
>>
>> MUTATION 0
>>
>> COUNTER_MUTATION 0
>>
>> BATCH_STORE  0
>>
>> BATCH_REMOVE 0
>>
>> REQUEST_RESPONSE 0
>>
>> PAGED_RANGE  0
>>
>> READ_REPAIR  0
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> On Thu, Jul 7, 2016 at 5:24 PM, Riccardo Ferrari 
>> wrote:
>>
>> Hi Yuan,
>>
>>
>>
>> You machine instance is 4 vcpus that is 4 threads (not cores!!!), aside
>> from any Cassandra specific discussion a system load of 10 on a 4 threads
>> machine is way too much in my opinion. If that is the running average
>> system load I would look deeper into system details. Is that IO wait? Is
>> that CPU Stolen? Is 

Re: NoHostAvailableException coming up on our server

[Abhinav Solan]

Thanks, Johnny
Actually, they were running .. it went through a series of read and writes
.. and recovered after the error.
Is there any settings I can specify in preparing the Session at java client
driver level, here are my current settings -

PoolingOptions poolingOptions = new PoolingOptions()
 .setConnectionsPerHost(HostDistance.LOCAL, 20, 20)
 .setMaxRequestsPerConnection(HostDistance.LOCAL, 128)
 .setNewConnectionThreshold(HostDistance.LOCAL, 100);

 Cluster.Builder builder = Cluster.builder()
 .addContactPoints(cp)
 .withPoolingOptions(poolingOptions)
 .withProtocolVersion(ProtocolVersion.NEWEST_SUPPORTED)
 .withPort(port);



On Tue, Jul 12, 2016 at 11:47 AM Johnny Miller 
wrote:

> Abhinav - your getting that as the driver isn’t finding any hosts up for
> your query. You probably need to check if all the nodes in your cluster are
> running.
>
> See:
> http://docs.datastax.com/en/drivers/java/3.0/com/datastax/driver/core/exceptions/NoHostAvailableException.html
>
>
> Johnny
>
> On 12 Jul 2016, at 18:46, Abhinav Solan  wrote:
>
> Hi Everyone,
>
> I am getting this error on our server, it comes and goes seems the
> connection drops a comes back after a while -
>
> Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All 
> host(s) tried for query failed (tried: :9042 
> (com.datastax.driver.core.exceptions.ConnectionException: [] 
> Pool is CLOSING))
>   at 
> com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:218)
>   at 
> com.datastax.driver.core.RequestHandler.access$1000(RequestHandler.java:43)
>   at 
> com.datastax.driver.core.RequestHandler$SpeculativeExecution.sendRequest(RequestHandler.java:284)
>   at 
> com.datastax.driver.core.RequestHandler.startNewExecution(RequestHandler.java:115)
>   at 
> com.datastax.driver.core.RequestHandler.sendRequest(RequestHandler.java:91)
>   at 
> com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:129)
>
> Can anyone suggest me what can be done to handle this error ?
>
>
> Thanks,
>
> Abhinav
>
>
>

Re: NoHostAvailableException coming up on our server

[Johnny Miller]

Abhinav - your getting that as the driver isn’t finding any hosts up for your 
query. You probably need to check if all the nodes in your cluster are running.

See: 
http://docs.datastax.com/en/drivers/java/3.0/com/datastax/driver/core/exceptions/NoHostAvailableException.html

Johnny

> On 12 Jul 2016, at 18:46, Abhinav Solan  wrote:
> 
> Hi Everyone,
> 
> I am getting this error on our server, it comes and goes seems the connection 
> drops a comes back after a while -
> Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All 
> host(s) tried for query failed (tried: :9042 
> (com.datastax.driver.core.exceptions.ConnectionException: [] 
> Pool is CLOSING))
>   at 
> com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:218)
>   at 
> com.datastax.driver.core.RequestHandler.access$1000(RequestHandler.java:43)
>   at 
> com.datastax.driver.core.RequestHandler$SpeculativeExecution.sendRequest(RequestHandler.java:284)
>   at 
> com.datastax.driver.core.RequestHandler.startNewExecution(RequestHandler.java:115)
>   at 
> com.datastax.driver.core.RequestHandler.sendRequest(RequestHandler.java:91)
>   at 
> com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:129)
> Can anyone suggest me what can be done to handle this error ? 
> 
> Thanks,
> Abhinav

Re: Is my cluster normal?

[Yuan Fang]

Hi Jeff,

The read being low is because we do not have much read operations right now.

The heap is only 4GB.

MAX_HEAP_SIZE=4GB

On Thu, Jul 7, 2016 at 7:17 PM, Jeff Jirsa 
wrote:

> EBS iops scale with volume size.
>
>
>
> A 600G EBS volume only guarantees 1800 iops – if you’re exhausting those
> on writes, you’re going to suffer on reads.
>
>
>
> You have a 16G server, and probably a good chunk of that allocated to
> heap. Consequently, you have almost no page cache, so your reads are going
> to hit the disk. Your reads being very low is not uncommon if you have no
> page cache – the default settings for Cassandra (64k compression chunks)
> are really inefficient for small reads served off of disk. If you drop the
> compression chunk size (4k, for example), you’ll probably see your read
> throughput increase significantly, which will give you more iops for
> commitlog, so write throughput likely goes up, too.
>
>
>
>
>
>
>
> *From: *Jonathan Haddad 
> *Reply-To: *"user@cassandra.apache.org" 
> *Date: *Thursday, July 7, 2016 at 6:54 PM
> *To: *"user@cassandra.apache.org" 
> *Subject: *Re: Is my cluster normal?
>
>
>
> What's your CPU looking like? If it's low, check your IO with iostat or
> dstat. I know some people have used Ebs and say it's fine but ive been
> burned too many times.
>
> On Thu, Jul 7, 2016 at 6:12 PM Yuan Fang  wrote:
>
> Hi Riccardo,
>
>
>
> Very low IO-wait. About 0.3%.
>
> No stolen CPU. It is a casssandra only instance. I did not see any dropped
> messages.
>
>
>
>
>
> ubuntu@cassandra1:/mnt/data$ nodetool tpstats
>
> Pool NameActive   Pending  Completed   Blocked
>  All time blocked
>
> MutationStage 1 1  929509244 0
> 0
>
> ViewMutationStage 0 0  0 0
> 0
>
> ReadStage 4 04021570 0
> 0
>
> RequestResponseStage  0 0  731477999 0
> 0
>
> ReadRepairStage   0 0 165603 0
> 0
>
> CounterMutationStage  0 0  0 0
> 0
>
> MiscStage 0 0  0 0
> 0
>
> CompactionExecutor255  92022 0
> 0
>
> MemtableReclaimMemory 0 0   1736 0
> 0
>
> PendingRangeCalculator0 0  6 0
> 0
>
> GossipStage   0 0 345474 0
> 0
>
> SecondaryIndexManagement  0 0  0 0
> 0
>
> HintsDispatcher   0 0  4 0
> 0
>
> MigrationStage0 0 35 0
> 0
>
> MemtablePostFlush 0 0   1973 0
> 0
>
> ValidationExecutor0 0  0 0
> 0
>
> Sampler   0 0  0 0
> 0
>
> MemtableFlushWriter   0 0   1736 0
> 0
>
> InternalResponseStage 0 0   5311 0
> 0
>
> AntiEntropyStage  0 0  0 0
> 0
>
> CacheCleanupExecutor  0 0  0 0
> 0
>
> Native-Transport-Requests   128   128  347508531 2
>  15891862
>
>
>
> Message type   Dropped
>
> READ 0
>
> RANGE_SLICE  0
>
> _TRACE   0
>
> HINT 0
>
> MUTATION 0
>
> COUNTER_MUTATION 0
>
> BATCH_STORE  0
>
> BATCH_REMOVE 0
>
> REQUEST_RESPONSE 0
>
> PAGED_RANGE  0
>
> READ_REPAIR  0
>
>
>
>
>
>
>
>
>
>
>
> On Thu, Jul 7, 2016 at 5:24 PM, Riccardo Ferrari 
> wrote:
>
> Hi Yuan,
>
>
>
> You machine instance is 4 vcpus that is 4 threads (not cores!!!), aside
> from any Cassandra specific discussion a system load of 10 on a 4 threads
> machine is way too much in my opinion. If that is the running average
> system load I would look deeper into system details. Is that IO wait? Is
> that CPU Stolen? Is that a Cassandra only instance or are there other
> processes pushing the load?
>
> What does your "nodetool tpstats" say? Hoe many dropped messages do you
> have?
>
>
>
> Best,
>
>
>
> On Fri, Jul 8, 2016 at 12:34 AM, Yuan Fang  wrote:
>
> Thanks Ben! For the post, it seems they got a little better but similar
> result 

Re: Is my cluster normal?

[Yuan Fang]

Hi Jonathan,

The IOs are like below. I am not sure why one node always has a much bigger
KB_read/s than other nodes. It seems not good.


==
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
  54.78   24.489.350.960.08   10.35

Device:tpskB_read/skB_wrtn/skB_readkB_wrtn
xvda  2.3114.6417.9514153481734856
xvdf252.68 11789.51  6394.15 1139459318  617996710

=

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
  22.716.573.960.500.19   66.07

Device:tpskB_read/skB_wrtn/skB_readkB_wrtn
xvda  1.12 3.6310.593993540   11648848
xvdf 68.20   923.51  2526.86 1016095212 2780187819

===
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
  22.318.083.700.260.23   65.42

Device:tpskB_read/skB_wrtn/skB_readkB_wrtn
xvda  1.07 2.8710.893153996   11976704
xvdf 34.48   498.21  2293.70  547844196 257746


avg-cpu:  %user   %nice %system %iowait  %steal   %idle
  22.758.133.820.360.21   64.73

Device:tpskB_read/skB_wrtn/skB_readkB_wrtn
xvda  1.10 3.2011.333515752   12442344
xvdf 44.45   474.30  2511.71  520758840 2757732583






On Thu, Jul 7, 2016 at 6:54 PM, Jonathan Haddad  wrote:

> What's your CPU looking like? If it's low, check your IO with iostat or
> dstat. I know some people have used Ebs and say it's fine but ive been
> burned too many times.
> On Thu, Jul 7, 2016 at 6:12 PM Yuan Fang  wrote:
>
>> Hi Riccardo,
>>
>> Very low IO-wait. About 0.3%.
>> No stolen CPU. It is a casssandra only instance. I did not see any
>> dropped messages.
>>
>>
>> ubuntu@cassandra1:/mnt/data$ nodetool tpstats
>> Pool NameActive   Pending  Completed   Blocked
>>  All time blocked
>> MutationStage 1 1  929509244 0
>>   0
>> ViewMutationStage 0 0  0 0
>>   0
>> ReadStage 4 04021570 0
>>   0
>> RequestResponseStage  0 0  731477999 0
>>   0
>> ReadRepairStage   0 0 165603 0
>>   0
>> CounterMutationStage  0 0  0 0
>>   0
>> MiscStage 0 0  0 0
>>   0
>> CompactionExecutor255  92022 0
>>   0
>> MemtableReclaimMemory 0 0   1736 0
>>   0
>> PendingRangeCalculator0 0  6 0
>>   0
>> GossipStage   0 0 345474 0
>>   0
>> SecondaryIndexManagement  0 0  0 0
>>   0
>> HintsDispatcher   0 0  4 0
>>   0
>> MigrationStage0 0 35 0
>>   0
>> MemtablePostFlush 0 0   1973 0
>>   0
>> ValidationExecutor0 0  0 0
>>   0
>> Sampler   0 0  0 0
>>   0
>> MemtableFlushWriter   0 0   1736 0
>>   0
>> InternalResponseStage 0 0   5311 0
>>   0
>> AntiEntropyStage  0 0  0 0
>>   0
>> CacheCleanupExecutor  0 0  0 0
>>   0
>> Native-Transport-Requests   128   128  347508531 2
>>15891862
>>
>> Message type   Dropped
>> READ 0
>> RANGE_SLICE  0
>> _TRACE   0
>> HINT 0
>> MUTATION 0
>> COUNTER_MUTATION 0
>> BATCH_STORE  0
>> BATCH_REMOVE 0
>> REQUEST_RESPONSE 0
>> PAGED_RANGE  0
>> READ_REPAIR  0
>>
>>
>>
>>
>>
>> On Thu, Jul 7, 2016 at 5:24 PM, Riccardo Ferrari 
>> wrote:
>>
>>> Hi Yuan,
>>>
>>> You machine instance is 4 vcpus that is 4 threads (not cores!!!), aside
>>> from any Cassandra specific discussion a system load of 10 on a 4 threads
>>> machine is way too much in my opinion. If that is the running average
>>> system load I would look deeper into system details. Is 

NoHostAvailableException coming up on our server

[Abhinav Solan]

Hi Everyone,

I am getting this error on our server, it comes and goes seems the
connection drops a comes back after a while -

Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException:
All host(s) tried for query failed (tried: :9042
(com.datastax.driver.core.exceptions.ConnectionException:
[] Pool is CLOSING))
at 
com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:218)
at 
com.datastax.driver.core.RequestHandler.access$1000(RequestHandler.java:43)
at 
com.datastax.driver.core.RequestHandler$SpeculativeExecution.sendRequest(RequestHandler.java:284)
at 
com.datastax.driver.core.RequestHandler.startNewExecution(RequestHandler.java:115)
at 
com.datastax.driver.core.RequestHandler.sendRequest(RequestHandler.java:91)
at 
com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:129)

Can anyone suggest me what can be done to handle this error ?


Thanks,

Abhinav

Re: (C)* stable version after 3.5

[Riccardo Ferrari]

You may want to read more about Cassandra release process, find:
http://www.planetcassandra.org/blog/cassandra-2-2-3-0-and-beyond/

On Tue, Jul 12, 2016 at 4:01 PM, Alain RODRIGUEZ  wrote:

> Hi,
>
> The only "fix" release after 3.5 is 3.7. Yet hard to say if it is more
> stable, we can hope so.
>
> For Tic-Toc releases (on 3.X)
>
> Odd numbers are fix releases.
> Even numbers are feature releases.
>
> Not sure why you want something above 3.5, but take care, those versions
> are really recent, and less tested so maybe not that "stable". If you want
> something more stable, I believe you can go with 3.0.8.
>
> Yet I am not telling you not to do that, some people need to start testing
> new things right... So if you choose 3.7 because you want some feature from
> there, it is perfectly ok, just move carefully, maybe read some opened
> tickets and previous experiences from the community and test the upgrade
> process first on a dev cluster.
>
> Also I am not sure if the 2.2 major version is something you can skip
> while upgrading through a rolling restart. I believe you can, but it is not
> what is recommended. Testing will let you know anyway.
>
> Good luck and tell us how it went :-).
>
> C*heers,
> ---
> Alain Rodriguez - al...@thelastpickle.com
> France
>
> The Last Pickle - Apache Cassandra Consulting
> http://www.thelastpickle.com
>
> 2016-07-12 11:05 GMT+02:00 Varun Barala :
>
>> Hi all users,
>>
>>
>> Currently we are using cassandra-2.1.13 but we want to upgrade *2.1.13
>> to 3.x* in production.
>>
>> Could anyone please tell me which is the most stable *cassandra version
>> after 3.5*.
>>
>> Thanking You!!
>>
>>
>> Regards,
>> Varun Barala
>>
>
>

Re: (C)* stable version after 3.5

[Alain RODRIGUEZ]

Hi,

The only "fix" release after 3.5 is 3.7. Yet hard to say if it is more
stable, we can hope so.

For Tic-Toc releases (on 3.X)

Odd numbers are fix releases.
Even numbers are feature releases.

Not sure why you want something above 3.5, but take care, those versions
are really recent, and less tested so maybe not that "stable". If you want
something more stable, I believe you can go with 3.0.8.

Yet I am not telling you not to do that, some people need to start testing
new things right... So if you choose 3.7 because you want some feature from
there, it is perfectly ok, just move carefully, maybe read some opened
tickets and previous experiences from the community and test the upgrade
process first on a dev cluster.

Also I am not sure if the 2.2 major version is something you can skip while
upgrading through a rolling restart. I believe you can, but it is not what
is recommended. Testing will let you know anyway.

Good luck and tell us how it went :-).

C*heers,
---
Alain Rodriguez - al...@thelastpickle.com
France

The Last Pickle - Apache Cassandra Consulting
http://www.thelastpickle.com

2016-07-12 11:05 GMT+02:00 Varun Barala :

> Hi all users,
>
>
> Currently we are using cassandra-2.1.13 but we want to upgrade *2.1.13 to
> 3.x* in production.
>
> Could anyone please tell me which is the most stable *cassandra version
> after 3.5*.
>
> Thanking You!!
>
>
> Regards,
> Varun Barala
>

Re: how to start a embed cassandra instance?

[DuyHai Doan]

If you're looking something similar to cassandra-unit with Apache 2
licence, there is a module in Achilles project that provides the same
thing:
https://github.com/doanduyhai/Achilles/wiki/CQL-embedded-cassandra-server

On Tue, Jul 12, 2016 at 12:56 PM, Peddi, Praveen  wrote:

> We do something similar by starting CassandraDaemon class directly (you
> would need to provide a yaml file though). You can start and stop
> CassandraDaemon class from your unit test (typically @BeforeClass).
>
> Praveen
>
> On Jul 12, 2016, at 3:30 AM, Stone Fang  wrote:
>
> Hi,
> how to start a embed cassandra instance?so we can do a unit test on
> local,dont need to start a
> cassandra server.
>
> https://github.com/jsevellec/cassandra-unit this project is good,but the
> license is not suitable.
> how do you achieve this?
>
> thanks in advance
>
> stone
>
>

Re: Are updates on map columns causing tombstones?

[Sebastian Estevez]

I did a short write-up on this.

http://www.sestevez.com/on-cassandra-collections-updates-and-tombstones/

all the best,

Sebastián
On Jul 11, 2016 4:22 PM, "Alain RODRIGUEZ"  wrote:

> Hi Jan,
>
>
>> when I replace the content of a map-valued column (when I replace the
>> complete map), will this create tombstones for those map entries that are
>> not present in the new map?
>
>
> This is almost correct, I would be more precise and say that it will
> create a range tombstone on this map, and create a new one. No merge. To
> have the behaviour you described, you have to set the map though:
>
> // Updating (or inserting)
> UPDATE users SET favs['author'] = 'Ed Poe' WHERE id = 'jsmith'
> UPDATE users SET favs = favs -  { 'movie' : 'Cassablanca' } WHERE id = 
> 'jsmith'
>
> ==> Tombstone on some specific cells or adding a value in a collection. 
> Previous data is kept.
>
> *Instead of*
>
> // Updating (or inserting)
>
> INSERT INTO users (id, given, surname, favs) VALUES ('jsmith', 'John', 
> 'Smith', { 'fruit' : 'apple', 'band' : 'Beatles' })
> UPDATE users SET favs = { 'movie' : 'Cassablanca' } WHERE id = 'jsmith'
>
> ==> Range tombstone on previous collection (the whole thing). Tombstone + new 
> write (no merge). No explicit delete but dropping data this can be unexpected 
> and hard to troubleshot.
>
> C*heers,
>
> ---
> Alain Rodriguez - al...@thelastpickle.com
> France
>
> The Last Pickle - Apache Cassandra Consultinghttp://www.thelastpickle.com
>
> 2016-07-11 16:51 GMT+02:00 Matthias Niehoff <
> matthias.nieh...@codecentric.de>:
>
>> Hi,
>>
>> it depends.
>> - If you defined the type as a frozen there will not be tombstone,
>> as the map is stored as one binary blob. The update is handled as a normal
>> upsert.
>> - If you do not use the frozen keyword you are right. There will be range
>> tombstones for all columns that have been deleted or updated.
>>
>> 2016-07-11 16:16 GMT+02:00 Jan Algermissen :
>>
>>> Hi,
>>>
>>> when I replace the content of a map-valued column (when I replace the
>>> complete map), will this create tombstones for those map entries that are
>>> not present in the new map?
>>>
>>> My expectation is 'yes', because the map is laid out as normal columns
>>> internally so keys not in the new map should lead to a delete.
>>>
>>> Is that correct?
>>>
>>> Jan
>>>
>>
>>
>>
>> --
>> Matthias Niehoff | IT-Consultant | Agile Software Factory  | Consulting
>> codecentric AG | Zeppelinstr 2 | 76185 Karlsruhe | Deutschland
>> tel: +49 (0) 721.9595-681 | fax: +49 (0) 721.9595-666 | mobil: +49 (0)
>> 172.1702676
>> www.codecentric.de | blog.codecentric.de | www.meettheexperts.de |
>> www.more4fi.de
>>
>> Sitz der Gesellschaft: Solingen | HRB 25917| Amtsgericht Wuppertal
>> Vorstand: Michael Hochgürtel . Mirko Novakovic . Rainer Vehns
>> Aufsichtsrat: Patric Fedlmeier (Vorsitzender) . Klaus Jäger . Jürgen
>> Schütz
>>
>> Diese E-Mail einschließlich evtl. beigefügter Dateien enthält
>> vertrauliche und/oder rechtlich geschützte Informationen. Wenn Sie nicht
>> der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben,
>> informieren Sie bitte sofort den Absender und löschen Sie diese E-Mail und
>> evtl. beigefügter Dateien umgehend. Das unerlaubte Kopieren, Nutzen oder
>> Öffnen evtl. beigefügter Dateien sowie die unbefugte Weitergabe dieser
>> E-Mail ist nicht gestattet
>>
>
>

Re: how to start a embed cassandra instance?

[Peddi, Praveen]

We do something similar by starting CassandraDaemon class directly (you would 
need to provide a yaml file though). You can start and stop CassandraDaemon 
class from your unit test (typically @BeforeClass).

Praveen

On Jul 12, 2016, at 3:30 AM, Stone Fang 
> wrote:

Hi,
how to start a embed cassandra instance?so we can do a unit test on local,dont 
need to start a
cassandra server.

https://github.com/jsevellec/cassandra-unit this project is good,but the 
license is not suitable.
how do you achieve this?

thanks in advance

stone

Re: Can't get performance for scanning the column that is not row key.

[Alain RODRIGUEZ]

Wow.

I have cassandra 1.2 and about fifty nodes.


This Cassandra version is no longer supported, 2.0 isn't either and 2.1 is
reaching end of life...

You could use materialised views, SASI indexes or this kind of new
features. Also CQL performances are better in recent versions etc.

You don't expose data about the table you want to read and how you want to
read it, it is hard to tell.

But I am almost sure you could take profit of some new features, bug fixes
and optimisation. Plus I would bet that no one is going to open the
Cassandra 1.2 code anymore to help you. So if you want free or paid
support, go to 2.1 at least, and keep in mind this one is EOL too...

someone advised to read sstables directly. is it helpful?


It would probably be harder than upgrading... And that's what Cassandra
does. I doubt you will be able to do it better than Cassandra does,
excepted, maybe on some rare corner cases...

Good luck,
C*heers,
---
Alain Rodriguez - al...@thelastpickle.com
France

The Last Pickle - Apache Cassandra Consulting
http://www.thelastpickle.com

2016-07-07 2:39 GMT+02:00 Sungju Hong :

> Hello,
>
> I have cassandra 1.2 and about fifty nodes.
>
> I sometime meet requirements for scanning the column that is not row key
> fast.
> I know that secondary index doesn't help.
>
> someone advised to read sstables directly. is it helpful?
>
> I will appreciate for any idea handling this topic.
>
> Thanks.
>
>
>

Re: How to print out the metrics information, like compaction and garbage collection?

[Alain RODRIGUEZ]

Hi Jun,

In case you did not solved your issue yet:

Metrics are exposed through JMX. Any jmx tool will allow you to read those
metrics.

For a more automated and interesting usage (creating dashboards etc), you
can use a metric reporter as described for graphite there:
http://www.datastax.com/dev/blog/pluggable-metrics-reporting-in-cassandra-2-0-2
.

Also as Otis mentioned, some "agents" from monitoring as a service
platforms take care of this.

It depends on what you have in mind :-).

C*heers,
---
Alain Rodriguez - al...@thelastpickle.com
France

The Last Pickle - Apache Cassandra Consulting
http://www.thelastpickle.com



2016-06-14 21:24 GMT+02:00 Jun Wu :

> Hi Otis,
>
>Thank you so much for the reply. I do appreciate it.
>
>Actually, I've tried the sematext spm few days ago:) It works well and
> is easy to deploy.
>
> However, in the monitoring output figure for different metrics, the
> interval is 1 minute, which is longer than I want. What I want is something
> like one metric value / second and I want to have a close eye on each
> metric.
>
> That's why I post question and want to print out the metrics
> information in the console.
>
>  I'm wondering whether you have any other solution for my question.
>
> Thanks!
>
> Jun
>
> --
> From: otis.gospodne...@gmail.com
> Date: Tue, 14 Jun 2016 15:01:20 -0400
> Subject: Re: How to print out the metrics information, like compaction and
> garbage collection?
> To: user@cassandra.apache.org
>
>
> Hi Jun,
>
> Here's a tool for dumping JMX contents: https://github.com/sematext/jmxc
> Here's a tool/service for monitoring Cassandra:
> https://sematext.com/spm/integrations/cassandra-monitoring/
>
> Otis
> --
> Monitoring - Log Management - Alerting - Anomaly Detection
> Solr & Elasticsearch Consulting Support Training - http://sematext.com/
>
>
> On Mon, Jun 13, 2016 at 5:17 PM, Jun Wu  wrote:
>
> Hi there,
>
>I've deployed 6 node in Amazon EC2. I'm trying to monitor some metrics
> for each node and print them out when I write/read data into Cassandra.
> Specifically I want to print out the information about garbage collection
> and compaction.
>
>I do notice that there's metrics for compaction in o.a.c.metrics.
> CompactionMetrics.java. But I don't know how to get them, I've tried method
> below:
>
>CompactionMetrics metrics = new CompactionMetrics();
>System.out.println(metrics.pendingtasks);
>
>But it doesn't work.
>
> *
>   Another/same question is about the garbage collection, any idea on where
> should I refer to and get the information about garbage collection, like
> garbage collection count, collection time, etc.
>
>Any hint will be appreciated.
>
> Thanks!
>
> Jun
>
>
>

Re: Can you enable Thrift on a node and still connect remotely to it using CQL/port 9042?

[Alain RODRIGUEZ]

My understanding is this should work. As this is a small issue from a month
ago, please shout out if you are still facing this issue. I'll try to help.

C*heers,
---
Alain Rodriguez - al...@thelastpickle.com
France

The Last Pickle - Apache Cassandra Consulting
http://www.thelastpickle.com

2016-06-08 1:33 GMT+02:00 Carolina Simoes Gomes :

> Hello,
>
> I have a question. I have changed the Cassandra driver (3.0.0) so we can
> connect to and query two nodes (Cassandra 3.2.1), the local one and a
> remote node, using the driver's native protocol (CQL, port 9042).
>
> But it seems that when I enable Thrift in cassandra.yaml (start_rpc=true,
> port 9160), I cannot subsequently connect remotely to the remote node using
> CQL, from inside the driver.
>
> Is there a way of having Thrift enabled on a node but also being able to
> connect remotely to it using CQL/port 9042?
>
> Thanks!
>
>
> *Carolina Gomes, M.Sc. *Systems Researcher
> Huawei Canada Research Center
> 19 Allstate Parkway, Suite 200
> Markham ON   L3R 5A
> carolina.go...@huawei.com 
> *www.huawei.ca* 
>

(C)* stable version after 3.5

[Varun Barala]

Hi all users,


Currently we are using cassandra-2.1.13 but we want to upgrade *2.1.13 to
3.x* in production.

Could anyone please tell me which is the most stable *cassandra version
after 3.5*.

Thanking You!!


Regards,
Varun Barala

Re: Help regarding cassandra node networking..

[Alain RODRIGUEZ]

Hi,

A few comments as I read:


> I'm using node0 as seeder


It is recommended to use at least 2 nodes per DC as seeds, for fault
tolerance and good gossip propagation.

Also I think you are a bit confused about what a seed node is and is not.
See:
https://docs.datastax.com/en/cassandra/2.1/cassandra/architecture/architectureGossipAbout_c.html

I want to use the node0 as seeder and use the public ip so that i can
> access it remotely and can join with other Datacenter nodes later time. But
> my rest of the nodes should not have any public interfaces. Now how to
> configure it?


This does not look right to me. All the nodes must receive the same
operational care. It is a lot easier to reason about an homogeneous cluster
and I do not recommend this kind of approaches at all.

To use public IP on a node, set the broadcast_address to public IP, and
make sure ports are open on whatever IP address you are trying to connect
to.

Sorry for the month delay, hope this is still useful.

C*heers,
---
Alain Rodriguez - al...@thelastpickle.com
France

The Last Pickle - Apache Cassandra Consulting
http://www.thelastpickle.com

2016-06-06 9:27 GMT+02:00 sahriar rahman supto :

> I'm creating a cassandra cluster. I've three node with centos 7 and
> cassandra 3.4 installed.
>
> Node0 interfaces:
> eth0 169.254.169.xx1 (Public IP)
> eth1 192.168.56.101(Private IP)
>
> Node1 interfaces:
> eth0 169.254.169.xx2 (Public IP)
> eth1 192.168.56.102(Private IP)
>
> Node2 interfaces:
> eth0 169.254.169.xx3 (Public IP)
> eth1 192.168.56.103(Private IP)
>
> I'm using node0 as seeder. My cassandra.yaml Configuration files for each
> of the nodes are as follows:
>
> Node0:
> cluster_name: 'DB Cluster'
> seeds: "162.253.42.xx1"
> listen_address: 162.253.42.xx1
> rpc_address: 162.253.42.xx1
> endpoint_snitch: GossipingPropertyFileSnitch
>
> Node1:
> cluster_name: 'DB Cluster'
> seeds: "162.253.42.xx1"
> listen_address: 162.253.42.xx2
> rpc_address: 162.253.42.xx2
> endpoint_snitch: GossipingPropertyFileSnitch
>
> Node2:
> cluster_name: 'DB Cluster'
> seeds: "162.253.42.xx1"
> listen_address: 162.253.42.xx3
> rpc_address: 162.253.42.xx3
> endpoint_snitch: GossipingPropertyFileSnitch
>
> Then my cassandra cluster started and running smoothly.
>
> Now my requirement is to configure it into private network. Suppose Only
> one of my node (seeder) has public ip and the rest of the nodes are
> connected with private ip only. Then how to configure it??
>
> Node0 interfaces:
> eth0 169.254.169.xx1 (Public IP)
> eth1 192.168.56.101(Private IP)
>
> Node1 interfaces:
> eth0 192.168.56.102(Private IP)
>
> Node2 interfaces:
> eth0 192.168.56.103(Private IP)
>
> I want to use the node0 as seeder and use the public ip so that i can
> access it remotely and can join with other Datacenter nodes later time. But
> my rest of the nodes should not have any public interfaces. Now how to
> configure it?
>
>

how to start a embed cassandra instance?

[Stone Fang]

Hi,
how to start a embed cassandra instance?so we can do a unit test on
local,dont need to start a
cassandra server.

https://github.com/jsevellec/cassandra-unit this project is good,but the
license is not suitable.
how do you achieve this?

thanks in advance

stone

Re: How I can ignore clustering key order and read latests writes first?

[Reza Samee]

Thanks for response.

Note: I'm not good in English (I'm not native and I'm learning); Please
forgive me if it words aren't easy to understand ;)

I didn't satisfied for some reasons:
 - as you mentioned, it don't (logically) override the last value! and it
will cause lot of side effects (from affecting compaction strategy, to
storage capcaity and even a new component for gc old signals). I realy
don't need event sourcing here. I wan't a fast and effient cache for last
updated signals, not more, not less.

 - I can achieve better result (in my assumtion) than what was in the
answer (of course base on my domain). for example I can have a
`last_day_signals` or `last_hour_signals` table plus a simple table (like
what I presented in my question). In this case I can use one table for very
hot sensors with less effects on compaction (because most of values would
update in memtable) and read query!

 - I prefer to use a range query to get values with less costs (application
complexity and i/o effiency).

But the main question is this: "Can I ignore clustering-keys effects on
rows order?" that seems the response is false!

And one more question: `ALLOW FILTERING` is not neccessary for my query! I
can use `sensors` clustering order too. But my assumtion is that when I set
partition-key in my query (and maybe clustering-key too) I will avoid  most
of `ALLOW FILTERING` costs! But it seems that I'm wrong! Am I? Why?