Re: select query on entire primary key returning more than one row in result

[Atul Saroha]

After further debug, this issue is found in in-memory memtable as doing
nodetool flush + compact resolve the issue. And there is no batch write
used for this table which is showing issue.
Table properties:

WITH CLUSTERING ORDER BY (f_name ASC)
> AND bloom_filter_fp_chance = 0.01
> AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
> AND comment = ''
> AND compaction = {'class':
> 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy',
> 'max_threshold': '32', 'min_threshold': '4'}
> AND compression = {'chunk_length_in_kb': '64', 'class':
> 'org.apache.cassandra.io.compress.LZ4Compressor'}
> AND crc_check_chance = 1.0
> AND dclocal_read_repair_chance = 0.1
> AND default_time_to_live = 0
> AND gc_grace_seconds = 864000
> AND max_index_interval = 2048
> AND memtable_flush_period_in_ms = 0
> AND min_index_interval = 128
> AND read_repair_chance = 0.0
> AND speculative_retry = '99PERCENTILE';
> CREATE CUSTOM INDEX nbf_index ON nbf () USING
> 'com.stratio.cassandra.lucene.Index' WITH OPTIONS = {'refresh_seconds':
> '1', 'schema': '{
> fields : {
> id  : {type : "bigint"},
> f_d_name : {
> type   : "string",
> indexed: true,
> sorted : false,
> validated  : true,
> case_sensitive : false
> }
> }
> }'};
>


-
Atul Saroha
*Lead Software Engineer*
*M*: +91 8447784271 *T*: +91 124-415-6069 *EXT*: 12369
Plot # 362, ASF Centre - Tower A, Udyog Vihar,
 Phase -4, Sector 18, Gurgaon, Haryana 122016, INDIA

On Mon, Jun 13, 2016 at 11:11 PM, Siddharth Verma <
verma.siddha...@snapdeal.com> wrote:

> No, all rows were not the same.
> Querying only on the partition key gives 20 rows.
> In the erroneous result, while querying on partition key and clustering
> key, we got 16 of those 20 rows.
>
> And for "*tombstone_threshold"* there isn't any entry at column family
> level.
>
> Thanks,
> Siddharth Verma
>
>
>

Re: Streaming from 1 node only when adding a new DC

[kurt Greaves]

What version of Cassandra are you using? Also what command are you using to
run the rebuilds? Are you using vnodes?

On 13 June 2016 at 09:01, Fabien Rousseau  wrote:

> Hello,
>
> We've tested adding a new DC from an existing DC having 3 nodes and RF=3
> (ie all nodes have all data).
> During the rebuild process, only one node of the first DC streamed data to
> the 3 nodes of the second DC.
>
> Our goal is to minimise the time it takes to rebuild a DC and would like
> to be able to stream from all nodes.
>
> Starting C* with debug logs, it appears that all nodes, when computing
> their "streaming plan" returns the same node for all ranges.
> This is probably because all nodes in DC2 have the same view of the ring.
>
> I understand that when bootstrapping a new node, it's preferable to stream
> from the node being replaced, but when rebuilding a new DC, it should
> probably select sources "randomly" (rather than always selecting the same
> source for a specific range).
> What do you think ?
>
> Best Regards,
> Fabien
>



-- 
Kurt Greaves
k...@instaclustr.com
www.instaclustr.com

[RELEASE] Apache Cassandra 3.0.7 released

[Jake Luciani]

The Cassandra team is pleased to announce the release of Apache Cassandra
version 3.0.7.

Apache Cassandra is a fully distributed database. It is the right choice
when you need scalability and high availability without compromising
performance.

 http://cassandra.apache.org/

Downloads of source and binary distributions are listed in our download
section:

 http://cassandra.apache.org/download/

This version is a bug fix release[1] on the 3.0 series. As always, please
pay
attention to the release notes[2] and Let us know[3] if you were to
encounter
any problem.

Enjoy!

[1]: http://goo.gl/yPJaXi (CHANGES.txt)
[2]: http://goo.gl/Jph9Fh (NEWS.txt)
[3]: https://issues.apache.org/jira/browse/CASSANDRA

[RELEASE] Apache Cassandra 3.7 released

[Jake Luciani]

The Cassandra team is pleased to announce the release of Apache Cassandra
version 3.7.

Apache Cassandra is a fully distributed database. It is the right choice
when you need scalability and high availability without compromising
performance.

 http://cassandra.apache.org/

Downloads of source and binary distributions are listed in our download
section:

 http://cassandra.apache.org/download/

This version is a tick-tock bug fix release[1] on the 3.x series. As
always, please pay
attention to the release notes[2] and Let us know[3] if you were to
encounter
any problem.

Enjoy!

[1]: http://goo.gl/k1abJV (CHANGES.txt)
[2]: http://goo.gl/3ENJIz (NEWS.txt)
[3]: https://issues.apache.org/jira/browse/CASSANDRA

Re: Tick Tock version numbers

[Tyler Hobbs]

On Mon, Jun 13, 2016 at 11:59 AM, Francisco Reyes  wrote:

>
>
> Can I upgrade them to 3.6 from 3.2? Or is it advisable to upgrade to each
> intermediary version?
>

You can (and should) upgrade directly to 3.6 or 3.7.  The 3.7 release is
just 3.6 + bugfixes.


>
> Based on what I have gather seems like it is matter of:
> bring node down
> install new version
> bring up
> run nodetool upgradesstables -a
>

For upgrades within the 3.x line, you don't need to run upgradesstables.
Other than that, this is correct.


-- 
Tyler Hobbs
DataStax 

Re: Installing Cassandra from Tarball

[Tyler Hobbs]

On Mon, Jun 13, 2016 at 11:49 AM, Bhuvan Rawal  wrote:

>
> WARN  15:41:58 Cassandra server running in degraded mode. Is swap
>> disabled? : true,  Address space adequate? : true,  nofile limit adequate?
>> : false, nproc limit adequate? : false
>>
> You need to disable swap in order to avoid this message, using swap space
> can have serious performance implications. Make sure you disable fstab
> entry as well for swap partition.
>

It looks like swap is actually disabled, but the nofile and nproc limits
are too low.


-- 
Tyler Hobbs
DataStax 

Re: select query on entire primary key returning more than one row in result

[Tyler Hobbs]

Is 'id' your partition key? I'm not familiar with the stratio indexes, but
it looks like the primary key columns are both indexed.  Perhaps this is
related?

On Tue, Jun 14, 2016 at 1:25 AM, Atul Saroha 
wrote:

> After further debug, this issue is found in in-memory memtable as doing
> nodetool flush + compact resolve the issue. And there is no batch write
> used for this table which is showing issue.
> Table properties:
>
> WITH CLUSTERING ORDER BY (f_name ASC)
>> AND bloom_filter_fp_chance = 0.01
>> AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
>> AND comment = ''
>> AND compaction = {'class':
>> 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy',
>> 'max_threshold': '32', 'min_threshold': '4'}
>> AND compression = {'chunk_length_in_kb': '64', 'class':
>> 'org.apache.cassandra.io.compress.LZ4Compressor'}
>> AND crc_check_chance = 1.0
>> AND dclocal_read_repair_chance = 0.1
>> AND default_time_to_live = 0
>> AND gc_grace_seconds = 864000
>> AND max_index_interval = 2048
>> AND memtable_flush_period_in_ms = 0
>> AND min_index_interval = 128
>> AND read_repair_chance = 0.0
>> AND speculative_retry = '99PERCENTILE';
>> CREATE CUSTOM INDEX nbf_index ON nbf () USING
>> 'com.stratio.cassandra.lucene.Index' WITH OPTIONS = {'refresh_seconds':
>> '1', 'schema': '{
>> fields : {
>> id  : {type : "bigint"},
>> f_d_name : {
>> type   : "string",
>> indexed: true,
>> sorted : false,
>> validated  : true,
>> case_sensitive : false
>> }
>> }
>> }'};
>>
>
>
>
> -
> Atul Saroha
> *Lead Software Engineer*
> *M*: +91 8447784271 *T*: +91 124-415-6069 *EXT*: 12369
> Plot # 362, ASF Centre - Tower A, Udyog Vihar,
>  Phase -4, Sector 18, Gurgaon, Haryana 122016, INDIA
>
> On Mon, Jun 13, 2016 at 11:11 PM, Siddharth Verma <
> verma.siddha...@snapdeal.com> wrote:
>
>> No, all rows were not the same.
>> Querying only on the partition key gives 20 rows.
>> In the erroneous result, while querying on partition key and clustering
>> key, we got 16 of those 20 rows.
>>
>> And for "*tombstone_threshold"* there isn't any entry at column family
>> level.
>>
>> Thanks,
>> Siddharth Verma
>>
>>
>>
>


-- 
Tyler Hobbs
DataStax 

Re: select query on entire primary key returning more than one row in result

[Siddharth Verma]

id is partition key,
f_name is clustering key

We weren't querying on lucene indexes.
lucene index is on id, and f_d_name (another column).


We were facing this issue on production in one column family, due to which
we had to downgrade to 3.0.3

Re: select query on entire primary key returning more than one row in result

[Atul Saroha]

Hi Tyler,

This issue is mainly visible for tables having static columns, still
investigating.
We will try to test after removing lucene index but I don’t think this
plug-in could led to change in behaviour of cassandra write to table's
memtable.

-
Atul Saroha
*Lead Software Engineer*
*M*: +91 8447784271 *T*: +91 124-415-6069 *EXT*: 12369
Plot # 362, ASF Centre - Tower A, Udyog Vihar,
 Phase -4, Sector 18, Gurgaon, Haryana 122016, INDIA

On Tue, Jun 14, 2016 at 9:54 PM, Tyler Hobbs  wrote:

> Is 'id' your partition key? I'm not familiar with the stratio indexes, but
> it looks like the primary key columns are both indexed.  Perhaps this is
> related?
>
> On Tue, Jun 14, 2016 at 1:25 AM, Atul Saroha 
> wrote:
>
>> After further debug, this issue is found in in-memory memtable as doing
>> nodetool flush + compact resolve the issue. And there is no batch write
>> used for this table which is showing issue.
>> Table properties:
>>
>> WITH CLUSTERING ORDER BY (f_name ASC)
>>> AND bloom_filter_fp_chance = 0.01
>>> AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
>>> AND comment = ''
>>> AND compaction = {'class':
>>> 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy',
>>> 'max_threshold': '32', 'min_threshold': '4'}
>>> AND compression = {'chunk_length_in_kb': '64', 'class':
>>> 'org.apache.cassandra.io.compress.LZ4Compressor'}
>>> AND crc_check_chance = 1.0
>>> AND dclocal_read_repair_chance = 0.1
>>> AND default_time_to_live = 0
>>> AND gc_grace_seconds = 864000
>>> AND max_index_interval = 2048
>>> AND memtable_flush_period_in_ms = 0
>>> AND min_index_interval = 128
>>> AND read_repair_chance = 0.0
>>> AND speculative_retry = '99PERCENTILE';
>>> CREATE CUSTOM INDEX nbf_index ON nbf () USING
>>> 'com.stratio.cassandra.lucene.Index' WITH OPTIONS = {'refresh_seconds':
>>> '1', 'schema': '{
>>> fields : {
>>> id  : {type : "bigint"},
>>> f_d_name : {
>>> type   : "string",
>>> indexed: true,
>>> sorted : false,
>>> validated  : true,
>>> case_sensitive : false
>>> }
>>> }
>>> }'};
>>>
>>
>>
>>
>> -
>> Atul Saroha
>> *Lead Software Engineer*
>> *M*: +91 8447784271 *T*: +91 124-415-6069 *EXT*: 12369
>> Plot # 362, ASF Centre - Tower A, Udyog Vihar,
>>  Phase -4, Sector 18, Gurgaon, Haryana 122016, INDIA
>>
>> On Mon, Jun 13, 2016 at 11:11 PM, Siddharth Verma <
>> verma.siddha...@snapdeal.com> wrote:
>>
>>> No, all rows were not the same.
>>> Querying only on the partition key gives 20 rows.
>>> In the erroneous result, while querying on partition key and clustering
>>> key, we got 16 of those 20 rows.
>>>
>>> And for "*tombstone_threshold"* there isn't any entry at column family
>>> level.
>>>
>>> Thanks,
>>> Siddharth Verma
>>>
>>>
>>>
>>
>
>
> --
> Tyler Hobbs
> DataStax 
>

Re: Installing Cassandra from Tarball

[Steve Anderson]

I think you’re right Tyler; the warning does not appear after making the 
changes suggested by Bhuvan.

Steve
—
"Surely, those who believe, those who are Jewish, the Christians, and the 
converts; anyone who (1) believes in God, (2) believes in the Last Day, and (3) 
leads a righteous life, will receive their recompense from their Lord; they 
have nothing to fear, nor will they grieve." (Quran 2:62, 5:69) …learn more at 
www.masjidtucson.org


> On 15/06/2016, at 12:26 AM, Tyler Hobbs  wrote:
> 
> 
> On Mon, Jun 13, 2016 at 11:49 AM, Bhuvan Rawal  > wrote:
> 
> WARN  15:41:58 Cassandra server running in degraded mode. Is swap disabled? : 
> true,  Address space adequate? : true,  nofile limit adequate? : false, nproc 
> limit adequate? : false
> You need to disable swap in order to avoid this message, using swap space can 
> have serious performance implications. Make sure you disable fstab entry as 
> well for swap partition.
> 
> It looks like swap is actually disabled, but the nofile and nproc limits are 
> too low.
> 
> 
> -- 
> Tyler Hobbs
> DataStax 

Re: Installing Cassandra from Tarball

[Steve Anderson]

Awesone, thanks Bhuvan! 

I have not worried about the JMX warning at this stage.

There were two other warnings, but I assume these are due to the size of my 
Amazon Linux Image (and are not worth worrying about at this stage).

WARN  17:24:20 Small commitlog volume detected at /var/lib/cassandra/commitlog; 
setting commitlog_total_space_in_mb to 1983.  You can override this in 
cassandra.yaml

WARN  17:24:20 Only 5.809GiB free across all data volumes. Consider adding more 
capacity to your cluster or removing obsolete snapshots

Regarding the commitlog_total_space_in_mb property, I see from the 
cassandra.yaml file:

The default value is the smaller of 8192, and 1/4 of the total space of 
the commitlog volume.

Awesome, thanks again.

Steve
—
"Surely, those who believe, those who are Jewish, the Christians, and the 
converts; anyone who (1) believes in God, (2) believes in the Last Day, and (3) 
leads a righteous life, will receive their recompense from their Lord; they 
have nothing to fear, nor will they grieve." (Quran 2:62, 5:69) …learn more at 
www.masjidtucson.org


> On 14/06/2016, at 12:49 AM, Bhuvan Rawal  wrote:
> 
> 
> Hi Steve,
> 
> Please find the responses in line:
> 
> WARN  15:41:58 Unable to lock JVM memory (ENOMEM). This can result in part of 
> the JVM being swapped out, especially with mmapped I/O enabled. Increase 
> RLIMIT_MEMLOCK or run Cassandra as root.
>  You can edit - /etc/security/limits.conf and put these lines in there
> 
> * - memlock unlimited
> * - nofile 10
> * - nproc 32768
> * - as unlimited
> 
> and reload the properties by command $ sudo sysctl -p
> and then checking it :
> $ ulimit -l
> and for the cassandra process by :
> $ cat /proc//limits
> Source - Datastax Troubleshooting 
> 
> 
> WARN  15:41:58 jemalloc shared library could not be preloaded to speed up 
> memory allocations
> If you want to allocate off heap memory using jemalloc then uncomment this 
> line in cassandra-env.sh and provide appropriate jemalloc path
> JVM_OPTS="$JVM_OPTS -Djava.library.path=/lib/"
> 
> WARN  15:41:58 JMX is not enabled to receive remote connections. Please see 
> cassandra-env.sh for more info.
> By default JMX is enabled only for local connections, if you want to debug 
> from a remote machine mark LOCAL_JMX=no  in cassandra-env.sh
> 
> WARN  15:41:58 Cassandra server running in degraded mode. Is swap disabled? : 
> true,  Address space adequate? : true,  nofile limit adequate? : false, nproc 
> limit adequate? : false
> You need to disable swap in order to avoid this message, using swap space can 
> have serious performance implications. Make sure you disable fstab entry as 
> well for swap partition.
> 
> Thanks & Regards,
> Bhuvan
> 

Re: select query on entire primary key returning more than one row in result

[Joel Knighton]

There's some precedent for similar issues with static columns in 3.5 with
https://issues.apache.org/jira/browse/CASSANDRA-11513 - a deterministic (or
somewhat deterministic) path for reproduction would help narrow the issue
down farther. I've played around locally with similar schemas (sans the
stratio indices) and couldn't reproduce the issue.

On Tue, Jun 14, 2016 at 1:41 PM, Bhuvan Rawal  wrote:

> Jira CASSANDRA-12003
>  Has been created
> for the same.
>
> On Tue, Jun 14, 2016 at 11:54 PM, Atul Saroha 
> wrote:
>
>> Hi Tyler,
>>
>> This issue is mainly visible for tables having static columns, still
>> investigating.
>> We will try to test after removing lucene index but I don’t think this
>> plug-in could led to change in behaviour of cassandra write to table's
>> memtable.
>>
>>
>> -
>> Atul Saroha
>> *Lead Software Engineer*
>> *M*: +91 8447784271 *T*: +91 124-415-6069 *EXT*: 12369
>> Plot # 362, ASF Centre - Tower A, Udyog Vihar,
>>  Phase -4, Sector 18, Gurgaon, Haryana 122016, INDIA
>>
>> On Tue, Jun 14, 2016 at 9:54 PM, Tyler Hobbs  wrote:
>>
>>> Is 'id' your partition key? I'm not familiar with the stratio indexes,
>>> but it looks like the primary key columns are both indexed.  Perhaps this
>>> is related?
>>>
>>> On Tue, Jun 14, 2016 at 1:25 AM, Atul Saroha 
>>> wrote:
>>>
 After further debug, this issue is found in in-memory memtable as doing
 nodetool flush + compact resolve the issue. And there is no batch write
 used for this table which is showing issue.
 Table properties:

 WITH CLUSTERING ORDER BY (f_name ASC)
> AND bloom_filter_fp_chance = 0.01
> AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
> AND comment = ''
> AND compaction = {'class':
> 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy',
> 'max_threshold': '32', 'min_threshold': '4'}
> AND compression = {'chunk_length_in_kb': '64', 'class':
> 'org.apache.cassandra.io.compress.LZ4Compressor'}
> AND crc_check_chance = 1.0
> AND dclocal_read_repair_chance = 0.1
> AND default_time_to_live = 0
> AND gc_grace_seconds = 864000
> AND max_index_interval = 2048
> AND memtable_flush_period_in_ms = 0
> AND min_index_interval = 128
> AND read_repair_chance = 0.0
> AND speculative_retry = '99PERCENTILE';
> CREATE CUSTOM INDEX nbf_index ON nbf () USING
> 'com.stratio.cassandra.lucene.Index' WITH OPTIONS = {'refresh_seconds':
> '1', 'schema': '{
> fields : {
> id  : {type : "bigint"},
> f_d_name : {
> type   : "string",
> indexed: true,
> sorted : false,
> validated  : true,
> case_sensitive : false
> }
> }
> }'};
>



 -
 Atul Saroha
 *Lead Software Engineer*
 *M*: +91 8447784271 *T*: +91 124-415-6069 *EXT*: 12369
 Plot # 362, ASF Centre - Tower A, Udyog Vihar,
  Phase -4, Sector 18, Gurgaon, Haryana 122016, INDIA

 On Mon, Jun 13, 2016 at 11:11 PM, Siddharth Verma <
 verma.siddha...@snapdeal.com> wrote:

> No, all rows were not the same.
> Querying only on the partition key gives 20 rows.
> In the erroneous result, while querying on partition key and
> clustering key, we got 16 of those 20 rows.
>
> And for "*tombstone_threshold"* there isn't any entry at column
> family level.
>
> Thanks,
> Siddharth Verma
>
>
>

>>>
>>>
>>> --
>>> Tyler Hobbs
>>> DataStax 
>>>
>>
>>
>


-- 



Joel Knighton
Cassandra Developer | joel.knigh...@datastax.com


 

 

Re: Cassandra monitoring

[Arun Ramakrishnan]

Thanks Jonathan.

Out of curiosity, does opscenter support some later version of cassandra
that is not OSS ?

Well, the most minimal requirement is that, I want to be able to monitor
for cluster health and hook this info to some alerting platform. We are AWS
heavy. We just really heavily on AWS cloud watch for our metrics as of now.
We prefer to not spend our time setting up additional tools if we can help
it. So, if we needed a 3rd party service we would consider an APM or
monitoring service that is on the cheaper side.




On Tue, Jun 14, 2016 at 12:20 PM, Jonathan Haddad  wrote:

> Depends what you want to monitor.  I wouldn't use a lesser version of
> Cassandra for OpsCenter, it doesn't give you a ton you can't get elsewhere
> and it's not ever going to support OSS > 2.1, so you kind of limit yourself
> to a pretty old version of Cassandra for a non-good reason.
>
> What else do you use for monitoring in your infra?  I've used a mix of OSS
> tools (nagios, statsd, graphite, ELK), and hosted solutions. The nice part
> about them is that you can monitor your whole stack in a single UI not just
> your database.
>
> On Tue, Jun 14, 2016 at 12:10 PM Arun Ramakrishnan <
> sinchronized.a...@gmail.com> wrote:
>
>> What are the options for a very small and nimble startup to do keep a
>> cassandra cluster running well oiled. We are on AWS. We are interested in a
>> monitoring tool and potentially also cluster management tools.
>>
>> We are currently on apache cassandra 3.7. We were hoping the datastax
>> opscenter would be it (It is free for startups our size). But, looks like
>> it does not support cassandra versions greater than v2.1. It is pretty
>> surprising considering cassandra v2.1  came out in 2014.
>>
>> We would consider downgrading to datastax cassandra 2.1 just to have
>> robust monitoring tools. But, I am not sure if having opscenter offsets all
>> the improvements that have been added to cassandra since 2.1.
>>
>> Sematext has a integrations for monitoring cassandra. Does anyone have
>> good experience with it ?
>>
>> How much work would be involved to setup Ganglia or some such option for
>> cassandra ?
>>
>> Thanks,
>> Arun
>>
>>
>>
>>
>>
>>
>>

Re: Cassandra monitoring

[Jonathan Haddad]

OpsCenter going forward is limited to datastax enterprise versions.

I know a lot of people like DataDog, but I haven't used it.  Maybe other
people on the list can speak from recent first hand experience on it's pros
and cons.


On Tue, Jun 14, 2016 at 1:20 PM Arun Ramakrishnan <
sinchronized.a...@gmail.com> wrote:

> Thanks Jonathan.
>
> Out of curiosity, does opscenter support some later version of cassandra
> that is not OSS ?
>
> Well, the most minimal requirement is that, I want to be able to monitor
> for cluster health and hook this info to some alerting platform. We are AWS
> heavy. We just really heavily on AWS cloud watch for our metrics as of now.
> We prefer to not spend our time setting up additional tools if we can help
> it. So, if we needed a 3rd party service we would consider an APM or
> monitoring service that is on the cheaper side.
>
>
>
>
> On Tue, Jun 14, 2016 at 12:20 PM, Jonathan Haddad 
> wrote:
>
>> Depends what you want to monitor.  I wouldn't use a lesser version of
>> Cassandra for OpsCenter, it doesn't give you a ton you can't get elsewhere
>> and it's not ever going to support OSS > 2.1, so you kind of limit yourself
>> to a pretty old version of Cassandra for a non-good reason.
>>
>> What else do you use for monitoring in your infra?  I've used a mix of
>> OSS tools (nagios, statsd, graphite, ELK), and hosted solutions. The nice
>> part about them is that you can monitor your whole stack in a single UI not
>> just your database.
>>
>> On Tue, Jun 14, 2016 at 12:10 PM Arun Ramakrishnan <
>> sinchronized.a...@gmail.com> wrote:
>>
>>> What are the options for a very small and nimble startup to do keep a
>>> cassandra cluster running well oiled. We are on AWS. We are interested in a
>>> monitoring tool and potentially also cluster management tools.
>>>
>>> We are currently on apache cassandra 3.7. We were hoping the datastax
>>> opscenter would be it (It is free for startups our size). But, looks like
>>> it does not support cassandra versions greater than v2.1. It is pretty
>>> surprising considering cassandra v2.1  came out in 2014.
>>>
>>> We would consider downgrading to datastax cassandra 2.1 just to have
>>> robust monitoring tools. But, I am not sure if having opscenter offsets all
>>> the improvements that have been added to cassandra since 2.1.
>>>
>>> Sematext has a integrations for monitoring cassandra. Does anyone have
>>> good experience with it ?
>>>
>>> How much work would be involved to setup Ganglia or some such option for
>>> cassandra ?
>>>
>>> Thanks,
>>> Arun
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>

Re: Streaming from 1 node only when adding a new DC

[Fabien Rousseau]

We've tested with C* 2.1.14 version
Yes VNodes with 256 tokens
Once all the nodes in dc2 are added, schema is modified to have RF=3 in dc1
and RF=3 in dc2.
Then on each nodes of dc2:
nodetool rebuild dc1
Le 14 juin 2016 10:39, "kurt Greaves"  a écrit :

> What version of Cassandra are you using? Also what command are you using
> to run the rebuilds? Are you using vnodes?
>
> On 13 June 2016 at 09:01, Fabien Rousseau  wrote:
>
>> Hello,
>>
>> We've tested adding a new DC from an existing DC having 3 nodes and RF=3
>> (ie all nodes have all data).
>> During the rebuild process, only one node of the first DC streamed data
>> to the 3 nodes of the second DC.
>>
>> Our goal is to minimise the time it takes to rebuild a DC and would like
>> to be able to stream from all nodes.
>>
>> Starting C* with debug logs, it appears that all nodes, when computing
>> their "streaming plan" returns the same node for all ranges.
>> This is probably because all nodes in DC2 have the same view of the ring.
>>
>> I understand that when bootstrapping a new node, it's preferable to
>> stream from the node being replaced, but when rebuilding a new DC, it
>> should probably select sources "randomly" (rather than always selecting the
>> same source for a specific range).
>> What do you think ?
>>
>> Best Regards,
>> Fabien
>>
>
>
>
> --
> Kurt Greaves
> k...@instaclustr.com
> www.instaclustr.com
>

Re: select query on entire primary key returning more than one row in result

[Bhuvan Rawal]

Jira CASSANDRA-12003
 Has
been created for the same.

On Tue, Jun 14, 2016 at 11:54 PM, Atul Saroha 
wrote:

> Hi Tyler,
>
> This issue is mainly visible for tables having static columns, still
> investigating.
> We will try to test after removing lucene index but I don’t think this
> plug-in could led to change in behaviour of cassandra write to table's
> memtable.
>
>
> -
> Atul Saroha
> *Lead Software Engineer*
> *M*: +91 8447784271 *T*: +91 124-415-6069 *EXT*: 12369
> Plot # 362, ASF Centre - Tower A, Udyog Vihar,
>  Phase -4, Sector 18, Gurgaon, Haryana 122016, INDIA
>
> On Tue, Jun 14, 2016 at 9:54 PM, Tyler Hobbs  wrote:
>
>> Is 'id' your partition key? I'm not familiar with the stratio indexes,
>> but it looks like the primary key columns are both indexed.  Perhaps this
>> is related?
>>
>> On Tue, Jun 14, 2016 at 1:25 AM, Atul Saroha 
>> wrote:
>>
>>> After further debug, this issue is found in in-memory memtable as doing
>>> nodetool flush + compact resolve the issue. And there is no batch write
>>> used for this table which is showing issue.
>>> Table properties:
>>>
>>> WITH CLUSTERING ORDER BY (f_name ASC)
 AND bloom_filter_fp_chance = 0.01
 AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
 AND comment = ''
 AND compaction = {'class':
 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy',
 'max_threshold': '32', 'min_threshold': '4'}
 AND compression = {'chunk_length_in_kb': '64', 'class':
 'org.apache.cassandra.io.compress.LZ4Compressor'}
 AND crc_check_chance = 1.0
 AND dclocal_read_repair_chance = 0.1
 AND default_time_to_live = 0
 AND gc_grace_seconds = 864000
 AND max_index_interval = 2048
 AND memtable_flush_period_in_ms = 0
 AND min_index_interval = 128
 AND read_repair_chance = 0.0
 AND speculative_retry = '99PERCENTILE';
 CREATE CUSTOM INDEX nbf_index ON nbf () USING
 'com.stratio.cassandra.lucene.Index' WITH OPTIONS = {'refresh_seconds':
 '1', 'schema': '{
 fields : {
 id  : {type : "bigint"},
 f_d_name : {
 type   : "string",
 indexed: true,
 sorted : false,
 validated  : true,
 case_sensitive : false
 }
 }
 }'};

>>>
>>>
>>>
>>> -
>>> Atul Saroha
>>> *Lead Software Engineer*
>>> *M*: +91 8447784271 *T*: +91 124-415-6069 *EXT*: 12369
>>> Plot # 362, ASF Centre - Tower A, Udyog Vihar,
>>>  Phase -4, Sector 18, Gurgaon, Haryana 122016, INDIA
>>>
>>> On Mon, Jun 13, 2016 at 11:11 PM, Siddharth Verma <
>>> verma.siddha...@snapdeal.com> wrote:
>>>
 No, all rows were not the same.
 Querying only on the partition key gives 20 rows.
 In the erroneous result, while querying on partition key and clustering
 key, we got 16 of those 20 rows.

 And for "*tombstone_threshold"* there isn't any entry at column family
 level.

 Thanks,
 Siddharth Verma



>>>
>>
>>
>> --
>> Tyler Hobbs
>> DataStax 
>>
>
>

Cassandra monitoring

[Arun Ramakrishnan]

What are the options for a very small and nimble startup to do keep a
cassandra cluster running well oiled. We are on AWS. We are interested in a
monitoring tool and potentially also cluster management tools.

We are currently on apache cassandra 3.7. We were hoping the datastax
opscenter would be it (It is free for startups our size). But, looks like
it does not support cassandra versions greater than v2.1. It is pretty
surprising considering cassandra v2.1  came out in 2014.

We would consider downgrading to datastax cassandra 2.1 just to have robust
monitoring tools. But, I am not sure if having opscenter offsets all the
improvements that have been added to cassandra since 2.1.

Sematext has a integrations for monitoring cassandra. Does anyone have good
experience with it ?

How much work would be involved to setup Ganglia or some such option for
cassandra ?

Thanks,
Arun

Re: How to print out the metrics information, like compaction and garbage collection?

[Otis Gospodnetić]

Hi Jun,

Here's a tool for dumping JMX contents: https://github.com/sematext/jmxc
Here's a tool/service for monitoring Cassandra:
https://sematext.com/spm/integrations/cassandra-monitoring/

Otis
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/


On Mon, Jun 13, 2016 at 5:17 PM, Jun Wu  wrote:

> Hi there,
>
>I've deployed 6 node in Amazon EC2. I'm trying to monitor some metrics
> for each node and print them out when I write/read data into Cassandra.
> Specifically I want to print out the information about garbage collection
> and compaction.
>
>I do notice that there's metrics for compaction in o.a.c.metrics.
> CompactionMetrics.java. But I don't know how to get them, I've tried method
> below:
>
>CompactionMetrics metrics = new CompactionMetrics();
>System.out.println(metrics.pendingtasks);
>
>But it doesn't work.
>
> *
>   Another/same question is about the garbage collection, any idea on where
> should I refer to and get the information about garbage collection, like
> garbage collection count, collection time, etc.
>
>Any hint will be appreciated.
>
> Thanks!
>
> Jun
>

Re: select query on entire primary key returning more than one row in result

[Bhuvan Rawal]

I can reproduce CASSANDRA-11513
 locally on 3.5,
possible duplicate.

On Wed, Jun 15, 2016 at 12:29 AM, Joel Knighton 
wrote:

> There's some precedent for similar issues with static columns in 3.5 with
> https://issues.apache.org/jira/browse/CASSANDRA-11513 - a deterministic
> (or somewhat deterministic) path for reproduction would help narrow the
> issue down farther. I've played around locally with similar schemas (sans
> the stratio indices) and couldn't reproduce the issue.
>
> On Tue, Jun 14, 2016 at 1:41 PM, Bhuvan Rawal  wrote:
>
>> Jira CASSANDRA-12003
>>  Has been created
>> for the same.
>>
>> On Tue, Jun 14, 2016 at 11:54 PM, Atul Saroha 
>> wrote:
>>
>>> Hi Tyler,
>>>
>>> This issue is mainly visible for tables having static columns, still
>>> investigating.
>>> We will try to test after removing lucene index but I don’t think this
>>> plug-in could led to change in behaviour of cassandra write to table's
>>> memtable.
>>>
>>>
>>> -
>>> Atul Saroha
>>> *Lead Software Engineer*
>>> *M*: +91 8447784271 *T*: +91 124-415-6069 *EXT*: 12369
>>> Plot # 362, ASF Centre - Tower A, Udyog Vihar,
>>>  Phase -4, Sector 18, Gurgaon, Haryana 122016, INDIA
>>>
>>> On Tue, Jun 14, 2016 at 9:54 PM, Tyler Hobbs  wrote:
>>>
 Is 'id' your partition key? I'm not familiar with the stratio indexes,
 but it looks like the primary key columns are both indexed.  Perhaps this
 is related?

 On Tue, Jun 14, 2016 at 1:25 AM, Atul Saroha 
 wrote:

> After further debug, this issue is found in in-memory memtable as
> doing nodetool flush + compact resolve the issue. And there is no batch
> write used for this table which is showing issue.
> Table properties:
>
> WITH CLUSTERING ORDER BY (f_name ASC)
>> AND bloom_filter_fp_chance = 0.01
>> AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
>> AND comment = ''
>> AND compaction = {'class':
>> 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy',
>> 'max_threshold': '32', 'min_threshold': '4'}
>> AND compression = {'chunk_length_in_kb': '64', 'class':
>> 'org.apache.cassandra.io.compress.LZ4Compressor'}
>> AND crc_check_chance = 1.0
>> AND dclocal_read_repair_chance = 0.1
>> AND default_time_to_live = 0
>> AND gc_grace_seconds = 864000
>> AND max_index_interval = 2048
>> AND memtable_flush_period_in_ms = 0
>> AND min_index_interval = 128
>> AND read_repair_chance = 0.0
>> AND speculative_retry = '99PERCENTILE';
>> CREATE CUSTOM INDEX nbf_index ON nbf () USING
>> 'com.stratio.cassandra.lucene.Index' WITH OPTIONS = {'refresh_seconds':
>> '1', 'schema': '{
>> fields : {
>> id  : {type : "bigint"},
>> f_d_name : {
>> type   : "string",
>> indexed: true,
>> sorted : false,
>> validated  : true,
>> case_sensitive : false
>> }
>> }
>> }'};
>>
>
>
>
> -
> Atul Saroha
> *Lead Software Engineer*
> *M*: +91 8447784271 *T*: +91 124-415-6069 *EXT*: 12369
> Plot # 362, ASF Centre - Tower A, Udyog Vihar,
>  Phase -4, Sector 18, Gurgaon, Haryana 122016, INDIA
>
> On Mon, Jun 13, 2016 at 11:11 PM, Siddharth Verma <
> verma.siddha...@snapdeal.com> wrote:
>
>> No, all rows were not the same.
>> Querying only on the partition key gives 20 rows.
>> In the erroneous result, while querying on partition key and
>> clustering key, we got 16 of those 20 rows.
>>
>> And for "*tombstone_threshold"* there isn't any entry at column
>> family level.
>>
>> Thanks,
>> Siddharth Verma
>>
>>
>>
>


 --
 Tyler Hobbs
 DataStax 

>>>
>>>
>>
>
>
> --
>
> 
>
> Joel Knighton
> Cassandra Developer | joel.knigh...@datastax.com
>
> 
>  
> 
>  
>
> 
>

Re: select query on entire primary key returning more than one row in result

[Bhuvan Rawal]

Joel,

If we look at the schema carefully:

CREATE TABLE test0 (
pk int,
a int,
b text,
s text static,
PRIMARY KEY (*pk, a)*
);

and filtering is performed on clustering column a and its not a static
column:

select * from test0 where pk=0 and a=2;



On Wed, Jun 15, 2016 at 12:39 AM, Joel Knighton 
wrote:

> It doesn't seem to be an exact duplicate - CASSANDRA-11513 relies on you
> selecting a static column, which you weren't doing in the reported issue.
> That said, I haven't looked too closely.
>
> On Tue, Jun 14, 2016 at 2:07 PM, Bhuvan Rawal  wrote:
>
>> I can reproduce CASSANDRA-11513
>>  locally on 3.5,
>> possible duplicate.
>>
>> On Wed, Jun 15, 2016 at 12:29 AM, Joel Knighton <
>> joel.knigh...@datastax.com> wrote:
>>
>>> There's some precedent for similar issues with static columns in 3.5
>>> with https://issues.apache.org/jira/browse/CASSANDRA-11513 - a
>>> deterministic (or somewhat deterministic) path for reproduction would help
>>> narrow the issue down farther. I've played around locally with similar
>>> schemas (sans the stratio indices) and couldn't reproduce the issue.
>>>
>>> On Tue, Jun 14, 2016 at 1:41 PM, Bhuvan Rawal 
>>> wrote:
>>>
 Jira CASSANDRA-12003
  Has been
 created for the same.

 On Tue, Jun 14, 2016 at 11:54 PM, Atul Saroha  wrote:

> Hi Tyler,
>
> This issue is mainly visible for tables having static columns, still
> investigating.
> We will try to test after removing lucene index but I don’t think this
> plug-in could led to change in behaviour of cassandra write to table's
> memtable.
>
>
> -
> Atul Saroha
> *Lead Software Engineer*
> *M*: +91 8447784271 *T*: +91 124-415-6069 *EXT*: 12369
> Plot # 362, ASF Centre - Tower A, Udyog Vihar,
>  Phase -4, Sector 18, Gurgaon, Haryana 122016, INDIA
>
> On Tue, Jun 14, 2016 at 9:54 PM, Tyler Hobbs 
> wrote:
>
>> Is 'id' your partition key? I'm not familiar with the stratio
>> indexes, but it looks like the primary key columns are both indexed.
>> Perhaps this is related?
>>
>> On Tue, Jun 14, 2016 at 1:25 AM, Atul Saroha <
>> atul.sar...@snapdeal.com> wrote:
>>
>>> After further debug, this issue is found in in-memory memtable as
>>> doing nodetool flush + compact resolve the issue. And there is no batch
>>> write used for this table which is showing issue.
>>> Table properties:
>>>
>>> WITH CLUSTERING ORDER BY (f_name ASC)
 AND bloom_filter_fp_chance = 0.01
 AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
 AND comment = ''
 AND compaction = {'class':
 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy',
 'max_threshold': '32', 'min_threshold': '4'}
 AND compression = {'chunk_length_in_kb': '64', 'class':
 'org.apache.cassandra.io.compress.LZ4Compressor'}
 AND crc_check_chance = 1.0
 AND dclocal_read_repair_chance = 0.1
 AND default_time_to_live = 0
 AND gc_grace_seconds = 864000
 AND max_index_interval = 2048
 AND memtable_flush_period_in_ms = 0
 AND min_index_interval = 128
 AND read_repair_chance = 0.0
 AND speculative_retry = '99PERCENTILE';
 CREATE CUSTOM INDEX nbf_index ON nbf () USING
 'com.stratio.cassandra.lucene.Index' WITH OPTIONS = {'refresh_seconds':
 '1', 'schema': '{
 fields : {
 id  : {type : "bigint"},
 f_d_name : {
 type   : "string",
 indexed: true,
 sorted : false,
 validated  : true,
 case_sensitive : false
 }
 }
 }'};

>>>
>>>
>>>
>>> -
>>> Atul Saroha
>>> *Lead Software Engineer*
>>> *M*: +91 8447784271 *T*: +91 124-415-6069 *EXT*: 12369
>>> Plot # 362, ASF Centre - Tower A, Udyog Vihar,
>>>  Phase -4, Sector 18, Gurgaon, Haryana 122016, INDIA
>>>
>>> On Mon, Jun 13, 2016 at 11:11 PM, Siddharth Verma <
>>> verma.siddha...@snapdeal.com> wrote:
>>>
 No, all rows were not the same.
 Querying only on the partition key gives 20 rows.
 In the erroneous result, while querying on partition key and
 clustering key, 

Re: select query on entire primary key returning more than one row in result

[Bhuvan Rawal]

I have verified this issue to be fixed in 3.6 and 3.7.
And the issue mentioned on this thread is fixed as well.

On Wed, Jun 15, 2016 at 12:43 AM, Bhuvan Rawal  wrote:

> Joel,
>
> If we look at the schema carefully:
>
> CREATE TABLE test0 (
> pk int,
> a int,
> b text,
> s text static,
> PRIMARY KEY (*pk, a)*
> );
>
> and filtering is performed on clustering column a and its not a static
> column:
>
> select * from test0 where pk=0 and a=2;
>
>
>
> On Wed, Jun 15, 2016 at 12:39 AM, Joel Knighton <
> joel.knigh...@datastax.com> wrote:
>
>> It doesn't seem to be an exact duplicate - CASSANDRA-11513 relies on you
>> selecting a static column, which you weren't doing in the reported issue.
>> That said, I haven't looked too closely.
>>
>> On Tue, Jun 14, 2016 at 2:07 PM, Bhuvan Rawal 
>> wrote:
>>
>>> I can reproduce CASSANDRA-11513
>>>  locally on 3.5,
>>> possible duplicate.
>>>
>>> On Wed, Jun 15, 2016 at 12:29 AM, Joel Knighton <
>>> joel.knigh...@datastax.com> wrote:
>>>
 There's some precedent for similar issues with static columns in 3.5
 with https://issues.apache.org/jira/browse/CASSANDRA-11513 - a
 deterministic (or somewhat deterministic) path for reproduction would help
 narrow the issue down farther. I've played around locally with similar
 schemas (sans the stratio indices) and couldn't reproduce the issue.

 On Tue, Jun 14, 2016 at 1:41 PM, Bhuvan Rawal 
 wrote:

> Jira CASSANDRA-12003
>  Has been
> created for the same.
>
> On Tue, Jun 14, 2016 at 11:54 PM, Atul Saroha <
> atul.sar...@snapdeal.com> wrote:
>
>> Hi Tyler,
>>
>> This issue is mainly visible for tables having static columns, still
>> investigating.
>> We will try to test after removing lucene index but I don’t think
>> this plug-in could led to change in behaviour of cassandra write to 
>> table's
>> memtable.
>>
>>
>> -
>> Atul Saroha
>> *Lead Software Engineer*
>> *M*: +91 8447784271 *T*: +91 124-415-6069 *EXT*: 12369
>> Plot # 362, ASF Centre - Tower A, Udyog Vihar,
>>  Phase -4, Sector 18, Gurgaon, Haryana 122016, INDIA
>>
>> On Tue, Jun 14, 2016 at 9:54 PM, Tyler Hobbs 
>> wrote:
>>
>>> Is 'id' your partition key? I'm not familiar with the stratio
>>> indexes, but it looks like the primary key columns are both indexed.
>>> Perhaps this is related?
>>>
>>> On Tue, Jun 14, 2016 at 1:25 AM, Atul Saroha <
>>> atul.sar...@snapdeal.com> wrote:
>>>
 After further debug, this issue is found in in-memory memtable as
 doing nodetool flush + compact resolve the issue. And there is no batch
 write used for this table which is showing issue.
 Table properties:

 WITH CLUSTERING ORDER BY (f_name ASC)
> AND bloom_filter_fp_chance = 0.01
> AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
> AND comment = ''
> AND compaction = {'class':
> 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy',
> 'max_threshold': '32', 'min_threshold': '4'}
> AND compression = {'chunk_length_in_kb': '64', 'class':
> 'org.apache.cassandra.io.compress.LZ4Compressor'}
> AND crc_check_chance = 1.0
> AND dclocal_read_repair_chance = 0.1
> AND default_time_to_live = 0
> AND gc_grace_seconds = 864000
> AND max_index_interval = 2048
> AND memtable_flush_period_in_ms = 0
> AND min_index_interval = 128
> AND read_repair_chance = 0.0
> AND speculative_retry = '99PERCENTILE';
> CREATE CUSTOM INDEX nbf_index ON nbf () USING
> 'com.stratio.cassandra.lucene.Index' WITH OPTIONS = 
> {'refresh_seconds':
> '1', 'schema': '{
> fields : {
> id  : {type : "bigint"},
> f_d_name : {
> type   : "string",
> indexed: true,
> sorted : false,
> validated  : true,
> case_sensitive : false
> }
> }
> }'};
>



 -
 Atul Saroha
 *Lead Software Engineer*
 *M*: +91 8447784271 *T*: +91 124-415-6069 *EXT*: 12369
 Plot # 362, ASF Centre - Tower A, Udyog Vihar,
  Phase -4, Sector 

Re: select query on entire primary key returning more than one row in result

[Joel Knighton]

It doesn't seem to be an exact duplicate - CASSANDRA-11513 relies on you
selecting a static column, which you weren't doing in the reported issue.
That said, I haven't looked too closely.

On Tue, Jun 14, 2016 at 2:07 PM, Bhuvan Rawal  wrote:

> I can reproduce CASSANDRA-11513
>  locally on 3.5,
> possible duplicate.
>
> On Wed, Jun 15, 2016 at 12:29 AM, Joel Knighton <
> joel.knigh...@datastax.com> wrote:
>
>> There's some precedent for similar issues with static columns in 3.5 with
>> https://issues.apache.org/jira/browse/CASSANDRA-11513 - a deterministic
>> (or somewhat deterministic) path for reproduction would help narrow the
>> issue down farther. I've played around locally with similar schemas (sans
>> the stratio indices) and couldn't reproduce the issue.
>>
>> On Tue, Jun 14, 2016 at 1:41 PM, Bhuvan Rawal 
>> wrote:
>>
>>> Jira CASSANDRA-12003
>>>  Has been
>>> created for the same.
>>>
>>> On Tue, Jun 14, 2016 at 11:54 PM, Atul Saroha 
>>> wrote:
>>>
 Hi Tyler,

 This issue is mainly visible for tables having static columns, still
 investigating.
 We will try to test after removing lucene index but I don’t think this
 plug-in could led to change in behaviour of cassandra write to table's
 memtable.


 -
 Atul Saroha
 *Lead Software Engineer*
 *M*: +91 8447784271 *T*: +91 124-415-6069 *EXT*: 12369
 Plot # 362, ASF Centre - Tower A, Udyog Vihar,
  Phase -4, Sector 18, Gurgaon, Haryana 122016, INDIA

 On Tue, Jun 14, 2016 at 9:54 PM, Tyler Hobbs 
 wrote:

> Is 'id' your partition key? I'm not familiar with the stratio indexes,
> but it looks like the primary key columns are both indexed.  Perhaps this
> is related?
>
> On Tue, Jun 14, 2016 at 1:25 AM, Atul Saroha  > wrote:
>
>> After further debug, this issue is found in in-memory memtable as
>> doing nodetool flush + compact resolve the issue. And there is no batch
>> write used for this table which is showing issue.
>> Table properties:
>>
>> WITH CLUSTERING ORDER BY (f_name ASC)
>>> AND bloom_filter_fp_chance = 0.01
>>> AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
>>> AND comment = ''
>>> AND compaction = {'class':
>>> 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy',
>>> 'max_threshold': '32', 'min_threshold': '4'}
>>> AND compression = {'chunk_length_in_kb': '64', 'class':
>>> 'org.apache.cassandra.io.compress.LZ4Compressor'}
>>> AND crc_check_chance = 1.0
>>> AND dclocal_read_repair_chance = 0.1
>>> AND default_time_to_live = 0
>>> AND gc_grace_seconds = 864000
>>> AND max_index_interval = 2048
>>> AND memtable_flush_period_in_ms = 0
>>> AND min_index_interval = 128
>>> AND read_repair_chance = 0.0
>>> AND speculative_retry = '99PERCENTILE';
>>> CREATE CUSTOM INDEX nbf_index ON nbf () USING
>>> 'com.stratio.cassandra.lucene.Index' WITH OPTIONS = {'refresh_seconds':
>>> '1', 'schema': '{
>>> fields : {
>>> id  : {type : "bigint"},
>>> f_d_name : {
>>> type   : "string",
>>> indexed: true,
>>> sorted : false,
>>> validated  : true,
>>> case_sensitive : false
>>> }
>>> }
>>> }'};
>>>
>>
>>
>>
>> -
>> Atul Saroha
>> *Lead Software Engineer*
>> *M*: +91 8447784271 *T*: +91 124-415-6069 *EXT*: 12369
>> Plot # 362, ASF Centre - Tower A, Udyog Vihar,
>>  Phase -4, Sector 18, Gurgaon, Haryana 122016, INDIA
>>
>> On Mon, Jun 13, 2016 at 11:11 PM, Siddharth Verma <
>> verma.siddha...@snapdeal.com> wrote:
>>
>>> No, all rows were not the same.
>>> Querying only on the partition key gives 20 rows.
>>> In the erroneous result, while querying on partition key and
>>> clustering key, we got 16 of those 20 rows.
>>>
>>> And for "*tombstone_threshold"* there isn't any entry at column
>>> family level.
>>>
>>> Thanks,
>>> Siddharth Verma
>>>
>>>
>>>
>>
>
>
> --
> Tyler Hobbs
> DataStax 
>


>>>
>>
>>
>> --
>>
>> 
>>
>> Joel Knighton
>> Cassandra Developer | joel.knigh...@datastax.com
>>
>> 

Re: Cassandra monitoring

[Jonathan Haddad]

Depends what you want to monitor.  I wouldn't use a lesser version of
Cassandra for OpsCenter, it doesn't give you a ton you can't get elsewhere
and it's not ever going to support OSS > 2.1, so you kind of limit yourself
to a pretty old version of Cassandra for a non-good reason.

What else do you use for monitoring in your infra?  I've used a mix of OSS
tools (nagios, statsd, graphite, ELK), and hosted solutions. The nice part
about them is that you can monitor your whole stack in a single UI not just
your database.

On Tue, Jun 14, 2016 at 12:10 PM Arun Ramakrishnan <
sinchronized.a...@gmail.com> wrote:

> What are the options for a very small and nimble startup to do keep a
> cassandra cluster running well oiled. We are on AWS. We are interested in a
> monitoring tool and potentially also cluster management tools.
>
> We are currently on apache cassandra 3.7. We were hoping the datastax
> opscenter would be it (It is free for startups our size). But, looks like
> it does not support cassandra versions greater than v2.1. It is pretty
> surprising considering cassandra v2.1  came out in 2014.
>
> We would consider downgrading to datastax cassandra 2.1 just to have
> robust monitoring tools. But, I am not sure if having opscenter offsets all
> the improvements that have been added to cassandra since 2.1.
>
> Sematext has a integrations for monitoring cassandra. Does anyone have
> good experience with it ?
>
> How much work would be involved to setup Ganglia or some such option for
> cassandra ?
>
> Thanks,
> Arun
>
>
>
>
>
>
>

RE: How to print out the metrics information, like compaction and garbage collection?

[Jun Wu]

Hi Otis,
   Thank you so much for the reply. I do appreciate it.
   Actually, I've tried the sematext spm few days ago:) It works well and is 
easy to deploy. 
However, in the monitoring output figure for different metrics, the 
interval is 1 minute, which is longer than I want. What I want is something 
like one metric value / second and I want to have a close eye on each metric.
That's why I post question and want to print out the metrics information in 
the console.
 I'm wondering whether you have any other solution for my question.
Thanks!
Jun

From: otis.gospodne...@gmail.com
Date: Tue, 14 Jun 2016 15:01:20 -0400
Subject: Re: How to print out the metrics information, like compaction and 
garbage collection?
To: user@cassandra.apache.org

Hi Jun,
Here's a tool for dumping JMX contents: https://github.com/sematext/jmxcHere's 
a tool/service for monitoring Cassandra: 
https://sematext.com/spm/integrations/cassandra-monitoring/Otis
--
Monitoring - Log Management - Alerting - Anomaly DetectionSolr & Elasticsearch 
Consulting Support Training - http://sematext.com/



On Mon, Jun 13, 2016 at 5:17 PM, Jun Wu  wrote:



Hi there,
   I've deployed 6 node in Amazon EC2. I'm trying to monitor some metrics for 
each node and print them out when I write/read data into Cassandra. 
Specifically I want to print out the information about garbage collection and 
compaction.
   I do notice that there's metrics for compaction in o.a.c.metrics. 
CompactionMetrics.java. But I don't know how to get them, I've tried method 
below:  CompactionMetrics metrics = new CompactionMetrics();   
System.out.println(metrics.pendingtasks);
   But it doesn't work. 
*  Another/same question is about the garbage collection, any 
idea on where should I refer to and get the information about garbage 
collection, like garbage collection count, collection time, etc.
   Any hint will be appreciated.
Thanks!
Jun   

  

Re: Cassandra monitoring

[Michał Łowicki]

My team ended up with Diamond / StatsD / Graphite / Grafana (more
background in
medium.com/@mlowicki/alternatives-to-datastax-opscenter-8ad893efe063).
We're relying on such stack heavily in other projects and our infra in
general.

On Tue, Jun 14, 2016 at 10:29 PM, Jonathan Haddad  wrote:

> OpsCenter going forward is limited to datastax enterprise versions.
>
> I know a lot of people like DataDog, but I haven't used it.  Maybe other
> people on the list can speak from recent first hand experience on it's pros
> and cons.
>
>
> On Tue, Jun 14, 2016 at 1:20 PM Arun Ramakrishnan <
> sinchronized.a...@gmail.com> wrote:
>
>> Thanks Jonathan.
>>
>> Out of curiosity, does opscenter support some later version of cassandra
>> that is not OSS ?
>>
>> Well, the most minimal requirement is that, I want to be able to monitor
>> for cluster health and hook this info to some alerting platform. We are AWS
>> heavy. We just really heavily on AWS cloud watch for our metrics as of now.
>> We prefer to not spend our time setting up additional tools if we can help
>> it. So, if we needed a 3rd party service we would consider an APM or
>> monitoring service that is on the cheaper side.
>>
>>
>>
>>
>> On Tue, Jun 14, 2016 at 12:20 PM, Jonathan Haddad 
>> wrote:
>>
>>> Depends what you want to monitor.  I wouldn't use a lesser version of
>>> Cassandra for OpsCenter, it doesn't give you a ton you can't get elsewhere
>>> and it's not ever going to support OSS > 2.1, so you kind of limit yourself
>>> to a pretty old version of Cassandra for a non-good reason.
>>>
>>> What else do you use for monitoring in your infra?  I've used a mix of
>>> OSS tools (nagios, statsd, graphite, ELK), and hosted solutions. The nice
>>> part about them is that you can monitor your whole stack in a single UI not
>>> just your database.
>>>
>>> On Tue, Jun 14, 2016 at 12:10 PM Arun Ramakrishnan <
>>> sinchronized.a...@gmail.com> wrote:
>>>
 What are the options for a very small and nimble startup to do keep a
 cassandra cluster running well oiled. We are on AWS. We are interested in a
 monitoring tool and potentially also cluster management tools.

 We are currently on apache cassandra 3.7. We were hoping the datastax
 opscenter would be it (It is free for startups our size). But, looks like
 it does not support cassandra versions greater than v2.1. It is pretty
 surprising considering cassandra v2.1  came out in 2014.

 We would consider downgrading to datastax cassandra 2.1 just to have
 robust monitoring tools. But, I am not sure if having opscenter offsets all
 the improvements that have been added to cassandra since 2.1.

 Sematext has a integrations for monitoring cassandra. Does anyone have
 good experience with it ?

 How much work would be involved to setup Ganglia or some such option
 for cassandra ?

 Thanks,
 Arun







>>


-- 
BR,
Michał Łowicki

Re: Data lost in Cassandra 3.5 single instance via Erlang driver

[linbo liao]

I am not sure, but looks it will cause the update other than insert. If it
is true, the only way is request includes IF NOT EXISTS, inform the client
it failed?

Thanks,
Linbo

2016-06-15 10:59 GMT+08:00 Ben Slater :

> Is it possible that your pub_timestamp values are colliding (which would
> result in an update rather than an insert)?
>
> On Wed, 15 Jun 2016 at 12:55 linbo liao  wrote:
>
>> Hi,
>>
>> I use Erlang driver to send data to Cassandra, do testing at local
>> environment meet data lost issue. I have no idea what step is wrong.
>>
>> *Environment:*
>>
>> 1. Ubuntu  12.04 LTS  x64bit
>> 2. Cassandra 3.5 single instance, not a cluster, installed via the
>> offical installation document, and didn't change any configuration except
>> enable the authenticator and authorizer.
>> 3. Cassandra binary protocol v4
>> 3. Latest Erlang driver https://github.com/matehat/cqerl
>> 4. Erlang OTP 18.3
>>
>> *Schema:*
>>
>> > DESCRIBE TABLE message.history
>>>
>>> CREATE TABLE message.history (
>>> appkey text,
>>> pub_date text,
>>> pub_timestamp timestamp,
>>> apns text,
>>> message blob,
>>> message_id text,
>>> pub_method smallint,
>>> qos smallint,
>>> recv_type smallint,
>>> topic text,
>>> PRIMARY KEY ((appkey, pub_date), pub_timestamp)
>>> ) WITH CLUSTERING ORDER BY (pub_timestamp ASC)
>>> AND bloom_filter_fp_chance = 0.01
>>> AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
>>> AND comment = ''
>>> AND compaction = {'class':
>>> 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy',
>>> 'max_threshold': '32', 'min_threshold': '4',
>>> 'unchecked_tombstone_compaction': 'true'}
>>> AND compression = {'chunk_length_in_kb': '64', 'class':
>>> 'org.apache.cassandra.io.compress.LZ4Compressor'}
>>> AND crc_check_chance = 1.0
>>> AND dclocal_read_repair_chance = 0.1
>>> AND default_time_to_live = 0
>>> AND gc_grace_seconds = 3600
>>> AND max_index_interval = 2048
>>> AND memtable_flush_period_in_ms = 0
>>> AND min_index_interval = 128
>>> AND read_repair_chance = 0.0
>>> AND speculative_retry = '99PERCENTILE';
>>>
>>
>>
>> *Issue:*
>>
>>  The client send 6 insert request to Server (ttl is 1000s), check the TCP
>> package every thing works fine, but data will miss random at Server
>> side(select insert data was missing).
>>
>>
>> > select * from history;
>>>
>>>  appkey   | pub_date   | pub_timestamp
>>> | apns | message
>>>   | message_id   | pub_method |
>>> qos | recv_type | topic
>>>
>>> --++-+--+
>>>
>>> --+--++-+---+-
>>>  52fcc04c4dc903d66d6f8f92 | 2016-06-13 | 2016-06-13 09:16:24.70+
>>> |  |
>>> 0x68656c6c6f20746f20616c6961732066726f6d207
>>> 075626c697368325f746f5f616c696173 |   589020307122032641 |  2
>>> |   1 | 1 | alias_mqttc_sub
>>>  52fcc04c4dc903d66d6f8f92 | 2016-06-13 | 2016-06-13 09:16:24.817000+
>>> | {"aps":{"sound":"bingbong.aiff","badge":3,"alert":"douban"}}
>>> |   0x7b22612
>>> 23a2066726f6d207075626c697368327d | 11833652203486491113 |  2
>>> |   1 | 0 |   t2thi
>>>  52fcc04c4dc903d66d6f8f92 | 2016-06-13 | 2016-06-13 09:16:24.818000+
>>> |  |
>>>  0x66726f6d20707974686f6e |   589020307579211776 |  2
>>> |   1 | 0 |  testtopic2
>>>  52fcc04c4dc903d66d6f8f92 | 2016-06-13 | 2016-06-13 09:16:24.89+
>>> | {"aps":{"sound":"bingbong.aiff","badge":3,"alert":"douban"}} |
>>>  0x66726f6d207075626c69736832 |   589020307814092800 |  2
>>> |   1 | 0 |  testtopic2
>>>  52fcc04c4dc903d66d6f8f92 | 2016-06-13 | 2016-06-13 09:16:25.024000+
>>> |  |
>>>0x68656c6c6f20746f20616c696173 |   589020307818287105 |  2
>>> |   1 | 1 |mytestalias1
>>>
>>
>>
>> *TCP package inserted succeed flow:*
>>
>> 17:16:24.818210 IP localhost.38918 > localhost.9042: Flags [P.], seq
>>> 1953472577:1953472814, ack 1963420469, win 530, options [nop,nop,TS val
>>> 72982868 ecr 72940042], length 237
>>> 0x:  4500 0121 e5b1 4000 4006 5623 7f00 0001  E..!..@.@.V#
>>> 0x0010:  7f00 0001 9806 2352 746f a041 7507 6b35  ..#Rto.Au.k5
>>> 0x0020:  8018 0212 ff15  0101 080a 0459 a154  .Y.T
>>> 0x0030:  0458 fa0a *0400  0a00  e400 1005*  .X..
>>> 0x0040:  6e06 c1fc 222c 813f 6228 61c5 7364 

Re: select query on entire primary key returning more than one row in result

[Bhuvan Rawal]

Joel,

Id rather thank you for naming 11513 earlier in the mail, I would have been
lost in the code for a much longer time otherwise.

Repeating what Tianshi mentioned in 11513 - "*Cassandra community is
awesome! Should buy you a beer, Joel."* :)

On Wed, Jun 15, 2016 at 6:01 AM, Joel Knighton 
wrote:

> Great work, Bhuvan - I sat down after work to look at this more carefully.
>
> For a short summary, you are correct.
>
> For a longer summary, I initially thought the reproduction you provided
> would not run into the issue from 3.4/3.5 because it didn't select any
> static columns, which meant that it wouldn't have statics in its
> ColumnFilter (basically, the filter we apply when deciding if we need to
> look for the requested data in more SSTables). This was an incorrect
> understanding - in order to preserve the CQL semantic (see CASSANDRA-6588
> for details), we are including all columns, including the static columns,
> in the fetched columns, which means they are part of the ColumnFilter. I
> believe there may be an opportunity for an optimization here, but that's a
> whole different discussion. I now agree that these are the same issue.
>
> You are correct in your analysis that 3.4/3.5 are the only affected
> versions. It has been patched in release 3.6 forward and was not introduced
> until 3.4
>
> Thanks for sticking with me on this - I'm going to resolve CASSANDRA-12003
> as a duplicate of CASSANDRA-11513.
>
> On Tue, Jun 14, 2016 at 4:21 PM, Bhuvan Rawal  wrote:
>
>> Joel,
>>
>> Thanks for your reply, I have checked and found that the behavior is same
>> in case of CASSANDRA-11513
>> . I have verified
>> this behavior (for both 11513 & 12003) to occur in case of 3.4 & 3.5. They
>> both don't occur in 3.0.4, 3.6 & 3.7.
>>
>> Please find below the results of selecting only pk and clustering key
>> from 11513. It has also been verified that both issues occur while
>> selecting all / filtered rows therefore selection criteria is not an issue
>> filtering by WHERE is:
>>
>> cqlsh:ks> select pk,a from test0 where pk=0 and a=2;
>>
>>  pk | a
>> +---
>>   0 | 1
>>   0 | 2
>>   0 | 3
>>
>> We can verify this claim by applying 11513 Patch to 3.5 Tag and build &
>> test for 12003. If it is fixed then we can guarantee the claim. Let me
>> know if any further input may possibly be required here.
>>
>> On Wed, Jun 15, 2016 at 2:23 AM, Joel Knighton <
>> joel.knigh...@datastax.com> wrote:
>>
>>> The important part of that query is that it's selecting a static column
>>> (with select *), not whether it is filtering on one. In CASSANDRA-12003 and
>>> this thread, it looks like you're only selecting the primary and clustering
>>> columns. I'd be cautious about concluding that CASSANDRA-12003 and
>>> CASSANDRA-11513 are the same issue and that CASSANDRA-12003 is fixed.
>>>
>>> If you have a reproduction path for CASSANDRA-12003, I'd recommend
>>> attaching it to a ticket, and someone can investigate internals to see if
>>> CASSANDRA-11513 (or something else entirely) fixed the issue.
>>>
>>> On Tue, Jun 14, 2016 at 2:13 PM, Bhuvan Rawal 
>>> wrote:
>>>
 Joel,

 If we look at the schema carefully:

 CREATE TABLE test0 (
 pk int,
 a int,
 b text,
 s text static,
 PRIMARY KEY (*pk, a)*
 );

 and filtering is performed on clustering column a and its not a static
 column:

 select * from test0 where pk=0 and a=2;



 On Wed, Jun 15, 2016 at 12:39 AM, Joel Knighton <
 joel.knigh...@datastax.com> wrote:

> It doesn't seem to be an exact duplicate - CASSANDRA-11513 relies on
> you selecting a static column, which you weren't doing in the reported
> issue. That said, I haven't looked too closely.
>
> On Tue, Jun 14, 2016 at 2:07 PM, Bhuvan Rawal 
> wrote:
>
>> I can reproduce CASSANDRA-11513
>>  locally on
>> 3.5, possible duplicate.
>>
>> On Wed, Jun 15, 2016 at 12:29 AM, Joel Knighton <
>> joel.knigh...@datastax.com> wrote:
>>
>>> There's some precedent for similar issues with static columns in 3.5
>>> with https://issues.apache.org/jira/browse/CASSANDRA-11513 - a
>>> deterministic (or somewhat deterministic) path for reproduction would 
>>> help
>>> narrow the issue down farther. I've played around locally with similar
>>> schemas (sans the stratio indices) and couldn't reproduce the issue.
>>>
>>> On Tue, Jun 14, 2016 at 1:41 PM, Bhuvan Rawal 
>>> wrote:
>>>
 Jira CASSANDRA-12003
  Has been
 created for the same.

 On Tue, Jun 14, 2016 at 11:54 PM, Atul Saroha <
 atul.sar...@snapdeal.com> wrote:

Data lost in Cassandra 3.5 single instance via Erlang driver

[linbo liao]

Hi,

I use Erlang driver to send data to Cassandra, do testing at local
environment meet data lost issue. I have no idea what step is wrong.

*Environment:*

1. Ubuntu  12.04 LTS  x64bit
2. Cassandra 3.5 single instance, not a cluster, installed via the offical
installation document, and didn't change any configuration except enable
the authenticator and authorizer.
3. Cassandra binary protocol v4
3. Latest Erlang driver https://github.com/matehat/cqerl
4. Erlang OTP 18.3

*Schema:*

> DESCRIBE TABLE message.history
>
> CREATE TABLE message.history (
> appkey text,
> pub_date text,
> pub_timestamp timestamp,
> apns text,
> message blob,
> message_id text,
> pub_method smallint,
> qos smallint,
> recv_type smallint,
> topic text,
> PRIMARY KEY ((appkey, pub_date), pub_timestamp)
> ) WITH CLUSTERING ORDER BY (pub_timestamp ASC)
> AND bloom_filter_fp_chance = 0.01
> AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
> AND comment = ''
> AND compaction = {'class':
> 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy',
> 'max_threshold': '32', 'min_threshold': '4',
> 'unchecked_tombstone_compaction': 'true'}
> AND compression = {'chunk_length_in_kb': '64', 'class':
> 'org.apache.cassandra.io.compress.LZ4Compressor'}
> AND crc_check_chance = 1.0
> AND dclocal_read_repair_chance = 0.1
> AND default_time_to_live = 0
> AND gc_grace_seconds = 3600
> AND max_index_interval = 2048
> AND memtable_flush_period_in_ms = 0
> AND min_index_interval = 128
> AND read_repair_chance = 0.0
> AND speculative_retry = '99PERCENTILE';
>


*Issue:*

 The client send 6 insert request to Server (ttl is 1000s), check the TCP
package every thing works fine, but data will miss random at Server
side(select insert data was missing).


> select * from history;
>
>  appkey   | pub_date   | pub_timestamp   |
> apns | message
>   | message_id   | pub_method |
> qos | recv_type | topic
>
> --++-+--+
>
> --+--++-+---+-
>  52fcc04c4dc903d66d6f8f92 | 2016-06-13 | 2016-06-13 09:16:24.70+
> |  |
> 0x68656c6c6f20746f20616c6961732066726f6d207
> 075626c697368325f746f5f616c696173 |   589020307122032641 |  2 |
> 1 | 1 | alias_mqttc_sub
>  52fcc04c4dc903d66d6f8f92 | 2016-06-13 | 2016-06-13 09:16:24.817000+ |
> {"aps":{"sound":"bingbong.aiff","badge":3,"alert":"douban"}}
> |   0x7b22612
> 23a2066726f6d207075626c697368327d | 11833652203486491113 |  2 |
> 1 | 0 |   t2thi
>  52fcc04c4dc903d66d6f8f92 | 2016-06-13 | 2016-06-13 09:16:24.818000+
> |  |
>  0x66726f6d20707974686f6e |   589020307579211776 |  2 |
> 1 | 0 |  testtopic2
>  52fcc04c4dc903d66d6f8f92 | 2016-06-13 | 2016-06-13 09:16:24.89+ |
> {"aps":{"sound":"bingbong.aiff","badge":3,"alert":"douban"}} |
>  0x66726f6d207075626c69736832 |   589020307814092800 |  2 |
> 1 | 0 |  testtopic2
>  52fcc04c4dc903d66d6f8f92 | 2016-06-13 | 2016-06-13 09:16:25.024000+
> |  |
>0x68656c6c6f20746f20616c696173 |   589020307818287105 |  2 |
> 1 | 1 |mytestalias1
>


*TCP package inserted succeed flow:*

17:16:24.818210 IP localhost.38918 > localhost.9042: Flags [P.], seq
> 1953472577:1953472814, ack 1963420469, win 530, options [nop,nop,TS val
> 72982868 ecr 72940042], length 237
> 0x:  4500 0121 e5b1 4000 4006 5623 7f00 0001  E..!..@.@.V#
> 0x0010:  7f00 0001 9806 2352 746f a041 7507 6b35  ..#Rto.Au.k5
> 0x0020:  8018 0212 ff15  0101 080a 0459 a154  .Y.T
> 0x0030:  0458 fa0a *0400  0a00  e400 1005*  .X..
> 0x0040:  6e06 c1fc 222c 813f 6228 61c5 7364 6500  n...",.?b(a.sde.
> 0x0050:  0105 000b  0018 3532 6663 6330 3463  52fcc04c
> 0x0060:  3464 6339 3033 6436 3664 3666 3866 3932  4dc903d66d6f8f92
> 0x0070:   000a 3230 3136 2d30 362d 3133   2016-06-13..
> 0x0080:  0008  0155 490c 3571  0002 0001  .UI.5q..
> 0x0090:   0002 0002  0014 3131 3833 3336  ..118336
> 0x00a0:  3532 3230 3334 3836 3439 3131 3133   52203486491113..
> 0x00b0:  0014 7b22 6122 3a20 6672 6f6d 2070 7562  ..{"a":.from.pub
> 0x00c0:  6c69 7368 327d  003c 7b22 6170 7322  lish2}...<{"aps"
> 0x00d0:  3a7b 2273 6f75 6e64 223a 2262 696e 6762  

Re: Data lost in Cassandra 3.5 single instance via Erlang driver

[Paul Fife]

If pub_timestamp could possibly match I'd suggest making it a timeuuid type
instead. With the above schema it's not a failure or data loss if the
 timestamp is duplicated - your writes all probably made it - the
duplicates just got overwritten.

On Tue, Jun 14, 2016 at 9:40 PM, linbo liao  wrote:

> I am not sure, but looks it will cause the update other than insert. If it
> is true, the only way is request includes IF NOT EXISTS, inform the
> client it failed?
>
> Thanks,
> Linbo
>
> 2016-06-15 10:59 GMT+08:00 Ben Slater :
>
>> Is it possible that your pub_timestamp values are colliding (which would
>> result in an update rather than an insert)?
>>
>> On Wed, 15 Jun 2016 at 12:55 linbo liao  wrote:
>>
>>> Hi,
>>>
>>> I use Erlang driver to send data to Cassandra, do testing at local
>>> environment meet data lost issue. I have no idea what step is wrong.
>>>
>>> *Environment:*
>>>
>>> 1. Ubuntu  12.04 LTS  x64bit
>>> 2. Cassandra 3.5 single instance, not a cluster, installed via the
>>> offical installation document, and didn't change any configuration except
>>> enable the authenticator and authorizer.
>>> 3. Cassandra binary protocol v4
>>> 3. Latest Erlang driver https://github.com/matehat/cqerl
>>> 4. Erlang OTP 18.3
>>>
>>> *Schema:*
>>>
>>> > DESCRIBE TABLE message.history

 CREATE TABLE message.history (
 appkey text,
 pub_date text,
 pub_timestamp timestamp,
 apns text,
 message blob,
 message_id text,
 pub_method smallint,
 qos smallint,
 recv_type smallint,
 topic text,
 PRIMARY KEY ((appkey, pub_date), pub_timestamp)
 ) WITH CLUSTERING ORDER BY (pub_timestamp ASC)
 AND bloom_filter_fp_chance = 0.01
 AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
 AND comment = ''
 AND compaction = {'class':
 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy',
 'max_threshold': '32', 'min_threshold': '4',
 'unchecked_tombstone_compaction': 'true'}
 AND compression = {'chunk_length_in_kb': '64', 'class':
 'org.apache.cassandra.io.compress.LZ4Compressor'}
 AND crc_check_chance = 1.0
 AND dclocal_read_repair_chance = 0.1
 AND default_time_to_live = 0
 AND gc_grace_seconds = 3600
 AND max_index_interval = 2048
 AND memtable_flush_period_in_ms = 0
 AND min_index_interval = 128
 AND read_repair_chance = 0.0
 AND speculative_retry = '99PERCENTILE';

>>>
>>>
>>> *Issue:*
>>>
>>>  The client send 6 insert request to Server (ttl is 1000s), check the
>>> TCP package every thing works fine, but data will miss random at Server
>>> side(select insert data was missing).
>>>
>>>
>>> > select * from history;

  appkey   | pub_date   |
 pub_timestamp   |
 apns | message
   | message_id   | pub_method |
 qos | recv_type | topic

 --++-+--+

 --+--++-+---+-
  52fcc04c4dc903d66d6f8f92 | 2016-06-13 | 2016-06-13
 09:16:24.70+
 |  |
 0x68656c6c6f20746f20616c6961732066726f6d207
 075626c697368325f746f5f616c696173 |   589020307122032641 |  2
 |   1 | 1 | alias_mqttc_sub
  52fcc04c4dc903d66d6f8f92 | 2016-06-13 | 2016-06-13
 09:16:24.817000+ |
 {"aps":{"sound":"bingbong.aiff","badge":3,"alert":"douban"}}
 |   0x7b22612
 23a2066726f6d207075626c697368327d | 11833652203486491113 |  2
 |   1 | 0 |   t2thi
  52fcc04c4dc903d66d6f8f92 | 2016-06-13 | 2016-06-13
 09:16:24.818000+
 |  |
  0x66726f6d20707974686f6e |   589020307579211776 |  2
 |   1 | 0 |  testtopic2
  52fcc04c4dc903d66d6f8f92 | 2016-06-13 | 2016-06-13
 09:16:24.89+ |
 {"aps":{"sound":"bingbong.aiff","badge":3,"alert":"douban"}} |
  0x66726f6d207075626c69736832 |   589020307814092800 |  2
 |   1 | 0 |  testtopic2
  52fcc04c4dc903d66d6f8f92 | 2016-06-13 | 2016-06-13
 09:16:25.024000+
 |  |
0x68656c6c6f20746f20616c696173 |   589020307818287105 |  2
 |   1 | 1 |mytestalias1

>>>
>>>
>>> *TCP package inserted succeed flow:*
>>>
>>> 17:16:24.818210 IP localhost.38918 > localhost.9042: 

Re: Data lost in Cassandra 3.5 single instance via Erlang driver

[Ben Slater]

Is it possible that your pub_timestamp values are colliding (which would
result in an update rather than an insert)?

On Wed, 15 Jun 2016 at 12:55 linbo liao  wrote:

> Hi,
>
> I use Erlang driver to send data to Cassandra, do testing at local
> environment meet data lost issue. I have no idea what step is wrong.
>
> *Environment:*
>
> 1. Ubuntu  12.04 LTS  x64bit
> 2. Cassandra 3.5 single instance, not a cluster, installed via the offical
> installation document, and didn't change any configuration except enable
> the authenticator and authorizer.
> 3. Cassandra binary protocol v4
> 3. Latest Erlang driver https://github.com/matehat/cqerl
> 4. Erlang OTP 18.3
>
> *Schema:*
>
> > DESCRIBE TABLE message.history
>>
>> CREATE TABLE message.history (
>> appkey text,
>> pub_date text,
>> pub_timestamp timestamp,
>> apns text,
>> message blob,
>> message_id text,
>> pub_method smallint,
>> qos smallint,
>> recv_type smallint,
>> topic text,
>> PRIMARY KEY ((appkey, pub_date), pub_timestamp)
>> ) WITH CLUSTERING ORDER BY (pub_timestamp ASC)
>> AND bloom_filter_fp_chance = 0.01
>> AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
>> AND comment = ''
>> AND compaction = {'class':
>> 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy',
>> 'max_threshold': '32', 'min_threshold': '4',
>> 'unchecked_tombstone_compaction': 'true'}
>> AND compression = {'chunk_length_in_kb': '64', 'class':
>> 'org.apache.cassandra.io.compress.LZ4Compressor'}
>> AND crc_check_chance = 1.0
>> AND dclocal_read_repair_chance = 0.1
>> AND default_time_to_live = 0
>> AND gc_grace_seconds = 3600
>> AND max_index_interval = 2048
>> AND memtable_flush_period_in_ms = 0
>> AND min_index_interval = 128
>> AND read_repair_chance = 0.0
>> AND speculative_retry = '99PERCENTILE';
>>
>
>
> *Issue:*
>
>  The client send 6 insert request to Server (ttl is 1000s), check the TCP
> package every thing works fine, but data will miss random at Server
> side(select insert data was missing).
>
>
> > select * from history;
>>
>>  appkey   | pub_date   | pub_timestamp
>> | apns | message
>>   | message_id   | pub_method |
>> qos | recv_type | topic
>>
>> --++-+--+
>>
>> --+--++-+---+-
>>  52fcc04c4dc903d66d6f8f92 | 2016-06-13 | 2016-06-13 09:16:24.70+
>> |  |
>> 0x68656c6c6f20746f20616c6961732066726f6d207
>> 075626c697368325f746f5f616c696173 |   589020307122032641 |  2 |
>> 1 | 1 | alias_mqttc_sub
>>  52fcc04c4dc903d66d6f8f92 | 2016-06-13 | 2016-06-13 09:16:24.817000+
>> | {"aps":{"sound":"bingbong.aiff","badge":3,"alert":"douban"}}
>> |   0x7b22612
>> 23a2066726f6d207075626c697368327d | 11833652203486491113 |  2 |
>> 1 | 0 |   t2thi
>>  52fcc04c4dc903d66d6f8f92 | 2016-06-13 | 2016-06-13 09:16:24.818000+
>> |  |
>>  0x66726f6d20707974686f6e |   589020307579211776 |  2 |
>> 1 | 0 |  testtopic2
>>  52fcc04c4dc903d66d6f8f92 | 2016-06-13 | 2016-06-13 09:16:24.89+
>> | {"aps":{"sound":"bingbong.aiff","badge":3,"alert":"douban"}} |
>>  0x66726f6d207075626c69736832 |   589020307814092800 |  2 |
>> 1 | 0 |  testtopic2
>>  52fcc04c4dc903d66d6f8f92 | 2016-06-13 | 2016-06-13 09:16:25.024000+
>> |  |
>>0x68656c6c6f20746f20616c696173 |   589020307818287105 |  2 |
>> 1 | 1 |mytestalias1
>>
>
>
> *TCP package inserted succeed flow:*
>
> 17:16:24.818210 IP localhost.38918 > localhost.9042: Flags [P.], seq
>> 1953472577:1953472814, ack 1963420469, win 530, options [nop,nop,TS val
>> 72982868 ecr 72940042], length 237
>> 0x:  4500 0121 e5b1 4000 4006 5623 7f00 0001  E..!..@.@.V#
>> 0x0010:  7f00 0001 9806 2352 746f a041 7507 6b35  ..#Rto.Au.k5
>> 0x0020:  8018 0212 ff15  0101 080a 0459 a154  .Y.T
>> 0x0030:  0458 fa0a *0400  0a00  e400 1005*  .X..
>> 0x0040:  6e06 c1fc 222c 813f 6228 61c5 7364 6500  n...",.?b(a.sde.
>> 0x0050:  0105 000b  0018 3532 6663 6330 3463  52fcc04c
>> 0x0060:  3464 6339 3033 6436 3664 3666 3866 3932  4dc903d66d6f8f92
>> 0x0070:   000a 3230 3136 2d30 362d 3133   2016-06-13..
>> 0x0080:  0008  0155 490c 3571  0002 0001  .UI.5q..
>> 0x0090:   0002 0002  0014 3131 3833 

Re: Data lost in Cassandra 3.5 single instance via Erlang driver

[Alain Rastoul]

On 15/06/2016 06:40, linbo liao wrote:

I am not sure, but looks it will cause the update other than insert. If
it is true, the only way is request includes IF NOT EXISTS, inform the
client it failed?

Thanks,
Linbo


Hi Linbo,

+1 with what Ben said, timestamp has a millisecond precision and is a 
bad choice for making PK unicity.
If your client and server are on the same physical machine (both on same 
computer or different vms on same hypervisor), insert duration can go 
down to very few microseconds (2~3 on a recent computer).

Your insert will/should often become "update".
The reason is that update does not exists in cassandra, neither delete, 
they are just "appends":  append with same key for update or append of a 
tombstone for delete.
You should try to use a timeuuid instead, it has a node, clock sequence, 
a counter plus the timestamp part that you can get with cql functions, 
and it exists for that use.

see here for the functions
https://docs.datastax.com/en/cql/3.3/cql/cql_reference/timeuuid_functions_r.html


--
best,
Alain

Re: Spark Memory Error - Not enough space to cache broadcast

[Gaurav Bhatnagar]

try setting the option --driver-memory 4G

On Tue, Jun 14, 2016 at 3:52 PM, Ben Slater 
wrote:

> A high level shot in the dark but in our testing we found Spark 1.6 a lot
> more reliable in low memory situations (presumably due to
> https://issues.apache.org/jira/browse/SPARK-1). If it’s an option,
> probably worth a try.
>
> Cheers
> Ben
>
> On Wed, 15 Jun 2016 at 08:48 Cassa L  wrote:
>
>> Hi,
>> I would appreciate any clue on this. It has become a bottleneck for our
>> spark job.
>>
>> On Mon, Jun 13, 2016 at 2:56 PM, Cassa L  wrote:
>>
>>> Hi,
>>>
>>> I'm using spark 1.5.1 version. I am reading data from Kafka into Spark and 
>>> writing it into Cassandra after processing it. Spark job starts fine and 
>>> runs all good for some time until I start getting below errors. Once these 
>>> errors come, job start to lag behind and I see that job has scheduling and 
>>> processing delays in streaming  UI.
>>>
>>> Worker memory is 6GB, executor-memory is 5GB, I also tried to tweak 
>>> memoryFraction parameters. Nothing works.
>>>
>>>
>>> 16/06/13 21:26:02 INFO MemoryStore: ensureFreeSpace(4044) called with 
>>> curMem=565394, maxMem=2778495713
>>> 16/06/13 21:26:02 INFO MemoryStore: Block broadcast_69652_piece0 stored as 
>>> bytes in memory (estimated size 3.9 KB, free 2.6 GB)
>>> 16/06/13 21:26:02 INFO TorrentBroadcast: Reading broadcast variable 69652 
>>> took 2 ms
>>> 16/06/13 21:26:02 WARN MemoryStore: Failed to reserve initial memory 
>>> threshold of 1024.0 KB for computing block broadcast_69652 in memory.
>>> 16/06/13 21:26:02 WARN MemoryStore: Not enough space to cache 
>>> broadcast_69652 in memory! (computed 496.0 B so far)
>>> 16/06/13 21:26:02 INFO MemoryStore: Memory use = 556.1 KB (blocks) + 2.6 GB 
>>> (scratch space shared across 0 tasks(s)) = 2.6 GB. Storage limit = 2.6 GB.
>>> 16/06/13 21:26:02 WARN MemoryStore: Persisting block broadcast_69652 to 
>>> disk instead.
>>> 16/06/13 21:26:02 INFO BlockManager: Found block rdd_100761_1 locally
>>> 16/06/13 21:26:02 INFO Executor: Finished task 0.0 in stage 71577.0 (TID 
>>> 452316). 2043 bytes result sent to driver
>>>
>>>
>>> Thanks,
>>>
>>> L
>>>
>>>
>> --
> 
> Ben Slater
> Chief Product Officer
> Instaclustr: Cassandra + Spark - Managed | Consulting | Support
> +61 437 929 798
>

Re: select query on entire primary key returning more than one row in result

[Bhuvan Rawal]

Joel,

Thanks for your reply, I have checked and found that the behavior is same
in case of CASSANDRA-11513
. I have verified
this behavior (for both 11513 & 12003) to occur in case of 3.4 & 3.5. They
both don't occur in 3.0.4, 3.6 & 3.7.

Please find below the results of selecting only pk and clustering key
from 11513.
It has also been verified that both issues occur while selecting all /
filtered rows therefore selection criteria is not an issue filtering by
WHERE is:

cqlsh:ks> select pk,a from test0 where pk=0 and a=2;

 pk | a
+---
  0 | 1
  0 | 2
  0 | 3

We can verify this claim by applying 11513 Patch to 3.5 Tag and build &
test for 12003. If it is fixed then we can guarantee the claim. Let me know
if any further input may possibly be required here.

On Wed, Jun 15, 2016 at 2:23 AM, Joel Knighton 
wrote:

> The important part of that query is that it's selecting a static column
> (with select *), not whether it is filtering on one. In CASSANDRA-12003 and
> this thread, it looks like you're only selecting the primary and clustering
> columns. I'd be cautious about concluding that CASSANDRA-12003 and
> CASSANDRA-11513 are the same issue and that CASSANDRA-12003 is fixed.
>
> If you have a reproduction path for CASSANDRA-12003, I'd recommend
> attaching it to a ticket, and someone can investigate internals to see if
> CASSANDRA-11513 (or something else entirely) fixed the issue.
>
> On Tue, Jun 14, 2016 at 2:13 PM, Bhuvan Rawal  wrote:
>
>> Joel,
>>
>> If we look at the schema carefully:
>>
>> CREATE TABLE test0 (
>> pk int,
>> a int,
>> b text,
>> s text static,
>> PRIMARY KEY (*pk, a)*
>> );
>>
>> and filtering is performed on clustering column a and its not a static
>> column:
>>
>> select * from test0 where pk=0 and a=2;
>>
>>
>>
>> On Wed, Jun 15, 2016 at 12:39 AM, Joel Knighton <
>> joel.knigh...@datastax.com> wrote:
>>
>>> It doesn't seem to be an exact duplicate - CASSANDRA-11513 relies on you
>>> selecting a static column, which you weren't doing in the reported issue.
>>> That said, I haven't looked too closely.
>>>
>>> On Tue, Jun 14, 2016 at 2:07 PM, Bhuvan Rawal 
>>> wrote:
>>>
 I can reproduce CASSANDRA-11513
  locally on
 3.5, possible duplicate.

 On Wed, Jun 15, 2016 at 12:29 AM, Joel Knighton <
 joel.knigh...@datastax.com> wrote:

> There's some precedent for similar issues with static columns in 3.5
> with https://issues.apache.org/jira/browse/CASSANDRA-11513 - a
> deterministic (or somewhat deterministic) path for reproduction would help
> narrow the issue down farther. I've played around locally with similar
> schemas (sans the stratio indices) and couldn't reproduce the issue.
>
> On Tue, Jun 14, 2016 at 1:41 PM, Bhuvan Rawal 
> wrote:
>
>> Jira CASSANDRA-12003
>>  Has been
>> created for the same.
>>
>> On Tue, Jun 14, 2016 at 11:54 PM, Atul Saroha <
>> atul.sar...@snapdeal.com> wrote:
>>
>>> Hi Tyler,
>>>
>>> This issue is mainly visible for tables having static columns, still
>>> investigating.
>>> We will try to test after removing lucene index but I don’t think
>>> this plug-in could led to change in behaviour of cassandra write to 
>>> table's
>>> memtable.
>>>
>>>
>>> -
>>> Atul Saroha
>>> *Lead Software Engineer*
>>> *M*: +91 8447784271 *T*: +91 124-415-6069 *EXT*: 12369
>>> Plot # 362, ASF Centre - Tower A, Udyog Vihar,
>>>  Phase -4, Sector 18, Gurgaon, Haryana 122016, INDIA
>>>
>>> On Tue, Jun 14, 2016 at 9:54 PM, Tyler Hobbs 
>>> wrote:
>>>
 Is 'id' your partition key? I'm not familiar with the stratio
 indexes, but it looks like the primary key columns are both indexed.
 Perhaps this is related?

 On Tue, Jun 14, 2016 at 1:25 AM, Atul Saroha <
 atul.sar...@snapdeal.com> wrote:

> After further debug, this issue is found in in-memory memtable as
> doing nodetool flush + compact resolve the issue. And there is no 
> batch
> write used for this table which is showing issue.
> Table properties:
>
> WITH CLUSTERING ORDER BY (f_name ASC)
>> AND bloom_filter_fp_chance = 0.01
>> AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
>> AND comment = ''
>> AND compaction = {'class':
>> 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy',
>> 'max_threshold': '32', 'min_threshold': '4'}
>> AND 

Re: Spark Memory Error - Not enough space to cache broadcast

[Cassa L]

Hi,
I would appreciate any clue on this. It has become a bottleneck for our
spark job.

On Mon, Jun 13, 2016 at 2:56 PM, Cassa L  wrote:

> Hi,
>
> I'm using spark 1.5.1 version. I am reading data from Kafka into Spark and 
> writing it into Cassandra after processing it. Spark job starts fine and runs 
> all good for some time until I start getting below errors. Once these errors 
> come, job start to lag behind and I see that job has scheduling and 
> processing delays in streaming  UI.
>
> Worker memory is 6GB, executor-memory is 5GB, I also tried to tweak 
> memoryFraction parameters. Nothing works.
>
>
> 16/06/13 21:26:02 INFO MemoryStore: ensureFreeSpace(4044) called with 
> curMem=565394, maxMem=2778495713
> 16/06/13 21:26:02 INFO MemoryStore: Block broadcast_69652_piece0 stored as 
> bytes in memory (estimated size 3.9 KB, free 2.6 GB)
> 16/06/13 21:26:02 INFO TorrentBroadcast: Reading broadcast variable 69652 
> took 2 ms
> 16/06/13 21:26:02 WARN MemoryStore: Failed to reserve initial memory 
> threshold of 1024.0 KB for computing block broadcast_69652 in memory.
> 16/06/13 21:26:02 WARN MemoryStore: Not enough space to cache broadcast_69652 
> in memory! (computed 496.0 B so far)
> 16/06/13 21:26:02 INFO MemoryStore: Memory use = 556.1 KB (blocks) + 2.6 GB 
> (scratch space shared across 0 tasks(s)) = 2.6 GB. Storage limit = 2.6 GB.
> 16/06/13 21:26:02 WARN MemoryStore: Persisting block broadcast_69652 to disk 
> instead.
> 16/06/13 21:26:02 INFO BlockManager: Found block rdd_100761_1 locally
> 16/06/13 21:26:02 INFO Executor: Finished task 0.0 in stage 71577.0 (TID 
> 452316). 2043 bytes result sent to driver
>
>
> Thanks,
>
> L
>
>

Re: select query on entire primary key returning more than one row in result

[Joel Knighton]

Great work, Bhuvan - I sat down after work to look at this more carefully.

For a short summary, you are correct.

For a longer summary, I initially thought the reproduction you provided
would not run into the issue from 3.4/3.5 because it didn't select any
static columns, which meant that it wouldn't have statics in its
ColumnFilter (basically, the filter we apply when deciding if we need to
look for the requested data in more SSTables). This was an incorrect
understanding - in order to preserve the CQL semantic (see CASSANDRA-6588
for details), we are including all columns, including the static columns,
in the fetched columns, which means they are part of the ColumnFilter. I
believe there may be an opportunity for an optimization here, but that's a
whole different discussion. I now agree that these are the same issue.

You are correct in your analysis that 3.4/3.5 are the only affected
versions. It has been patched in release 3.6 forward and was not introduced
until 3.4

Thanks for sticking with me on this - I'm going to resolve CASSANDRA-12003
as a duplicate of CASSANDRA-11513.

On Tue, Jun 14, 2016 at 4:21 PM, Bhuvan Rawal  wrote:

> Joel,
>
> Thanks for your reply, I have checked and found that the behavior is same
> in case of CASSANDRA-11513
> . I have verified
> this behavior (for both 11513 & 12003) to occur in case of 3.4 & 3.5. They
> both don't occur in 3.0.4, 3.6 & 3.7.
>
> Please find below the results of selecting only pk and clustering key from 
> 11513.
> It has also been verified that both issues occur while selecting all /
> filtered rows therefore selection criteria is not an issue filtering by
> WHERE is:
>
> cqlsh:ks> select pk,a from test0 where pk=0 and a=2;
>
>  pk | a
> +---
>   0 | 1
>   0 | 2
>   0 | 3
>
> We can verify this claim by applying 11513 Patch to 3.5 Tag and build &
> test for 12003. If it is fixed then we can guarantee the claim. Let me
> know if any further input may possibly be required here.
>
> On Wed, Jun 15, 2016 at 2:23 AM, Joel Knighton  > wrote:
>
>> The important part of that query is that it's selecting a static column
>> (with select *), not whether it is filtering on one. In CASSANDRA-12003 and
>> this thread, it looks like you're only selecting the primary and clustering
>> columns. I'd be cautious about concluding that CASSANDRA-12003 and
>> CASSANDRA-11513 are the same issue and that CASSANDRA-12003 is fixed.
>>
>> If you have a reproduction path for CASSANDRA-12003, I'd recommend
>> attaching it to a ticket, and someone can investigate internals to see if
>> CASSANDRA-11513 (or something else entirely) fixed the issue.
>>
>> On Tue, Jun 14, 2016 at 2:13 PM, Bhuvan Rawal 
>> wrote:
>>
>>> Joel,
>>>
>>> If we look at the schema carefully:
>>>
>>> CREATE TABLE test0 (
>>> pk int,
>>> a int,
>>> b text,
>>> s text static,
>>> PRIMARY KEY (*pk, a)*
>>> );
>>>
>>> and filtering is performed on clustering column a and its not a static
>>> column:
>>>
>>> select * from test0 where pk=0 and a=2;
>>>
>>>
>>>
>>> On Wed, Jun 15, 2016 at 12:39 AM, Joel Knighton <
>>> joel.knigh...@datastax.com> wrote:
>>>
 It doesn't seem to be an exact duplicate - CASSANDRA-11513 relies on
 you selecting a static column, which you weren't doing in the reported
 issue. That said, I haven't looked too closely.

 On Tue, Jun 14, 2016 at 2:07 PM, Bhuvan Rawal 
 wrote:

> I can reproduce CASSANDRA-11513
>  locally on
> 3.5, possible duplicate.
>
> On Wed, Jun 15, 2016 at 12:29 AM, Joel Knighton <
> joel.knigh...@datastax.com> wrote:
>
>> There's some precedent for similar issues with static columns in 3.5
>> with https://issues.apache.org/jira/browse/CASSANDRA-11513 - a
>> deterministic (or somewhat deterministic) path for reproduction would 
>> help
>> narrow the issue down farther. I've played around locally with similar
>> schemas (sans the stratio indices) and couldn't reproduce the issue.
>>
>> On Tue, Jun 14, 2016 at 1:41 PM, Bhuvan Rawal 
>> wrote:
>>
>>> Jira CASSANDRA-12003
>>>  Has been
>>> created for the same.
>>>
>>> On Tue, Jun 14, 2016 at 11:54 PM, Atul Saroha <
>>> atul.sar...@snapdeal.com> wrote:
>>>
 Hi Tyler,

 This issue is mainly visible for tables having static columns,
 still investigating.
 We will try to test after removing lucene index but I don’t think
 this plug-in could led to change in behaviour of cassandra write to 
 table's
 memtable.


 

Re: Spark Memory Error - Not enough space to cache broadcast

[Ben Slater]

A high level shot in the dark but in our testing we found Spark 1.6 a lot
more reliable in low memory situations (presumably due to
https://issues.apache.org/jira/browse/SPARK-1). If it’s an option,
probably worth a try.

Cheers
Ben

On Wed, 15 Jun 2016 at 08:48 Cassa L  wrote:

> Hi,
> I would appreciate any clue on this. It has become a bottleneck for our
> spark job.
>
> On Mon, Jun 13, 2016 at 2:56 PM, Cassa L  wrote:
>
>> Hi,
>>
>> I'm using spark 1.5.1 version. I am reading data from Kafka into Spark and 
>> writing it into Cassandra after processing it. Spark job starts fine and 
>> runs all good for some time until I start getting below errors. Once these 
>> errors come, job start to lag behind and I see that job has scheduling and 
>> processing delays in streaming  UI.
>>
>> Worker memory is 6GB, executor-memory is 5GB, I also tried to tweak 
>> memoryFraction parameters. Nothing works.
>>
>>
>> 16/06/13 21:26:02 INFO MemoryStore: ensureFreeSpace(4044) called with 
>> curMem=565394, maxMem=2778495713
>> 16/06/13 21:26:02 INFO MemoryStore: Block broadcast_69652_piece0 stored as 
>> bytes in memory (estimated size 3.9 KB, free 2.6 GB)
>> 16/06/13 21:26:02 INFO TorrentBroadcast: Reading broadcast variable 69652 
>> took 2 ms
>> 16/06/13 21:26:02 WARN MemoryStore: Failed to reserve initial memory 
>> threshold of 1024.0 KB for computing block broadcast_69652 in memory.
>> 16/06/13 21:26:02 WARN MemoryStore: Not enough space to cache 
>> broadcast_69652 in memory! (computed 496.0 B so far)
>> 16/06/13 21:26:02 INFO MemoryStore: Memory use = 556.1 KB (blocks) + 2.6 GB 
>> (scratch space shared across 0 tasks(s)) = 2.6 GB. Storage limit = 2.6 GB.
>> 16/06/13 21:26:02 WARN MemoryStore: Persisting block broadcast_69652 to disk 
>> instead.
>> 16/06/13 21:26:02 INFO BlockManager: Found block rdd_100761_1 locally
>> 16/06/13 21:26:02 INFO Executor: Finished task 0.0 in stage 71577.0 (TID 
>> 452316). 2043 bytes result sent to driver
>>
>>
>> Thanks,
>>
>> L
>>
>>
> --

Ben Slater
Chief Product Officer
Instaclustr: Cassandra + Spark - Managed | Consulting | Support
+61 437 929 798