Re: Showing a new property in DESCRIBE TABLE output

[Murukesh Mohanan]

Yes, that was it. The file to modify was cassandra/metadata.py in
cassandra-driver, which has a list of recognized table options (
https://github.com/datastax/python-driver/blob/master/cassandra/metadata.py#L2169).
Sorry for not posting the implementation, but I've followed the changes
made for adding the CDC option (
https://github.com/apache/cassandra/commit/e31e216234c6b57a531cae607e0355666007deb2).
Naturally, it didn't contain the changes made in another repository.

OT: I wonder if cassandra-driver can be added as a submodule to cassandra,
instead of embedding it as zip file. Since submodule updates are also part
of the commit history, that will make it easier to spot related changes in
the driver.

On Wed, 25 Jan 2017 at 07:56 Blake Eggleston  wrote:

I haven't seen your implementation, but the likely cause of your problem is
either that the new parameter isn't being sent over the client protocol, or
that cqlsh is ignoring it. The cqlsh output of DESCRIBE TABLE seems to be
generated by the TableMetadata class in the python driver (see the
as_cql_query method). Dropping a breakpoint in there would probably be a
good place to start.
On January 24, 2017 at 7:07:38 AM, Murukesh Mohanan (
murukesh.moha...@gmail.com) wrote:

I'm having a go at CASSANDRA-13002 (
https://issues.apache.org/jira/browse/CASSANDRA-12403), by adding a new
table property which will override the global slow_query_log_timeout_in_ms
setting. It works, but I can't get it to show up in cqlsh DESCRIBE TABLE
output. For example, this is what I get:

cqlsh> DESCRIBE TABLE foo.bar;

CREATE TABLE foo.bar (
id uuid PRIMARY KEY,
name text
) WITH bloom_filter_fp_chance = 0.01
AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
AND cdc = true
AND comment = ''
AND compaction = {'class':
'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy',
'max_threshold': '32', 'min_threshold': '4'}
AND compression = {'chunk_length_in_kb': '64', 'class':
'org.apache.cassandra.io.compress.LZ4Compressor'}
AND crc_check_chance = 1.0
AND dclocal_read_repair_chance = 0.1
AND default_time_to_live = 0
AND gc_grace_seconds = 864000
AND max_index_interval = 2048
AND memtable_flush_period_in_ms = 1001
AND min_index_interval = 128
AND read_repair_chance = 0.0
AND speculative_retry = '99PERCENTILE';

cqlsh> select table_name, slow_query_log_timeout_in_ms from
system_schema.tables where table_name = 'bar' allow filtering;

table_name | slow_query_log_timeout_in_ms
+--
bar | 103

The property (which is also called `slow_query_log_timeout_in_ms`) shows up
in the system_schema table.

It seems that the file to modify would be
src/java/org/apache/cassandra/db/ColumnFamilyStoreCQLHelper.java, but I
didn't have any luck modifying it.

Any pointers, please?



--

Murukesh Mohanan,
Yahoo! Japan

-- 

Murukesh Mohanan,
Yahoo! Japan

Re: Showing a new property in DESCRIBE TABLE output

[Blake Eggleston]

I haven't seen your implementation, but the likely cause of your problem is 
either that the new parameter isn't being sent over the client protocol, or 
that cqlsh is ignoring it. The cqlsh output of DESCRIBE TABLE seems to be 
generated by the TableMetadata class in the python driver (see the as_cql_query 
method). Dropping a breakpoint in there would probably be a good place to start.
On January 24, 2017 at 7:07:38 AM, Murukesh Mohanan 
(murukesh.moha...@gmail.com) wrote:

I'm having a go at CASSANDRA-13002 ( 
https://issues.apache.org/jira/browse/CASSANDRA-12403), by adding a new 
table property which will override the global slow_query_log_timeout_in_ms 
setting. It works, but I can't get it to show up in cqlsh DESCRIBE TABLE 
output. For example, this is what I get: 

cqlsh> DESCRIBE TABLE foo.bar; 

CREATE TABLE foo.bar ( 
id uuid PRIMARY KEY, 
name text 
) WITH bloom_filter_fp_chance = 0.01 
AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'} 
AND cdc = true 
AND comment = '' 
AND compaction = {'class': 
'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 
'max_threshold': '32', 'min_threshold': '4'} 
AND compression = {'chunk_length_in_kb': '64', 'class': 
'org.apache.cassandra.io.compress.LZ4Compressor'} 
AND crc_check_chance = 1.0 
AND dclocal_read_repair_chance = 0.1 
AND default_time_to_live = 0 
AND gc_grace_seconds = 864000 
AND max_index_interval = 2048 
AND memtable_flush_period_in_ms = 1001 
AND min_index_interval = 128 
AND read_repair_chance = 0.0 
AND speculative_retry = '99PERCENTILE'; 

cqlsh> select table_name, slow_query_log_timeout_in_ms from 
system_schema.tables where table_name = 'bar' allow filtering; 

table_name | slow_query_log_timeout_in_ms 
+-- 
bar | 103 

The property (which is also called `slow_query_log_timeout_in_ms`) shows up 
in the system_schema table. 

It seems that the file to modify would be 
src/java/org/apache/cassandra/db/ColumnFamilyStoreCQLHelper.java, but I 
didn't have any luck modifying it. 

Any pointers, please? 



-- 

Murukesh Mohanan, 
Yahoo! Japan 

Re: Dropped messages on random nodes.

[Dikang Gu]

Thanks guys! Jeff Jirsa helped me take a look, and I found a 10sec young gc
pause in the GC log.

3071128K->282000K(3495296K), 0.1144648 secs]
25943529K->23186623K(66409856K), 9.8971781 secs] [Times: user=2.33
sys=0.00, real=9.89 secs]

I'm trying to get a histogram or heap dump.


Thanks!



On Mon, Jan 23, 2017 at 7:08 PM, Brandon Williams  wrote:

> The lion's share of your drops are from cross-node timeouts, which require
> clock synchronization, so check that first.  If your clocks are synced,
> that means not only are you showing eager dropping based on time, but
> despite the eager dropping you are still facing overload.
>
> That local, non-gc pause is also troubling. (I assume non-gc since there
> wasn't anything logged by the GC inspector.)
>
> On Mon, Jan 23, 2017 at 12:36 AM, Dikang Gu  wrote:
>
> > Hello there,
> >
> > We have a 100 nodes ish cluster, I find that there are dropped messages
> on
> > random nodes in the cluster, which caused error spikes and P99 latency
> > spikes as well.
> >
> > I tried to figure out the cause. I do not see any obvious bottleneck in
> > the cluster, the C* nodes still have plenty of cpu idle/disk io. But I do
> > see some suspicious gossip events around that time, not sure if it's
> > related.
> >
> > 2017-01-21_16:43:56.71033 WARN  16:43:56 [GossipTasks:1]: Not marking
> > nodes down due to local pause of 13079498815 > 50
> > 2017-01-21_16:43:56.85532 INFO  16:43:56 [ScheduledTasks:1]: MUTATION
> > messages were dropped in last 5000 ms: 65 for internal timeout and 10895
> > for cross node timeout
> > 2017-01-21_16:43:56.85533 INFO  16:43:56 [ScheduledTasks:1]: READ
> messages
> > were dropped in last 5000 ms: 33 for internal timeout and 7867 for cross
> > node timeout
> > 2017-01-21_16:43:56.85534 INFO  16:43:56 [ScheduledTasks:1]: Pool Name
> >Active   Pending  Completed   Blocked  All Time
> Blocked
> > 2017-01-21_16:43:56.85534 INFO  16:43:56 [ScheduledTasks:1]:
> MutationStage
> >   128 47794 1015525068 0
>  0
> > 2017-01-21_16:43:56.85535
> > 2017-01-21_16:43:56.85535 INFO  16:43:56 [ScheduledTasks:1]: ReadStage
> >64 20202  450508940 0
>  0
> >
> > Any suggestions?
> >
> > Thanks!
> >
> > --
> > Dikang
> >
> >
>



-- 
Dikang

Any idea when 3.0.11 will be released?

[Johnny Miller]

Just wondering when 3.0.11 will be released?

Thanks,

Johnny

Showing a new property in DESCRIBE TABLE output

[Murukesh Mohanan]

I'm having a go at CASSANDRA-13002 (
https://issues.apache.org/jira/browse/CASSANDRA-12403), by adding a new
table property which will override the global slow_query_log_timeout_in_ms
setting. It works, but I can't get it to show up in cqlsh DESCRIBE TABLE
output. For example, this is what I get:

cqlsh> DESCRIBE TABLE foo.bar;

CREATE TABLE foo.bar (
id uuid PRIMARY KEY,
name text
) WITH bloom_filter_fp_chance = 0.01
AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
AND cdc = true
AND comment = ''
AND compaction = {'class':
'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy',
'max_threshold': '32', 'min_threshold': '4'}
AND compression = {'chunk_length_in_kb': '64', 'class':
'org.apache.cassandra.io.compress.LZ4Compressor'}
AND crc_check_chance = 1.0
AND dclocal_read_repair_chance = 0.1
AND default_time_to_live = 0
AND gc_grace_seconds = 864000
AND max_index_interval = 2048
AND memtable_flush_period_in_ms = 1001
AND min_index_interval = 128
AND read_repair_chance = 0.0
AND speculative_retry = '99PERCENTILE';

cqlsh> select table_name, slow_query_log_timeout_in_ms from
system_schema.tables  where table_name = 'bar' allow filtering;

 table_name | slow_query_log_timeout_in_ms
+--
bar |  103

The property (which is also called `slow_query_log_timeout_in_ms`) shows up
in the system_schema table.

It seems that the file to modify would be
src/java/org/apache/cassandra/db/ColumnFamilyStoreCQLHelper.java, but I
didn't have any luck modifying it.

Any pointers, please?



-- 

Murukesh Mohanan,
Yahoo! Japan