[jira] [Comment Edited] (CASSANDRA-14908) Deadlock occurs when executing a file selection in levelcompact
[ https://issues.apache.org/jira/browse/CASSANDRA-14908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16704580#comment-16704580 ] wu taiyin edited comment on CASSANDRA-14908 at 1/12/19 6:24 AM: Thank you for you reply. and the environment as follows: CPU：48C memory：385G disk and data size per disk: /dev/sda3 754G 9.2G 706G 2% /opt/sds /dev/sdb 1.5T 223G 1.3T 15% /disk/data1 /dev/sdc 1.5T 223G 1.3T 15% /disk/data2 /dev/sdd 1.5T 213G 1.3T 15% /disk/data3 /dev/sde 1.5T 216G 1.3T 15% /disk/data4 /dev/sdf 1.5T 214G 1.3T 15% /disk/data5 Cassandra.yaml : data_file_directories: - /disk/data1/cassandra - /disk/data2/cassandra - /disk/data3/cassandra - /disk/data4/cassandra - /disk/data5/cassandra commitlog_directory: /opt/sds/logs/cassandra/commitlog ./nodetool cfstats app99 Keyspace: app99 Read Count: 2789855752 Read Latency: 0.6304968936945955 ms. Write Count: 1490782843 Write Latency: 0.03916580592616869 ms. Pending Flushes: 0 Table: dn SSTable count: 1 SSTables in each level: [1, 0, 0, 0, 0, 0, 0, 0, 0] Space used (live): 6092 Space used (total): 6092 Space used by snapshots (total): 0 Off heap memory used (total): 127 SSTable Compression Ratio: 0.14210985178727115 Number of keys (estimate): 11 Memtable cell count: 483 Memtable data size: 2352 Memtable off heap memory used: 0 Memtable switch count: 2 Local read count: 10133694 Local read latency: 0.136 ms Local write count: 270 Local write latency: 0.135 ms Pending flushes: 0 Bloom filter false positives: 0 Bloom filter false ratio: 0.0 Bloom filter space used: 96 Bloom filter off heap memory used: 88 Index summary off heap memory used: 31 Compression metadata off heap memory used: 8 Compacted partition minimum bytes: 125 Compacted partition maximum bytes: 1916 Compacted partition mean bytes: 974 Average live cells per slice (last five minutes): 1.0 Maximum live cells per slice (last five minutes): 1.0 Average tombstones per slice (last five minutes): 0.0 Maximum tombstones per slice (last five minutes): 0.0 Table: file SSTable count: 4295 SSTables in each level: [0, 10, 100, 1016/1000, 3169, 0, 0, 0, 0] Space used (live): 1166100476265 Space used (total): 1169018887208 Space used by snapshots (total): 0 Off heap memory used (total): 9184778389 SSTable Compression Ratio: 0.38421502411531816 Number of keys (estimate): 8849691024 Memtable cell count: 8238732 Memtable data size: 301883508 Memtable off heap memory used: 0 Memtable switch count: 521 Local read count: 2779722909 Local read latency: 0.633 ms Local write count: 1490783015 Local write latency: 0.040 ms Pending flushes: 0 Bloom filter false positives: 5092 Bloom filter false ratio: 0.0 Bloom filter space used: 6387737864 Bloom filter off heap memory used: 6387703504 Index summary off heap memory used: 2575905581 Compression metadata off heap memory used: 221169304 Compacted partition minimum bytes: 43 Compacted partition maximum bytes: 88148 Compacted partition mean bytes: 215 Average live cells per slice (last five minutes): 0.4247412120152055 Maximum live cells per slice (last five minutes): 1.0 Average tombstones per slice (last five minutes): 0.0 Maximum tombstones per slice (last five minutes): 0.0 was (Author: dockerwu): Thank you for you reply. and the environment as follows: CPU：48C memory：385G disk and data size per disk: /dev/sda3 754G 9.2G 706G 2% /opt/huawei /dev/sdb 1.5T 223G 1.3T 15% /disk/data1 /dev/sdc 1.5T 223G 1.3T 15% /disk/data2 /dev/sdd 1.5T 213G 1.3T 15% /disk/data3 /dev/sde 1.5T 216G 1.3T 15% /disk/data4 /dev/sdf 1.5T 214G 1.3T 15% /disk/data5 Cassandra.yaml : data_file_directories: - /disk/data1/cassandra - /disk/data2/cassandra - /disk/data3/cassandra - /disk/data4/cassandra - /disk/data5/cassandra commitlog_directory: /opt/huawei/logs/cassandra/commitlog ./nodetool cfstats app99 Keyspace: app99 Read Count: 2789855752 Read Latency: 0.6304968936945955 ms. Write Count: 1490782843 Write Latency: 0.03916580592616869 ms. Pending Flushes: 0 Table: dn SSTable count: 1 SSTables in each level: [1, 0, 0, 0, 0, 0, 0, 0, 0] Space used (live): 6092 Space used (total): 6092 Space used by snapshots (total): 0 Off heap memory used (total): 127 SSTable Compression Ratio: 0.14210985178727115 Number of keys (estimate): 11 Memtable cell count: 483 Memtable data size: 2352 Memtable off heap memory used: 0 Memtable switch count: 2 Local read count: 10133694 Local read latency: 0.136 ms Local write count: 270 Local write latency: 0.135 ms Pending flushes: 0 Bloom filter false positives: 0 Bloom filter false ratio: 0.0 Bloom filter space used: 96 Bloom filter off heap memory used: 88 Index summary off heap memory used: 31 Compression metadata off heap memory used: 8 Compacted partition minimum bytes: 125 Compacted partition maxim
[jira] [Commented] (CASSANDRA-14526) dtest to validate Cassandra state post failed/successful bootstrap
[
https://issues.apache.org/jira/browse/CASSANDRA-14526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740779#comment-16740779
]
Jaydeepkumar Chovatia commented on CASSANDRA-14526:
---
[~jay.zhuang] I was able to reproduce when I ran the test with count=10. I
think I have found a problem and fix for it. With this fix I retried with
{{count=50}} and it seems test is passing now.
{code:java}
python3 -m pytest --count=50 --cassandra-dir=~/cassandra/
secondary_indexes_test.py::TestPreJoinCallback::test_resume
...
...
==
50 passed in 4221.96 seconds
===
{code}
Following change I've made:
{code:java}
diff --git a/secondary_indexes_test.py b/secondary_indexes_test.py
index 9a9346d9..044c583d 100644
--- a/secondary_indexes_test.py
+++ b/secondary_indexes_test.py
@@ -1205,6 +1205,7 @@ class TestPreJoinCallback(Tester):
node2.set_configuration_options(values=yaml_opts)
node2.start(wait_other_notice=True, wait_for_binary_proto=False)
+node2.watch_log_for('Some data streaming failed. Use nodetool to
check bootstrap state and resume.')
node2.nodetool("bootstrap resume")
{code}
I've updated the code, can you please retry?
[https://github.com/apache/cassandra-dtest/compare/master...jaydeepkumar1984:14526-trunk]
> dtest to validate Cassandra state post failed/successful bootstrap
> --
>
> Key: CASSANDRA-14526
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14526
> Project: Cassandra
> Issue Type: Sub-task
> Components: Test/dtest
>Reporter: Jaydeepkumar Chovatia
>Assignee: Jaydeepkumar Chovatia
>Priority: Major
> Labels: dtest
>
> Please find dtest here:
> || dtest ||
> | [patch
> |https://github.com/apache/cassandra-dtest/compare/master...jaydeepkumar1984:14526-trunk]|
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14975) cqlsh dtests are not running in CircleCI
[ https://issues.apache.org/jira/browse/CASSANDRA-14975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740735#comment-16740735 ] Ariel Weisberg commented on CASSANDRA-14975: Those are not the same tests. These are the dtests. I have them mostly working it's not that bad. Just annoying because they are broken in a lot of little different ways. Dinesh is working on the tests in the Cassandra repo in CASSANDRA-14976. I'm not sure you need to port it to python 3. Dinesh says the tests don't invoke cqlsh directly, but we'll see. > cqlsh dtests are not running in CircleCI > > > Key: CASSANDRA-14975 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14975 > Project: Cassandra > Issue Type: Test > Components: Test/dtest >Reporter: Ariel Weisberg >Assignee: Ariel Weisberg >Priority: Major > > Right now the dtests skip these tests because most don't pass. Originally > they weren't being collected because the test files end in_tests.py instead > of _test.py, but I fixed that by adding _tests.py to the list of recognized > patterns. > Get them all running in CircleCI. Nothing special they will be part of the > existing dtest jobs. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-14958) Counters fail to increment in 2.1/2.2 to 3.X mixed version clusters
[
https://issues.apache.org/jira/browse/CASSANDRA-14958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740648#comment-16740648
]
Aleksey Yeschenko edited comment on CASSANDRA-14958 at 1/11/19 6:40 PM:
Code: [3.0|https://github.com/iamaleksey/cassandra/commits/14958-3.0],
[3.11|https://github.com/iamaleksey/cassandra/commits/14958-3.11].
CI:
[3.0|https://circleci.com/workflow-run/ad96e947-dff3-4d63-9c34-3e5220e550e4],
[3.11|https://circleci.com/workflow-run/8378d5c3-a517-486f-8df0-b8d36c05bd79].
To fix another compatibility issue, CASSANDRA-13691 switched from using local
shards to stash counter update values to a special sentinel id.
{{LegacyLayout}} code was in one place unfortunately not updated to reflect the
change, breaking counter update cell serialisation to legacy nodes, always
sending 0-increments.
The linked branches address the issue. There is no new test as this scenario is
already covered pretty well by the (now working) upgrade tests.
was (Author: iamaleksey):
Code: [3.0|https://github.com/iamaleksey/cassandra/commits/14958-3.0],
[3.11|https://github.com/iamaleksey/cassandra/commits/14958-3.11].
CI:
[3.0|https://circleci.com/workflow-run/ad96e947-dff3-4d63-9c34-3e5220e550e4],
[3.11|https://circleci.com/workflow-run/8378d5c3-a517-486f-8df0-b8d36c05bd79].
To fix another compatibility issue, CASSANDRA-13691 switched from using local
shards to stash counter update values to a special sentinel id.
{{LegacyLayout}} code was one place unfortunately not updated to reflect the
change, breaking counter update cell serialisation to legacy nodes, always
sending 0-increments.
The linked branches address the issue. There is no new test as this scenario is
already covered pretty well by the (now working) upgrade tests.
> Counters fail to increment in 2.1/2.2 to 3.X mixed version clusters
> ---
>
> Key: CASSANDRA-14958
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14958
> Project: Cassandra
> Issue Type: Bug
> Components: Feature/Counters
>Reporter: Ariel Weisberg
>Assignee: Aleksey Yeschenko
>Priority: Critical
> Fix For: 3.0.x, 3.11.x
>
>
> The upgrade test for this is failing
> https://circleci.com/gh/aweisberg/cassandra/2362#tests/containers/1
> I confirmed that this is occurring manually using cqlsh against the cluster
> constructed by the dtest.
> {noformat}
> cqlsh> describe schema;
> CREATE KEYSPACE ks WITH replication = {'class': 'SimpleStrategy',
> 'replication_factor': '1'} AND durable_writes = true;
> CREATE TABLE ks.clicks (
> userid int,
> url text,
> total counter,
> PRIMARY KEY (userid, url)
> ) WITH COMPACT STORAGE
> AND CLUSTERING ORDER BY (url ASC)
> AND bloom_filter_fp_chance = 0.01
> AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
> AND comment = ''
> AND compaction = {'class':
> 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy',
> 'max_threshold': '32', 'min_threshold': '4'}
> AND compression = {'chunk_length_in_kb': '64', 'class':
> 'org.apache.cassandra.io.compress.LZ4Compressor'}
> AND crc_check_chance = 1.0
> AND dclocal_read_repair_chance = 0.1
> AND default_time_to_live = 0
> AND gc_grace_seconds = 864000
> AND max_index_interval = 2048
> AND memtable_flush_period_in_ms = 0
> AND min_index_interval = 128
> AND read_repair_chance = 0.0
> AND speculative_retry = '99PERCENTILE';
> cqlsh> use ks;
> cqlsh:ks> UPDATE clicks SET total = total + 1 WHERE userid = 1 AND url =
> 'http://foo.com';
> cqlsh:ks> SELECT total FROM clicks WHERE userid = 1 AND url = 'http://foo.com'
> ... ;
> total
> ---
> 0
> (1 rows)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14958) Counters fail to increment in 2.1/2.2 to 3.X mixed version clusters
[
https://issues.apache.org/jira/browse/CASSANDRA-14958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740648#comment-16740648
]
Aleksey Yeschenko commented on CASSANDRA-14958:
---
Code: [3.0|https://github.com/iamaleksey/cassandra/commits/14958-3.0],
[3.11|https://github.com/iamaleksey/cassandra/commits/14958-3.11].
CI:
[3.0|https://circleci.com/workflow-run/ad96e947-dff3-4d63-9c34-3e5220e550e4],
[3.11|https://circleci.com/workflow-run/8378d5c3-a517-486f-8df0-b8d36c05bd79].
To fix another compatibility issue, CASSANDRA-13691 switched from using local
shards to stash counter update values to a special sentinel id.
{{LegacyLayout}} code was one place unfortunately not updated to reflect the
change, breaking counter update cell serialisation to legacy nodes, always
sending 0-increments.
The linked branches address the issue. There is no new test as this scenario is
already covered pretty well by the (now working) upgrade tests.
> Counters fail to increment in 2.1/2.2 to 3.X mixed version clusters
> ---
>
> Key: CASSANDRA-14958
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14958
> Project: Cassandra
> Issue Type: Bug
> Components: Feature/Counters
>Reporter: Ariel Weisberg
>Assignee: Aleksey Yeschenko
>Priority: Critical
> Fix For: 3.0.x, 3.11.x
>
>
> The upgrade test for this is failing
> https://circleci.com/gh/aweisberg/cassandra/2362#tests/containers/1
> I confirmed that this is occurring manually using cqlsh against the cluster
> constructed by the dtest.
> {noformat}
> cqlsh> describe schema;
> CREATE KEYSPACE ks WITH replication = {'class': 'SimpleStrategy',
> 'replication_factor': '1'} AND durable_writes = true;
> CREATE TABLE ks.clicks (
> userid int,
> url text,
> total counter,
> PRIMARY KEY (userid, url)
> ) WITH COMPACT STORAGE
> AND CLUSTERING ORDER BY (url ASC)
> AND bloom_filter_fp_chance = 0.01
> AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
> AND comment = ''
> AND compaction = {'class':
> 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy',
> 'max_threshold': '32', 'min_threshold': '4'}
> AND compression = {'chunk_length_in_kb': '64', 'class':
> 'org.apache.cassandra.io.compress.LZ4Compressor'}
> AND crc_check_chance = 1.0
> AND dclocal_read_repair_chance = 0.1
> AND default_time_to_live = 0
> AND gc_grace_seconds = 864000
> AND max_index_interval = 2048
> AND memtable_flush_period_in_ms = 0
> AND min_index_interval = 128
> AND read_repair_chance = 0.0
> AND speculative_retry = '99PERCENTILE';
> cqlsh> use ks;
> cqlsh:ks> UPDATE clicks SET total = total + 1 WHERE userid = 1 AND url =
> 'http://foo.com';
> cqlsh:ks> SELECT total FROM clicks WHERE userid = 1 AND url = 'http://foo.com'
> ... ;
> total
> ---
> 0
> (1 rows)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14958) Counters fail to increment in 2.1/2.2 to 3.X mixed version clusters
[
https://issues.apache.org/jira/browse/CASSANDRA-14958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aleksey Yeschenko updated CASSANDRA-14958:
--
Reviewer: Benedict
Status: Patch Available (was: In Progress)
> Counters fail to increment in 2.1/2.2 to 3.X mixed version clusters
> ---
>
> Key: CASSANDRA-14958
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14958
> Project: Cassandra
> Issue Type: Bug
> Components: Feature/Counters
>Reporter: Ariel Weisberg
>Assignee: Aleksey Yeschenko
>Priority: Critical
> Fix For: 3.0.x, 3.11.x
>
>
> The upgrade test for this is failing
> https://circleci.com/gh/aweisberg/cassandra/2362#tests/containers/1
> I confirmed that this is occurring manually using cqlsh against the cluster
> constructed by the dtest.
> {noformat}
> cqlsh> describe schema;
> CREATE KEYSPACE ks WITH replication = {'class': 'SimpleStrategy',
> 'replication_factor': '1'} AND durable_writes = true;
> CREATE TABLE ks.clicks (
> userid int,
> url text,
> total counter,
> PRIMARY KEY (userid, url)
> ) WITH COMPACT STORAGE
> AND CLUSTERING ORDER BY (url ASC)
> AND bloom_filter_fp_chance = 0.01
> AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
> AND comment = ''
> AND compaction = {'class':
> 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy',
> 'max_threshold': '32', 'min_threshold': '4'}
> AND compression = {'chunk_length_in_kb': '64', 'class':
> 'org.apache.cassandra.io.compress.LZ4Compressor'}
> AND crc_check_chance = 1.0
> AND dclocal_read_repair_chance = 0.1
> AND default_time_to_live = 0
> AND gc_grace_seconds = 864000
> AND max_index_interval = 2048
> AND memtable_flush_period_in_ms = 0
> AND min_index_interval = 128
> AND read_repair_chance = 0.0
> AND speculative_retry = '99PERCENTILE';
> cqlsh> use ks;
> cqlsh:ks> UPDATE clicks SET total = total + 1 WHERE userid = 1 AND url =
> 'http://foo.com';
> cqlsh:ks> SELECT total FROM clicks WHERE userid = 1 AND url = 'http://foo.com'
> ... ;
> total
> ---
> 0
> (1 rows)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14975) cqlsh dtests are not running in CircleCI
[ https://issues.apache.org/jira/browse/CASSANDRA-14975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740632#comment-16740632 ] Stefan Podkowinski commented on CASSANDRA-14975: There's been progress on that already in CASSANDRA-14298 It's not as trivial. Also some tests won't run at all, before porting cqlsh/cqlshlib to Python3 IIRC. > cqlsh dtests are not running in CircleCI > > > Key: CASSANDRA-14975 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14975 > Project: Cassandra > Issue Type: Test > Components: Test/dtest >Reporter: Ariel Weisberg >Assignee: Ariel Weisberg >Priority: Major > > Right now the dtests skip these tests because most don't pass. Originally > they weren't being collected because the test files end in_tests.py instead > of _test.py, but I fixed that by adding _tests.py to the list of recognized > patterns. > Get them all running in CircleCI. Nothing special they will be part of the > existing dtest jobs. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14958) Counters fail to increment in 2.1/2.2 to 3.X mixed version clusters
[
https://issues.apache.org/jira/browse/CASSANDRA-14958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aleksey Yeschenko updated CASSANDRA-14958:
--
Fix Version/s: (was: 2.0.x)
3.11.x
> Counters fail to increment in 2.1/2.2 to 3.X mixed version clusters
> ---
>
> Key: CASSANDRA-14958
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14958
> Project: Cassandra
> Issue Type: Bug
> Components: Feature/Counters
>Reporter: Ariel Weisberg
>Assignee: Aleksey Yeschenko
>Priority: Critical
> Fix For: 3.0.x, 3.11.x
>
>
> The upgrade test for this is failing
> https://circleci.com/gh/aweisberg/cassandra/2362#tests/containers/1
> I confirmed that this is occurring manually using cqlsh against the cluster
> constructed by the dtest.
> {noformat}
> cqlsh> describe schema;
> CREATE KEYSPACE ks WITH replication = {'class': 'SimpleStrategy',
> 'replication_factor': '1'} AND durable_writes = true;
> CREATE TABLE ks.clicks (
> userid int,
> url text,
> total counter,
> PRIMARY KEY (userid, url)
> ) WITH COMPACT STORAGE
> AND CLUSTERING ORDER BY (url ASC)
> AND bloom_filter_fp_chance = 0.01
> AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
> AND comment = ''
> AND compaction = {'class':
> 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy',
> 'max_threshold': '32', 'min_threshold': '4'}
> AND compression = {'chunk_length_in_kb': '64', 'class':
> 'org.apache.cassandra.io.compress.LZ4Compressor'}
> AND crc_check_chance = 1.0
> AND dclocal_read_repair_chance = 0.1
> AND default_time_to_live = 0
> AND gc_grace_seconds = 864000
> AND max_index_interval = 2048
> AND memtable_flush_period_in_ms = 0
> AND min_index_interval = 128
> AND read_repair_chance = 0.0
> AND speculative_retry = '99PERCENTILE';
> cqlsh> use ks;
> cqlsh:ks> UPDATE clicks SET total = total + 1 WHERE userid = 1 AND url =
> 'http://foo.com';
> cqlsh:ks> SELECT total FROM clicks WHERE userid = 1 AND url = 'http://foo.com'
> ... ;
> total
> ---
> 0
> (1 rows)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14958) Counters fail to increment in 2.1/2.2 to 3.X mixed version clusters
[
https://issues.apache.org/jira/browse/CASSANDRA-14958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aleksey Yeschenko updated CASSANDRA-14958:
--
Summary: Counters fail to increment in 2.1/2.2 to 3.X mixed version
clusters (was: Counters fail to increment on 2.1/2.2 to 3.X mixed version
clusters)
> Counters fail to increment in 2.1/2.2 to 3.X mixed version clusters
> ---
>
> Key: CASSANDRA-14958
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14958
> Project: Cassandra
> Issue Type: Bug
> Components: Feature/Counters
>Reporter: Ariel Weisberg
>Assignee: Aleksey Yeschenko
>Priority: Critical
> Fix For: 3.0.x, 2.0.x
>
>
> The upgrade test for this is failing
> https://circleci.com/gh/aweisberg/cassandra/2362#tests/containers/1
> I confirmed that this is occurring manually using cqlsh against the cluster
> constructed by the dtest.
> {noformat}
> cqlsh> describe schema;
> CREATE KEYSPACE ks WITH replication = {'class': 'SimpleStrategy',
> 'replication_factor': '1'} AND durable_writes = true;
> CREATE TABLE ks.clicks (
> userid int,
> url text,
> total counter,
> PRIMARY KEY (userid, url)
> ) WITH COMPACT STORAGE
> AND CLUSTERING ORDER BY (url ASC)
> AND bloom_filter_fp_chance = 0.01
> AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
> AND comment = ''
> AND compaction = {'class':
> 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy',
> 'max_threshold': '32', 'min_threshold': '4'}
> AND compression = {'chunk_length_in_kb': '64', 'class':
> 'org.apache.cassandra.io.compress.LZ4Compressor'}
> AND crc_check_chance = 1.0
> AND dclocal_read_repair_chance = 0.1
> AND default_time_to_live = 0
> AND gc_grace_seconds = 864000
> AND max_index_interval = 2048
> AND memtable_flush_period_in_ms = 0
> AND min_index_interval = 128
> AND read_repair_chance = 0.0
> AND speculative_retry = '99PERCENTILE';
> cqlsh> use ks;
> cqlsh:ks> UPDATE clicks SET total = total + 1 WHERE userid = 1 AND url =
> 'http://foo.com';
> cqlsh:ks> SELECT total FROM clicks WHERE userid = 1 AND url = 'http://foo.com'
> ... ;
> total
> ---
> 0
> (1 rows)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-14977) Make it possible to run dtests with transient replication in as many cases as possible
Ariel Weisberg created CASSANDRA-14977: -- Summary: Make it possible to run dtests with transient replication in as many cases as possible Key: CASSANDRA-14977 URL: https://issues.apache.org/jira/browse/CASSANDRA-14977 Project: Cassandra Issue Type: Test Components: Feature/Transient Replication, Test/dtest Reporter: Ariel Weisberg Assignee: Ariel Weisberg Fix For: 4.0 The dtests already test the database pretty extensively. It would be nice if you could set a flag and the tests would automatically modify replication to use transient replication as much as possible so the existing tests could also validate transient replication. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-14976) cqlsh tests in cassandra repo aren't running/runnable
Ariel Weisberg created CASSANDRA-14976: -- Summary: cqlsh tests in cassandra repo aren't running/runnable Key: CASSANDRA-14976 URL: https://issues.apache.org/jira/browse/CASSANDRA-14976 Project: Cassandra Issue Type: Test Components: Test/dtest Reporter: Ariel Weisberg Assignee: Dinesh Joshi These are still python 2/nosetests. I think we should upgrade them to python 3/pytest and move them to the dtest repo. No one is running them in the Cassandra repo. There aren't even instructions. People do run the dtests so we should put them there and use the regular dtest conventions. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-14975) cqlsh dtests are not running in CircleCI
Ariel Weisberg created CASSANDRA-14975: -- Summary: cqlsh dtests are not running in CircleCI Key: CASSANDRA-14975 URL: https://issues.apache.org/jira/browse/CASSANDRA-14975 Project: Cassandra Issue Type: Test Components: Test/dtest Reporter: Ariel Weisberg Assignee: Ariel Weisberg Right now the dtests skip these tests because most don't pass. Originally they weren't being collected because the test files end in_tests.py instead of _test.py, but I fixed that by adding _tests.py to the list of recognized patterns. Get them all running in CircleCI. Nothing special they will be part of the existing dtest jobs. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14958) Counters fail to increment on 2.1/2.2 to 3.X mixed version clusters
[
https://issues.apache.org/jira/browse/CASSANDRA-14958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740541#comment-16740541
]
Aleksey Yeschenko commented on CASSANDRA-14958:
---
Unfortunately there is indeed an issue.
In a mixed-version 2.1 + 3.0 cluster, when 3.0 coordinates a counter update
request, and chooses to forward the counter mutation to a different leader,
such update would be lost.
For that to happen you need to have a mixed mode cluster with 3.0 node both
coordinating and opting to *not* be a leader itself, which is actually a rather
likely scenario. I'm (very) surprised that this upgrade bug hasn't been noticed
before (before CASSANDRA-14881 that is).
> Counters fail to increment on 2.1/2.2 to 3.X mixed version clusters
> ---
>
> Key: CASSANDRA-14958
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14958
> Project: Cassandra
> Issue Type: Bug
> Components: Feature/Counters
>Reporter: Ariel Weisberg
>Assignee: Aleksey Yeschenko
>Priority: Critical
> Fix For: 3.0.x, 2.0.x
>
>
> The upgrade test for this is failing
> https://circleci.com/gh/aweisberg/cassandra/2362#tests/containers/1
> I confirmed that this is occurring manually using cqlsh against the cluster
> constructed by the dtest.
> {noformat}
> cqlsh> describe schema;
> CREATE KEYSPACE ks WITH replication = {'class': 'SimpleStrategy',
> 'replication_factor': '1'} AND durable_writes = true;
> CREATE TABLE ks.clicks (
> userid int,
> url text,
> total counter,
> PRIMARY KEY (userid, url)
> ) WITH COMPACT STORAGE
> AND CLUSTERING ORDER BY (url ASC)
> AND bloom_filter_fp_chance = 0.01
> AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
> AND comment = ''
> AND compaction = {'class':
> 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy',
> 'max_threshold': '32', 'min_threshold': '4'}
> AND compression = {'chunk_length_in_kb': '64', 'class':
> 'org.apache.cassandra.io.compress.LZ4Compressor'}
> AND crc_check_chance = 1.0
> AND dclocal_read_repair_chance = 0.1
> AND default_time_to_live = 0
> AND gc_grace_seconds = 864000
> AND max_index_interval = 2048
> AND memtable_flush_period_in_ms = 0
> AND min_index_interval = 128
> AND read_repair_chance = 0.0
> AND speculative_retry = '99PERCENTILE';
> cqlsh> use ks;
> cqlsh:ks> UPDATE clicks SET total = total + 1 WHERE userid = 1 AND url =
> 'http://foo.com';
> cqlsh:ks> SELECT total FROM clicks WHERE userid = 1 AND url = 'http://foo.com'
> ... ;
> total
> ---
> 0
> (1 rows)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-14958) Counters fail to increment on 2.1/2.2 to 3.X mixed version clusters
[
https://issues.apache.org/jira/browse/CASSANDRA-14958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740541#comment-16740541
]
Aleksey Yeschenko edited comment on CASSANDRA-14958 at 1/11/19 4:33 PM:
Unfortunately there is indeed an issue.
In a mixed-version 2.1 + 3.0 cluster, when 3.0 coordinates a counter update
request, and chooses to forward the counter mutation to a different leader,
such an update would be lost.
For that to happen you need to have a mixed mode cluster with 3.0 node both
coordinating and opting to *not* be a leader itself, which is actually a rather
likely scenario. I'm (very) surprised that this upgrade bug hasn't been noticed
before (before CASSANDRA-14881 that is).
was (Author: iamaleksey):
Unfortunately there is indeed an issue.
In a mixed-version 2.1 + 3.0 cluster, when 3.0 coordinates a counter update
request, and chooses to forward the counter mutation to a different leader,
such update would be lost.
For that to happen you need to have a mixed mode cluster with 3.0 node both
coordinating and opting to *not* be a leader itself, which is actually a rather
likely scenario. I'm (very) surprised that this upgrade bug hasn't been noticed
before (before CASSANDRA-14881 that is).
> Counters fail to increment on 2.1/2.2 to 3.X mixed version clusters
> ---
>
> Key: CASSANDRA-14958
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14958
> Project: Cassandra
> Issue Type: Bug
> Components: Feature/Counters
>Reporter: Ariel Weisberg
>Assignee: Aleksey Yeschenko
>Priority: Critical
> Fix For: 3.0.x, 2.0.x
>
>
> The upgrade test for this is failing
> https://circleci.com/gh/aweisberg/cassandra/2362#tests/containers/1
> I confirmed that this is occurring manually using cqlsh against the cluster
> constructed by the dtest.
> {noformat}
> cqlsh> describe schema;
> CREATE KEYSPACE ks WITH replication = {'class': 'SimpleStrategy',
> 'replication_factor': '1'} AND durable_writes = true;
> CREATE TABLE ks.clicks (
> userid int,
> url text,
> total counter,
> PRIMARY KEY (userid, url)
> ) WITH COMPACT STORAGE
> AND CLUSTERING ORDER BY (url ASC)
> AND bloom_filter_fp_chance = 0.01
> AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
> AND comment = ''
> AND compaction = {'class':
> 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy',
> 'max_threshold': '32', 'min_threshold': '4'}
> AND compression = {'chunk_length_in_kb': '64', 'class':
> 'org.apache.cassandra.io.compress.LZ4Compressor'}
> AND crc_check_chance = 1.0
> AND dclocal_read_repair_chance = 0.1
> AND default_time_to_live = 0
> AND gc_grace_seconds = 864000
> AND max_index_interval = 2048
> AND memtable_flush_period_in_ms = 0
> AND min_index_interval = 128
> AND read_repair_chance = 0.0
> AND speculative_retry = '99PERCENTILE';
> cqlsh> use ks;
> cqlsh:ks> UPDATE clicks SET total = total + 1 WHERE userid = 1 AND url =
> 'http://foo.com';
> cqlsh:ks> SELECT total FROM clicks WHERE userid = 1 AND url = 'http://foo.com'
> ... ;
> total
> ---
> 0
> (1 rows)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14922) In JVM dtests need to clean up after instance shutdown
[
https://issues.apache.org/jira/browse/CASSANDRA-14922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740530#comment-16740530
]
Benedict commented on CASSANDRA-14922:
--
I've pushed a branch
[here|https://github.com/belliottsmith/cassandra/tree/14922-followup] that
removes the cleanup of thread locals, and removes the cluster-wide executor,
instead introducing a node-specific executor on which all invocations to that
node happen. This might introduce some slight penalty for evaluating tests, as
we have multiple synchronisation points where control-flow is handed between
threads, but it guarantees that all state remains isolated without ever
burdening the user of the API to restrict their own program design.
We can elaborate on this later to make it ergonomic to use the executor, or
perhaps to skip the executor, in order to provide efficiency where it matters.
I think it makes sense to classify this as part of the work for this ticket,
and I will then backport it alongside the original patch for this ticket and
the jvm-dtests, however it's primarily necessary for _future_ reasons:
* 3.0 and earlier use {{ThreadLocal}}, not {{FastThreadLocal}}, so we need the
earlier hackier reflection to fix on these
* The current approach only cleans up the main thread - any other threads may
themselves retain state indefinitely
* Most importantly, when we begin stopping and starting nodes, and/or
upgrading them, the cluster-wide executor service will retain thread local
state from the stopped nodes, potentially indefinitely, probably leading to a
steady leak of memory.
It seems easiest to introduce the future-proof approach now and port it to all
the versions at once.
On a related note, there is CASSANDRA-14974, which simultaneously fixes a bug
in our stdout capture, _and_ the impact this has to the cleanup of a
{{TestCluster}} that occurs once this bug is fixed. If somebody watching this
ticket fancies reviewing that ticket, it would also be appreciated.
> In JVM dtests need to clean up after instance shutdown
> --
>
> Key: CASSANDRA-14922
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14922
> Project: Cassandra
> Issue Type: Bug
> Components: Test/dtest
>Reporter: Joseph Lynch
>Assignee: Joseph Lynch
>Priority: Minor
> Fix For: 4.0
>
> Attachments: AllThreadsStopped.png, ClassLoadersRetaining.png,
> Leaking_Metrics_On_Shutdown.png, MainClassRetaining.png,
> MemoryReclaimedFix.png, Metaspace_Actually_Collected.png,
> OnlyThreeRootsLeft.png, no_more_references.png
>
>
> Currently the unit tests are failing on circleci ([example
> one|https://circleci.com/gh/jolynch/cassandra/300#tests/containers/1],
> [example
> two|https://circleci.com/gh/rustyrazorblade/cassandra/44#tests/containers/1])
> because we use a small container (medium) for unit tests by default and the
> in JVM dtests are leaking a few hundred megabytes of memory per test right
> now. This is not a big deal because the dtest runs with the larger containers
> continue to function fine as well as local testing as the number of in JVM
> dtests is not yet high enough to cause a problem with more than 2GB of
> available heap. However we should fix the memory leak so that going forwards
> we can add more in JVM dtests without worry.
> I've been working with [~ifesdjeen] to debug, and the issue appears to be
> unreleased Table/Keyspace metrics (screenshot showing the leak attached). I
> believe that we have a few potential issues that are leading to the leaks:
> 1. The
> [{{Instance::shutdown}}|https://github.com/apache/cassandra/blob/f22fec927de7ac29120c2f34de5b8cc1c695/test/distributed/org/apache/cassandra/distributed/Instance.java#L328-L354]
> method is not successfully cleaning up all the metrics created by the
> {{CassandraMetricsRegistry}}
> 2. The
> [{{TestCluster::close}}|https://github.com/apache/cassandra/blob/f22fec927de7ac29120c2f34de5b8cc1c695/test/distributed/org/apache/cassandra/distributed/TestCluster.java#L283]
> method is not waiting for all the instances to finish shutting down and
> cleaning up before continuing on
> 3. I'm not sure if this is an issue assuming we clear all metrics, but
> [{{TableMetrics::release}}|https://github.com/apache/cassandra/blob/4ae229f5cd270c2b43475b3f752a7b228de260ea/src/java/org/apache/cassandra/metrics/TableMetrics.java#L951]
> does not release all the metric references (which could leak them)
> I am working on a patch which shuts down everything and assures that we do
> not leak memory.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additio
[jira] [Updated] (CASSANDRA-14974) Unit test stdout capture is malfunctioning in 4.0
[ https://issues.apache.org/jira/browse/CASSANDRA-14974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benedict updated CASSANDRA-14974: - Status: Patch Available (was: Open) [patch|https://github.com/belliottsmith/cassandra/tree/14974], [CI|https://circleci.com/workflow-run/01d40d0f-52e1-429d-8cc9-22aa7fd8f07b] > Unit test stdout capture is malfunctioning in 4.0 > - > > Key: CASSANDRA-14974 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14974 > Project: Cassandra > Issue Type: Bug > Components: Test/dtest, Test/unit >Reporter: Benedict >Assignee: Benedict >Priority: Major > Fix For: 4.0 > > > In 3.x unit tests we make sure to capture stdout to the unit test files, in > case tests log to stdout and not to a logger. However, in 4.0 due to a > configuration parameter that is deprecated in logback, the logic is > short-circuited silently. > Once fixed, this affects the cleanup of in-jvm dtests which would register an > infinite chain of System.out overrides (one for each node), so we make the > functionality robust to multiple instantiations, as well as improve its > startup/shutdown sequence guarantees. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14866) Issue a CQL native protocol warning if SASI indexes are enabled on a table
[ https://issues.apache.org/jira/browse/CASSANDRA-14866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740426#comment-16740426 ] Aleksey Yeschenko commented on CASSANDRA-14866: --- It's one thing to issue a warning here - which, to be clear, I'm not against at all - but an altogether qualitatively different thing to disable the feature by default. I would argue that the latter, in 2019 Cassandra universe, should require a discussion on the dev list first. > Issue a CQL native protocol warning if SASI indexes are enabled on a table > -- > > Key: CASSANDRA-14866 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14866 > Project: Cassandra > Issue Type: Bug > Components: Feature/SASI >Reporter: Andrés de la Peña >Assignee: Andrés de la Peña >Priority: Major > Fix For: 4.0, 3.11.x, 4.x > > > If someone enables SASI indexes then we should return a native protocol > warning that will be printed by cqlsh saying that they are beta quality still > and you need to be careful with using them in production. > This is motivated not only by [the existing bugs and > limitations|https://issues.apache.org/jira/browse/CASSANDRA-12674?jql=project%20%3D%20CASSANDRA%20AND%20status%20%3D%20Open%20AND%20component%20%3D%20sasi] > but for the fact that they haven't been extensively tested yet. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14866) Issue a CQL native protocol warning if SASI indexes are enabled on a table
[ https://issues.apache.org/jira/browse/CASSANDRA-14866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740418#comment-16740418 ] Jeff Jirsa commented on CASSANDRA-14866: As already mentioned in a previous comment, this deserves a mailing list thread. That’s doubly true if you’re proposing changing the default > Issue a CQL native protocol warning if SASI indexes are enabled on a table > -- > > Key: CASSANDRA-14866 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14866 > Project: Cassandra > Issue Type: Bug > Components: Feature/SASI >Reporter: Andrés de la Peña >Assignee: Andrés de la Peña >Priority: Major > Fix For: 4.0, 3.11.x, 4.x > > > If someone enables SASI indexes then we should return a native protocol > warning that will be printed by cqlsh saying that they are beta quality still > and you need to be careful with using them in production. > This is motivated not only by [the existing bugs and > limitations|https://issues.apache.org/jira/browse/CASSANDRA-12674?jql=project%20%3D%20CASSANDRA%20AND%20status%20%3D%20Open%20AND%20component%20%3D%20sasi] > but for the fact that they haven't been extensively tested yet. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14866) Issue a CQL native protocol warning if SASI indexes are enabled on a table
[ https://issues.apache.org/jira/browse/CASSANDRA-14866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740377#comment-16740377 ] Andrés de la Peña commented on CASSANDRA-14866: --- [~snazy] I have just updated the patch set SASI as disabled by default on trunk. This has involved a few minor changes in unit tests. I think that with the new unit tests we don't require the extra dtests anymore. CI looks good to me. > Issue a CQL native protocol warning if SASI indexes are enabled on a table > -- > > Key: CASSANDRA-14866 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14866 > Project: Cassandra > Issue Type: Bug > Components: Feature/SASI >Reporter: Andrés de la Peña >Assignee: Andrés de la Peña >Priority: Major > Fix For: 4.0, 3.11.x, 4.x > > > If someone enables SASI indexes then we should return a native protocol > warning that will be printed by cqlsh saying that they are beta quality still > and you need to be careful with using them in production. > This is motivated not only by [the existing bugs and > limitations|https://issues.apache.org/jira/browse/CASSANDRA-12674?jql=project%20%3D%20CASSANDRA%20AND%20status%20%3D%20Open%20AND%20component%20%3D%20sasi] > but for the fact that they haven't been extensively tested yet. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-14974) Unit test stdout capture is malfunctioning in 4.0
Benedict created CASSANDRA-14974: Summary: Unit test stdout capture is malfunctioning in 4.0 Key: CASSANDRA-14974 URL: https://issues.apache.org/jira/browse/CASSANDRA-14974 Project: Cassandra Issue Type: Bug Components: Test/dtest, Test/unit Reporter: Benedict Assignee: Benedict Fix For: 4.0 In 3.x unit tests we make sure to capture stdout to the unit test files, in case tests log to stdout and not to a logger. However, in 4.0 due to a configuration parameter that is deprecated in logback, the logic is short-circuited silently. Once fixed, this affects the cleanup of in-jvm dtests which would register an infinite chain of System.out overrides (one for each node), so we make the functionality robust to multiple instantiations, as well as improve its startup/shutdown sequence guarantees. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13292) Replace MessagingService usage of MD5 with something more modern
[
https://issues.apache.org/jira/browse/CASSANDRA-13292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740157#comment-16740157
]
JinhuaLuo commented on CASSANDRA-13292:
---
I have a question: adler or murmur3 is not _cryptographic_ hash, so there may
be collision hash for different inputs. That is, given two different query
result, it may give the same digest value. But digest request is used to check
if all replica contains the same data for the specific query, so if the hash
does not reflect the actual difference, it would give wrong result and do not
trigger read repair.
But I also think the digest is heavyweight, which brings in unnecessary
overhead, especially when it calculates the digest upon the unchanged large
data.
I'm thinking that whether it could bring in a digest cache, then if the schema
or query columns (or fields in complex columns) was not mutated, then it could
fulfill the digest request directly from the cache.
> Replace MessagingService usage of MD5 with something more modern
>
>
> Key: CASSANDRA-13292
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13292
> Project: Cassandra
> Issue Type: Improvement
> Components: Legacy/Core
>Reporter: Michael Kjellman
>Assignee: Michael Kjellman
>Priority: Major
>
> While profiling C* via multiple profilers, I've consistently seen a
> significant amount of time being spent calculating MD5 digests.
> {code}
> Stack Trace Sample CountPercentage(%)
> sun.security.provider.MD5.implCompress(byte[], int) 264 1.566
>sun.security.provider.DigestBase.implCompressMultiBlock(byte[], int, int)
> 200 1.187
> sun.security.provider.DigestBase.engineUpdate(byte[], int, int) 200
> 1.187
> java.security.MessageDigestSpi.engineUpdate(ByteBuffer) 200
> 1.187
> java.security.MessageDigest$Delegate.engineUpdate(ByteBuffer)
> 200 1.187
>java.security.MessageDigest.update(ByteBuffer) 200 1.187
> org.apache.cassandra.db.Column.updateDigest(MessageDigest)
> 193 1.145
>
> org.apache.cassandra.db.ColumnFamily.updateDigest(MessageDigest) 193 1.145
>
> org.apache.cassandra.db.ColumnFamily.digest(ColumnFamily) 193 1.145
>
> org.apache.cassandra.service.RowDigestResolver.resolve() 106 0.629
>
> org.apache.cassandra.service.RowDigestResolver.resolve()106 0.629
>
> org.apache.cassandra.service.ReadCallback.get() 88 0.522
>
> org.apache.cassandra.service.AbstractReadExecutor.get() 88 0.522
>
> org.apache.cassandra.service.StorageProxy.fetchRows(List, ConsistencyLevel)
> 88 0.522
>
> org.apache.cassandra.service.StorageProxy.read(List, ConsistencyLevel)
> 88 0.522
>
> org.apache.cassandra.service.pager.SliceQueryPager.queryNextPage(int,
> ConsistencyLevel, boolean) 88 0.522
>
> org.apache.cassandra.service.pager.AbstractQueryPager.fetchPage(int) 88
> 0.522
>
> org.apache.cassandra.service.pager.SliceQueryPager.fetchPage(int) 88
> 0.522
>
> org.apache.cassandra.cql3.statements.SelectStatement.execute(QueryState,
> QueryOptions) 88 0.522
>
> org.apache.cassandra.cql3.statements.SelectStatement.execute(QueryState,
> QueryOptions) 88 0.522
>
> org.apache.cassandra.cql3.QueryProcessor.processStatement(CQLStatement,
> QueryState, QueryOptions) 88 0.522
>
> org.apache.cassandra.cql3.QueryProcessor.process(String, QueryState,
> QueryOptions) 88 0.522
>
> org.apache.cassandra.transport.messages.QueryMessage.execute(QueryState)
> 88 0.522
>
> org.apache.cassandra.transport.Message$Dispatcher.messageReceived(ChannelHandlerContext,
> MessageEvent) 88 0.522
>
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(ChannelHandlerContext,
> ChannelEvent) 88 0.522
>
