[jira] [Commented] (CASSANDRA-15410) Avoid over-allocation of bytes for UTF8 string serialization
[
https://issues.apache.org/jira/browse/CASSANDRA-15410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16975536#comment-16975536
]
Yifan Cai commented on CASSANDRA-15410:
---
Good idea! I have updated the PR accordingly. The benchmark result indicates
the new {{CBUtil.writeAsciiString}} method has a similar performance.
{code:java}
[java] Benchmark Mode Cnt
ScoreError Units
[java] StringsEncodeBench.writeLongTextavgt6
570.920 ± 21.376 ns/op
[java] StringsEncodeBench.writeLongTextAsASCII avgt6
291.466 ± 9.508 ns/op
[java] StringsEncodeBench.writeLongTextWithExactSize avgt6
467.222 ± 25.140 ns/op
[java] StringsEncodeBench.writeLongTextWithExactSizeSkipCalc avgt6
285.320 ± 10.883 ns/op
[java] StringsEncodeBench.writeShortText avgt6
62.076 ± 2.107 ns/op
[java] StringsEncodeBench.writeShortTextAsASCIIavgt6
32.121 ± 0.403 ns/op
[java] StringsEncodeBench.writeShortTextWithExactSize avgt6
41.929 ± 1.783 ns/op
[java] StringsEncodeBench.writeShortTextWithExactSizeSkipCalc avgt6
34.638 ± 0.455 ns/op
{code}
Changes included in the latest commit:
* added {{CBUtil.writeAsciiString}}.
* updated the code to call the new method whenever applicable, i.e. the string
is certain to only contain char from (0, 127].
* added unit test cases for the new method.
One thing to note is that {{ByteBufUtil#writeAscii()}} allows the extend 8-bits
ASCII character, meanwhile, cassandra's ASCII data type follows the US-ASCII,
which is 7-bits. If the input string contains any character with value >
{{0x007F}}, {{ByteBufUtil#writeAscii()}} is going to encode it as 1 byte.
However, cassandra's encoding considers it as 2 bytes. It leads to errors when
reading the string.
Client that calls {{CBUtil.writeAsciiString}} needs to be certain that the data
type is ascii.
I tried to add the validation step to check if the characters are US-ASCII
compatible before writing, and add a boolean parameter to skip validation for
the cases that I am sure the string is compatible. It ends up with all
references of the method have the parameter set to skip, which looks slightly
messy.
I think for the places we want to use {{writeAsciiString}} over
{{writeString}}, we already know the encoding of the string. So the parameter
was omitted.
Last but not the least, the root issue is that the {{ecodeSize}} is not a
strong constraint when encoding. The encoding method cannot safely rely on the
information. One improvement could be automate the calculation of the message
size. Right now, the size was calculated manually by adding up the size of each
field and *it is really error-prone*. Surely, it is outside the scope of this
ticket.
> Avoid over-allocation of bytes for UTF8 string serialization
> -
>
> Key: CASSANDRA-15410
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15410
> Project: Cassandra
> Issue Type: Improvement
> Components: Messaging/Client
>Reporter: Yifan Cai
>Assignee: Yifan Cai
>Priority: Normal
> Fix For: 4.0
>
>
> In the current message encoding implementation, it first calculates the
> `encodeSize` and allocates the bytebuffer with that size.
> However, during encoding, it assumes the worst case of writing UTF8 string to
> allocate bytes, i.e. assuming each letter takes 3 bytes.
> The over-estimation further leads to resizing the underlying array and data
> copy.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15431) add table options required_ttl and max_ttl
[ https://issues.apache.org/jira/browse/CASSANDRA-15431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benedict Elliott Smith updated CASSANDRA-15431: --- Fix Version/s: 4.x > add table options required_ttl and max_ttl > -- > > Key: CASSANDRA-15431 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15431 > Project: Cassandra > Issue Type: Improvement > Components: Consistency/Repair, Legacy/Local Write-Read Paths, > Local/Compaction >Reporter: David Capwell >Priority: Normal > Fix For: 4.x > > > This is related to CASSANDRA-9420 > For use cases where applications know an upper bound on TTL or know that all > TTLs are the same value, if this is put into the schema then we can choose to > skip cells past these TTL values rather than converting them to tombstones. > For this to work, we would need these fixed and immutable once set into the > table, but would need to allow tables to add this after the fact (alter > table). > For all new data, the TTLs would need to be checked to make sure that they > respect the expected property (the ttl is required_ttl if defined, or <= > max_ttl if defined). For existing data, we could ether validate the TTL > matches the expected setting and drop the cell, or we could detect that it > doesn’t match, and fall back to tombstone (see > org.apache.cassandra.db.rows.AbstractCell#purge). -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-15431) add table options required_ttl and max_ttl
David Capwell created CASSANDRA-15431: - Summary: add table options required_ttl and max_ttl Key: CASSANDRA-15431 URL: https://issues.apache.org/jira/browse/CASSANDRA-15431 Project: Cassandra Issue Type: Improvement Components: Consistency/Repair, Legacy/Local Write-Read Paths, Local/Compaction Reporter: David Capwell This is related to CASSANDRA-9420 For use cases where applications know an upper bound on TTL or know that all TTLs are the same value, if this is put into the schema then we can choose to skip cells past these TTL values rather than converting them to tombstones. For this to work, we would need these fixed and immutable once set into the table, but would need to allow tables to add this after the fact (alter table). For all new data, the TTLs would need to be checked to make sure that they respect the expected property (the ttl is required_ttl if defined, or <= max_ttl if defined). For existing data, we could ether validate the TTL matches the expected setting and drop the cell, or we could detect that it doesn’t match, and fall back to tombstone (see org.apache.cassandra.db.rows.AbstractCell#purge). -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12197) Integrate top threads command in nodetool
[ https://issues.apache.org/jira/browse/CASSANDRA-12197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16975429#comment-16975429 ] Chris Lohfink commented on CASSANDRA-12197: --- It was actually idea behind CASSANDRA-8341 which could be exposed in the thread_pools virtual table or tpstats. Its more efficient to hit the Thread mbean locally than to make individual RMI calls on each thread at high rates but to capture it cross stages gets complicated with SEP as its switches between stages. This also is something that sjk makes a bit confusing as a MutationStage thread can switch to a ReadStage from 1 query to another with same threadid making it inaccurately report. Of course in practice it doesn't matter and is incredibly valuable even with inaccuracies. > Integrate top threads command in nodetool > - > > Key: CASSANDRA-12197 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12197 > Project: Cassandra > Issue Type: Improvement > Components: Tool/nodetool >Reporter: J.B. Langston >Assignee: Ekaterina Dimitrova >Priority: Low > > SJK (https://github.com/aragozin/jvm-tools) has a command called ttop that > displays the top threads within the JVM, sorted either by CPU utilization or > heap allocation rate. When diagnosing garbage collection or high cpu > utilization, this is very helpful information. It would be great if users > can get this directly with nodetool without having to download something > else. SJK is Apache 2.0 licensed so it might be possible leverage its code. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12197) Integrate top threads command in nodetool
[ https://issues.apache.org/jira/browse/CASSANDRA-12197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16975418#comment-16975418 ] Brandon Williams commented on CASSANDRA-12197: -- In nodetool we can just invoke sjk directly and it's done. For virtual tables you'd have to programmatically manipulate the data into a virtual table format, and then to use it you'd have to repeatedly query the table since ttop's values constantly change over time. > Integrate top threads command in nodetool > - > > Key: CASSANDRA-12197 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12197 > Project: Cassandra > Issue Type: Improvement > Components: Tool/nodetool >Reporter: J.B. Langston >Assignee: Ekaterina Dimitrova >Priority: Low > > SJK (https://github.com/aragozin/jvm-tools) has a command called ttop that > displays the top threads within the JVM, sorted either by CPU utilization or > heap allocation rate. When diagnosing garbage collection or high cpu > utilization, this is very helpful information. It would be great if users > can get this directly with nodetool without having to download something > else. SJK is Apache 2.0 licensed so it might be possible leverage its code. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12197) Integrate top threads command in nodetool
[ https://issues.apache.org/jira/browse/CASSANDRA-12197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16975415#comment-16975415 ] Sankalp Kohli commented on CASSANDRA-12197: --- Should we not expose this via virtual tables rather than nodetool? > Integrate top threads command in nodetool > - > > Key: CASSANDRA-12197 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12197 > Project: Cassandra > Issue Type: Improvement > Components: Tool/nodetool >Reporter: J.B. Langston >Assignee: Ekaterina Dimitrova >Priority: Low > > SJK (https://github.com/aragozin/jvm-tools) has a command called ttop that > displays the top threads within the JVM, sorted either by CPU utilization or > heap allocation rate. When diagnosing garbage collection or high cpu > utilization, this is very helpful information. It would be great if users > can get this directly with nodetool without having to download something > else. SJK is Apache 2.0 licensed so it might be possible leverage its code. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15278) User's password for sstableloader tool is visible in ps command output.
[ https://issues.apache.org/jira/browse/CASSANDRA-15278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16975295#comment-16975295 ] Bhargav Joshi commented on CASSANDRA-15278: --- In addition, sstableloader should be able to read username and password from cqlrshrc > User's password for sstableloader tool is visible in ps command output. > --- > > Key: CASSANDRA-15278 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15278 > Project: Cassandra > Issue Type: Bug > Components: Tool/bulk load >Reporter: Niket Vilas Bagwe >Priority: Normal > > As of now, the password is visible in ps auxww output to any of the system > user if the command line utility for sstableloader is used. This seems to be > a security flaw. There should be an alternate option to pass the user's > password other than as a command line argument. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-15081) LegacyLayout does not have same behavior as 2.x when handling unknown column names
[
https://issues.apache.org/jira/browse/CASSANDRA-15081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16975187#comment-16975187
]
Michael Semb Wever edited comment on CASSANDRA-15081 at 11/15/19 6:08 PM:
--
We have no CI currently that runs [upgrade
tests|https://github.com/apache/cassandra-dtest/tree/master/upgrade_tests]
(unless you have a paid circleci account).
To run the upgrade test in question i did the following (first time i've ever
used the upgrade dtests locally).
{code}
# $cassandra_src needs to be a git clone, checked out to patched branch
cd cassandra-dtest
source ~/dtest/bin/activate
export LOCAL_GIT_REPO=$cassandra_src
python -m pytest --execute-upgrade-tests --cassandra-dir=$cassandra_src
upgrade_tests/storage_engine_upgrade_test.py::TestStorageEngineUpgrade::test_update_and_drop_column
python -m pytest --execute-upgrade-tests --cassandra-dir=$cassandra_src -s
--keep-test-dir
upgrade_tests/storage_engine_upgrade_test.py::TestBootstrapAfterUpgrade::test_update_and_drop_column
# when done
deactivate
{code}
was (Author: michaelsembwever):
We have no CI currently that runs [upgrade
tests|https://github.com/apache/cassandra-dtest/tree/master/upgrade_tests]
(unless you have a paid circleci account).
To run the upgrade test in question i did the following (first time i've ever
used the upgrade dtests locally).
{code}
# $cassandra_src needs to be a git clone, checked out to patched branch
cd cassandra-dtest
source ~/dtest/bin/activate
export LOCAL_GIT_REPO=$cassandra_src
python -m pytest --execute-upgrade-tests --cassandra-dir=$cassandra_src
upgrade_tests/storage_engine_upgrade_test.py::TestStorageEngineUpgrade::test_update_and_drop_column
python -m pytest --execute-upgrade-tests --cassandra-dir=$cassandra_src -s
--keep-test-dir
upgrade_tests/storage_engine_upgrade_test.py::TestBootstrapAfterUpgrade::test_update_and_drop_column
// when done
deactivate
{code}
> LegacyLayout does not have same behavior as 2.x when handling unknown column
> names
> --
>
> Key: CASSANDRA-15081
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15081
> Project: Cassandra
> Issue Type: Bug
> Components: Legacy/Local Write-Read Paths
>Reporter: Cameron Zemek
>Assignee: Cameron Zemek
>Priority: High
> Labels: patch, pull-request-available
> Fix For: 3.11.6
>
> Attachments: 15081.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Due to a bug I haven't been able to reproduce the production cluster had
> unknown column names. To replicate the issue for this test I did the
> following:
> {noformat}
> $ ccm create -v 2.1.19 -n 1 -s bug
> $ cat > schema.cql << 'EOF'
> CREATE KEYSPACE test WITH replication = {'class': 'SimpleStrategy',
> 'replication_factor': '1'} AND durable_writes = true;
> CREATE TABLE test.unknowntest (id int primary key, payload text, "paylo!d"
> text);
> EOF
> $ ccm node1 cqlsh -f schema.cql
> $ export CASSANDRA_INCLUDE=~/.ccm/bug/node1/bin/cassandra.in.sh
> $ cat > bug.json << 'EOF'
> [
> {"key": "1",
> "cells": [["","",1554432501209207],
> ["paylo!d","hello world",1554432501209207],
> ["payload","hello world",1554432501209207]]}
> ]
> EOF
> $ ~/.ccm/repository/2.1.19/tools/bin/json2sstable -K test -c unknowntest
> ~/bug.json
> ~/.ccm/bug/node1/data0/test/unknowntest-/test-unknowntest-ka-1-Data.db{noformat}
> Then test the behavior of unknown columns in 2.1:
> {noformat}
> $ ccm stop
> $ ccm create -v 2.1.19 -n 1 -s bug2_1_19
> $ cat > schema2.cql << 'EOF'
> CREATE KEYSPACE test WITH replication = {'class': 'SimpleStrategy',
> 'replication_factor': '1'} AND durable_writes = true;
> CREATE TABLE test.unknowntest (id int primary key, payload text);
> EOF
> $ ccm node1 cqlsh -f schema2.cql
> $ ccm stop
> $ cp ~/.ccm/bug/node1/data0/test/unknowntest-/test-unknowntest-ka-1-*
> ~/.ccm/bug2_1_19/node1/data0/test/unknowntest-/
> $ ccm start
> $ ccm node1 cqlsh
> Connected to bug2_1_19 at 127.0.0.1:9042.
> [cqlsh 5.0.1 | Cassandra 2.1.19 | CQL spec 3.2.1 | Native protocol v3]
> Use HELP for help.
> cqlsh> select * from test.unknowntest where id = 1;
> id | payload
> +-
> 1 | hello world
> (1 rows){noformat}
> Compared to 3.11.4 which did the following:
> {noformat}
> $ ccm stop
> $ ccm create -v 3.11.4 -n 1 -s bug3_11_4
> $ ccm node1 cqlsh -f schema2.cql
> $ ccm stop
> $ cp ~/.ccm/bug/node1/data0/test/unknowntest-/test-unknowntest-ka-1-*
> ~/.ccm/bug3_11_4/node1/data0/test/unknowntest-/
> $ ccm start
> $ ccm node1 cqlsh
> Connected to bug3_11_4 at 127.0.0.1:9042.
> [cqlsh 5.0.1 | Cassandra 3.11.4 | CQL spec 3.4.4 | Native protocol v4]
> Use HELP for help.
> cqlsh> select * from test.unknowntest where id = 1;
> ReadFailure:
[jira] [Updated] (CASSANDRA-15081) LegacyLayout does not have same behavior as 2.x when handling unknown column names
[
https://issues.apache.org/jira/browse/CASSANDRA-15081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Semb Wever updated CASSANDRA-15081:
---
Source Control Link:
https://github.com/apache/cassandra/commit/31a86f891b00ec0db20fcef4919dce63be7bf31d
Status: Resolved (was: Ready to Commit)
Committed as 31a86f891b00ec0db20fcef4919dce63be7bf31d
> LegacyLayout does not have same behavior as 2.x when handling unknown column
> names
> --
>
> Key: CASSANDRA-15081
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15081
> Project: Cassandra
> Issue Type: Bug
> Components: Legacy/Local Write-Read Paths
>Reporter: Cameron Zemek
>Assignee: Cameron Zemek
>Priority: High
> Labels: patch, pull-request-available
> Fix For: 3.11.6
>
> Attachments: 15081.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Due to a bug I haven't been able to reproduce the production cluster had
> unknown column names. To replicate the issue for this test I did the
> following:
> {noformat}
> $ ccm create -v 2.1.19 -n 1 -s bug
> $ cat > schema.cql << 'EOF'
> CREATE KEYSPACE test WITH replication = {'class': 'SimpleStrategy',
> 'replication_factor': '1'} AND durable_writes = true;
> CREATE TABLE test.unknowntest (id int primary key, payload text, "paylo!d"
> text);
> EOF
> $ ccm node1 cqlsh -f schema.cql
> $ export CASSANDRA_INCLUDE=~/.ccm/bug/node1/bin/cassandra.in.sh
> $ cat > bug.json << 'EOF'
> [
> {"key": "1",
> "cells": [["","",1554432501209207],
> ["paylo!d","hello world",1554432501209207],
> ["payload","hello world",1554432501209207]]}
> ]
> EOF
> $ ~/.ccm/repository/2.1.19/tools/bin/json2sstable -K test -c unknowntest
> ~/bug.json
> ~/.ccm/bug/node1/data0/test/unknowntest-/test-unknowntest-ka-1-Data.db{noformat}
> Then test the behavior of unknown columns in 2.1:
> {noformat}
> $ ccm stop
> $ ccm create -v 2.1.19 -n 1 -s bug2_1_19
> $ cat > schema2.cql << 'EOF'
> CREATE KEYSPACE test WITH replication = {'class': 'SimpleStrategy',
> 'replication_factor': '1'} AND durable_writes = true;
> CREATE TABLE test.unknowntest (id int primary key, payload text);
> EOF
> $ ccm node1 cqlsh -f schema2.cql
> $ ccm stop
> $ cp ~/.ccm/bug/node1/data0/test/unknowntest-/test-unknowntest-ka-1-*
> ~/.ccm/bug2_1_19/node1/data0/test/unknowntest-/
> $ ccm start
> $ ccm node1 cqlsh
> Connected to bug2_1_19 at 127.0.0.1:9042.
> [cqlsh 5.0.1 | Cassandra 2.1.19 | CQL spec 3.2.1 | Native protocol v3]
> Use HELP for help.
> cqlsh> select * from test.unknowntest where id = 1;
> id | payload
> +-
> 1 | hello world
> (1 rows){noformat}
> Compared to 3.11.4 which did the following:
> {noformat}
> $ ccm stop
> $ ccm create -v 3.11.4 -n 1 -s bug3_11_4
> $ ccm node1 cqlsh -f schema2.cql
> $ ccm stop
> $ cp ~/.ccm/bug/node1/data0/test/unknowntest-/test-unknowntest-ka-1-*
> ~/.ccm/bug3_11_4/node1/data0/test/unknowntest-/
> $ ccm start
> $ ccm node1 cqlsh
> Connected to bug3_11_4 at 127.0.0.1:9042.
> [cqlsh 5.0.1 | Cassandra 3.11.4 | CQL spec 3.4.4 | Native protocol v4]
> Use HELP for help.
> cqlsh> select * from test.unknowntest where id = 1;
> ReadFailure: Error from server: code=1300 [Replica(s) failed to execute read]
> message="Operation failed - received 0 responses and 1 failures"
> info={'failures': 1, 'received_responses': 0, 'required_responses': 1,
> 'consistency': 'ONE'}
> {noformat}
> In the logs this resulted in an IllegalStateException from LegacyLayout line
> 1127
> The expected behavior would be to ignore the column and return results the
> same as in 2.1
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] branch trunk updated (c98a31b -> 05d2474)
This is an automated email from the ASF dual-hosted git repository. mck pushed a change to branch trunk in repository https://gitbox.apache.org/repos/asf/cassandra.git. from c98a31b Make sure all exceptions are propagated in DebuggableThreadPoolExecutor new 31a86f8 Fix LegacyLayout to have same behavior as 2.x when handling unknown column names new 8387999 Merge branch 'cassandra-3.11' into trunk new 05d2474 update netbeans project file for latest dependencies The 3 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: ide/nbproject/project.xml | 2 +- ide/nbproject/update-netbeans-classpaths.sh | 2 ++ 2 files changed, 3 insertions(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] branch cassandra-3.11 updated: Fix LegacyLayout to have same behavior as 2.x when handling unknown column names
This is an automated email from the ASF dual-hosted git repository.
mck pushed a commit to branch cassandra-3.11
in repository https://gitbox.apache.org/repos/asf/cassandra.git
The following commit(s) were added to refs/heads/cassandra-3.11 by this push:
new 31a86f8 Fix LegacyLayout to have same behavior as 2.x when handling
unknown column names
31a86f8 is described below
commit 31a86f891b00ec0db20fcef4919dce63be7bf31d
Author: Cameron Zemek
AuthorDate: Fri Nov 1 15:36:03 2019 +1000
Fix LegacyLayout to have same behavior as 2.x when handling unknown column
names
patch by Cameron Zemek; reviewed by Mick Semb Wever for CASSANDRA-15081
---
CHANGES.txt | 1 +
src/java/org/apache/cassandra/db/LegacyLayout.java | 8
...legacy_tables-legacy_ka_15081-ka-1-CompressionInfo.db | Bin 0 -> 43 bytes
.../legacy_tables-legacy_ka_15081-ka-1-Data.db | Bin 0 -> 76 bytes
.../legacy_tables-legacy_ka_15081-ka-1-Digest.sha1 | 1 +
.../legacy_tables-legacy_ka_15081-ka-1-Filter.db | Bin 0 -> 16 bytes
.../legacy_tables-legacy_ka_15081-ka-1-Index.db | Bin 0 -> 18 bytes
.../legacy_tables-legacy_ka_15081-ka-1-Statistics.db | Bin 0 -> 4438 bytes
.../legacy_tables-legacy_ka_15081-ka-1-Summary.db| Bin 0 -> 92 bytes
.../legacy_tables-legacy_ka_15081-ka-1-TOC.txt | 8
.../apache/cassandra/io/sstable/LegacySSTableTest.java | 13 -
11 files changed, 26 insertions(+), 5 deletions(-)
diff --git a/CHANGES.txt b/CHANGES.txt
index c1480a4..8c52f34 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
3.11.6
+ * Fix LegacyLayout to have same behavior as 2.x when handling unknown column
names (CASSANDRA-15081)
Merged from 3.0:
* Minimize clustering values in metadata collector (CASSANDRA-15400)
* Avoid over-trimming of results in mixed mode clusters (CASSANDRA-15405)
diff --git a/src/java/org/apache/cassandra/db/LegacyLayout.java
b/src/java/org/apache/cassandra/db/LegacyLayout.java
index bbef11d..09f9cfa 100644
--- a/src/java/org/apache/cassandra/db/LegacyLayout.java
+++ b/src/java/org/apache/cassandra/db/LegacyLayout.java
@@ -1213,10 +1213,10 @@ public abstract class LegacyLayout
// but we don't do this here and re-throw the exception because
the calling code sometimes has to know
// about this happening. This does mean code calling this method
should handle this case properly.
if (!metadata.ksName.equals(SchemaConstants.SYSTEM_KEYSPACE_NAME)
&& metadata.getDroppedColumnDefinition(e.columnName) == null)
-throw new IllegalStateException(String.format("Got cell for
unknown column %s in sstable of %s.%s: " +
- "This suggest a
problem with the schema which doesn't list " +
- "this column.
Even if that column was dropped, it should have " +
- "been listed as
such", metadata.ksName, metadata.cfName,
UTF8Type.instance.compose(e.columnName)), e);
+logger.warn(String.format("Got cell for unknown column %s in
sstable of %s.%s: " +
+ "This suggest a problem with the
schema which doesn't list " +
+ "this column. Even if that column
was dropped, it should have " +
+ "been listed as such",
metadata.ksName, metadata.cfName, UTF8Type.instance.compose(e.columnName)), e);
throw e;
}
diff --git
a/test/data/legacy-sstables/ka/legacy_tables/legacy_ka_15081/legacy_tables-legacy_ka_15081-ka-1-CompressionInfo.db
b/test/data/legacy-sstables/ka/legacy_tables/legacy_ka_15081/legacy_tables-legacy_ka_15081-ka-1-CompressionInfo.db
new file mode 100644
index 000..3793e50
Binary files /dev/null and
b/test/data/legacy-sstables/ka/legacy_tables/legacy_ka_15081/legacy_tables-legacy_ka_15081-ka-1-CompressionInfo.db
differ
diff --git
a/test/data/legacy-sstables/ka/legacy_tables/legacy_ka_15081/legacy_tables-legacy_ka_15081-ka-1-Data.db
b/test/data/legacy-sstables/ka/legacy_tables/legacy_ka_15081/legacy_tables-legacy_ka_15081-ka-1-Data.db
new file mode 100644
index 000..94c6f93
Binary files /dev/null and
b/test/data/legacy-sstables/ka/legacy_tables/legacy_ka_15081/legacy_tables-legacy_ka_15081-ka-1-Data.db
differ
diff --git
a/test/data/legacy-sstables/ka/legacy_tables/legacy_ka_15081/legacy_tables-legacy_ka_15081-ka-1-Digest.sha1
b/test/data/legacy-sstables/ka/legacy_tables/legacy_ka_15081/legacy_tables-legacy_ka_15081-ka-1-Digest.sha1
new file mode 100644
index 000..60bd60d
--- /dev/null
+++
b/test/data/legacy-sstables/ka/legacy_tables/legacy_ka_15081/legacy_tables-legacy_ka_15081-ka-1-Digest.sha1
@@ -0,0 +1 @@
+718738748
\ No newline at end of file
diff --git
[cassandra] 01/02: Merge branch 'cassandra-3.11' into trunk
This is an automated email from the ASF dual-hosted git repository. mck pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/cassandra.git commit 83879993dbff5ff3031e5756c8dece7ee173afdf Merge: c98a31b 31a86f8 Author: Mick Semb Wever AuthorDate: Fri Nov 15 18:59:23 2019 +0100 Merge branch 'cassandra-3.11' into trunk - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] 02/02: update netbeans project file for latest dependencies
This is an automated email from the ASF dual-hosted git repository.
mck pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra.git
commit 05d24745b482a993c2588f515dda326b8f1cd80e
Author: Mick Semb Wever
AuthorDate: Sun Aug 4 23:53:47 2019 +0200
update netbeans project file for latest dependencies
---
ide/nbproject/project.xml | 2 +-
ide/nbproject/update-netbeans-classpaths.sh | 2 ++
2 files changed, 3 insertions(+), 1 deletion(-)
diff --git a/ide/nbproject/project.xml b/ide/nbproject/project.xml
index 9ca0910..94a0f3f 100644
--- a/ide/nbproject/project.xml
+++ b/ide/nbproject/project.xml
@@ -7,7 +7,7 @@
..
-${project.dir}/lib/HdrHistogram-2.1.9.jar:${project.dir}/lib/ST4-4.0.8.jar:${project.dir}/lib/airline-0.8.jar:${project.dir}/lib/antlr-runtime-3.5.2.jar:${project.dir}/lib/asm-6.2.jar:${project.dir}/lib/caffeine-2.3.5.jar:${project.dir}/lib/cassandra-driver-core-3.4.0-SNAPSHOT-shaded.jar:${project.dir}/lib/chronicle-bytes-1.16.3.jar:${project.dir}/lib/chronicle-core-1.16.3-SNAPSHOT.jar:${project.dir}/lib/chronicle-queue-4.16.3.jar
[...]
+${project.dir}/lib/HdrHistogram-2.1.9.jar:${project.dir}/lib/ST4-4.0.8.jar:${project.dir}/lib/airline-0.8.jar:${project.dir}/lib/antlr-runtime-3.5.2.jar:${project.dir}/lib/asm-7.1.jar:${project.dir}/lib/caffeine-2.3.5.jar:${project.dir}/lib/cassandra-driver-core-3.6.0-shaded.jar:${project.dir}/lib/chronicle-bytes-1.16.3.jar:${project.dir}/lib/chronicle-core-1.16.4.jar:${project.dir}/lib/chronicle-queue-4.16.3.jar:${project.dir}/li
[...]
diff --git a/ide/nbproject/update-netbeans-classpaths.sh
b/ide/nbproject/update-netbeans-classpaths.sh
index 8d1ba00..afef864 100755
--- a/ide/nbproject/update-netbeans-classpaths.sh
+++ b/ide/nbproject/update-netbeans-classpaths.sh
@@ -6,6 +6,8 @@
#
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
+
+cd $DIR/../..
CLASSPATH=`for f in lib/*.jar ; do echo -n '${project.dir}/'$f: ; done ; for f
in build/lib/jars/*.jar ; do echo -n '${project.dir}/'$f: ; done ;`
sed -i ''
's/cassandra\.classpath\.jars\">.*<\/property>/cassandra\.classpath\.jars\">NEW_CLASSPATH<\/property>/'
$DIR/project.xml
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15332) When repair is running with tracing, if a CorruptSSTableException is thrown while building Merkle Trees the DiskFailurePolicy does not get applied
[ https://issues.apache.org/jira/browse/CASSANDRA-15332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcus Eriksson updated CASSANDRA-15332: Since Version: 2.1 beta1 Source Control Link: https://github.com/apache/cassandra/commit/c98a31bef0d3071bc8ebfd358584aadf9e787fb8 Resolution: Fixed Status: Resolved (was: Ready to Commit) committed, thanks! > When repair is running with tracing, if a CorruptSSTableException is thrown > while building Merkle Trees the DiskFailurePolicy does not get applied > -- > > Key: CASSANDRA-15332 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15332 > Project: Cassandra > Issue Type: Bug > Components: Consistency/Repair, Observability/Tracing >Reporter: David Capwell >Assignee: David Capwell >Priority: Normal > Labels: pull-request-available > Fix For: 4.0 > > Time Spent: 1h 50m > Remaining Estimate: 0h > > When a repair is in the validation phase and is building MerkleTrees, if a > corrupt SSTable exception is thrown the disk failure policy does not get > applied -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15332) When repair is running with tracing, if a CorruptSSTableException is thrown while building Merkle Trees the DiskFailurePolicy does not get applied
[ https://issues.apache.org/jira/browse/CASSANDRA-15332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcus Eriksson updated CASSANDRA-15332: Status: Ready to Commit (was: Review In Progress) > When repair is running with tracing, if a CorruptSSTableException is thrown > while building Merkle Trees the DiskFailurePolicy does not get applied > -- > > Key: CASSANDRA-15332 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15332 > Project: Cassandra > Issue Type: Bug > Components: Consistency/Repair, Observability/Tracing >Reporter: David Capwell >Assignee: David Capwell >Priority: Normal > Labels: pull-request-available > Fix For: 4.0 > > Time Spent: 1h 50m > Remaining Estimate: 0h > > When a repair is in the validation phase and is building MerkleTrees, if a > corrupt SSTable exception is thrown the disk failure policy does not get > applied -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15332) When repair is running with tracing, if a CorruptSSTableException is thrown while building Merkle Trees the DiskFailurePolicy does not get applied
[ https://issues.apache.org/jira/browse/CASSANDRA-15332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcus Eriksson updated CASSANDRA-15332: Reviewers: Jordan West, Marcus Eriksson, Marcus Eriksson (was: Jordan West, Marcus Eriksson) Jordan West, Marcus Eriksson, Marcus Eriksson (was: Blake Eggleston, Jordan West, Marcus Eriksson) Status: Review In Progress (was: Patch Available) > When repair is running with tracing, if a CorruptSSTableException is thrown > while building Merkle Trees the DiskFailurePolicy does not get applied > -- > > Key: CASSANDRA-15332 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15332 > Project: Cassandra > Issue Type: Bug > Components: Consistency/Repair, Observability/Tracing >Reporter: David Capwell >Assignee: David Capwell >Priority: Normal > Labels: pull-request-available > Fix For: 4.0 > > Time Spent: 1h 50m > Remaining Estimate: 0h > > When a repair is in the validation phase and is building MerkleTrees, if a > corrupt SSTable exception is thrown the disk failure policy does not get > applied -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] branch trunk updated: Make sure all exceptions are propagated in DebuggableThreadPoolExecutor
This is an automated email from the ASF dual-hosted git repository.
marcuse pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra.git
The following commit(s) were added to refs/heads/trunk by this push:
new c98a31b Make sure all exceptions are propagated in
DebuggableThreadPoolExecutor
c98a31b is described below
commit c98a31bef0d3071bc8ebfd358584aadf9e787fb8
Author: David Capwell
AuthorDate: Mon Oct 14 11:45:29 2019 -0700
Make sure all exceptions are propagated in DebuggableThreadPoolExecutor
Patch by David Capwell; reviewed by Jordan West and marcuse for
CASSANDRA-15332
---
CHANGES.txt| 1 +
.../concurrent/DebuggableThreadPoolExecutor.java | 141 -
.../apache/cassandra/service/CassandraDaemon.java | 55 +-
.../cassandra/distributed/api/IInstance.java | 10 +
.../distributed/impl/AbstractCluster.java | 39 +-
.../cassandra/distributed/impl/Instance.java | 21 +-
.../distributed/impl/InstanceClassLoader.java | 5 +
.../cassandra/distributed/impl/InstanceKiller.java | 50 ++
.../distributed/test/FailingRepairTest.java| 346 +++
.../io/sstable/format/ForwardingSSTableReader.java | 679 +
.../DebuggableThreadPoolExecutorTest.java | 161 +
11 files changed, 1458 insertions(+), 50 deletions(-)
diff --git a/CHANGES.txt b/CHANGES.txt
index d3e0b89..7cc960d 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
4.0-alpha3
+ * Make sure all exceptions are propagated in DebuggableThreadPoolExecutor
(CASSANDRA-15332)
* Make it possible to resize concurrent read / write thread pools at runtime
(CASSANDRA-15277)
Merged from 2.2:
* In-JVM DTest: Set correct internode message version for upgrade test
(CASSANDRA-15371)
diff --git
a/src/java/org/apache/cassandra/concurrent/DebuggableThreadPoolExecutor.java
b/src/java/org/apache/cassandra/concurrent/DebuggableThreadPoolExecutor.java
index ef0f43c..a2de775 100644
--- a/src/java/org/apache/cassandra/concurrent/DebuggableThreadPoolExecutor.java
+++ b/src/java/org/apache/cassandra/concurrent/DebuggableThreadPoolExecutor.java
@@ -17,7 +17,21 @@
*/
package org.apache.cassandra.concurrent;
-import java.util.concurrent.*;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.Callable;
+import java.util.concurrent.CancellationException;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.FutureTask;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.RejectedExecutionHandler;
+import java.util.concurrent.RunnableFuture;
+import java.util.concurrent.SynchronousQueue;
+import java.util.concurrent.ThreadFactory;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -147,7 +161,7 @@ public class DebuggableThreadPoolExecutor extends
ThreadPoolExecutor implements
{
super.execute(locals == null || command instanceof LocalSessionWrapper
? command
- : new LocalSessionWrapper(command, locals));
+ : LocalSessionWrapper.create(command, null, locals));
}
public void maybeExecuteImmediately(Runnable command)
@@ -160,7 +174,7 @@ public class DebuggableThreadPoolExecutor extends
ThreadPoolExecutor implements
public void execute(Runnable command)
{
super.execute(isTracing() && !(command instanceof LocalSessionWrapper)
- ? new
LocalSessionWrapper(Executors.callable(command, null))
+ ? LocalSessionWrapper.create(command)
: command);
}
@@ -168,9 +182,9 @@ public class DebuggableThreadPoolExecutor extends
ThreadPoolExecutor implements
protected RunnableFuture newTaskFor(Runnable runnable, T result)
{
if (isTracing() && !(runnable instanceof LocalSessionWrapper))
-{
-return new LocalSessionWrapper(Executors.callable(runnable,
result));
-}
+return LocalSessionWrapper.create(runnable, result);
+if (runnable instanceof RunnableFuture)
+return new ForwardingRunnableFuture<>((RunnableFuture) runnable,
result);
return super.newTaskFor(runnable, result);
}
@@ -178,9 +192,7 @@ public class DebuggableThreadPoolExecutor extends
ThreadPoolExecutor implements
protected RunnableFuture newTaskFor(Callable callable)
{
if (isTracing() && !(callable instanceof LocalSessionWrapper))
-{
-return new LocalSessionWrapper(callable);
-}
+return LocalSessionWrapper.create(callable);
return super.newTaskFor(callable);
}
@@ -258,16 +270,32 @@ public class
[jira] [Updated] (CASSANDRA-15081) LegacyLayout does not have same behavior as 2.x when handling unknown column names
[
https://issues.apache.org/jira/browse/CASSANDRA-15081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Semb Wever updated CASSANDRA-15081:
---
Fix Version/s: (was: 4.0-alpha)
> LegacyLayout does not have same behavior as 2.x when handling unknown column
> names
> --
>
> Key: CASSANDRA-15081
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15081
> Project: Cassandra
> Issue Type: Bug
> Components: Legacy/Local Write-Read Paths
>Reporter: Cameron Zemek
>Assignee: Cameron Zemek
>Priority: High
> Labels: patch, pull-request-available
> Fix For: 3.11.6
>
> Attachments: 15081.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Due to a bug I haven't been able to reproduce the production cluster had
> unknown column names. To replicate the issue for this test I did the
> following:
> {noformat}
> $ ccm create -v 2.1.19 -n 1 -s bug
> $ cat > schema.cql << 'EOF'
> CREATE KEYSPACE test WITH replication = {'class': 'SimpleStrategy',
> 'replication_factor': '1'} AND durable_writes = true;
> CREATE TABLE test.unknowntest (id int primary key, payload text, "paylo!d"
> text);
> EOF
> $ ccm node1 cqlsh -f schema.cql
> $ export CASSANDRA_INCLUDE=~/.ccm/bug/node1/bin/cassandra.in.sh
> $ cat > bug.json << 'EOF'
> [
> {"key": "1",
> "cells": [["","",1554432501209207],
> ["paylo!d","hello world",1554432501209207],
> ["payload","hello world",1554432501209207]]}
> ]
> EOF
> $ ~/.ccm/repository/2.1.19/tools/bin/json2sstable -K test -c unknowntest
> ~/bug.json
> ~/.ccm/bug/node1/data0/test/unknowntest-/test-unknowntest-ka-1-Data.db{noformat}
> Then test the behavior of unknown columns in 2.1:
> {noformat}
> $ ccm stop
> $ ccm create -v 2.1.19 -n 1 -s bug2_1_19
> $ cat > schema2.cql << 'EOF'
> CREATE KEYSPACE test WITH replication = {'class': 'SimpleStrategy',
> 'replication_factor': '1'} AND durable_writes = true;
> CREATE TABLE test.unknowntest (id int primary key, payload text);
> EOF
> $ ccm node1 cqlsh -f schema2.cql
> $ ccm stop
> $ cp ~/.ccm/bug/node1/data0/test/unknowntest-/test-unknowntest-ka-1-*
> ~/.ccm/bug2_1_19/node1/data0/test/unknowntest-/
> $ ccm start
> $ ccm node1 cqlsh
> Connected to bug2_1_19 at 127.0.0.1:9042.
> [cqlsh 5.0.1 | Cassandra 2.1.19 | CQL spec 3.2.1 | Native protocol v3]
> Use HELP for help.
> cqlsh> select * from test.unknowntest where id = 1;
> id | payload
> +-
> 1 | hello world
> (1 rows){noformat}
> Compared to 3.11.4 which did the following:
> {noformat}
> $ ccm stop
> $ ccm create -v 3.11.4 -n 1 -s bug3_11_4
> $ ccm node1 cqlsh -f schema2.cql
> $ ccm stop
> $ cp ~/.ccm/bug/node1/data0/test/unknowntest-/test-unknowntest-ka-1-*
> ~/.ccm/bug3_11_4/node1/data0/test/unknowntest-/
> $ ccm start
> $ ccm node1 cqlsh
> Connected to bug3_11_4 at 127.0.0.1:9042.
> [cqlsh 5.0.1 | Cassandra 3.11.4 | CQL spec 3.4.4 | Native protocol v4]
> Use HELP for help.
> cqlsh> select * from test.unknowntest where id = 1;
> ReadFailure: Error from server: code=1300 [Replica(s) failed to execute read]
> message="Operation failed - received 0 responses and 1 failures"
> info={'failures': 1, 'received_responses': 0, 'required_responses': 1,
> 'consistency': 'ONE'}
> {noformat}
> In the logs this resulted in an IllegalStateException from LegacyLayout line
> 1127
> The expected behavior would be to ignore the column and return results the
> same as in 2.1
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15081) LegacyLayout does not have same behavior as 2.x when handling unknown column names
[
https://issues.apache.org/jira/browse/CASSANDRA-15081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Semb Wever updated CASSANDRA-15081:
---
Status: Ready to Commit (was: Review In Progress)
> LegacyLayout does not have same behavior as 2.x when handling unknown column
> names
> --
>
> Key: CASSANDRA-15081
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15081
> Project: Cassandra
> Issue Type: Bug
> Components: Legacy/Local Write-Read Paths
>Reporter: Cameron Zemek
>Assignee: Cameron Zemek
>Priority: High
> Labels: patch, pull-request-available
> Attachments: 15081.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Due to a bug I haven't been able to reproduce the production cluster had
> unknown column names. To replicate the issue for this test I did the
> following:
> {noformat}
> $ ccm create -v 2.1.19 -n 1 -s bug
> $ cat > schema.cql << 'EOF'
> CREATE KEYSPACE test WITH replication = {'class': 'SimpleStrategy',
> 'replication_factor': '1'} AND durable_writes = true;
> CREATE TABLE test.unknowntest (id int primary key, payload text, "paylo!d"
> text);
> EOF
> $ ccm node1 cqlsh -f schema.cql
> $ export CASSANDRA_INCLUDE=~/.ccm/bug/node1/bin/cassandra.in.sh
> $ cat > bug.json << 'EOF'
> [
> {"key": "1",
> "cells": [["","",1554432501209207],
> ["paylo!d","hello world",1554432501209207],
> ["payload","hello world",1554432501209207]]}
> ]
> EOF
> $ ~/.ccm/repository/2.1.19/tools/bin/json2sstable -K test -c unknowntest
> ~/bug.json
> ~/.ccm/bug/node1/data0/test/unknowntest-/test-unknowntest-ka-1-Data.db{noformat}
> Then test the behavior of unknown columns in 2.1:
> {noformat}
> $ ccm stop
> $ ccm create -v 2.1.19 -n 1 -s bug2_1_19
> $ cat > schema2.cql << 'EOF'
> CREATE KEYSPACE test WITH replication = {'class': 'SimpleStrategy',
> 'replication_factor': '1'} AND durable_writes = true;
> CREATE TABLE test.unknowntest (id int primary key, payload text);
> EOF
> $ ccm node1 cqlsh -f schema2.cql
> $ ccm stop
> $ cp ~/.ccm/bug/node1/data0/test/unknowntest-/test-unknowntest-ka-1-*
> ~/.ccm/bug2_1_19/node1/data0/test/unknowntest-/
> $ ccm start
> $ ccm node1 cqlsh
> Connected to bug2_1_19 at 127.0.0.1:9042.
> [cqlsh 5.0.1 | Cassandra 2.1.19 | CQL spec 3.2.1 | Native protocol v3]
> Use HELP for help.
> cqlsh> select * from test.unknowntest where id = 1;
> id | payload
> +-
> 1 | hello world
> (1 rows){noformat}
> Compared to 3.11.4 which did the following:
> {noformat}
> $ ccm stop
> $ ccm create -v 3.11.4 -n 1 -s bug3_11_4
> $ ccm node1 cqlsh -f schema2.cql
> $ ccm stop
> $ cp ~/.ccm/bug/node1/data0/test/unknowntest-/test-unknowntest-ka-1-*
> ~/.ccm/bug3_11_4/node1/data0/test/unknowntest-/
> $ ccm start
> $ ccm node1 cqlsh
> Connected to bug3_11_4 at 127.0.0.1:9042.
> [cqlsh 5.0.1 | Cassandra 3.11.4 | CQL spec 3.4.4 | Native protocol v4]
> Use HELP for help.
> cqlsh> select * from test.unknowntest where id = 1;
> ReadFailure: Error from server: code=1300 [Replica(s) failed to execute read]
> message="Operation failed - received 0 responses and 1 failures"
> info={'failures': 1, 'received_responses': 0, 'required_responses': 1,
> 'consistency': 'ONE'}
> {noformat}
> In the logs this resulted in an IllegalStateException from LegacyLayout line
> 1127
> The expected behavior would be to ignore the column and return results the
> same as in 2.1
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15081) LegacyLayout does not have same behavior as 2.x when handling unknown column names
[
https://issues.apache.org/jira/browse/CASSANDRA-15081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Semb Wever updated CASSANDRA-15081:
---
Fix Version/s: 4.0-alpha
3.11.6
> LegacyLayout does not have same behavior as 2.x when handling unknown column
> names
> --
>
> Key: CASSANDRA-15081
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15081
> Project: Cassandra
> Issue Type: Bug
> Components: Legacy/Local Write-Read Paths
>Reporter: Cameron Zemek
>Assignee: Cameron Zemek
>Priority: High
> Labels: patch, pull-request-available
> Fix For: 3.11.6, 4.0-alpha
>
> Attachments: 15081.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Due to a bug I haven't been able to reproduce the production cluster had
> unknown column names. To replicate the issue for this test I did the
> following:
> {noformat}
> $ ccm create -v 2.1.19 -n 1 -s bug
> $ cat > schema.cql << 'EOF'
> CREATE KEYSPACE test WITH replication = {'class': 'SimpleStrategy',
> 'replication_factor': '1'} AND durable_writes = true;
> CREATE TABLE test.unknowntest (id int primary key, payload text, "paylo!d"
> text);
> EOF
> $ ccm node1 cqlsh -f schema.cql
> $ export CASSANDRA_INCLUDE=~/.ccm/bug/node1/bin/cassandra.in.sh
> $ cat > bug.json << 'EOF'
> [
> {"key": "1",
> "cells": [["","",1554432501209207],
> ["paylo!d","hello world",1554432501209207],
> ["payload","hello world",1554432501209207]]}
> ]
> EOF
> $ ~/.ccm/repository/2.1.19/tools/bin/json2sstable -K test -c unknowntest
> ~/bug.json
> ~/.ccm/bug/node1/data0/test/unknowntest-/test-unknowntest-ka-1-Data.db{noformat}
> Then test the behavior of unknown columns in 2.1:
> {noformat}
> $ ccm stop
> $ ccm create -v 2.1.19 -n 1 -s bug2_1_19
> $ cat > schema2.cql << 'EOF'
> CREATE KEYSPACE test WITH replication = {'class': 'SimpleStrategy',
> 'replication_factor': '1'} AND durable_writes = true;
> CREATE TABLE test.unknowntest (id int primary key, payload text);
> EOF
> $ ccm node1 cqlsh -f schema2.cql
> $ ccm stop
> $ cp ~/.ccm/bug/node1/data0/test/unknowntest-/test-unknowntest-ka-1-*
> ~/.ccm/bug2_1_19/node1/data0/test/unknowntest-/
> $ ccm start
> $ ccm node1 cqlsh
> Connected to bug2_1_19 at 127.0.0.1:9042.
> [cqlsh 5.0.1 | Cassandra 2.1.19 | CQL spec 3.2.1 | Native protocol v3]
> Use HELP for help.
> cqlsh> select * from test.unknowntest where id = 1;
> id | payload
> +-
> 1 | hello world
> (1 rows){noformat}
> Compared to 3.11.4 which did the following:
> {noformat}
> $ ccm stop
> $ ccm create -v 3.11.4 -n 1 -s bug3_11_4
> $ ccm node1 cqlsh -f schema2.cql
> $ ccm stop
> $ cp ~/.ccm/bug/node1/data0/test/unknowntest-/test-unknowntest-ka-1-*
> ~/.ccm/bug3_11_4/node1/data0/test/unknowntest-/
> $ ccm start
> $ ccm node1 cqlsh
> Connected to bug3_11_4 at 127.0.0.1:9042.
> [cqlsh 5.0.1 | Cassandra 3.11.4 | CQL spec 3.4.4 | Native protocol v4]
> Use HELP for help.
> cqlsh> select * from test.unknowntest where id = 1;
> ReadFailure: Error from server: code=1300 [Replica(s) failed to execute read]
> message="Operation failed - received 0 responses and 1 failures"
> info={'failures': 1, 'received_responses': 0, 'required_responses': 1,
> 'consistency': 'ONE'}
> {noformat}
> In the logs this resulted in an IllegalStateException from LegacyLayout line
> 1127
> The expected behavior would be to ignore the column and return results the
> same as in 2.1
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15081) LegacyLayout does not have same behavior as 2.x when handling unknown column names
[
https://issues.apache.org/jira/browse/CASSANDRA-15081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16975187#comment-16975187
]
Michael Semb Wever commented on CASSANDRA-15081:
We have no CI currently that runs [upgrade
tests|https://github.com/apache/cassandra-dtest/tree/master/upgrade_tests]
(unless you have a paid circleci account).
To run the upgrade test in question i did the following (first time i've ever
used the upgrade dtests locally).
{code}
# $cassandra_src needs to be a git clone, checked out to patched branch
cd cassandra-dtest
source ~/dtest/bin/activate
export LOCAL_GIT_REPO=$cassandra_src
python -m pytest --execute-upgrade-tests --cassandra-dir=$cassandra_src
upgrade_tests/storage_engine_upgrade_test.py::TestStorageEngineUpgrade::test_update_and_drop_column
python -m pytest --execute-upgrade-tests --cassandra-dir=$cassandra_src -s
--keep-test-dir
upgrade_tests/storage_engine_upgrade_test.py::TestBootstrapAfterUpgrade::test_update_and_drop_column
// when done
deactivate
{code}
> LegacyLayout does not have same behavior as 2.x when handling unknown column
> names
> --
>
> Key: CASSANDRA-15081
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15081
> Project: Cassandra
> Issue Type: Bug
> Components: Legacy/Local Write-Read Paths
>Reporter: Cameron Zemek
>Assignee: Cameron Zemek
>Priority: High
> Labels: patch, pull-request-available
> Attachments: 15081.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Due to a bug I haven't been able to reproduce the production cluster had
> unknown column names. To replicate the issue for this test I did the
> following:
> {noformat}
> $ ccm create -v 2.1.19 -n 1 -s bug
> $ cat > schema.cql << 'EOF'
> CREATE KEYSPACE test WITH replication = {'class': 'SimpleStrategy',
> 'replication_factor': '1'} AND durable_writes = true;
> CREATE TABLE test.unknowntest (id int primary key, payload text, "paylo!d"
> text);
> EOF
> $ ccm node1 cqlsh -f schema.cql
> $ export CASSANDRA_INCLUDE=~/.ccm/bug/node1/bin/cassandra.in.sh
> $ cat > bug.json << 'EOF'
> [
> {"key": "1",
> "cells": [["","",1554432501209207],
> ["paylo!d","hello world",1554432501209207],
> ["payload","hello world",1554432501209207]]}
> ]
> EOF
> $ ~/.ccm/repository/2.1.19/tools/bin/json2sstable -K test -c unknowntest
> ~/bug.json
> ~/.ccm/bug/node1/data0/test/unknowntest-/test-unknowntest-ka-1-Data.db{noformat}
> Then test the behavior of unknown columns in 2.1:
> {noformat}
> $ ccm stop
> $ ccm create -v 2.1.19 -n 1 -s bug2_1_19
> $ cat > schema2.cql << 'EOF'
> CREATE KEYSPACE test WITH replication = {'class': 'SimpleStrategy',
> 'replication_factor': '1'} AND durable_writes = true;
> CREATE TABLE test.unknowntest (id int primary key, payload text);
> EOF
> $ ccm node1 cqlsh -f schema2.cql
> $ ccm stop
> $ cp ~/.ccm/bug/node1/data0/test/unknowntest-/test-unknowntest-ka-1-*
> ~/.ccm/bug2_1_19/node1/data0/test/unknowntest-/
> $ ccm start
> $ ccm node1 cqlsh
> Connected to bug2_1_19 at 127.0.0.1:9042.
> [cqlsh 5.0.1 | Cassandra 2.1.19 | CQL spec 3.2.1 | Native protocol v3]
> Use HELP for help.
> cqlsh> select * from test.unknowntest where id = 1;
> id | payload
> +-
> 1 | hello world
> (1 rows){noformat}
> Compared to 3.11.4 which did the following:
> {noformat}
> $ ccm stop
> $ ccm create -v 3.11.4 -n 1 -s bug3_11_4
> $ ccm node1 cqlsh -f schema2.cql
> $ ccm stop
> $ cp ~/.ccm/bug/node1/data0/test/unknowntest-/test-unknowntest-ka-1-*
> ~/.ccm/bug3_11_4/node1/data0/test/unknowntest-/
> $ ccm start
> $ ccm node1 cqlsh
> Connected to bug3_11_4 at 127.0.0.1:9042.
> [cqlsh 5.0.1 | Cassandra 3.11.4 | CQL spec 3.4.4 | Native protocol v4]
> Use HELP for help.
> cqlsh> select * from test.unknowntest where id = 1;
> ReadFailure: Error from server: code=1300 [Replica(s) failed to execute read]
> message="Operation failed - received 0 responses and 1 failures"
> info={'failures': 1, 'received_responses': 0, 'required_responses': 1,
> 'consistency': 'ONE'}
> {noformat}
> In the logs this resulted in an IllegalStateException from LegacyLayout line
> 1127
> The expected behavior would be to ignore the column and return results the
> same as in 2.1
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15409) nodetool compactionstats showing extra pending task for TWCS
[
https://issues.apache.org/jira/browse/CASSANDRA-15409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16975175#comment-16975175
]
Stefania Alborghetti commented on CASSANDRA-15409:
--
Patch LGTM
Moving forward, it's perhaps easier to post the direct link to the branches:
[CASSANDRA-15409-3.11|https://github.com/ekaterinadimitrova2/cassandra/tree/CASSANDRA-15409-3.11]
[CASSANDRA-15409-trunk|https://github.com/ekaterinadimitrova2/cassandra/tree/CASSANDRA-15409-trunk]
Also, before merging, we need to post the CI results.
> nodetool compactionstats showing extra pending task for TWCS
>
>
> Key: CASSANDRA-15409
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15409
> Project: Cassandra
> Issue Type: Bug
> Components: Tool/nodetool
>Reporter: Ekaterina Dimitrova
>Assignee: Ekaterina Dimitrova
>Priority: Normal
> Fix For: 3.11.6, 4.x
>
>
> Summary: nodetool compactionstats showing extra pending task for TWCS
> -
> The output of {{nodetool compactionstats}}can show "pending tasks: 1" when
> there are actually none. This seems to be a consistent problem in testing C*
> trunk. In my testing, it looks like the {{nodetool compactionstats}} counter
> output is consistently off by 1 as compared to the table output of the tasks
>
> testing with {{concurrent_compactors: 8}}
> In 12 hours it never ended, always showing 1 pending job
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15409) nodetool compactionstats showing extra pending task for TWCS
[
https://issues.apache.org/jira/browse/CASSANDRA-15409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefania Alborghetti updated CASSANDRA-15409:
-
Reviewers: Stefania Alborghetti, Stefania Alborghetti (was: Stefania
Alborghetti)
Stefania Alborghetti, Stefania Alborghetti
Status: Review In Progress (was: Patch Available)
> nodetool compactionstats showing extra pending task for TWCS
>
>
> Key: CASSANDRA-15409
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15409
> Project: Cassandra
> Issue Type: Bug
> Components: Tool/nodetool
>Reporter: Ekaterina Dimitrova
>Assignee: Ekaterina Dimitrova
>Priority: Normal
> Fix For: 3.11.6, 4.x
>
>
> Summary: nodetool compactionstats showing extra pending task for TWCS
> -
> The output of {{nodetool compactionstats}}can show "pending tasks: 1" when
> there are actually none. This seems to be a consistent problem in testing C*
> trunk. In my testing, it looks like the {{nodetool compactionstats}} counter
> output is consistently off by 1 as compared to the table output of the tasks
>
> testing with {{concurrent_compactors: 8}}
> In 12 hours it never ended, always showing 1 pending job
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Assigned] (CASSANDRA-15081) LegacyLayout does not have same behavior as 2.x when handling unknown column names
[
https://issues.apache.org/jira/browse/CASSANDRA-15081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Semb Wever reassigned CASSANDRA-15081:
--
Assignee: Cameron Zemek
> LegacyLayout does not have same behavior as 2.x when handling unknown column
> names
> --
>
> Key: CASSANDRA-15081
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15081
> Project: Cassandra
> Issue Type: Bug
> Components: Legacy/Local Write-Read Paths
>Reporter: Cameron Zemek
>Assignee: Cameron Zemek
>Priority: High
> Labels: patch, pull-request-available
> Attachments: 15081.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Due to a bug I haven't been able to reproduce the production cluster had
> unknown column names. To replicate the issue for this test I did the
> following:
> {noformat}
> $ ccm create -v 2.1.19 -n 1 -s bug
> $ cat > schema.cql << 'EOF'
> CREATE KEYSPACE test WITH replication = {'class': 'SimpleStrategy',
> 'replication_factor': '1'} AND durable_writes = true;
> CREATE TABLE test.unknowntest (id int primary key, payload text, "paylo!d"
> text);
> EOF
> $ ccm node1 cqlsh -f schema.cql
> $ export CASSANDRA_INCLUDE=~/.ccm/bug/node1/bin/cassandra.in.sh
> $ cat > bug.json << 'EOF'
> [
> {"key": "1",
> "cells": [["","",1554432501209207],
> ["paylo!d","hello world",1554432501209207],
> ["payload","hello world",1554432501209207]]}
> ]
> EOF
> $ ~/.ccm/repository/2.1.19/tools/bin/json2sstable -K test -c unknowntest
> ~/bug.json
> ~/.ccm/bug/node1/data0/test/unknowntest-/test-unknowntest-ka-1-Data.db{noformat}
> Then test the behavior of unknown columns in 2.1:
> {noformat}
> $ ccm stop
> $ ccm create -v 2.1.19 -n 1 -s bug2_1_19
> $ cat > schema2.cql << 'EOF'
> CREATE KEYSPACE test WITH replication = {'class': 'SimpleStrategy',
> 'replication_factor': '1'} AND durable_writes = true;
> CREATE TABLE test.unknowntest (id int primary key, payload text);
> EOF
> $ ccm node1 cqlsh -f schema2.cql
> $ ccm stop
> $ cp ~/.ccm/bug/node1/data0/test/unknowntest-/test-unknowntest-ka-1-*
> ~/.ccm/bug2_1_19/node1/data0/test/unknowntest-/
> $ ccm start
> $ ccm node1 cqlsh
> Connected to bug2_1_19 at 127.0.0.1:9042.
> [cqlsh 5.0.1 | Cassandra 2.1.19 | CQL spec 3.2.1 | Native protocol v3]
> Use HELP for help.
> cqlsh> select * from test.unknowntest where id = 1;
> id | payload
> +-
> 1 | hello world
> (1 rows){noformat}
> Compared to 3.11.4 which did the following:
> {noformat}
> $ ccm stop
> $ ccm create -v 3.11.4 -n 1 -s bug3_11_4
> $ ccm node1 cqlsh -f schema2.cql
> $ ccm stop
> $ cp ~/.ccm/bug/node1/data0/test/unknowntest-/test-unknowntest-ka-1-*
> ~/.ccm/bug3_11_4/node1/data0/test/unknowntest-/
> $ ccm start
> $ ccm node1 cqlsh
> Connected to bug3_11_4 at 127.0.0.1:9042.
> [cqlsh 5.0.1 | Cassandra 3.11.4 | CQL spec 3.4.4 | Native protocol v4]
> Use HELP for help.
> cqlsh> select * from test.unknowntest where id = 1;
> ReadFailure: Error from server: code=1300 [Replica(s) failed to execute read]
> message="Operation failed - received 0 responses and 1 failures"
> info={'failures': 1, 'received_responses': 0, 'required_responses': 1,
> 'consistency': 'ONE'}
> {noformat}
> In the logs this resulted in an IllegalStateException from LegacyLayout line
> 1127
> The expected behavior would be to ignore the column and return results the
> same as in 2.1
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15430) Cassandra 3.0.18: BatchMessage.execute - 10x more on-heap allocations compared to 2.1.18
[
https://issues.apache.org/jira/browse/CASSANDRA-15430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Steinmaurer updated CASSANDRA-15430:
---
Description:
In a 6 node loadtest cluster, we have been running with 2.1.18 a certain
production-like workload constantly and sufficiently. After upgrading one node
to 3.0.18 (remaining 5 still on 2.1.18 after we have seen that sort of
regression described below), 3.0.18 is showing increased CPU usage, increase
GC, high mutation stage pending tasks, dropped mutation messages ...
Some spec. All 6 nodes equally sized:
* Bare metal, 32 physical cores, 512G RAM
* Xmx31G, G1, max pause millis = 2000ms
* cassandra.yaml basically unchanged, thus same settings in regard to number
of threads, compaction throttling etc.
Following dashboard shows highlighted areas (CPU, suspension) with metrics for
all 6 nodes and the one outlier being the node upgraded to Cassandra 3.0.18.
!dashboard.png|width=1280!
Additionally we see a large increase on pending tasks in the mutation stage
after the upgrade:
!mutation_stage.png!
And dropped mutation messages, also confirmed in the Cassandra log:
{noformat}
INFO [ScheduledTasks:1] 2019-11-15 08:24:24,780 MessagingService.java:1022 -
MUTATION messages were dropped in last 5000 ms: 41552 for internal timeout and
0 for cross node timeout
INFO [ScheduledTasks:1] 2019-11-15 08:24:25,157 StatusLogger.java:52 - Pool
NameActive Pending Completed Blocked All Time
Blocked
INFO [ScheduledTasks:1] 2019-11-15 08:24:25,168 StatusLogger.java:56 -
MutationStage 256 81824 3360532756 0
0
INFO [ScheduledTasks:1] 2019-11-15 08:24:25,168 StatusLogger.java:56 -
ViewMutationStage 0 0 0 0
0
INFO [ScheduledTasks:1] 2019-11-15 08:24:25,168 StatusLogger.java:56 -
ReadStage 0 0 62862266 0
0
INFO [ScheduledTasks:1] 2019-11-15 08:24:25,169 StatusLogger.java:56 -
RequestResponseStage 0 0 2176659856 0
0
INFO [ScheduledTasks:1] 2019-11-15 08:24:25,169 StatusLogger.java:56 -
ReadRepairStage 0 0 0 0
0
INFO [ScheduledTasks:1] 2019-11-15 08:24:25,169 StatusLogger.java:56 -
CounterMutationStage 0 0 0 0
0
...
{noformat}
Judging from a 15min JFR session for both, 3.0.18 and 2.1.18 on a different
node, high-level, it looks like the code path underneath
{{BatchMessage.execute}} is producing ~ 10x more on-heap allocations in 3.0.18
compared to 2.1.18.
!jfr_allocations.png!
Left => 3.0.18
Right => 2.1.18
JFRs zipped are exceeding the 60MB limit to directly attach to the ticket. I
can upload them, if there is another destination available.
was:
In a 6 node loadtest cluster, we have been running with 2.1.18 a certain
production-like workload constantly and sufficiently. After upgrading one node
to 3.0.18 (remaining 5 still on 2.1.18 after we have seen that sort of
regression described below), 3.0.18 is showing increased CPU usage, increase
GC, high mutation stage pending tasks, dropped mutation messages ...
Some spec. All 6 nodes equally sized:
* Bare metal, 32 physical cores, 512G RAM
* Xmx31G, G1, max pause millis = 2000ms
* cassandra.yaml basically unchanged, thus some settings in regard to number
of threads, compaction throttling etc.
Following dashboard shows highlighted areas (CPU, suspension) with metrics for
all 6 nodes and the one outlier being the node upgraded to Cassandra 3.0.18.
!dashboard.png|width=1280!
Additionally we see a large increase on pending tasks in the mutation stage
after the upgrade:
!mutation_stage.png!
And dropped mutation messages, also confirmed in the Cassandra log:
{noformat}
INFO [ScheduledTasks:1] 2019-11-15 08:24:24,780 MessagingService.java:1022 -
MUTATION messages were dropped in last 5000 ms: 41552 for internal timeout and
0 for cross node timeout
INFO [ScheduledTasks:1] 2019-11-15 08:24:25,157 StatusLogger.java:52 - Pool
NameActive Pending Completed Blocked All Time
Blocked
INFO [ScheduledTasks:1] 2019-11-15 08:24:25,168 StatusLogger.java:56 -
MutationStage 256 81824 3360532756 0
0
INFO [ScheduledTasks:1] 2019-11-15 08:24:25,168 StatusLogger.java:56 -
ViewMutationStage 0 0 0 0
0
INFO [ScheduledTasks:1] 2019-11-15 08:24:25,168 StatusLogger.java:56 -
ReadStage 0 0 62862266 0
0
INFO [ScheduledTasks:1] 2019-11-15 08:24:25,169 StatusLogger.java:56 -
RequestResponseStage 0 0 2176659856
[jira] [Comment Edited] (CASSANDRA-14365) Commit log replay failure for static columns with collections in clustering keys
[
https://issues.apache.org/jira/browse/CASSANDRA-14365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16975056#comment-16975056
]
Michael Semb Wever edited comment on CASSANDRA-14365 at 11/15/19 1:24 PM:
--
||branch||circleci||asf jenkins tests||asf jenkins dtests||
|[cassandra-2.2_14365|https://github.com/apache/cassandra/compare/trunk...vincewhite:zero_length_collection]|[circleci|https://circleci.com/workflow-run/d500cc5f-1d87-4beb-815e-9931f8e84d95]|[!https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-pipeline/29//badge/icon!|https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-pipeline/29/]|[!https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/703//badge/icon!|https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/703]|
[~VincentWhite], can you update the unit test on a cassandra-3.0 based branch
as well…
you see can
[here|https://github.com/thelastpickle/cassandra/commit/af2ca79b1001150d39a6b97cfadc0ce97f4e99b4]
the api the tests use have changed a bit.
was (Author: michaelsembwever):
||branch||circleci||asf jenkins tests||asf jenkins dtests||
|[cassandra-2.2_14365|https://github.com/apache/cassandra/compare/trunk...vincewhite:zero_length_collection]|[circleci|https://circleci.com/workflow-run/d500cc5f-1d87-4beb-815e-9931f8e84d95]|[!https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-pipeline/29//badge/icon!|https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-pipeline/29/]|[!https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/703//badge/icon!|https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/703]|
| wip… | | | |
| wip… | | | |
| wip… | | | |
> Commit log replay failure for static columns with collections in clustering
> keys
>
>
> Key: CASSANDRA-14365
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14365
> Project: Cassandra
> Issue Type: Bug
> Components: Legacy/Core
>Reporter: Vincent White
>Assignee: Vincent White
>Priority: Normal
>
> In the old storage engine, static cells with a collection as part of the
> clustering key fail to validate because a 0 byte collection (like in the cell
> name of a static cell) isn't valid.
> To reproduce:
> 1.
> {code:java}
> CREATE TABLE test.x (
> id int,
> id2 frozen>,
> st int static,
> PRIMARY KEY (id, id2)
> );
> INSERT INTO test.x (id, st) VALUES (1, 2);
> {code}
> 2.
> Kill the cassandra process
> 3.
> Restart cassandra to replay the commitlog
> Outcome:
> {noformat}
> ERROR [main] 2018-04-05 04:58:23,741 JVMStabilityInspector.java:99 - Exiting
> due to error while processing commit log during initialization.
> org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException:
> Unexpected error deserializing mutation; saved to
> /tmp/mutation3825739904516830950dat. This may be caused by replaying a
> mutation against a table with the same name but incompatible schema.
> Exception follows: org.apache.cassandra.serializers.MarshalException: Not
> enough bytes to read a set
> at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.handleReplayError(CommitLogReplayer.java:638)
> [main/:na]
> at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.replayMutation(CommitLogReplayer.java:565)
> [main/:na]
> at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.replaySyncSection(CommitLogReplayer.java:517)
> [main/:na]
> at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:397)
> [main/:na]
> at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:143)
> [main/:na]
> at
> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:181)
> [main/:na]
> at
> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:161)
> [main/:na]
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:284)
> [main/:na]
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:533)
> [main/:na]
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:642)
> [main/:na]
> {noformat}
> I haven't investigated if there are other more subtle issues caused by these
> cells failing to validate other places in the code, but I believe the fix for
> this is to check for 0 byte length collections and accept them as valid as we
> do with other types.
> I haven't had a chance for any extensive testing but this naive patch seems
> to have the desired affect.
> ||Patch||
> |[2.2
>
[jira] [Created] (CASSANDRA-15430) Cassandra 3.0.18: BatchMessage.execute - 10x more on-heap allocations compared to 2.1.18
Thomas Steinmaurer created CASSANDRA-15430:
--
Summary: Cassandra 3.0.18: BatchMessage.execute - 10x more on-heap
allocations compared to 2.1.18
Key: CASSANDRA-15430
URL: https://issues.apache.org/jira/browse/CASSANDRA-15430
Project: Cassandra
Issue Type: Bug
Reporter: Thomas Steinmaurer
Attachments: dashboard.png, jfr_allocations.png, mutation_stage.png
In a 6 node loadtest cluster, we have been running with 2.1.18 a certain
production-like workload constantly and sufficiently. After upgrading one node
to 3.0.18 (remaining 5 still on 2.1.18 after we have seen that sort of
regression described below), 3.0.18 is showing increased CPU usage, increase
GC, high mutation stage pending tasks, dropped mutation messages ...
Some spec. All 6 nodes equally sized:
* Bare metal, 32 physical cores, 512G RAM
* Xmx31G, G1, max pause millis = 2000ms
* cassandra.yaml basically unchanged, thus some settings in regard to number of
threads, compaction throttling etc.
Following dashboard shows highlighted areas (CPU, suspension) with metrics for
all 6 nodes and the outlier being the node upgraded to Cassandra 3.0.18.
!dashboard.png|width=1280!
Additionally we see a large increase on pending tasks in the mutation stage
after the upgrade:
!mutation_stage.png!
And dropped mutation messages, also confirmed in the Cassandra log:
{noformat}
INFO [ScheduledTasks:1] 2019-11-15 08:24:24,780 MessagingService.java:1022 -
MUTATION messages were dropped in last 5000 ms: 41552 for internal timeout and
0 for cross node timeout
INFO [ScheduledTasks:1] 2019-11-15 08:24:25,157 StatusLogger.java:52 - Pool
NameActive Pending Completed Blocked All Time
Blocked
INFO [ScheduledTasks:1] 2019-11-15 08:24:25,168 StatusLogger.java:56 -
MutationStage 256 81824 3360532756 0
0
INFO [ScheduledTasks:1] 2019-11-15 08:24:25,168 StatusLogger.java:56 -
ViewMutationStage 0 0 0 0
0
INFO [ScheduledTasks:1] 2019-11-15 08:24:25,168 StatusLogger.java:56 -
ReadStage 0 0 62862266 0
0
INFO [ScheduledTasks:1] 2019-11-15 08:24:25,169 StatusLogger.java:56 -
RequestResponseStage 0 0 2176659856 0
0
INFO [ScheduledTasks:1] 2019-11-15 08:24:25,169 StatusLogger.java:56 -
ReadRepairStage 0 0 0 0
0
INFO [ScheduledTasks:1] 2019-11-15 08:24:25,169 StatusLogger.java:56 -
CounterMutationStage 0 0 0 0
0
...
{noformat}
Judging from a 15min JFR session for both, 3.0.18 and 2.1.18 on a different
node, high-level, it looks like the code path underneath
{{BatchMessage.execute}} is producing ~ 10x more on-heap allocations in 3.0.18
compared to 2.1.18.
!jfr_allocations.png!
Left => 3.0.18
Right => 2.1.18
JFRs zipped are exceeding the 60MB limit to directly attach to the ticket. I
can upload them, if there is another destination available.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15430) Cassandra 3.0.18: BatchMessage.execute - 10x more on-heap allocations compared to 2.1.18
[
https://issues.apache.org/jira/browse/CASSANDRA-15430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Steinmaurer updated CASSANDRA-15430:
---
Description:
In a 6 node loadtest cluster, we have been running with 2.1.18 a certain
production-like workload constantly and sufficiently. After upgrading one node
to 3.0.18 (remaining 5 still on 2.1.18 after we have seen that sort of
regression described below), 3.0.18 is showing increased CPU usage, increase
GC, high mutation stage pending tasks, dropped mutation messages ...
Some spec. All 6 nodes equally sized:
* Bare metal, 32 physical cores, 512G RAM
* Xmx31G, G1, max pause millis = 2000ms
* cassandra.yaml basically unchanged, thus some settings in regard to number
of threads, compaction throttling etc.
Following dashboard shows highlighted areas (CPU, suspension) with metrics for
all 6 nodes and the one outlier being the node upgraded to Cassandra 3.0.18.
!dashboard.png|width=1280!
Additionally we see a large increase on pending tasks in the mutation stage
after the upgrade:
!mutation_stage.png!
And dropped mutation messages, also confirmed in the Cassandra log:
{noformat}
INFO [ScheduledTasks:1] 2019-11-15 08:24:24,780 MessagingService.java:1022 -
MUTATION messages were dropped in last 5000 ms: 41552 for internal timeout and
0 for cross node timeout
INFO [ScheduledTasks:1] 2019-11-15 08:24:25,157 StatusLogger.java:52 - Pool
NameActive Pending Completed Blocked All Time
Blocked
INFO [ScheduledTasks:1] 2019-11-15 08:24:25,168 StatusLogger.java:56 -
MutationStage 256 81824 3360532756 0
0
INFO [ScheduledTasks:1] 2019-11-15 08:24:25,168 StatusLogger.java:56 -
ViewMutationStage 0 0 0 0
0
INFO [ScheduledTasks:1] 2019-11-15 08:24:25,168 StatusLogger.java:56 -
ReadStage 0 0 62862266 0
0
INFO [ScheduledTasks:1] 2019-11-15 08:24:25,169 StatusLogger.java:56 -
RequestResponseStage 0 0 2176659856 0
0
INFO [ScheduledTasks:1] 2019-11-15 08:24:25,169 StatusLogger.java:56 -
ReadRepairStage 0 0 0 0
0
INFO [ScheduledTasks:1] 2019-11-15 08:24:25,169 StatusLogger.java:56 -
CounterMutationStage 0 0 0 0
0
...
{noformat}
Judging from a 15min JFR session for both, 3.0.18 and 2.1.18 on a different
node, high-level, it looks like the code path underneath
{{BatchMessage.execute}} is producing ~ 10x more on-heap allocations in 3.0.18
compared to 2.1.18.
!jfr_allocations.png!
Left => 3.0.18
Right => 2.1.18
JFRs zipped are exceeding the 60MB limit to directly attach to the ticket. I
can upload them, if there is another destination available.
was:
In a 6 node loadtest cluster, we have been running with 2.1.18 a certain
production-like workload constantly and sufficiently. After upgrading one node
to 3.0.18 (remaining 5 still on 2.1.18 after we have seen that sort of
regression described below), 3.0.18 is showing increased CPU usage, increase
GC, high mutation stage pending tasks, dropped mutation messages ...
Some spec. All 6 nodes equally sized:
* Bare metal, 32 physical cores, 512G RAM
* Xmx31G, G1, max pause millis = 2000ms
* cassandra.yaml basically unchanged, thus some settings in regard to number of
threads, compaction throttling etc.
Following dashboard shows highlighted areas (CPU, suspension) with metrics for
all 6 nodes and the outlier being the node upgraded to Cassandra 3.0.18.
!dashboard.png|width=1280!
Additionally we see a large increase on pending tasks in the mutation stage
after the upgrade:
!mutation_stage.png!
And dropped mutation messages, also confirmed in the Cassandra log:
{noformat}
INFO [ScheduledTasks:1] 2019-11-15 08:24:24,780 MessagingService.java:1022 -
MUTATION messages were dropped in last 5000 ms: 41552 for internal timeout and
0 for cross node timeout
INFO [ScheduledTasks:1] 2019-11-15 08:24:25,157 StatusLogger.java:52 - Pool
NameActive Pending Completed Blocked All Time
Blocked
INFO [ScheduledTasks:1] 2019-11-15 08:24:25,168 StatusLogger.java:56 -
MutationStage 256 81824 3360532756 0
0
INFO [ScheduledTasks:1] 2019-11-15 08:24:25,168 StatusLogger.java:56 -
ViewMutationStage 0 0 0 0
0
INFO [ScheduledTasks:1] 2019-11-15 08:24:25,168 StatusLogger.java:56 -
ReadStage 0 0 62862266 0
0
INFO [ScheduledTasks:1] 2019-11-15 08:24:25,169 StatusLogger.java:56 -
RequestResponseStage 0 0 2176659856 0
[jira] [Commented] (CASSANDRA-15410) Avoid over-allocation of bytes for UTF8 string serialization
[
https://issues.apache.org/jira/browse/CASSANDRA-15410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16975077#comment-16975077
]
Aleksey Yeschenko commented on CASSANDRA-15410:
---
I don't want to trust {{encodedSize()}} implementations, really (we had
mismatch bugs in those calculations before).
Now, if you inspect the usages of {{CBUtil#writeString()}} you'll notice that
absolute most of those calls are performed strictly on ASCII strings. Keyspace
names, table names, type names, function names, field names, enums - that can
*only* be ASCII due to our grammar limitations.
Of the strings in messages C* server writes - majority being in
{{ResultSet.ResultMetadata}} - none of the strings ever go beyond ASCII. So we
could introduce a special-case {{CBUtil#writeASCIIString()}} to use
{{ByteBufUtil#writeAscii()}} and use that throughout, that should resolve the
issue. And you'll get a further speed bump from the loop removal in ascii
string size calculation in all the {{encodedSize()}} methods.
> Avoid over-allocation of bytes for UTF8 string serialization
> -
>
> Key: CASSANDRA-15410
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15410
> Project: Cassandra
> Issue Type: Improvement
> Components: Messaging/Client
>Reporter: Yifan Cai
>Assignee: Yifan Cai
>Priority: Normal
> Fix For: 4.0
>
>
> In the current message encoding implementation, it first calculates the
> `encodeSize` and allocates the bytebuffer with that size.
> However, during encoding, it assumes the worst case of writing UTF8 string to
> allocate bytes, i.e. assuming each letter takes 3 bytes.
> The over-estimation further leads to resizing the underlying array and data
> copy.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14365) Commit log replay failure for static columns with collections in clustering keys
[
https://issues.apache.org/jira/browse/CASSANDRA-14365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16975056#comment-16975056
]
Michael Semb Wever commented on CASSANDRA-14365:
||branch||circleci||asf jenkins tests||asf jenkins dtests||
|[cassandra-2.2_14365|https://github.com/apache/cassandra/compare/trunk...vincewhite:zero_length_collection]|[circleci|https://circleci.com/workflow-run/d500cc5f-1d87-4beb-815e-9931f8e84d95]|[!https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-pipeline/29//badge/icon!|https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-pipeline/29/]|[!https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/703//badge/icon!|https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/703]|
| wip… | | | |
| wip… | | | |
| wip… | | | |
> Commit log replay failure for static columns with collections in clustering
> keys
>
>
> Key: CASSANDRA-14365
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14365
> Project: Cassandra
> Issue Type: Bug
> Components: Legacy/Core
>Reporter: Vincent White
>Assignee: Vincent White
>Priority: Normal
>
> In the old storage engine, static cells with a collection as part of the
> clustering key fail to validate because a 0 byte collection (like in the cell
> name of a static cell) isn't valid.
> To reproduce:
> 1.
> {code:java}
> CREATE TABLE test.x (
> id int,
> id2 frozen>,
> st int static,
> PRIMARY KEY (id, id2)
> );
> INSERT INTO test.x (id, st) VALUES (1, 2);
> {code}
> 2.
> Kill the cassandra process
> 3.
> Restart cassandra to replay the commitlog
> Outcome:
> {noformat}
> ERROR [main] 2018-04-05 04:58:23,741 JVMStabilityInspector.java:99 - Exiting
> due to error while processing commit log during initialization.
> org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException:
> Unexpected error deserializing mutation; saved to
> /tmp/mutation3825739904516830950dat. This may be caused by replaying a
> mutation against a table with the same name but incompatible schema.
> Exception follows: org.apache.cassandra.serializers.MarshalException: Not
> enough bytes to read a set
> at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.handleReplayError(CommitLogReplayer.java:638)
> [main/:na]
> at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.replayMutation(CommitLogReplayer.java:565)
> [main/:na]
> at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.replaySyncSection(CommitLogReplayer.java:517)
> [main/:na]
> at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:397)
> [main/:na]
> at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:143)
> [main/:na]
> at
> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:181)
> [main/:na]
> at
> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:161)
> [main/:na]
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:284)
> [main/:na]
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:533)
> [main/:na]
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:642)
> [main/:na]
> {noformat}
> I haven't investigated if there are other more subtle issues caused by these
> cells failing to validate other places in the code, but I believe the fix for
> this is to check for 0 byte length collections and accept them as valid as we
> do with other types.
> I haven't had a chance for any extensive testing but this naive patch seems
> to have the desired affect.
> ||Patch||
> |[2.2
> PoC|https://github.com/vincewhite/cassandra/commits/zero_length_collection]|
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14365) Commit log replay failure for static columns with collections in clustering keys
[
https://issues.apache.org/jira/browse/CASSANDRA-14365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Semb Wever updated CASSANDRA-14365:
---
Reviewers: Michael Semb Wever, Michael Semb Wever (was: Michael Semb Wever)
Michael Semb Wever, Michael Semb Wever
Status: Review In Progress (was: Patch Available)
> Commit log replay failure for static columns with collections in clustering
> keys
>
>
> Key: CASSANDRA-14365
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14365
> Project: Cassandra
> Issue Type: Bug
> Components: Legacy/Core
>Reporter: Vincent White
>Assignee: Vincent White
>Priority: Normal
>
> In the old storage engine, static cells with a collection as part of the
> clustering key fail to validate because a 0 byte collection (like in the cell
> name of a static cell) isn't valid.
> To reproduce:
> 1.
> {code:java}
> CREATE TABLE test.x (
> id int,
> id2 frozen>,
> st int static,
> PRIMARY KEY (id, id2)
> );
> INSERT INTO test.x (id, st) VALUES (1, 2);
> {code}
> 2.
> Kill the cassandra process
> 3.
> Restart cassandra to replay the commitlog
> Outcome:
> {noformat}
> ERROR [main] 2018-04-05 04:58:23,741 JVMStabilityInspector.java:99 - Exiting
> due to error while processing commit log during initialization.
> org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException:
> Unexpected error deserializing mutation; saved to
> /tmp/mutation3825739904516830950dat. This may be caused by replaying a
> mutation against a table with the same name but incompatible schema.
> Exception follows: org.apache.cassandra.serializers.MarshalException: Not
> enough bytes to read a set
> at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.handleReplayError(CommitLogReplayer.java:638)
> [main/:na]
> at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.replayMutation(CommitLogReplayer.java:565)
> [main/:na]
> at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.replaySyncSection(CommitLogReplayer.java:517)
> [main/:na]
> at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:397)
> [main/:na]
> at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:143)
> [main/:na]
> at
> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:181)
> [main/:na]
> at
> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:161)
> [main/:na]
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:284)
> [main/:na]
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:533)
> [main/:na]
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:642)
> [main/:na]
> {noformat}
> I haven't investigated if there are other more subtle issues caused by these
> cells failing to validate other places in the code, but I believe the fix for
> this is to check for 0 byte length collections and accept them as valid as we
> do with other types.
> I haven't had a chance for any extensive testing but this naive patch seems
> to have the desired affect.
> ||Patch||
> |[2.2
> PoC|https://github.com/vincewhite/cassandra/commits/zero_length_collection]|
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
