[jira] [Commented] (CASSANDRA-15350) Add CAS “uncertainty” and “contention" messages that are currently propagated as a WriteTimeoutException.

[Yifan Cai (Jira)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-15350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027916#comment-17027916
 ] 

Yifan Cai commented on CASSANDRA-15350:
---

Thanks Dinesh. Applied the most of the changes you suggested. 

> Add CAS “uncertainty” and “contention" messages that are currently propagated 
> as a WriteTimeoutException.
> -
>
> Key: CASSANDRA-15350
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15350
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Feature/Lightweight Transactions
>Reporter: Alex Petrov
>Assignee: Yifan Cai
>Priority: Normal
>  Labels: protocolv5, pull-request-available
> Attachments: Utf8StringEncodeBench.java
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Right now, CAS uncertainty introduced in 
> https://issues.apache.org/jira/browse/CASSANDRA-6013 is propagating as 
> WriteTimeout. One of this conditions it manifests is when there’s at least 
> one acceptor that has accepted the value, which means that this value _may_ 
> still get accepted during the later round, despite the proposer failure. 
> Similar problem happens with CAS contention, which is also indistinguishable 
> from the “regular” timeout, even though it is visible in metrics correctly.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-15350) Add CAS “uncertainty” and “contention" messages that are currently propagated as a WriteTimeoutException.

[Dinesh Joshi (Jira)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-15350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027909#comment-17027909
 ] 

Dinesh Joshi commented on CASSANDRA-15350:
--

I think the overall PR looks good. I just had a few minor nits that I've mocked 
[here|https://github.com/yifan-c/cassandra/commit/b9a58b38cf9ce85203021bd48f89d98d0f69acd5].
 Please feel free to cherry pick.

> Add CAS “uncertainty” and “contention" messages that are currently propagated 
> as a WriteTimeoutException.
> -
>
> Key: CASSANDRA-15350
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15350
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Feature/Lightweight Transactions
>Reporter: Alex Petrov
>Assignee: Yifan Cai
>Priority: Normal
>  Labels: protocolv5, pull-request-available
> Attachments: Utf8StringEncodeBench.java
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Right now, CAS uncertainty introduced in 
> https://issues.apache.org/jira/browse/CASSANDRA-6013 is propagating as 
> WriteTimeout. One of this conditions it manifests is when there’s at least 
> one acceptor that has accepted the value, which means that this value _may_ 
> still get accepted during the later round, despite the proposer failure. 
> Similar problem happens with CAS contention, which is also indistinguishable 
> from the “regular” timeout, even though it is visible in metrics correctly.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14970) New releases must supply SHA-256 and/or SHA-512 checksums

[Michael Semb Wever (Jira)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027871#comment-17027871
 ] 

Michael Semb Wever commented on CASSANDRA-14970:


In addition to the cassandra-builds 
[patch|https://github.com/apache/cassandra-builds/compare/master...thelastpickle:mck/14970_sha512-checksums],
 there is an added [patch for updated 
documentation|https://github.com/apache/cassandra/compare/trunk...thelastpickle:mck/trunk_14970_docs].

> New releases must supply SHA-256 and/or SHA-512 checksums
> -
>
> Key: CASSANDRA-14970
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14970
> Project: Cassandra
>  Issue Type: Bug
>  Components: Packaging
>Reporter: Michael Shuler
>Assignee: Michael Semb Wever
>Priority: Urgent
> Fix For: 2.2.16, 3.0.20, 3.11.6, 4.0
>
> Attachments: 
> 0001-Update-downloads-for-sha256-sha512-checksum-files.patch, 
> 0001-Update-release-checksum-algorithms-to-SHA-256-SHA-512.patch, 
> ant-publish-checksum-fail.jpg, build_cassandra-2.1.png, build_trunk.png, 
> cassandra-2.1_14970_updated.patch
>
>
> Release policy was updated around 9/2018 to state:
> "For new releases, PMCs MUST supply SHA-256 and/or SHA-512; and SHOULD NOT 
> supply MD5 or SHA-1. Existing releases do not need to be changed."
> build.xml needs to be updated from MD5 & SHA-1 to, at least, SHA-256 or both. 
> cassandra-builds/cassandra-release scripts need to be updated to work with 
> the new checksum files.
> http://www.apache.org/dev/release-distribution#sigs-and-sums



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13994) Remove COMPACT STORAGE internals before 4.0 release

[Ekaterina Dimitrova (Jira)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027865#comment-17027865
 ] 

Ekaterina Dimitrova commented on CASSANDRA-13994:
-

Hi [~jjirsa] and [~ifesdjeen],

Maybe I can help with this one?

> Remove COMPACT STORAGE internals before 4.0 release
> ---
>
> Key: CASSANDRA-13994
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13994
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Legacy/Local Write-Read Paths
>Reporter: Alex Petrov
>Assignee: Alex Petrov
>Priority: Low
> Fix For: 4.0, 4.0-rc
>
>
> 4.0 comes without thrift (after [CASSANDRA-5]) and COMPACT STORAGE (after 
> [CASSANDRA-10857]), and since Compact Storage flags are now disabled, all of 
> the related functionality is useless.
> There are still some things to consider:
> 1. One of the system tables (built indexes) was compact. For now, we just 
> added {{value}} column to it to make sure it's backwards-compatible, but we 
> might want to make sure it's just a "normal" table and doesn't have redundant 
> columns.
> 2. Compact Tables were building indexes in {{KEYS}} mode. Removing it is 
> trivial, but this would mean that all built indexes will be defunct. We could 
> log a warning for now and ask users to migrate off those for now and 
> completely remove it from future releases. It's just a couple of classes 
> though.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[cassandra-builds] branch master updated: Compress Jenkins build logs

[mck]

This is an automated email from the ASF dual-hosted git repository.

mck pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/cassandra-builds.git


The following commit(s) were added to refs/heads/master by this push:
 new 41261e3  Compress Jenkins build logs
41261e3 is described below

commit 41261e3351dc79a0af6ca5688fb889859d912b85
Author: mck 
AuthorDate: Fri Jan 31 23:02:24 2020 +0100

Compress Jenkins build logs
---
 jenkins-dsl/cassandra_job_dsl_seed.groovy | 10 ++
 1 file changed, 10 insertions(+)

diff --git a/jenkins-dsl/cassandra_job_dsl_seed.groovy 
b/jenkins-dsl/cassandra_job_dsl_seed.groovy
index 36a2c7b..cf6eabf 100644
--- a/jenkins-dsl/cassandra_job_dsl_seed.groovy
+++ b/jenkins-dsl/cassandra_job_dsl_seed.groovy
@@ -72,6 +72,7 @@ job('Cassandra-template-artifacts') {
 description(jobDescription)
 jdk(jdkLabel)
 label(slaveLabel)
+compressBuildLog()
 logRotator {
 numToKeep(25)
 artifactNumToKeep(1)
@@ -144,6 +145,7 @@ job('Cassandra-template-test') {
 description(jobDescription)
 jdk(jdkLabel)
 label(slaveLabel)
+compressBuildLog()
 logRotator {
 numToKeep(25)
 artifactNumToKeep(1)
@@ -204,6 +206,7 @@ job('Cassandra-template-dtest') {
 description(jobDescription)
 jdk(jdkLabel)
 label(slaveLabel)
+compressBuildLog()
 logRotator {
 numToKeep(25)
 artifactNumToKeep(1)
@@ -262,6 +265,7 @@ job('Cassandra-template-dtest') {
 matrixJob('Cassandra-template-cqlsh-tests') {
 disabled(true)
 description(jobDescription)
+compressBuildLog()
 logRotator {
 numToKeep(25)
 artifactNumToKeep(1)
@@ -425,6 +429,7 @@ cassandraBranches.each {
  */
 pipelineJob("${jobNamePrefix}") {
 description(jobDescription)
+compressBuildLog()
 logRotator {
 numToKeep(25)
 artifactNumToKeep(1)
@@ -464,6 +469,7 @@ job('Cassandra-devbranch-artifacts') {
 description(jobDescription)
 jdk(jdkLabel)
 label(slaveLabel)
+compressBuildLog()
 logRotator {
 numToKeep(25)
 artifactNumToKeep(1)
@@ -520,6 +526,7 @@ testTargets.each {
 concurrentBuild()
 jdk(jdkLabel)
 label(slaveLabel)
+compressBuildLog()
 logRotator {
 numToKeep(25)
 artifactNumToKeep(1)
@@ -586,6 +593,7 @@ job('Cassandra-devbranch-dtest') {
 concurrentBuild()
 jdk(jdkLabel)
 label(slaveLabel)
+compressBuildLog()
 logRotator {
 numToKeep(25)
 artifactNumToKeep(1)
@@ -653,6 +661,7 @@ job('Cassandra-devbranch-dtest') {
 matrixJob('Cassandra-devbranch-cqlsh-tests') {
 description(jobDescription)
 concurrentBuild()
+compressBuildLog()
 logRotator {
 numToKeep(25)
 artifactNumToKeep(1)
@@ -726,6 +735,7 @@ matrixJob('Cassandra-devbranch-cqlsh-tests') {
  */
 pipelineJob('Cassandra-devbranch') {
 description(jobDescription)
+compressBuildLog()
 logRotator {
 numToKeep(25)
 artifactNumToKeep(1)


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Comment Edited] (CASSANDRA-15350) Add CAS “uncertainty” and “contention" messages that are currently propagated as a WriteTimeoutException.

[Yifan Cai (Jira)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-15350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027827#comment-17027827
 ] 

Yifan Cai edited comment on CASSANDRA-15350 at 1/31/20 9:41 PM:


Thanks Alex. 

Using fuzz to reproduce contentions sounds OK to me.

Having deterministic test cases to precisely produce contentions is also 
valuable, IMO. But I am good to not have those test cases being merged as of 
now. 

I rebased my branch with your improvements commit and added/tweaked test cases 
in {{CasWriteTest}}.

Still the same [PR|https://github.com/apache/cassandra/pull/379] and [code 
branch|https://github.com/yifan-c/cassandra/tree/cas-exception-changes]. Test 
result is 
[here.|https://app.circleci.com/github/yifan-c/cassandra/pipelines/8d10bf4b-9f36-468e-922c-18a6dcf7df81/workflows/d20b5ae8-cd82-49cb-9026-97644359ee76]
 


was (Author: yifanc):
Thanks Alex. 

Using fuzz to reproduce contentions sounds OK to me.

Having deterministic test cases to precisely produce contentions is also 
valuable, IMO. But I am good to not have those test cases being merged as of 
now. 

I rebased my branch with your improvements commit and added/tweaked test cases 
in {{CasWriteTest}}.

Still the same [PR|https://github.com/apache/cassandra/pull/379] and [code 
branch|https://github.com/yifan-c/cassandra/tree/cas-exception-changes]. Test 
result is 
[here.|https://app.circleci.com/github/yifan-c/cassandra/pipelines/458cafcb-117b-443a-ae6c-c089efd6a929/workflows/7742c20d-1a82-47a8-93d0-cc40a542db5f]
 

> Add CAS “uncertainty” and “contention" messages that are currently propagated 
> as a WriteTimeoutException.
> -
>
> Key: CASSANDRA-15350
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15350
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Feature/Lightweight Transactions
>Reporter: Alex Petrov
>Assignee: Yifan Cai
>Priority: Normal
>  Labels: protocolv5, pull-request-available
> Attachments: Utf8StringEncodeBench.java
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Right now, CAS uncertainty introduced in 
> https://issues.apache.org/jira/browse/CASSANDRA-6013 is propagating as 
> WriteTimeout. One of this conditions it manifests is when there’s at least 
> one acceptor that has accepted the value, which means that this value _may_ 
> still get accepted during the later round, despite the proposer failure. 
> Similar problem happens with CAS contention, which is also indistinguishable 
> from the “regular” timeout, even though it is visible in metrics correctly.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Issue Comment Deleted] (CASSANDRA-14737) Limit the dependencies used by UDFs/UDAs

[Ryan Svihla (Jira)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ryan Svihla updated CASSANDRA-14737:

Comment: was deleted

(was: Changes look valid and the tests are passing +1)

> Limit the dependencies used by UDFs/UDAs
> 
>
> Key: CASSANDRA-14737
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14737
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Legacy/CQL
>Reporter: Robert Stupp
>Assignee: Robert Stupp
>Priority: Low
>  Labels: UDF
> Fix For: 4.0
>
>
> In an effort to clean up our hygiene and limit the dependencies used by 
> UDFs/UDAs, I think we should refactor the UDF code parts and remove the 
> dependency to the Java Driver in that area without breaking existing 
> UDFs/UDAs.
>   
> The patch is in [this 
> branch|https://github.com/snazy/cassandra/tree/feature/remove-udf-driver-dep-trunk].
>  The changes are rather trivial and provide 100% backwards compatibility for 
> existing UDFs.
>   
>  The prototype copies the necessary parts from the Java Driver into the C* 
> source tree to {{org.apache.cassandra.cql3.functions.types}} and adopts its 
> usages - i.e. UDF/UDA code plus {{CQLSSTableWriter}} + 
> {{StressCQLSSTableWriter}}. The latter two classes have a reference to UDF's 
> {{UDHelper}} and had to be changed as well.
>   
>  Some functionality, like type parsing & handling, is duplicated in the code 
> base with this prototype - once in the "current" source tree and once for 
> UDFs. However, unifying the code paths is not trivial, since the UDF sandbox 
> prohibits the use of internal classes (direct and likely indirect 
> dependencies).
>   
>  /cc [~jbellis] 
>   



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-15350) Add CAS “uncertainty” and “contention" messages that are currently propagated as a WriteTimeoutException.

[Dinesh Joshi (Jira)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-15350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dinesh Joshi updated CASSANDRA-15350:
-
Reviewers: Alex Petrov, Dinesh Joshi, Dinesh Joshi  (was: Alex Petrov, 
Dinesh Joshi)
   Alex Petrov, Dinesh Joshi, Dinesh Joshi  (was: Alex Petrov, 
Dinesh Joshi)
   Status: Review In Progress  (was: Patch Available)

> Add CAS “uncertainty” and “contention" messages that are currently propagated 
> as a WriteTimeoutException.
> -
>
> Key: CASSANDRA-15350
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15350
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Feature/Lightweight Transactions
>Reporter: Alex Petrov
>Assignee: Yifan Cai
>Priority: Normal
>  Labels: protocolv5, pull-request-available
> Attachments: Utf8StringEncodeBench.java
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Right now, CAS uncertainty introduced in 
> https://issues.apache.org/jira/browse/CASSANDRA-6013 is propagating as 
> WriteTimeout. One of this conditions it manifests is when there’s at least 
> one acceptor that has accepted the value, which means that this value _may_ 
> still get accepted during the later round, despite the proposer failure. 
> Similar problem happens with CAS contention, which is also indistinguishable 
> from the “regular” timeout, even though it is visible in metrics correctly.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14737) Limit the dependencies used by UDFs/UDAs

[Ryan Svihla (Jira)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027832#comment-17027832
 ] 

Ryan Svihla commented on CASSANDRA-14737:
-

Changes look valid and the tests are passing +1

> Limit the dependencies used by UDFs/UDAs
> 
>
> Key: CASSANDRA-14737
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14737
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Legacy/CQL
>Reporter: Robert Stupp
>Assignee: Robert Stupp
>Priority: Low
>  Labels: UDF
> Fix For: 4.0
>
>
> In an effort to clean up our hygiene and limit the dependencies used by 
> UDFs/UDAs, I think we should refactor the UDF code parts and remove the 
> dependency to the Java Driver in that area without breaking existing 
> UDFs/UDAs.
>   
> The patch is in [this 
> branch|https://github.com/snazy/cassandra/tree/feature/remove-udf-driver-dep-trunk].
>  The changes are rather trivial and provide 100% backwards compatibility for 
> existing UDFs.
>   
>  The prototype copies the necessary parts from the Java Driver into the C* 
> source tree to {{org.apache.cassandra.cql3.functions.types}} and adopts its 
> usages - i.e. UDF/UDA code plus {{CQLSSTableWriter}} + 
> {{StressCQLSSTableWriter}}. The latter two classes have a reference to UDF's 
> {{UDHelper}} and had to be changed as well.
>   
>  Some functionality, like type parsing & handling, is duplicated in the code 
> base with this prototype - once in the "current" source tree and once for 
> UDFs. However, unifying the code paths is not trivial, since the UDF sandbox 
> prohibits the use of internal classes (direct and likely indirect 
> dependencies).
>   
>  /cc [~jbellis] 
>   



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-15350) Add CAS “uncertainty” and “contention" messages that are currently propagated as a WriteTimeoutException.

[Yifan Cai (Jira)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-15350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027827#comment-17027827
 ] 

Yifan Cai commented on CASSANDRA-15350:
---

Thanks Alex. 

Using fuzz to reproduce contentions sounds OK to me.

Having deterministic test cases to precisely produce contentions is also 
valuable, IMO. But I am good to not have those test cases being merged as of 
now. 

I rebased my branch with your improvements commit and added/tweaked test cases 
in {{CasWriteTest}}.

Still the same [PR|https://github.com/apache/cassandra/pull/379] and [code 
branch|https://github.com/yifan-c/cassandra/tree/cas-exception-changes]. Test 
result is 
[here.|https://app.circleci.com/github/yifan-c/cassandra/pipelines/458cafcb-117b-443a-ae6c-c089efd6a929/workflows/7742c20d-1a82-47a8-93d0-cc40a542db5f]
 

> Add CAS “uncertainty” and “contention" messages that are currently propagated 
> as a WriteTimeoutException.
> -
>
> Key: CASSANDRA-15350
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15350
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Feature/Lightweight Transactions
>Reporter: Alex Petrov
>Assignee: Yifan Cai
>Priority: Normal
>  Labels: protocolv5, pull-request-available
> Attachments: Utf8StringEncodeBench.java
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Right now, CAS uncertainty introduced in 
> https://issues.apache.org/jira/browse/CASSANDRA-6013 is propagating as 
> WriteTimeout. One of this conditions it manifests is when there’s at least 
> one acceptor that has accepted the value, which means that this value _may_ 
> still get accepted during the later round, despite the proposer failure. 
> Similar problem happens with CAS contention, which is also indistinguishable 
> from the “regular” timeout, even though it is visible in metrics correctly.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-15539) Extract in-jvm API and tests out of Cassandra and into a separate repository

[David Capwell (Jira)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-15539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Capwell updated CASSANDRA-15539:
--
Change Category: Quality Assurance
 Complexity: Normal
Component/s: Test/dtest
 Status: Open  (was: Triage Needed)

> Extract in-jvm API and tests out of Cassandra and into a separate repository
> 
>
> Key: CASSANDRA-15539
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15539
> Project: Cassandra
>  Issue Type: Task
>  Components: Test/dtest
>Reporter: Alex Petrov
>Assignee: Alex Petrov
>Priority: Normal
>
> Extract in-jvm DTest _API_ and tests into a separate repository that is 
> shared between Cassandra branches. Tests themselves should be buildable using 
> just API, wwhich is not  the case now, since cluster creation relies on impl 
> package, since we do not have factories / constructors in API.
> Main goals we’re trying to achieve:
> 1. We should be able to fail a build on API incompatibility between versions 
> 2. Make it as easy as possible to detect break APIs between versions. 
> 3. Make development of _tests_ based on in-jvm framework simpler
> 4. Reduce surface area of impact when making modifications to tests 
> Potentially, we’d also like to use a plugin to detect API incompatibilities 
> between in-jvm DTest API and in-branch implementations, and start running 
> tests using shared in-jvm test repository with each existing implementation 
> in the branch. This entails both running tests for all branches whenever 
> there’s a change in tests jar and running tests for a specific branch 
> whenever the branch has changed.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-15539) Extract in-jvm API and tests out of Cassandra and into a separate repository

[David Capwell (Jira)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-15539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Capwell updated CASSANDRA-15539:
--
Description: 
Extract in-jvm DTest _API_ and tests into a separate repository that is shared 
between Cassandra branches. Tests themselves should be buildable using just 
API, which is not  the case now, since cluster creation relies on impl package, 
since we do not have factories / constructors in API.

Main goals we’re trying to achieve:
1. We should be able to fail a build on API incompatibility between versions 
2. Make it as easy as possible to detect break APIs between versions. 
3. Make development of _tests_ based on in-jvm framework simpler
4. Reduce surface area of impact when making modifications to tests 

Potentially, we’d also like to use a plugin to detect API incompatibilities 
between in-jvm DTest API and in-branch implementations, and start running tests 
using shared in-jvm test repository with each existing implementation in the 
branch. This entails both running tests for all branches whenever there’s a 
change in tests jar and running tests for a specific branch whenever the branch 
has changed.

  was:
Extract in-jvm DTest _API_ and tests into a separate repository that is shared 
between Cassandra branches. Tests themselves should be buildable using just 
API, wwhich is not  the case now, since cluster creation relies on impl 
package, since we do not have factories / constructors in API.

Main goals we’re trying to achieve:
1. We should be able to fail a build on API incompatibility between versions 
2. Make it as easy as possible to detect break APIs between versions. 
3. Make development of _tests_ based on in-jvm framework simpler
4. Reduce surface area of impact when making modifications to tests 

Potentially, we’d also like to use a plugin to detect API incompatibilities 
between in-jvm DTest API and in-branch implementations, and start running tests 
using shared in-jvm test repository with each existing implementation in the 
branch. This entails both running tests for all branches whenever there’s a 
change in tests jar and running tests for a specific branch whenever the branch 
has changed.


> Extract in-jvm API and tests out of Cassandra and into a separate repository
> 
>
> Key: CASSANDRA-15539
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15539
> Project: Cassandra
>  Issue Type: Task
>  Components: Test/dtest
>Reporter: Alex Petrov
>Assignee: Alex Petrov
>Priority: Normal
>
> Extract in-jvm DTest _API_ and tests into a separate repository that is 
> shared between Cassandra branches. Tests themselves should be buildable using 
> just API, which is not  the case now, since cluster creation relies on impl 
> package, since we do not have factories / constructors in API.
> Main goals we’re trying to achieve:
> 1. We should be able to fail a build on API incompatibility between versions 
> 2. Make it as easy as possible to detect break APIs between versions. 
> 3. Make development of _tests_ based on in-jvm framework simpler
> 4. Reduce surface area of impact when making modifications to tests 
> Potentially, we’d also like to use a plugin to detect API incompatibilities 
> between in-jvm DTest API and in-branch implementations, and start running 
> tests using shared in-jvm test repository with each existing implementation 
> in the branch. This entails both running tests for all branches whenever 
> there’s a change in tests jar and running tests for a specific branch 
> whenever the branch has changed.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-15532) python dtest function assert_stderr_clean should exclude JAVA_TOOL_OPTIONS

[Marcus Eriksson (Jira)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-15532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Marcus Eriksson updated CASSANDRA-15532:

Reviewers: Marcus Eriksson, Marcus Eriksson  (was: Marcus Eriksson)
   Marcus Eriksson, Marcus Eriksson
   Status: Review In Progress  (was: Patch Available)

> python dtest function assert_stderr_clean should exclude JAVA_TOOL_OPTIONS
> --
>
> Key: CASSANDRA-15532
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15532
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Test/dtest
>Reporter: David Capwell
>Assignee: David Capwell
>Priority: Normal
>
> JAVA_TOOL_OPTIONS is a JVM override to allow setting JVM flags without 
> modifying all code which creates the JVM, but for security reasons (see 
> [https://bugs.openjdk.java.net/browse/JDK-8039152]) it needs to log to 
> stderr. Python dtest uses the function assert_stderr_clean to check to see if 
> stderr had any unexpected errors, and should whitelist JAVA_TOOL_OPTIONS so 
> it doesn’t cause test failures.
> PR: https://github.com/apache/cassandra-dtest/pull/56
> Testing done
> {code}
> export JAVA_TOOL_OPTIONS='-ea'
> pytest \
> --log-level="DEBUG" \
> --log-cli-level="DEBUG" \
> --junit-xml=pytest_result.xml \
> --cassandra-dir="/cassandra" \
> --keep-test-dir \
> bootstrap_test::TestBootstrap::test_local_quorum_bootstrap
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-15532) python dtest function assert_stderr_clean should exclude JAVA_TOOL_OPTIONS

[Marcus Eriksson (Jira)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-15532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Marcus Eriksson updated CASSANDRA-15532:

Status: Ready to Commit  (was: Review In Progress)

> python dtest function assert_stderr_clean should exclude JAVA_TOOL_OPTIONS
> --
>
> Key: CASSANDRA-15532
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15532
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Test/dtest
>Reporter: David Capwell
>Assignee: David Capwell
>Priority: Normal
>
> JAVA_TOOL_OPTIONS is a JVM override to allow setting JVM flags without 
> modifying all code which creates the JVM, but for security reasons (see 
> [https://bugs.openjdk.java.net/browse/JDK-8039152]) it needs to log to 
> stderr. Python dtest uses the function assert_stderr_clean to check to see if 
> stderr had any unexpected errors, and should whitelist JAVA_TOOL_OPTIONS so 
> it doesn’t cause test failures.
> PR: https://github.com/apache/cassandra-dtest/pull/56
> Testing done
> {code}
> export JAVA_TOOL_OPTIONS='-ea'
> pytest \
> --log-level="DEBUG" \
> --log-cli-level="DEBUG" \
> --junit-xml=pytest_result.xml \
> --cassandra-dir="/cassandra" \
> --keep-test-dir \
> bootstrap_test::TestBootstrap::test_local_quorum_bootstrap
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-15532) python dtest function assert_stderr_clean should exclude JAVA_TOOL_OPTIONS

[Marcus Eriksson (Jira)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-15532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Marcus Eriksson updated CASSANDRA-15532:

  Fix Version/s: 4.0-alpha
Source Control Link: 
https://github.com/apache/cassandra-dtest/commit/9b93594ae66b04d64f08d1e5ef979e0520f5b3a6
 Resolution: Fixed
 Status: Resolved  (was: Ready to Commit)

+1, committed

> python dtest function assert_stderr_clean should exclude JAVA_TOOL_OPTIONS
> --
>
> Key: CASSANDRA-15532
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15532
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Test/dtest
>Reporter: David Capwell
>Assignee: David Capwell
>Priority: Normal
> Fix For: 4.0-alpha
>
>
> JAVA_TOOL_OPTIONS is a JVM override to allow setting JVM flags without 
> modifying all code which creates the JVM, but for security reasons (see 
> [https://bugs.openjdk.java.net/browse/JDK-8039152]) it needs to log to 
> stderr. Python dtest uses the function assert_stderr_clean to check to see if 
> stderr had any unexpected errors, and should whitelist JAVA_TOOL_OPTIONS so 
> it doesn’t cause test failures.
> PR: https://github.com/apache/cassandra-dtest/pull/56
> Testing done
> {code}
> export JAVA_TOOL_OPTIONS='-ea'
> pytest \
> --log-level="DEBUG" \
> --log-cli-level="DEBUG" \
> --junit-xml=pytest_result.xml \
> --cassandra-dir="/cassandra" \
> --keep-test-dir \
> bootstrap_test::TestBootstrap::test_local_quorum_bootstrap
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[cassandra-dtest] branch master updated: assert_stderr_clean should exclude JAVA_TOOL_OPTIONS

[marcuse]

This is an automated email from the ASF dual-hosted git repository.

marcuse pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/cassandra-dtest.git


The following commit(s) were added to refs/heads/master by this push:
 new 9b93594  assert_stderr_clean should exclude JAVA_TOOL_OPTIONS
9b93594 is described below

commit 9b93594ae66b04d64f08d1e5ef979e0520f5b3a6
Author: David Capwell 
AuthorDate: Thu Jan 30 15:18:39 2020 -0800

assert_stderr_clean should exclude JAVA_TOOL_OPTIONS

Patch by David Capwell; reviewed by marcuse for CASSANDRA-15532

closes #56
---
 tools/assertions.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/tools/assertions.py b/tools/assertions.py
index 1b82251..7491a4b 100644
--- a/tools/assertions.py
+++ b/tools/assertions.py
@@ -297,7 +297,8 @@ def assert_stderr_clean(err, acceptable_errors=None):
 acceptable_errors = ["WARN.*JNA link failure.*unavailable.",
  "objc.*Class JavaLaunchHelper.*?Which one is 
undefined.",
  # Stress tool JMX connection failure, see 
CASSANDRA-12437
- "Failed to connect over JMX; not collecting these 
stats"]
+ "Failed to connect over JMX; not collecting these 
stats",
+ "Picked up JAVA_TOOL_OPTIONS:.*"]
 
 regex_str = r"^({}|\s*|\n)*$".format("|".join(acceptable_errors))
 err_str = err.strip()


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-15532) python dtest function assert_stderr_clean should exclude JAVA_TOOL_OPTIONS

[David Capwell (Jira)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-15532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Capwell updated CASSANDRA-15532:
--
Description: 
JAVA_TOOL_OPTIONS is a JVM override to allow setting JVM flags without 
modifying all code which creates the JVM, but for security reasons (see 
[https://bugs.openjdk.java.net/browse/JDK-8039152]) it needs to log to stderr. 
Python dtest uses the function assert_stderr_clean to check to see if stderr 
had any unexpected errors, and should whitelist JAVA_TOOL_OPTIONS so it doesn’t 
cause test failures.

PR: https://github.com/apache/cassandra-dtest/pull/56

Testing done
{code}
export JAVA_TOOL_OPTIONS='-ea'
pytest \
--log-level="DEBUG" \
--log-cli-level="DEBUG" \
--junit-xml=pytest_result.xml \
--cassandra-dir="/cassandra" \
--keep-test-dir \
bootstrap_test::TestBootstrap::test_local_quorum_bootstrap
{code}

  was:JAVA_TOOL_OPTIONS is a JVM override to allow setting JVM flags without 
modifying all code which creates the JVM, but for security reasons (see 
https://bugs.openjdk.java.net/browse/JDK-8039152) it needs to log to stderr.  
Python dtest uses the function assert_stderr_clean to check to see if stderr 
had any unexpected errors, and should whitelist JAVA_TOOL_OPTIONS so it doesn’t 
cause test failures.


> python dtest function assert_stderr_clean should exclude JAVA_TOOL_OPTIONS
> --
>
> Key: CASSANDRA-15532
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15532
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Test/dtest
>Reporter: David Capwell
>Assignee: David Capwell
>Priority: Normal
>
> JAVA_TOOL_OPTIONS is a JVM override to allow setting JVM flags without 
> modifying all code which creates the JVM, but for security reasons (see 
> [https://bugs.openjdk.java.net/browse/JDK-8039152]) it needs to log to 
> stderr. Python dtest uses the function assert_stderr_clean to check to see if 
> stderr had any unexpected errors, and should whitelist JAVA_TOOL_OPTIONS so 
> it doesn’t cause test failures.
> PR: https://github.com/apache/cassandra-dtest/pull/56
> Testing done
> {code}
> export JAVA_TOOL_OPTIONS='-ea'
> pytest \
> --log-level="DEBUG" \
> --log-cli-level="DEBUG" \
> --junit-xml=pytest_result.xml \
> --cassandra-dir="/cassandra" \
> --keep-test-dir \
> bootstrap_test::TestBootstrap::test_local_quorum_bootstrap
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-15350) Add CAS “uncertainty” and “contention" messages that are currently propagated as a WriteTimeoutException.

[Alex Petrov (Jira)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-15350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027710#comment-17027710
 ] 

Alex Petrov commented on CASSANDRA-15350:
-

I've pushed a rebased branch with minor improvements 
[here|https://github.com/ifesdjeen/cassandra/tree/cas-exception-changes], tests 
running 
[here|https://circleci.com/gh/ifesdjeen/cassandra/tree/cas-exception-changes].

I'd propose to use fuzz tests instead of cascades of latches. If we could 
implement a simple test to reproduce (any) contention in this fashion, it'd be 
great. Could you implement it? 

Other than that, I'm +1 and ready to commit when contention fuzz test is added.

> Add CAS “uncertainty” and “contention" messages that are currently propagated 
> as a WriteTimeoutException.
> -
>
> Key: CASSANDRA-15350
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15350
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Feature/Lightweight Transactions
>Reporter: Alex Petrov
>Assignee: Yifan Cai
>Priority: Normal
>  Labels: protocolv5, pull-request-available
> Attachments: Utf8StringEncodeBench.java
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Right now, CAS uncertainty introduced in 
> https://issues.apache.org/jira/browse/CASSANDRA-6013 is propagating as 
> WriteTimeout. One of this conditions it manifests is when there’s at least 
> one acceptor that has accepted the value, which means that this value _may_ 
> still get accepted during the later round, despite the proposer failure. 
> Similar problem happens with CAS contention, which is also indistinguishable 
> from the “regular” timeout, even though it is visible in metrics correctly.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-15532) python dtest function assert_stderr_clean should exclude JAVA_TOOL_OPTIONS

[David Capwell (Jira)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-15532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Capwell updated CASSANDRA-15532:
--
Test and Documentation Plan: 
PR: https://github.com/apache/cassandra-dtest/pull/56

Testing done

{code}
export JAVA_TOOL_OPTIONS='-ea'
pytest \
--log-level="DEBUG" \
--log-cli-level="DEBUG" \
--junit-xml=pytest_result.xml \
--cassandra-dir="/cassandra" \
--keep-test-dir \
bootstrap_test::TestBootstrap::test_local_quorum_bootstrap
{code}
 Status: Patch Available  (was: In Progress)

> python dtest function assert_stderr_clean should exclude JAVA_TOOL_OPTIONS
> --
>
> Key: CASSANDRA-15532
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15532
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Test/dtest
>Reporter: David Capwell
>Assignee: David Capwell
>Priority: Normal
>
> JAVA_TOOL_OPTIONS is a JVM override to allow setting JVM flags without 
> modifying all code which creates the JVM, but for security reasons (see 
> https://bugs.openjdk.java.net/browse/JDK-8039152) it needs to log to stderr.  
> Python dtest uses the function assert_stderr_clean to check to see if stderr 
> had any unexpected errors, and should whitelist JAVA_TOOL_OPTIONS so it 
> doesn’t cause test failures.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-15367) Memtable memory allocations may deadlock

[Blake Eggleston (Jira)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-15367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027653#comment-17027653
 ] 

Blake Eggleston commented on CASSANDRA-15367:
-

bq. I don't think any deadlock is acceptable to ignore.

I was assuming this couldn’t be modified to eliminate that window for deadlock. 
It would be great if it could be. If not, I agree this needs to be fixed 
eventually, but given the extent that this mitigates the issue, I do think it 
should be committed, at least as a stop-gap for trunk, but possibly as the fix 
for 3.x.

bq. lock stuff

maybe we should pick this conversation up when we get back to removing the 
lock. I’m pretty unfamiliar with the specifics, and it’s probably not very 
useful for me to be arguing about stuff based on a bunch of assumptions.

> Memtable memory allocations may deadlock
> 
>
> Key: CASSANDRA-15367
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15367
> Project: Cassandra
>  Issue Type: Bug
>  Components: Local/Commit Log, Local/Memtable
>Reporter: Benedict Elliott Smith
>Assignee: Benedict Elliott Smith
>Priority: Normal
> Fix For: 4.0, 2.2.x, 3.0.x, 3.11.x
>
>
> * Under heavy contention, we guard modifications to a partition with a mutex, 
> for the lifetime of the memtable.
> * Memtables block for the completion of all {{OpOrder.Group}} started before 
> their flush began
> * Memtables permit operations from this cohort to fall-through to the 
> following Memtable, in order to guarantee a precise commitLogUpperBound
> * Memtable memory limits may be lifted for operations in the first cohort, 
> since they block flush (and hence block future memory allocation)
> With very unfortunate scheduling
> * A contended partition may rapidly escalate to a mutex
> * The system may reach memory limits that prevent allocations for the new 
> Memtable’s cohort (C2) 
> * An operation from C2 may hold the mutex when this occurs
> * Operations from a prior Memtable’s cohort (C1), for a contended partition, 
> may fall-through to the next Memtable
> * The operations from C1 may execute after the above is encountered by those 
> from C2



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-15505) Add message interceptors to in-jvm dtests

[Alex Petrov (Jira)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-15505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alex Petrov updated CASSANDRA-15505:

  Fix Version/s: 4.0
 3.11.6
 3.0.20
 2.2.16
Source Control Link: 
https://github.com/apache/cassandra/commit/b2f2c70e26a32253b0e58ee197c2f8abf01dd449
 Resolution: Fixed
 Status: Resolved  (was: Ready to Commit)

> Add message interceptors to in-jvm dtests
> -
>
> Key: CASSANDRA-15505
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15505
> Project: Cassandra
>  Issue Type: New Feature
>  Components: Test/dtest
>Reporter: Alex Petrov
>Assignee: Alex Petrov
>Priority: Normal
>  Labels: pull-request-available
> Fix For: 2.2.16, 3.0.20, 3.11.6, 4.0
>
>  Time Spent: 3h 50m
>  Remaining Estimate: 0h
>
> Currently we only have means to filter messages in in-jvm tests. We need a 
> facility to intercept and modify the messages between nodes for testing 
> purposes.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-15505) Add message interceptors to in-jvm dtests

[Alex Petrov (Jira)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-15505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027506#comment-17027506
 ] 

Alex Petrov commented on CASSANDRA-15505:
-

Thank you for the review!

Committed to 2.2 with 
[b2f2c70e26a32253b0e58ee197c2f8abf01dd449|https://github.com/apache/cassandra/commit/b2f2c70e26a32253b0e58ee197c2f8abf01dd449],
 and merged up to 
[3.0|https://github.com/apache/cassandra/commit/a27092926b42a83ec0a1e6188677329737c5a3f4],
 
[3.11|https://github.com/apache/cassandra/commit/f09e1be19e5368bf18b4a81c9bed3bf0fc6643eb],
 and 
[trunk|https://github.com/apache/cassandra/commit/aa9326b17b4e49b470033b762070e5838b7bcd58]
 with minor modifications.

Most notably, 2.2 to 3.11 had a slightly different signature for 
{{deserializeMessage}} to expose {{partial}} int which is read from the buffer 
but is not a part of the message itself. Since it's not a public API, and we 
expose it only for convenience, this should be ok.

> Add message interceptors to in-jvm dtests
> -
>
> Key: CASSANDRA-15505
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15505
> Project: Cassandra
>  Issue Type: New Feature
>  Components: Test/dtest
>Reporter: Alex Petrov
>Assignee: Alex Petrov
>Priority: Normal
>  Labels: pull-request-available
>  Time Spent: 3h 50m
>  Remaining Estimate: 0h
>
> Currently we only have means to filter messages in in-jvm tests. We need a 
> facility to intercept and modify the messages between nodes for testing 
> purposes.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[cassandra] branch trunk updated (7a7eece -> aa9326b)

[ifesdjeen]

This is an automated email from the ASF dual-hosted git repository.

ifesdjeen pushed a change to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra.git.


from 7a7eece  Merge branch 'cassandra-3.11' into trunk
 new b2f2c70  Add message interceptors to in-jvm dtests
 new a270929  Merge branch 'cassandra-2.2' into cassandra-3.0
 new f09e1be  Merge branch 'cassandra-3.0' into cassandra-3.11
 new aa9326b  Merge branch 'cassandra-3.11' into trunk

The 4 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../apache/cassandra/distributed/api/IMessage.java |   8 +-
 .../cassandra/distributed/api/IMessageFilters.java |  29 ++-
 .../distributed/impl/AbstractCluster.java  |  17 +-
 .../distributed/impl/IInvokableInstance.java   |   1 -
 .../cassandra/distributed/impl/Instance.java   |  93 -
 .../cassandra/distributed/impl/MessageFilters.java |  73 ---
 .../distributed/test/MessageFiltersTest.java   | 209 +
 7 files changed, 331 insertions(+), 99 deletions(-)
 create mode 100644 
test/distributed/org/apache/cassandra/distributed/test/MessageFiltersTest.java


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[cassandra] branch cassandra-2.2 updated: Add message interceptors to in-jvm dtests

[ifesdjeen]

This is an automated email from the ASF dual-hosted git repository.

ifesdjeen pushed a commit to branch cassandra-2.2
in repository https://gitbox.apache.org/repos/asf/cassandra.git


The following commit(s) were added to refs/heads/cassandra-2.2 by this push:
 new b2f2c70  Add message interceptors to in-jvm dtests
b2f2c70 is described below

commit b2f2c70e26a32253b0e58ee197c2f8abf01dd449
Author: Alex Petrov 
AuthorDate: Wed Jan 15 17:18:46 2020 +0100

Add message interceptors to in-jvm dtests

Patch by Alex Petrov; reviewed by Yifan Cai and David Capwell for 
CASSANDRA-15505.
---
 .../apache/cassandra/distributed/api/IMessage.java |   8 +-
 .../cassandra/distributed/api/IMessageFilters.java |  28 ++-
 .../distributed/impl/AbstractCluster.java  |  17 +-
 .../distributed/impl/IInvokableInstance.java   |   1 -
 .../cassandra/distributed/impl/Instance.java   | 187 ++
 .../cassandra/distributed/impl/MessageFilters.java |  76 
 .../distributed/test/MessageFiltersTest.java   | 210 +
 7 files changed, 391 insertions(+), 136 deletions(-)

diff --git 
a/test/distributed/org/apache/cassandra/distributed/api/IMessage.java 
b/test/distributed/org/apache/cassandra/distributed/api/IMessage.java
index 1e537ed..cd98543 100644
--- a/test/distributed/org/apache/cassandra/distributed/api/IMessage.java
+++ b/test/distributed/org/apache/cassandra/distributed/api/IMessage.java
@@ -18,12 +18,16 @@
 
 package org.apache.cassandra.distributed.api;
 
+import java.io.Serializable;
+
 import org.apache.cassandra.locator.InetAddressAndPort;
 
 /**
- * A cross-version interface for delivering internode messages via message 
sinks
+ * A cross-version interface for delivering internode messages via message 
sinks.
+ *
+ * Message implementations should be serializable so we could load into 
instances.
  */
-public interface IMessage
+public interface IMessage extends Serializable
 {
 int verb();
 byte[] bytes();
diff --git 
a/test/distributed/org/apache/cassandra/distributed/api/IMessageFilters.java 
b/test/distributed/org/apache/cassandra/distributed/api/IMessageFilters.java
index f7c8094..01fe972 100644
--- a/test/distributed/org/apache/cassandra/distributed/api/IMessageFilters.java
+++ b/test/distributed/org/apache/cassandra/distributed/api/IMessageFilters.java
@@ -18,29 +18,39 @@
 
 package org.apache.cassandra.distributed.api;
 
-import org.apache.cassandra.net.MessagingService;
-
 public interface IMessageFilters
 {
 public interface Filter
 {
-Filter restore();
-Filter drop();
+Filter off();
+Filter on();
 }
 
 public interface Builder
 {
 Builder from(int ... nums);
 Builder to(int ... nums);
-Filter ready();
+
+/**
+ * Every message for which matcher returns `true` will be _dropped_ 
(assuming all
+ * other matchers in the chain will return `true` as well).
+ */
+Builder messagesMatching(Matcher filter);
 Filter drop();
 }
 
-Builder verbs(MessagingService.Verb... verbs);
+public interface Matcher
+{
+boolean matches(int from, int to, IMessage message);
+}
+
+Builder verbs(int... verbs);
 Builder allVerbs();
 void reset();
 
-// internal
-boolean permit(IInstance from, IInstance to, int verb);
-
+/**
+ * {@code true} value returned by the implementation implies that the 
message was
+ * not matched by any filters and therefore should be delivered.
+ */
+boolean permit(int from, int to, IMessage msg);
 }
diff --git 
a/test/distributed/org/apache/cassandra/distributed/impl/AbstractCluster.java 
b/test/distributed/org/apache/cassandra/distributed/impl/AbstractCluster.java
index 45e9919..1ee0c14 100644
--- 
a/test/distributed/org/apache/cassandra/distributed/impl/AbstractCluster.java
+++ 
b/test/distributed/org/apache/cassandra/distributed/impl/AbstractCluster.java
@@ -39,7 +39,6 @@ import java.util.stream.IntStream;
 import java.util.stream.Stream;
 
 import com.google.common.collect.Sets;
-
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
@@ -48,6 +47,7 @@ import org.apache.cassandra.db.ColumnFamilyStore;
 import org.apache.cassandra.db.ConsistencyLevel;
 import org.apache.cassandra.db.Keyspace;
 import org.apache.cassandra.distributed.api.Feature;
+import org.apache.cassandra.distributed.api.ICluster;
 import org.apache.cassandra.distributed.api.ICoordinator;
 import org.apache.cassandra.distributed.api.IInstance;
 import org.apache.cassandra.distributed.api.IInstanceConfig;
@@ -55,7 +55,6 @@ import org.apache.cassandra.distributed.api.IIsolatedExecutor;
 import org.apache.cassandra.distributed.api.IListen;
 import org.apache.cassandra.distributed.api.IMessage;
 import org.apache.cassandra.distributed.api.IMessageFilters;
-import org.apache.cassandra.distributed.api.ICluster;
 import org.apache.cassandra.io.util.FileUtils;
 import 

[cassandra] 01/01: Merge branch 'cassandra-3.0' into cassandra-3.11

[ifesdjeen]

This is an automated email from the ASF dual-hosted git repository.

ifesdjeen pushed a commit to branch cassandra-3.11
in repository https://gitbox.apache.org/repos/asf/cassandra.git

commit f09e1be19e5368bf18b4a81c9bed3bf0fc6643eb
Merge: ffab2b8 a270929
Author: Alex Petrov 
AuthorDate: Fri Jan 31 14:39:52 2020 +0100

Merge branch 'cassandra-3.0' into cassandra-3.11

 .../apache/cassandra/distributed/api/IMessage.java |   8 +-
 .../cassandra/distributed/api/IMessageFilters.java |  28 ++-
 .../distributed/impl/AbstractCluster.java  |  17 +-
 .../distributed/impl/IInvokableInstance.java   |   1 -
 .../cassandra/distributed/impl/Instance.java   | 142 --
 .../cassandra/distributed/impl/MessageFilters.java |  76 
 .../distributed/test/MessageFiltersTest.java   | 210 +
 7 files changed, 366 insertions(+), 116 deletions(-)

diff --cc test/distributed/org/apache/cassandra/distributed/impl/Instance.java
index 986c864,5a4dcf4..bf4889e
--- a/test/distributed/org/apache/cassandra/distributed/impl/Instance.java
+++ b/test/distributed/org/apache/cassandra/distributed/impl/Instance.java
@@@ -236,6 -259,6 +260,7 @@@ public class Instance extends IsolatedE
  {
  private final BiConsumer deliver;
  private final Function 
lookupAddressAndPort;
++
  MessageDeliverySink(BiConsumer deliver, 
Function lookupAddressAndPort)
  {
  this.deliver = deliver;
@@@ -244,46 -267,34 +269,35 @@@
  
  public boolean allowOutgoingMessage(MessageOut messageOut, int id, 
InetAddress to)
  {
- try (DataOutputBuffer out = new DataOutputBuffer(1024))
++
+ InetAddressAndPort from = broadcastAddressAndPort();
+ assert from.equals(lookupAddressAndPort.apply(messageOut.from));
 -InetAddressAndPort toFull = lookupAddressAndPort.apply(to);
 -IMessage message = serializeMessage(messageOut, id, from, toFull);
+ 
+ // Tracing logic - similar to 
org.apache.cassandra.net.OutboundTcpConnection.writeConnected
+ byte[] sessionBytes = (byte[]) 
messageOut.parameters.get(Tracing.TRACE_HEADER);
+ if (sessionBytes != null)
  {
- InetAddressAndPort from = broadcastAddressAndPort();
- assert 
from.equals(lookupAddressAndPort.apply(messageOut.from));
- InetAddressAndPort toFull = lookupAddressAndPort.apply(to);
- int version = MessagingService.instance().getVersion(to);
- 
- // Tracing logic - similar to 
org.apache.cassandra.net.OutboundTcpConnection.writeConnected
- byte[] sessionBytes = (byte[]) 
messageOut.parameters.get(Tracing.TRACE_HEADER);
- if (sessionBytes != null)
+ UUID sessionId = 
UUIDGen.getUUID(ByteBuffer.wrap(sessionBytes));
+ TraceState state = Tracing.instance.get(sessionId);
 -String traceMessage = String.format("Sending %s message to 
%s", messageOut.verb, toFull.address);
++String message = String.format("Sending %s message to %s", 
messageOut.verb, to);
+ // session may have already finished; see CASSANDRA-5668
+ if (state == null)
  {
- UUID sessionId = 
UUIDGen.getUUID(ByteBuffer.wrap(sessionBytes));
- TraceState state = Tracing.instance.get(sessionId);
- String message = String.format("Sending %s message to 
%s", messageOut.verb, toFull.address);
- // session may have already finished; see CASSANDRA-5668
- if (state == null)
- {
- byte[] traceTypeBytes = (byte[]) 
messageOut.parameters.get(Tracing.TRACE_TYPE);
- Tracing.TraceType traceType = traceTypeBytes == null 
? Tracing.TraceType.QUERY : Tracing.TraceType.deserialize(traceTypeBytes[0]);
- Tracing.instance.trace(ByteBuffer.wrap(sessionBytes), 
message, traceType.getTTL());
- }
- else
- {
- state.trace(message);
- if (messageOut.verb == 
MessagingService.Verb.REQUEST_RESPONSE)
- Tracing.instance.doneWithNonLocalSession(state);
- }
+ byte[] traceTypeBytes = (byte[]) 
messageOut.parameters.get(Tracing.TRACE_TYPE);
+ Tracing.TraceType traceType = traceTypeBytes == null ? 
Tracing.TraceType.QUERY : Tracing.TraceType.deserialize(traceTypeBytes[0]);
 -
TraceState.mutateWithTracing(ByteBuffer.wrap(sessionBytes), traceMessage, -1, 
traceType.getTTL());
++Tracing.instance.trace(ByteBuffer.wrap(sessionBytes), 
message, traceType.getTTL());
+ }
+ else
+ {
 -

[cassandra] branch cassandra-3.11 updated (ffab2b8 -> f09e1be)

[ifesdjeen]

This is an automated email from the ASF dual-hosted git repository.

ifesdjeen pushed a change to branch cassandra-3.11
in repository https://gitbox.apache.org/repos/asf/cassandra.git.


from ffab2b8  C* 3.0 sstables w/ UDTs are corrupted in C* 3.11 and 4.0
 new b2f2c70  Add message interceptors to in-jvm dtests
 new a270929  Merge branch 'cassandra-2.2' into cassandra-3.0
 new f09e1be  Merge branch 'cassandra-3.0' into cassandra-3.11

The 3 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../apache/cassandra/distributed/api/IMessage.java |   8 +-
 .../cassandra/distributed/api/IMessageFilters.java |  28 ++-
 .../distributed/impl/AbstractCluster.java  |  17 +-
 .../distributed/impl/IInvokableInstance.java   |   1 -
 .../cassandra/distributed/impl/Instance.java   | 142 --
 .../cassandra/distributed/impl/MessageFilters.java |  76 
 .../distributed/test/MessageFiltersTest.java   | 210 +
 7 files changed, 366 insertions(+), 116 deletions(-)
 create mode 100644 
test/distributed/org/apache/cassandra/distributed/test/MessageFiltersTest.java


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[cassandra] branch cassandra-3.0 updated (cd82046 -> a270929)

[ifesdjeen]

This is an automated email from the ASF dual-hosted git repository.

ifesdjeen pushed a change to branch cassandra-3.0
in repository https://gitbox.apache.org/repos/asf/cassandra.git.


from cd82046  Merge branch 'cassandra-2.2' into cassandra-3.0
 new b2f2c70  Add message interceptors to in-jvm dtests
 new a270929  Merge branch 'cassandra-2.2' into cassandra-3.0

The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../apache/cassandra/distributed/api/IMessage.java |   8 +-
 .../cassandra/distributed/api/IMessageFilters.java |  28 ++-
 .../distributed/impl/AbstractCluster.java  |  17 +-
 .../distributed/impl/IInvokableInstance.java   |   1 -
 .../cassandra/distributed/impl/Instance.java   | 187 ++
 .../cassandra/distributed/impl/MessageFilters.java |  79 
 .../distributed/test/MessageFiltersTest.java   | 210 +
 7 files changed, 395 insertions(+), 135 deletions(-)
 create mode 100644 
test/distributed/org/apache/cassandra/distributed/test/MessageFiltersTest.java


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[cassandra] 01/01: Merge branch 'cassandra-3.11' into trunk

[ifesdjeen]

This is an automated email from the ASF dual-hosted git repository.

ifesdjeen pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra.git

commit aa9326b17b4e49b470033b762070e5838b7bcd58
Merge: 7a7eece f09e1be
Author: Alex Petrov 
AuthorDate: Fri Jan 31 14:42:12 2020 +0100

Merge branch 'cassandra-3.11' into trunk

 .../apache/cassandra/distributed/api/IMessage.java |   8 +-
 .../cassandra/distributed/api/IMessageFilters.java |  29 ++-
 .../distributed/impl/AbstractCluster.java  |  17 +-
 .../distributed/impl/IInvokableInstance.java   |   1 -
 .../cassandra/distributed/impl/Instance.java   |  93 -
 .../cassandra/distributed/impl/MessageFilters.java |  73 ---
 .../distributed/test/MessageFiltersTest.java   | 209 +
 7 files changed, 331 insertions(+), 99 deletions(-)

diff --cc 
test/distributed/org/apache/cassandra/distributed/impl/AbstractCluster.java
index 47f76fd,6370417..68cc6a2
--- 
a/test/distributed/org/apache/cassandra/distributed/impl/AbstractCluster.java
+++ 
b/test/distributed/org/apache/cassandra/distributed/impl/AbstractCluster.java
@@@ -54,10 -55,9 +54,9 @@@ import org.apache.cassandra.distributed
  import org.apache.cassandra.distributed.api.IListen;
  import org.apache.cassandra.distributed.api.IMessage;
  import org.apache.cassandra.distributed.api.IMessageFilters;
- import org.apache.cassandra.distributed.api.ICluster;
  import org.apache.cassandra.io.util.FileUtils;
  import org.apache.cassandra.locator.InetAddressAndPort;
 -import org.apache.cassandra.net.MessagingService;
 +import org.apache.cassandra.net.Verb;
  import org.apache.cassandra.utils.FBUtilities;
  import org.apache.cassandra.utils.Pair;
  import org.apache.cassandra.utils.concurrent.SimpleCondition;
@@@ -299,8 -280,18 +298,18 @@@ public abstract class AbstractCluster<
timeout, unit);
  }
  
- public IMessageFilters filters() { return filters; }
- public MessageFilters.Builder verbs(Verb... verbs) { return 
filters.verbs(verbs); }
+ public IMessageFilters filters()
+ {
+ return filters;
+ }
+ 
 -public MessageFilters.Builder verbs(MessagingService.Verb... verbs)
++public MessageFilters.Builder verbs(Verb... verbs)
+ {
+ int[] ids = new int[verbs.length];
+ for (int i = 0; i < verbs.length; ++i)
 -ids[i] = verbs[i].ordinal();
++ids[i] = verbs[i].id;
+ return filters.verbs(ids);
+ }
  
  public void disableAutoCompaction(String keyspace)
  {
diff --cc test/distributed/org/apache/cassandra/distributed/impl/Instance.java
index 6ec1a9e,bf4889e..b8ef25d
--- a/test/distributed/org/apache/cassandra/distributed/impl/Instance.java
+++ b/test/distributed/org/apache/cassandra/distributed/impl/Instance.java
@@@ -27,22 -31,21 +27,23 @@@ import java.util.UUID
  import java.util.concurrent.CompletableFuture;
  import java.util.concurrent.ExecutorService;
  import java.util.concurrent.Future;
 +import java.util.concurrent.TimeoutException;
  import java.util.function.BiConsumer;
- import java.util.function.BiPredicate;
 -import java.util.function.Function;
+ 
++import com.google.common.annotations.VisibleForTesting;
 +
 +import io.netty.util.concurrent.GlobalEventExecutor;
  import org.apache.cassandra.batchlog.BatchlogManager;
 +import org.apache.cassandra.concurrent.ExecutorLocals;
  import org.apache.cassandra.concurrent.ScheduledExecutors;
  import org.apache.cassandra.concurrent.SharedExecutorPool;
 -import org.apache.cassandra.concurrent.StageManager;
 +import org.apache.cassandra.concurrent.Stage;
  import org.apache.cassandra.config.Config;
  import org.apache.cassandra.config.DatabaseDescriptor;
 -import org.apache.cassandra.config.Schema;
 -import org.apache.cassandra.config.SchemaConstants;
  import org.apache.cassandra.cql3.CQLStatement;
 +import org.apache.cassandra.cql3.QueryHandler;
  import org.apache.cassandra.cql3.QueryOptions;
  import org.apache.cassandra.cql3.QueryProcessor;
 -import org.apache.cassandra.cql3.statements.ParsedStatement;
  import org.apache.cassandra.db.ColumnFamilyStore;
  import org.apache.cassandra.db.Keyspace;
  import org.apache.cassandra.db.Memtable;
@@@ -200,77 -194,170 +201,81 @@@ public class Instance extends IsolatedE
  {
  BiConsumer deliverToInstance = (to, 
message) -> cluster.get(to).receiveMessage(message);
  BiConsumer 
deliverToInstanceIfNotFiltered = (to, message) -> {
- if (cluster.filters().permit(this, cluster.get(to), 
message.verb()))
 -int fromNum = config().num();
 -int toNum = cluster.get(to).config().num();
 -
 -if (cluster.filters().permit(fromNum, toNum, message))
++if (permitMessage(cluster, to, message))
  deliverToInstance.accept(to, message);
  };
  
- MessagingService.instance().outboundSink.add(new 

[cassandra] 01/01: Merge branch 'cassandra-2.2' into cassandra-3.0

[ifesdjeen]

This is an automated email from the ASF dual-hosted git repository.

ifesdjeen pushed a commit to branch cassandra-3.0
in repository https://gitbox.apache.org/repos/asf/cassandra.git

commit a27092926b42a83ec0a1e6188677329737c5a3f4
Merge: cd82046 b2f2c70
Author: Alex Petrov 
AuthorDate: Fri Jan 31 14:37:25 2020 +0100

Merge branch 'cassandra-2.2' into cassandra-3.0

 .../apache/cassandra/distributed/api/IMessage.java |   8 +-
 .../cassandra/distributed/api/IMessageFilters.java |  28 ++-
 .../distributed/impl/AbstractCluster.java  |  17 +-
 .../distributed/impl/IInvokableInstance.java   |   1 -
 .../cassandra/distributed/impl/Instance.java   | 187 ++
 .../cassandra/distributed/impl/MessageFilters.java |  79 
 .../distributed/test/MessageFiltersTest.java   | 210 +
 7 files changed, 395 insertions(+), 135 deletions(-)

diff --cc test/distributed/org/apache/cassandra/distributed/impl/Instance.java
index 3de5ed8,0647198..5a4dcf4
--- a/test/distributed/org/apache/cassandra/distributed/impl/Instance.java
+++ b/test/distributed/org/apache/cassandra/distributed/impl/Instance.java
@@@ -192,7 -192,9 +193,10 @@@ public class Instance extends IsolatedE
  {
  BiConsumer deliverToInstance = (to, 
message) -> cluster.get(to).receiveMessage(message);
  BiConsumer 
deliverToInstanceIfNotFiltered = (to, message) -> {
- if (cluster.filters().permit(this, cluster.get(to), 
message.verb()))
+ int fromNum = config().num();
+ int toNum = cluster.get(to).config().num();
++
+ if (cluster.filters().permit(fromNum, toNum, message))
  deliverToInstance.accept(to, message);
  };
  
@@@ -242,46 -265,35 +267,34 @@@
  
  public boolean allowOutgoingMessage(MessageOut messageOut, int id, 
InetAddress to)
  {
- try (DataOutputBuffer out = new DataOutputBuffer(1024))
+ InetAddressAndPort from = broadcastAddressAndPort();
 -InetAddressAndPort toFull = lookupAddressAndPort.apply(to);
+ assert from.equals(lookupAddressAndPort.apply(messageOut.from));
 -
 -IMessage serialized = serializeMessage(messageOut, id, 
broadcastAddressAndPort(), lookupAddressAndPort.apply(messageOut.from));
++InetAddressAndPort toFull = lookupAddressAndPort.apply(to);
++IMessage message = serializeMessage(messageOut, id, from, toFull);
+ 
+ // Tracing logic - similar to 
org.apache.cassandra.net.OutboundTcpConnection.writeConnected
+ byte[] sessionBytes = (byte[]) 
messageOut.parameters.get(Tracing.TRACE_HEADER);
+ if (sessionBytes != null)
  {
- InetAddressAndPort from = broadcastAddressAndPort();
- assert 
from.equals(lookupAddressAndPort.apply(messageOut.from));
- InetAddressAndPort toFull = lookupAddressAndPort.apply(to);
- int version = MessagingService.instance().getVersion(to);
- 
- // Tracing logic - similar to 
org.apache.cassandra.net.OutboundTcpConnection.writeConnected
- byte[] sessionBytes = (byte[]) 
messageOut.parameters.get(Tracing.TRACE_HEADER);
- if (sessionBytes != null)
+ UUID sessionId = 
UUIDGen.getUUID(ByteBuffer.wrap(sessionBytes));
+ TraceState state = Tracing.instance.get(sessionId);
 -String message = String.format("Sending %s message to %s", 
messageOut.verb, to);
++String traceMessage = String.format("Sending %s message to 
%s", messageOut.verb, toFull.address);
+ // session may have already finished; see CASSANDRA-5668
+ if (state == null)
  {
- UUID sessionId = 
UUIDGen.getUUID(ByteBuffer.wrap(sessionBytes));
- TraceState state = Tracing.instance.get(sessionId);
- String message = String.format("Sending %s message to 
%s", messageOut.verb, toFull.address);
- // session may have already finished; see CASSANDRA-5668
- if (state == null)
- {
- byte[] traceTypeBytes = (byte[]) 
messageOut.parameters.get(Tracing.TRACE_TYPE);
- Tracing.TraceType traceType = traceTypeBytes == null 
? Tracing.TraceType.QUERY : Tracing.TraceType.deserialize(traceTypeBytes[0]);
- 
TraceState.mutateWithTracing(ByteBuffer.wrap(sessionBytes), message, -1, 
traceType.getTTL());
- }
- else
- {
- state.trace(message);
- if (messageOut.verb == 
MessagingService.Verb.REQUEST_RESPONSE)
- Tracing.instance.doneWithNonLocalSession(state);
- }
+ byte[] traceTypeBytes = (byte[]) 

[jira] [Commented] (CASSANDRA-15539) Extract in-jvm API and tests out of Cassandra and into a separate repository

[Alex Petrov (Jira)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-15539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027464#comment-17027464
 ] 

Alex Petrov commented on CASSANDRA-15539:
-

cc [~dcapwell] [~djoshi] [~drohrer] [~mck] 

> Extract in-jvm API and tests out of Cassandra and into a separate repository
> 
>
> Key: CASSANDRA-15539
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15539
> Project: Cassandra
>  Issue Type: Task
>Reporter: Alex Petrov
>Assignee: Alex Petrov
>Priority: Normal
>
> Extract in-jvm DTest _API_ and tests into a separate repository that is 
> shared between Cassandra branches. Tests themselves should be buildable using 
> just API, wwhich is not  the case now, since cluster creation relies on impl 
> package, since we do not have factories / constructors in API.
> Main goals we’re trying to achieve:
> 1. We should be able to fail a build on API incompatibility between versions 
> 2. Make it as easy as possible to detect break APIs between versions. 
> 3. Make development of _tests_ based on in-jvm framework simpler
> 4. Reduce surface area of impact when making modifications to tests 
> Potentially, we’d also like to use a plugin to detect API incompatibilities 
> between in-jvm DTest API and in-branch implementations, and start running 
> tests using shared in-jvm test repository with each existing implementation 
> in the branch. This entails both running tests for all branches whenever 
> there’s a change in tests jar and running tests for a specific branch 
> whenever the branch has changed.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Created] (CASSANDRA-15539) Extract in-jvm API and tests out of Cassandra and into a separate repository

[Alex Petrov (Jira)]

Alex Petrov created CASSANDRA-15539:
---

 Summary: Extract in-jvm API and tests out of Cassandra and into a 
separate repository
 Key: CASSANDRA-15539
 URL: https://issues.apache.org/jira/browse/CASSANDRA-15539
 Project: Cassandra
  Issue Type: Task
Reporter: Alex Petrov
Assignee: Alex Petrov


Extract in-jvm DTest _API_ and tests into a separate repository that is shared 
between Cassandra branches. Tests themselves should be buildable using just 
API, wwhich is not  the case now, since cluster creation relies on impl 
package, since we do not have factories / constructors in API.

Main goals we’re trying to achieve:
1. We should be able to fail a build on API incompatibility between versions 
2. Make it as easy as possible to detect break APIs between versions. 
3. Make development of _tests_ based on in-jvm framework simpler
4. Reduce surface area of impact when making modifications to tests 

Potentially, we’d also like to use a plugin to detect API incompatibilities 
between in-jvm DTest API and in-branch implementations, and start running tests 
using shared in-jvm test repository with each existing implementation in the 
branch. This entails both running tests for all branches whenever there’s a 
change in tests jar and running tests for a specific branch whenever the branch 
has changed.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-15035) C* 3.0 sstables w/ UDTs are corrupted in 3.11 + 4.0

[Robert Stupp (Jira)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-15035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Stupp updated CASSANDRA-15035:
-
Source Control Link: 
https://github.com/apache/cassandra/commit/ffab2b8dde0c7c40080c1c0b36831edd6965a042
 Resolution: Fixed
 Status: Resolved  (was: Ready to Commit)

Thanks for the review!

Committed as 
[ffab2b8dde0c7c40080c1c0b36831edd6965a042|https://github.com/apache/cassandra/commit/ffab2b8dde0c7c40080c1c0b36831edd6965a042]
 to [cassandra-3.11|https://github.com/apache/cassandra/tree/cassandra-3.11], 
[merged|https://github.com/apache/cassandra/commit/7a7eece9578312a2f9d77de6e0755a3c3c542e99]
 to [trunk|https://github.com/apache/cassandra/tree/trunk].

Committed manually as ffab2b8dde0c7c40080c1c0b36831edd6965a042
 Merged manually as 7a7eece9578312a2f9d77de6e0755a3c3c542e99

[dtest 
committed|https://github.com/apache/cassandra/commit/ffab2b8dde0c7c40080c1c0b36831edd6965a042]
 as well

> C* 3.0 sstables w/ UDTs are corrupted in 3.11 + 4.0
> ---
>
> Key: CASSANDRA-15035
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15035
> Project: Cassandra
>  Issue Type: Bug
>  Components: Feature/UDT, Local/SSTable
>Reporter: Robert Stupp
>Assignee: Robert Stupp
>Priority: Urgent
> Fix For: 3.11.6, 4.0
>
>
> OSS C* 3.0 writes incorrect type information for UDTs into the 
> serialization-header of each sstable.
> In C* 3.0, both UDTs and tuple are always frozen. A frozen type must be 
> enclosed in a {{frozen<...>}} via the {{CQL3Type}} hierarchy (resp 
> {{org.apache.cassandra.db.marshal.FrozenType(...)}} via the {{AbstractType}} 
> hierarchy) “bracket” in the schema and serialization-header.
> Since CASSANDRA-7423 (committed to C* 3.6) UDTs can also be non-frozen (= 
> multi-cell).
> Unfortunately, C* 3.0 does not write the 
> {{org.apache.cassandra.db.marshal.FrozenType(...)}} “bracket” for UDTs into 
> the {{SerializationHeader.Component}} in the {{-Stats.db}} sstable component.
> The order in which columns of a row are serialized depends on the concrete 
> {{AbstractType}}. Columns with variable length types (frozen types belong to 
> this category) are serialized before columns with multi-cell types 
> (non-frozen types belong to that category).
> If C* 3.6 (or any newer version) reads an sstable written by C* 3.0 (up to 
> 3.5), it will read the type information “non-frozen UDT” from the 
> serialization header, which is technically correct.
> This means, that upgrades from C* 3.0 to C* 3.11 and 4.0, using a schema that 
> uses UDTs, result in inaccessible data in those sstables. Reads against 3.0 
> sstables as well as attempts to scrub these sstables result in a wide variety 
> of errors/exceptions ({{CorruptSSTableException}}, {{EOFExcepiton}}, 
> {{OutOfMemoryError}}, etc etc), as usual in such cases.
> Mitigation strategy in the proposed patch:
> * Fix the broken serialization-headers automatically when an upgrade from C* 
> 3.0 is detected.
> * Enhance {{sstablescrub}} to verify the serialization-header against the 
> schema and allow {{sstablescrub}} to fix the UDT types according to the 
> information in the schema. This does not apply to "online scrub" (e.g. 
> nodetool scrub). The behavior of {{sstablescrub}} has been changed to first 
> inspect the serialization-header and verify the type information against the 
> schema. 
> Differences between the schema and the sstable serialization-headers cause 
> {{sstablescrub}} to error out and stop - i.e. safety first (there’s a way to 
> opt-out though).
> A new class {{SSTableHeaderFix}} can inspect the serialization-header 
> ({{SerializationHeader.Component}}) in the the {{-Statistics.db}} component 
> and fix the type information in those sstables for UDTs according to the 
> schema information.
> This new class could be used during verify and before sstables are imported. 
> But changes to “verify” and “import” are out of the scope of this ticket, as 
> the patch is already bigger than I originally expected.
> Another issue not tackled by this ticket is that the wrong ‘kind’ is written 
> to the type information in {{system_schema.dropped_columns}} when a 
> non-frozen UDT column is dropped. When a UDT column is dropped, the type of 
> the dropped column is converted from the UDT definition to its 
> “corresponding” tuple type definition. But all versions currently write 
> {{frozen>}}, but for non-frozen UDTs it should actually just be 
> {{tuple<...>}}. Unfortunately, there is nothing that could be done in this 
> ticket to fix (or even consider) the type information of a dropped column. 
> But for correctness, the tuple type should be a multi-cell one (only 
> accessible for dropped UDTs though - not as something that a user can create 
> as a 

[jira] [Updated] (CASSANDRA-15035) C* 3.0 sstables w/ UDTs are corrupted in 3.11 + 4.0

[Robert Stupp (Jira)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-15035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Stupp updated CASSANDRA-15035:
-
Reviewers: Brandon Williams, Robert Stupp
   Status: Review In Progress  (was: Patch Available)

> C* 3.0 sstables w/ UDTs are corrupted in 3.11 + 4.0
> ---
>
> Key: CASSANDRA-15035
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15035
> Project: Cassandra
>  Issue Type: Bug
>  Components: Feature/UDT, Local/SSTable
>Reporter: Robert Stupp
>Assignee: Robert Stupp
>Priority: Urgent
> Fix For: 3.11.6, 4.0
>
>
> OSS C* 3.0 writes incorrect type information for UDTs into the 
> serialization-header of each sstable.
> In C* 3.0, both UDTs and tuple are always frozen. A frozen type must be 
> enclosed in a {{frozen<...>}} via the {{CQL3Type}} hierarchy (resp 
> {{org.apache.cassandra.db.marshal.FrozenType(...)}} via the {{AbstractType}} 
> hierarchy) “bracket” in the schema and serialization-header.
> Since CASSANDRA-7423 (committed to C* 3.6) UDTs can also be non-frozen (= 
> multi-cell).
> Unfortunately, C* 3.0 does not write the 
> {{org.apache.cassandra.db.marshal.FrozenType(...)}} “bracket” for UDTs into 
> the {{SerializationHeader.Component}} in the {{-Stats.db}} sstable component.
> The order in which columns of a row are serialized depends on the concrete 
> {{AbstractType}}. Columns with variable length types (frozen types belong to 
> this category) are serialized before columns with multi-cell types 
> (non-frozen types belong to that category).
> If C* 3.6 (or any newer version) reads an sstable written by C* 3.0 (up to 
> 3.5), it will read the type information “non-frozen UDT” from the 
> serialization header, which is technically correct.
> This means, that upgrades from C* 3.0 to C* 3.11 and 4.0, using a schema that 
> uses UDTs, result in inaccessible data in those sstables. Reads against 3.0 
> sstables as well as attempts to scrub these sstables result in a wide variety 
> of errors/exceptions ({{CorruptSSTableException}}, {{EOFExcepiton}}, 
> {{OutOfMemoryError}}, etc etc), as usual in such cases.
> Mitigation strategy in the proposed patch:
> * Fix the broken serialization-headers automatically when an upgrade from C* 
> 3.0 is detected.
> * Enhance {{sstablescrub}} to verify the serialization-header against the 
> schema and allow {{sstablescrub}} to fix the UDT types according to the 
> information in the schema. This does not apply to "online scrub" (e.g. 
> nodetool scrub). The behavior of {{sstablescrub}} has been changed to first 
> inspect the serialization-header and verify the type information against the 
> schema. 
> Differences between the schema and the sstable serialization-headers cause 
> {{sstablescrub}} to error out and stop - i.e. safety first (there’s a way to 
> opt-out though).
> A new class {{SSTableHeaderFix}} can inspect the serialization-header 
> ({{SerializationHeader.Component}}) in the the {{-Statistics.db}} component 
> and fix the type information in those sstables for UDTs according to the 
> schema information.
> This new class could be used during verify and before sstables are imported. 
> But changes to “verify” and “import” are out of the scope of this ticket, as 
> the patch is already bigger than I originally expected.
> Another issue not tackled by this ticket is that the wrong ‘kind’ is written 
> to the type information in {{system_schema.dropped_columns}} when a 
> non-frozen UDT column is dropped. When a UDT column is dropped, the type of 
> the dropped column is converted from the UDT definition to its 
> “corresponding” tuple type definition. But all versions currently write 
> {{frozen>}}, but for non-frozen UDTs it should actually just be 
> {{tuple<...>}}. Unfortunately, there is nothing that could be done in this 
> ticket to fix (or even consider) the type information of a dropped column. 
> But for correctness, the tuple type should be a multi-cell one (only 
> accessible for dropped UDTs though - not as something that a user can create 
> as a type).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-15035) C* 3.0 sstables w/ UDTs are corrupted in 3.11 + 4.0

[Robert Stupp (Jira)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-15035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Stupp updated CASSANDRA-15035:
-
Status: Ready to Commit  (was: Review In Progress)

> C* 3.0 sstables w/ UDTs are corrupted in 3.11 + 4.0
> ---
>
> Key: CASSANDRA-15035
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15035
> Project: Cassandra
>  Issue Type: Bug
>  Components: Feature/UDT, Local/SSTable
>Reporter: Robert Stupp
>Assignee: Robert Stupp
>Priority: Urgent
> Fix For: 3.11.6, 4.0
>
>
> OSS C* 3.0 writes incorrect type information for UDTs into the 
> serialization-header of each sstable.
> In C* 3.0, both UDTs and tuple are always frozen. A frozen type must be 
> enclosed in a {{frozen<...>}} via the {{CQL3Type}} hierarchy (resp 
> {{org.apache.cassandra.db.marshal.FrozenType(...)}} via the {{AbstractType}} 
> hierarchy) “bracket” in the schema and serialization-header.
> Since CASSANDRA-7423 (committed to C* 3.6) UDTs can also be non-frozen (= 
> multi-cell).
> Unfortunately, C* 3.0 does not write the 
> {{org.apache.cassandra.db.marshal.FrozenType(...)}} “bracket” for UDTs into 
> the {{SerializationHeader.Component}} in the {{-Stats.db}} sstable component.
> The order in which columns of a row are serialized depends on the concrete 
> {{AbstractType}}. Columns with variable length types (frozen types belong to 
> this category) are serialized before columns with multi-cell types 
> (non-frozen types belong to that category).
> If C* 3.6 (or any newer version) reads an sstable written by C* 3.0 (up to 
> 3.5), it will read the type information “non-frozen UDT” from the 
> serialization header, which is technically correct.
> This means, that upgrades from C* 3.0 to C* 3.11 and 4.0, using a schema that 
> uses UDTs, result in inaccessible data in those sstables. Reads against 3.0 
> sstables as well as attempts to scrub these sstables result in a wide variety 
> of errors/exceptions ({{CorruptSSTableException}}, {{EOFExcepiton}}, 
> {{OutOfMemoryError}}, etc etc), as usual in such cases.
> Mitigation strategy in the proposed patch:
> * Fix the broken serialization-headers automatically when an upgrade from C* 
> 3.0 is detected.
> * Enhance {{sstablescrub}} to verify the serialization-header against the 
> schema and allow {{sstablescrub}} to fix the UDT types according to the 
> information in the schema. This does not apply to "online scrub" (e.g. 
> nodetool scrub). The behavior of {{sstablescrub}} has been changed to first 
> inspect the serialization-header and verify the type information against the 
> schema. 
> Differences between the schema and the sstable serialization-headers cause 
> {{sstablescrub}} to error out and stop - i.e. safety first (there’s a way to 
> opt-out though).
> A new class {{SSTableHeaderFix}} can inspect the serialization-header 
> ({{SerializationHeader.Component}}) in the the {{-Statistics.db}} component 
> and fix the type information in those sstables for UDTs according to the 
> schema information.
> This new class could be used during verify and before sstables are imported. 
> But changes to “verify” and “import” are out of the scope of this ticket, as 
> the patch is already bigger than I originally expected.
> Another issue not tackled by this ticket is that the wrong ‘kind’ is written 
> to the type information in {{system_schema.dropped_columns}} when a 
> non-frozen UDT column is dropped. When a UDT column is dropped, the type of 
> the dropped column is converted from the UDT definition to its 
> “corresponding” tuple type definition. But all versions currently write 
> {{frozen>}}, but for non-frozen UDTs it should actually just be 
> {{tuple<...>}}. Unfortunately, there is nothing that could be done in this 
> ticket to fix (or even consider) the type information of a dropped column. 
> But for correctness, the tuple type should be a multi-cell one (only 
> accessible for dropped UDTs though - not as something that a user can create 
> as a type).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[cassandra-dtest] branch master updated: dtest for UDT fix on upgrade

[snazy]

This is an automated email from the ASF dual-hosted git repository.

snazy pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/cassandra-dtest.git


The following commit(s) were added to refs/heads/master by this push:
 new c5f6d28  dtest for UDT fix on upgrade
c5f6d28 is described below

commit c5f6d28d88f314f70185de16777c52a79e1be053
Author: Robert Stupp 
AuthorDate: Sat Feb 9 10:47:05 2019 +0100

dtest for UDT fix on upgrade

patch by Robert Stupp; reviewed by Brandon Williams for CASSANDRA-15035
---
 upgrade_tests/upgrade_udtfix_test.py | 162 +++
 1 file changed, 162 insertions(+)

diff --git a/upgrade_tests/upgrade_udtfix_test.py 
b/upgrade_tests/upgrade_udtfix_test.py
new file mode 100644
index 000..a138755
--- /dev/null
+++ b/upgrade_tests/upgrade_udtfix_test.py
@@ -0,0 +1,162 @@
+import pytest
+import logging
+
+from dtest import RUN_STATIC_UPGRADE_MATRIX, Tester
+from distutils.version import LooseVersion
+from tools.misc import add_skip
+from .upgrade_manifest import build_upgrade_pairs
+
+since = pytest.mark.since
+logger = logging.getLogger(__name__)
+
+@pytest.mark.upgrade_test
+class UpgradeUDTFixTest(Tester):
+__test__ = False
+
+"""
+@jira_ticket: CASSANDRA-TBD
+
+3.0 sstable cannot be read by 3.11 and newer
+
+OSS C* 3.0 only support frozen UDTs. A frozen type is marked using
+"org.apache.cassandra.db.marshal.FrozenType(...)" using the AbstractType 
class hierarchy.
+But 3.0 does not write the "FrozenType()" type into the serialization 
header in the sstable
+metadata. CASSANDRA-7423 added support for non-frozen UDTs) in C* 3.6. 
Since then, UDTs can
+be either frozen or non-frozen, so it becomes important that the 
"FrozenType bracket" is
+present in the serialization header, because the serialization order of 
columns depends on
+its type (fixed length type column first, then variable length type 
columns, then multi-cell
+(non-frozen) type columns). This means, that C* 3.6 and newer will 
interpret the originally
+frozen UDT as non-frozen and deserialization fails.
+"""
+
+def test_udtfix_in_sstable(self):
+"""
+Verify that the fix to modify the serialization-header in the sstable 
metadata works for a single node.
+"""
+cluster = self.cluster
+cluster.set_install_dir(version=self.UPGRADE_PATH.starting_version)
+cluster.populate(1)
+node1, = cluster.nodelist()
+cluster.start()
+
+session = self.patient_cql_connection(node1)
+self._schema_and_data(node1, session, "ks", "{'class': 
'SimpleStrategy', 'replication_factor': 1}")
+references = self._read_rows(session, "ks")
+session.cluster.shutdown()
+
+logger.debug("Upgrading from {} to 
{}".format(self.UPGRADE_PATH.starting_version, 
self.UPGRADE_PATH.upgrade_version))
+cluster.stop()
+cluster.set_install_dir(version=self.UPGRADE_PATH.upgrade_version)
+
+log_mark = node1.mark_log()
+cluster.start()
+self._verify_upgrade_log(log_mark, node1)
+
+session = self.patient_cql_connection(node1)
+check = self._read_rows(session, "ks")
+assert references == check
+
+def test_udtfix_in_messaging(self):
+"""
+Paranoia test for CASSANDRA-TBD, but verify that internode reads are 
not affected - with the data
+both on the upgraded and non-upgraded node.
+"""
+cluster = self.cluster
+cluster.set_install_dir(version=self.UPGRADE_PATH.starting_version)
+cluster.populate([1, 1])
+node1, node2 = cluster.nodelist()
+cluster.start()
+
+logger.debug("Started 2-DC cluster")
+
+session = self.patient_cql_connection(node1)
+self._schema_and_data(node1, session, "ks1", "{'class': 
'NetworkTopologyStrategy', 'dc1': 1}")
+self._schema_and_data(node1, session, "ks2", "{'class': 
'NetworkTopologyStrategy', 'dc2': 1}")
+references11 = self._read_rows(session, "ks1")
+references12 = self._read_rows(session, "ks2")
+session.cluster.shutdown()
+
+session = self.patient_cql_connection(node2)
+references21 = self._read_rows(session, "ks1")
+references22 = self._read_rows(session, "ks2")
+session.cluster.shutdown()
+
+assert references11 == references21
+assert references12 == references22
+
+logger.debug("Upgrading {} from {} to {}".format(node1.name, 
self.UPGRADE_PATH.starting_version, self.UPGRADE_PATH.upgrade_version))
+node1.stop()
+node1.set_install_dir(version=self.UPGRADE_PATH.upgrade_version)
+cluster.set_configuration_options()  # re-write 
cassandra-topology.properties (it's overridden by Node.set_install_dir)
+log_mark = node1.mark_log()
+node1.start(wait_for_binary_proto=True)
+self._verify_upgrade_log(log_mark, node1)
+
+   

[cassandra] 01/02: C* 3.0 sstables w/ UDTs are corrupted in C* 3.11 and 4.0

[snazy]

This is an automated email from the ASF dual-hosted git repository.

snazy pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra.git

commit 4f27a37d7dd2750cc25261773a67ee8b4a07142c
Author: Robert Stupp 
AuthorDate: Fri Feb 15 14:24:39 2019 +0100

C* 3.0 sstables w/ UDTs are corrupted in C* 3.11 and 4.0

patch by Robert Stupp; reviewed by Brandon Williams for CASSANDRA-15035
---
 CHANGES.txt|   1 +
 NEWS.txt   |  18 +
 src/java/org/apache/cassandra/cql3/CQL3Type.java   |   4 +-
 .../apache/cassandra/db/SerializationHeader.java   |  12 +
 .../org/apache/cassandra/db/SystemKeyspace.java|   2 +
 .../org/apache/cassandra/db/rows/AbstractCell.java |  15 +-
 .../org/apache/cassandra/db/rows/AbstractRow.java  |  22 +-
 .../db/rows/AbstractTypeVersionComparator.java | 121 ---
 .../db/rows/ColumnMetadataVersionComparator.java   |  85 ++
 src/java/org/apache/cassandra/db/rows/Row.java |   2 +-
 src/java/org/apache/cassandra/db/rows/Rows.java|   2 +-
 .../cassandra/io/sstable/SSTableHeaderFix.java | 918 
 .../cassandra/io/sstable/format/SSTableReader.java |   5 +
 .../io/sstable/metadata/IMetadataSerializer.java   |   5 +
 .../io/sstable/metadata/MetadataSerializer.java|   2 +-
 .../apache/cassandra/service/CassandraDaemon.java  |   3 +
 .../apache/cassandra/tools/StandaloneScrubber.java | 144 ++-
 .../org/apache/cassandra/utils/FBUtilities.java|  12 +
 ...va => ColumnMetadataVersionComparatorTest.java} |  29 +-
 .../cassandra/io/sstable/SSTableHeaderFixTest.java | 964 +
 .../schema/TupleTypesRepresentationTest.java   | 403 +
 21 files changed, 2621 insertions(+), 148 deletions(-)

diff --git a/CHANGES.txt b/CHANGES.txt
index 98c189a..9554a00 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -25,6 +25,7 @@
otherwise synchronize their clocks, and that clocks are mostly in sync, 
since
this is a requirement for general correctness of last write wins. 
(CASSANDRA-15216)
 Merged from 3.11:
+ * Fix bad UDT sstable metadata serialization headers written by C* 3.0 on 
upgrade and in sstablescrub (CASSANDRA-15035)
  * Fix nodetool compactionstats showing extra pending task for TWCS - patch 
implemented (CASSANDRA-15409)
  * Fix SELECT JSON formatting for the "duration" type (CASSANDRA-15075)
  * Update nodetool help stop output (CASSANDRA-15401)
diff --git a/NEWS.txt b/NEWS.txt
index 9c6b43f..7d716fc 100644
--- a/NEWS.txt
+++ b/NEWS.txt
@@ -113,6 +113,24 @@ New features
 
 Upgrading
 -
+- Sstables for tables using with a frozen UDT written by C* 3.0 appear as 
corrupted.
+
+  Background: The serialization-header in the -Statistics.db sstable 
component contains the type information
+  of the table columns. C* 3.0 write incorrect type information for frozen 
UDTs by omitting the
+  "frozen" information. Non-frozen UDTs were introduced by CASSANDRA-7423 
in C* 3.6. Since then, the missing
+  "frozen" information leads to deserialization issues that result in 
CorruptSSTableExceptions, potentially other
+  exceptions as well.
+
+  As a mitigation, the sstable serialization-headers are rewritten to 
contain the missing "frozen" information for
+  UDTs once, when an upgrade from C* 3.0 is detected. This migration does 
not touch snapshots or backups.
+
+  The sstablescrub tool now performs a check of the sstable 
serialization-header against the schema. A mismatch of
+  the types in the serialization-header and the schema will cause 
sstablescrub to error out and stop by default.
+  See the new `-e` option. `-e off` disables the new validation code. `-e 
fix` or `-e fix-only`, e.g.
+  `sstablescrub -e fix keyspace table`, will validate the 
serialization-header, rewrite the non-frozen UDTs
+  in the serialzation-header to frozen UDTs, if that matches the schema, 
and continue with scrub.
+  See `sstablescrub -h`.
+  (CASSANDRA-15035)
 - CASSANDRA-13241 lowered the default chunk_lengh_in_kb for compresesd 
tables from
   64kb to 16kb. For highly compressible data this can have a noticeable 
impact
   on space utilization. You may want to consider manually specifying this 
value.
diff --git a/src/java/org/apache/cassandra/cql3/CQL3Type.java 
b/src/java/org/apache/cassandra/cql3/CQL3Type.java
index 340a992..ee2db68 100644
--- a/src/java/org/apache/cassandra/cql3/CQL3Type.java
+++ b/src/java/org/apache/cassandra/cql3/CQL3Type.java
@@ -704,8 +704,10 @@ public interface CQL3Type
 {
 if (innerType instanceof RawCollection)
 throw new InvalidRequestException("Non-frozen collections 
are not allowed inside collections: " + this);
-else
+else if (innerType.isUDT())
 throw new InvalidRequestException("Non-frozen UDTs are not 
allowed inside collections: 

[cassandra] branch trunk updated (5f7c886 -> 7a7eece)

[snazy]

This is an automated email from the ASF dual-hosted git repository.

snazy pushed a change to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra.git.


from 5f7c886  Merge branch 'cassandra-3.11' into trunk
 new 4f27a37  C* 3.0 sstables w/ UDTs are corrupted in C* 3.11 and 4.0
 new ffab2b8  C* 3.0 sstables w/ UDTs are corrupted in C* 3.11 and 4.0
 new 7a7eece  Merge branch 'cassandra-3.11' into trunk

The 3 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 CHANGES.txt|   1 +
 NEWS.txt   |  18 +
 src/java/org/apache/cassandra/cql3/CQL3Type.java   |   4 +-
 .../apache/cassandra/db/SerializationHeader.java   |  12 +
 .../org/apache/cassandra/db/SystemKeyspace.java|   2 +
 .../org/apache/cassandra/db/rows/AbstractCell.java |  15 +-
 .../org/apache/cassandra/db/rows/AbstractRow.java  |  22 +-
 .../db/rows/AbstractTypeVersionComparator.java | 121 ---
 .../db/rows/ColumnMetadataVersionComparator.java   |  85 ++
 src/java/org/apache/cassandra/db/rows/Row.java |   2 +-
 src/java/org/apache/cassandra/db/rows/Rows.java|   2 +-
 .../cassandra/io/sstable/SSTableHeaderFix.java | 918 
 .../cassandra/io/sstable/format/SSTableReader.java |   5 +
 .../io/sstable/metadata/IMetadataSerializer.java   |   5 +
 .../io/sstable/metadata/MetadataSerializer.java|   2 +-
 .../apache/cassandra/service/CassandraDaemon.java  |   3 +
 .../apache/cassandra/tools/StandaloneScrubber.java | 144 ++-
 .../org/apache/cassandra/utils/FBUtilities.java|  12 +
 ...va => ColumnMetadataVersionComparatorTest.java} |  29 +-
 .../cassandra/io/sstable/SSTableHeaderFixTest.java | 964 +
 .../schema/TupleTypesRepresentationTest.java   | 403 +
 21 files changed, 2621 insertions(+), 148 deletions(-)
 delete mode 100644 
src/java/org/apache/cassandra/db/rows/AbstractTypeVersionComparator.java
 create mode 100644 
src/java/org/apache/cassandra/db/rows/ColumnMetadataVersionComparator.java
 create mode 100644 
src/java/org/apache/cassandra/io/sstable/SSTableHeaderFix.java
 rename 
test/unit/org/apache/cassandra/db/rows/{AbstractTypeVersionComparatorTest.java 
=> ColumnMetadataVersionComparatorTest.java} (71%)
 create mode 100644 
test/unit/org/apache/cassandra/io/sstable/SSTableHeaderFixTest.java
 create mode 100644 
test/unit/org/apache/cassandra/schema/TupleTypesRepresentationTest.java


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[cassandra] 02/02: Merge branch 'cassandra-3.11' into trunk

[snazy]

This is an automated email from the ASF dual-hosted git repository.

snazy pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra.git

commit 7a7eece9578312a2f9d77de6e0755a3c3c542e99
Merge: 4f27a37 ffab2b8
Author: Robert Stupp 
AuthorDate: Fri Jan 31 10:08:33 2020 +0100

Merge branch 'cassandra-3.11' into trunk



-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[cassandra-builds] branch master updated: Jenkins jobs are to clean up tmp files and the workspace folder when they are finished.

[mck]

This is an automated email from the ASF dual-hosted git repository.

mck pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/cassandra-builds.git


The following commit(s) were added to refs/heads/master by this push:
 new 806018d  Jenkins jobs are to clean up tmp files and the workspace 
folder when they are finished.
806018d is described below

commit 806018d090b3339400e811f0d747ef95251b8206
Author: mck 
AuthorDate: Fri Jan 31 08:31:53 2020 +0100

Jenkins jobs are to clean up tmp files and the workspace folder when they 
are finished.

Jobs were generating a lot of test data under /tmp that wasn't getting 
cleaned up, as well as 1GB+ workspaces because of build directories.
And because the jobs in the pipeline don't re-use the built artifact for 
that SHA (from the first pipeline step), each job clones and builds its own 
cassandra codebase.

ref: https://issues.apache.org/jira/browse/INFRA-19793
---
 jenkins-dsl/cassandra_job_dsl_seed.groovy | 82 ---
 1 file changed, 76 insertions(+), 6 deletions(-)

diff --git a/jenkins-dsl/cassandra_job_dsl_seed.groovy 
b/jenkins-dsl/cassandra_job_dsl_seed.groovy
index c876ebf..36a2c7b 100644
--- a/jenkins-dsl/cassandra_job_dsl_seed.groovy
+++ b/jenkins-dsl/cassandra_job_dsl_seed.groovy
@@ -123,6 +123,16 @@ job('Cassandra-template-artifacts') {
 }
 }
 }
+postBuildTask {
+task('.', '''
+echo "Cleaning project…";
+ant realclean;
+echo "Reporting disk usage…";
+df -h ; du -hs `pwd` ; du -hs ../* ;
+echo "Cleaning tmp…";
+find /tmp -type -f -atime +3 -user jenkins -and -not -exec 
fuser -s {} ';' -and -delete 2>/dev/null
+''')
+}
 }
 }
 
@@ -173,7 +183,15 @@ job('Cassandra-template-test') {
 }
 }
 postBuildTask {
-task('.', 'echo "Finding job process orphans.."; if pgrep -af 
${JOB_BASE_NAME}; then pkill -9 -f ${JOB_BASE_NAME}; fi')
+task('.', '''
+echo "Finding job process orphans…"; if pgrep -af 
${JOB_BASE_NAME}; then pkill -9 -f ${JOB_BASE_NAME}; fi;
+echo "Cleaning project…";
+ant realclean;
+echo "Reporting disk usage…";
+df -h ; du -hs `pwd` ; du -hs ../* ;
+echo "Cleaning tmp…";
+find /tmp -type -f -atime +3 -user jenkins -and -not -exec 
fuser -s {} ';' -and -delete 2>/dev/null
+''')
 }
 }
 }
@@ -225,7 +243,15 @@ job('Cassandra-template-dtest') {
 }
 }
 postBuildTask {
-task('.', 'echo "Finding job process orphans.."; if pgrep -af 
${JOB_BASE_NAME}; then pkill -9 -f ${JOB_BASE_NAME}; fi')
+task('.', '''
+echo "Finding job process orphans…"; if pgrep -af 
${JOB_BASE_NAME}; then pkill -9 -f ${JOB_BASE_NAME}; fi;
+echo "Cleaning project…";
+ant realclean;
+echo "Reporting disk usage…";
+df -h ; du -hs `pwd` ; du -hs ../* ;
+echo "Cleaning tmp…";
+find /tmp -type -f -atime +3 -user jenkins -and -not -exec 
fuser -s {} ';' -and -delete 2>/dev/null
+''')
 }
 }
 }
@@ -282,7 +308,15 @@ matrixJob('Cassandra-template-cqlsh-tests') {
 }
 }
 postBuildTask {
-task('.', 'echo "Finding job process orphans.."; if pgrep -af 
"${JOB_BASE_NAME}"; then pkill -9 -f "${JOB_BASE_NAME}"; fi')
+task('.', '''
+echo "Finding job process orphans…"; if pgrep -af 
${JOB_BASE_NAME}; then pkill -9 -f ${JOB_BASE_NAME}; fi;
+echo "Cleaning project…";
+ant realclean;
+echo "Reporting disk usage…";
+df -h ; du -hs `pwd` ; du -hs ../* ;
+echo "Cleaning tmp…";
+find /tmp -type -f -atime +3 -user jenkins -and -not -exec 
fuser -s {} ';' -and -delete 2>/dev/null
+''')
 }
 }
 }
@@ -461,6 +495,18 @@ job('Cassandra-devbranch-artifacts') {
 buildDescription('', buildDescStr)
 shell("git clean -xdff ; git clone -b ${buildsBranch} ${buildsRepo}")
 }
+publishers {
+postBuildTask {
+task('.', '''
+echo "Cleaning project…";
+ant realclean;
+echo "Reporting disk usage…";
+df -h ; du -hs `pwd` ; du -hs ../* ;
+echo "Cleaning tmp…";
+find /tmp -type -f -atime +3 -user jenkins -and -not -exec 
fuser -s {} ';' -and -delete 2>/dev/null
+''')
+}
+}
 }
 
 /**
@@ -518,7 +564,15 @@ testTargets.each {
 }
 }
 postBuildTask {
-task('.', 'echo "Finding job process orphans..";