[jira] [Created] (CASSANDRA-15990) Running CQL command with non-ASCII values raises UnicodeDecodeError
Joseph Chu created CASSANDRA-15990:
--
Summary: Running CQL command with non-ASCII values raises
UnicodeDecodeError
Key: CASSANDRA-15990
URL: https://issues.apache.org/jira/browse/CASSANDRA-15990
Project: Cassandra
Issue Type: Bug
Components: Tool/cqlsh
Reporter: Joseph Chu
There are INSERT statements that contains non-ASCII values that have run fine
in Cassandra 3.11, but now raises a UnicodeDecodeError when I try executing
them in 4.0-alpha4 and 4.0-beta1.
Example input and output:
{code:java}
echo $LANG
en_US.UTF-8
$ cqlsh --debug
Using CQL driver:
Using connect timeout: 5 seconds
Using 'utf-8' encoding
Using ssl: False
Connected to Cassandra Cluster at 127.0.0.1:9042.
[cqlsh 5.0.1 | Cassandra 4.0-beta1 | CQL spec 3.4.5 | Native protocol v4]
Use HELP for help.
cqlsh> CREATE KEYSPACE killr_video WITH replication = {'class':
'NetworkTopologyStrategy', 'DC-Houston': 1};
cqlsh> USE killr_video;
cqlsh:killr_video> CREATE TABLE movies_by_genre ( genre TEXT, title TEXT, year
INT, duration INT, avg_rating FLOAT, country TEXT, PRIMARY KEY ((genre), title,
year));
cqlsh:killr_video> INSERT INTO movies_by_genre (genre, title, year, duration,
avg_rating, country)
... VALUES ('Action', 'The Extraordinary Adventures of Adèle Blanc-Sec', 2010,
107, 6.30, 'France');
Traceback (most recent call last):
File "/usr/share/cassandra/bin/cqlsh.py", line 937, in onecmd
self.handle_statement(st, statementtext)
File "/usr/share/cassandra/bin/cqlsh.py", line 962, in handle_statement
readline.add_history(new_hist)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe8' in position
134: ordinal not in range(128){code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15980) Improve log messages for socket connection/disconnection
[
https://issues.apache.org/jira/browse/CASSANDRA-15980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jon Meredith updated CASSANDRA-15980:
-
Test and Documentation Plan: I started up a local CCM instance using the
guide from Nate on [TLP
Blog|https://thelastpickle.com/blog/2015/09/30/hardening-cassandra-step-by-step-part-1-server-to-server.html]
to configure SSL, then checked the messages were sane & formatted.
Status: Patch Available (was: Open)
[Branch|https://github.com/jonmeredith/cassandra/tree/c15980]
[PR|https://github.com/apache/cassandra/pull/698]
[CircleCI|https://app.circleci.com/pipelines/github/jonmeredith/cassandra?branch=c15980]
> Improve log messages for socket connection/disconnection
>
>
> Key: CASSANDRA-15980
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15980
> Project: Cassandra
> Issue Type: Bug
> Components: Observability/Logging
>Reporter: Jon Meredith
>Assignee: Jon Meredith
>Priority: Normal
> Fix For: 4.0-beta
>
>
> Logging for inbound SSL connections can take place before protocol
> negotiation has taken place and logs a misleading cipher that could cause
> problems for security auditing.
>
>
> {code:java}
> INFO 2020-07-03T13:57:58,380 [Messaging-EventLoop-3-1]
> org.apache.cassandra.net.InboundConnectionInitiator:242 - connection from
> peer /1.1.1.1:57899 to /2.2.2.2:7000, protocol = TLSv1.2, cipher suite =
> SSL_NULL_WITH_NULL_NULL
> {code}
>
> Instead Cassandra should log the connection & protocol, then once the cipher
> has been negotiated log the agreed upon cipher.
>
>
> If the inbound SSL connection does not present a client certificate,
> Cassandra logs this error, even if the client wasn't required to.
> {code:java}
> ERROR 2020-07-14T11:58:45,925 [Native-Transport-Requests-1]
> org.apache.cassandra.transport.ServerConnection:140 - Failed to get peer
> certificates for peer /4.3.2.1:59263
> {code}
>
> Logging the absense of verified certificates should be a concern of the
> SaslNegotiator if it requires it, and not something worth alerting the
> operator for generally. Downgrade to debug message to make investigation
> possible if needed.
>
>
> Finally, to help with logging issues related to disconnection, add a log
> statement when an instance decides it no longer needs to keep a gossip
> connection open when cleaning up connections in
> org.apache.cassandra.net.OutboundConnections.UnusedConnectionMonitor#closeUnusedSinceLastRun
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15980) Improve log messages for socket connection/disconnection
[
https://issues.apache.org/jira/browse/CASSANDRA-15980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17166046#comment-17166046
]
Jon Meredith commented on CASSANDRA-15980:
--
Patch almost ready for submission, here are some sample log output lines.
Example of SSL enabled messaging connections being established
{code:java}
INFO [Messaging-EventLoop-3-11] 2020-07-27 16:36:14,746
InboundConnectionInitiator.java:241 - connection from peer /127.0.0.1:62955 to
/127.0.0.1:7000, protocol = TLSv1.2
INFO [Messaging-EventLoop-3-12] 2020-07-27 16:36:14,747
InboundConnectionInitiator.java:241 - connection from peer /127.0.0.1:62956 to
/127.0.0.1:7000, protocol = TLSv1.2
INFO [Messaging-EventLoop-3-12] 2020-07-27 16:36:14,761
InboundConnectionInitiator.java:457 -
127.0.0.3:7000(127.0.0.1:62956)->127.0.0.1:7000-LARGE_MESSAGES-7dc6697c
messaging connection established, version = 1
2, framing = CRC, encryption = enabled
(factory=openssl;protocol=TLSv1.2;cipher=TLS_RSA_WITH_AES_256_CBC_SHA)
INFO [Messaging-EventLoop-3-11] 2020-07-27 16:36:14,761
InboundConnectionInitiator.java:457 -
127.0.0.3:7000(127.0.0.1:62955)->127.0.0.1:7000-SMALL_MESSAGES-24cc4a4e
messaging connection established, version = 12, framing = CRC, encryption =
enabled (factory=openssl;protocol=TLSv1.2;cipher=TLS_RSA_WITH_AES_256_CBC_SHA)
{code}
And establishing streaming connections
{code:java}
INFO [Messaging-EventLoop-3-12] 2020-07-27 16:40:27,516
InboundConnectionInitiator.java:396 -
127.0.0.1:7000(127.0.0.3:63208)->127.0.0.3:7000-STREAMING-64b53f77 streaming
connection established, version = 12, fr
aming = UNPROTECTED, encryption = enabled
(factory=openssl;protocol=TLSv1.2;cipher=TLS_RSA_WITH_AES_256_CBC_SHA)
{code}
> Improve log messages for socket connection/disconnection
>
>
> Key: CASSANDRA-15980
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15980
> Project: Cassandra
> Issue Type: Bug
> Components: Observability/Logging
>Reporter: Jon Meredith
>Assignee: Jon Meredith
>Priority: Normal
> Fix For: 4.0-beta
>
>
> Logging for inbound SSL connections can take place before protocol
> negotiation has taken place and logs a misleading cipher that could cause
> problems for security auditing.
>
>
> {code:java}
> INFO 2020-07-03T13:57:58,380 [Messaging-EventLoop-3-1]
> org.apache.cassandra.net.InboundConnectionInitiator:242 - connection from
> peer /1.1.1.1:57899 to /2.2.2.2:7000, protocol = TLSv1.2, cipher suite =
> SSL_NULL_WITH_NULL_NULL
> {code}
>
> Instead Cassandra should log the connection & protocol, then once the cipher
> has been negotiated log the agreed upon cipher.
>
>
> If the inbound SSL connection does not present a client certificate,
> Cassandra logs this error, even if the client wasn't required to.
> {code:java}
> ERROR 2020-07-14T11:58:45,925 [Native-Transport-Requests-1]
> org.apache.cassandra.transport.ServerConnection:140 - Failed to get peer
> certificates for peer /4.3.2.1:59263
> {code}
>
> Logging the absense of verified certificates should be a concern of the
> SaslNegotiator if it requires it, and not something worth alerting the
> operator for generally. Downgrade to debug message to make investigation
> possible if needed.
>
>
> Finally, to help with logging issues related to disconnection, add a log
> statement when an instance decides it no longer needs to keep a gossip
> connection open when cleaning up connections in
> org.apache.cassandra.net.OutboundConnections.UnusedConnectionMonitor#closeUnusedSinceLastRun
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRASC-23) Set up structure for handling multiple Cassandra versions
[
https://issues.apache.org/jira/browse/CASSANDRASC-23?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dinesh Joshi updated CASSANDRASC-23:
Status: Ready to Commit (was: Changes Suggested)
Thanks for addressing my comments. LGTM. +1
> Set up structure for handling multiple Cassandra versions
> -
>
> Key: CASSANDRASC-23
> URL: https://issues.apache.org/jira/browse/CASSANDRASC-23
> Project: Sidecar for Apache Cassandra
> Issue Type: Improvement
> Components: Configuration
>Reporter: Jon Haddad
>Assignee: Jon Haddad
>Priority: Normal
>
> The first sidecar release will be for Cassandra 4.0, but one of the project
> goals is to be able to handle multiple versions. This will be especially
> important in mixed version clusters, or even mixed version 1:N sidecar to C*
> nodes (see CASSANDRASC-17).
> This JIRA is to lay the foundational work for supporting multiple Cassandra
> versions.
> Each Cassandra version (for example \{{cassandra40}}, \{{cassandra50}}, will
> be in a separate Gradle subproject, implementing a common interface defined
> in a \{{common}} subproject. A common cassandra integration testing
> framework will be able to test every version to ensure each version adheres
> to the expected interface.
> Each module will define the minimum compatibility it supports, so if someone
> (for example) wants to implement a Cassandra30 module for their own cluster,
> they will have the ability to do so.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRASC-23) Set up structure for handling multiple Cassandra versions
[
https://issues.apache.org/jira/browse/CASSANDRASC-23?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dinesh Joshi updated CASSANDRASC-23:
Status: Changes Suggested (was: Review In Progress)
> Set up structure for handling multiple Cassandra versions
> -
>
> Key: CASSANDRASC-23
> URL: https://issues.apache.org/jira/browse/CASSANDRASC-23
> Project: Sidecar for Apache Cassandra
> Issue Type: Improvement
> Components: Configuration
>Reporter: Jon Haddad
>Assignee: Jon Haddad
>Priority: Normal
>
> The first sidecar release will be for Cassandra 4.0, but one of the project
> goals is to be able to handle multiple versions. This will be especially
> important in mixed version clusters, or even mixed version 1:N sidecar to C*
> nodes (see CASSANDRASC-17).
> This JIRA is to lay the foundational work for supporting multiple Cassandra
> versions.
> Each Cassandra version (for example \{{cassandra40}}, \{{cassandra50}}, will
> be in a separate Gradle subproject, implementing a common interface defined
> in a \{{common}} subproject. A common cassandra integration testing
> framework will be able to test every version to ensure each version adheres
> to the expected interface.
> Each module will define the minimum compatibility it supports, so if someone
> (for example) wants to implement a Cassandra30 module for their own cluster,
> they will have the ability to do so.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRASC-23) Set up structure for handling multiple Cassandra versions
[
https://issues.apache.org/jira/browse/CASSANDRASC-23?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dinesh Joshi updated CASSANDRASC-23:
Status: Patch Available (was: In Progress)
> Set up structure for handling multiple Cassandra versions
> -
>
> Key: CASSANDRASC-23
> URL: https://issues.apache.org/jira/browse/CASSANDRASC-23
> Project: Sidecar for Apache Cassandra
> Issue Type: Improvement
> Components: Configuration
>Reporter: Jon Haddad
>Assignee: Jon Haddad
>Priority: Normal
>
> The first sidecar release will be for Cassandra 4.0, but one of the project
> goals is to be able to handle multiple versions. This will be especially
> important in mixed version clusters, or even mixed version 1:N sidecar to C*
> nodes (see CASSANDRASC-17).
> This JIRA is to lay the foundational work for supporting multiple Cassandra
> versions.
> Each Cassandra version (for example \{{cassandra40}}, \{{cassandra50}}, will
> be in a separate Gradle subproject, implementing a common interface defined
> in a \{{common}} subproject. A common cassandra integration testing
> framework will be able to test every version to ensure each version adheres
> to the expected interface.
> Each module will define the minimum compatibility it supports, so if someone
> (for example) wants to implement a Cassandra30 module for their own cluster,
> they will have the ability to do so.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRASC-23) Set up structure for handling multiple Cassandra versions
[
https://issues.apache.org/jira/browse/CASSANDRASC-23?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dinesh Joshi updated CASSANDRASC-23:
Reviewers: Dinesh Joshi, Vinay Chella, Dinesh Joshi (was: Dinesh Joshi,
Vinay Chella)
Dinesh Joshi, Vinay Chella, Dinesh Joshi (was: Dinesh Joshi,
Vinay Chella)
Status: Review In Progress (was: Patch Available)
> Set up structure for handling multiple Cassandra versions
> -
>
> Key: CASSANDRASC-23
> URL: https://issues.apache.org/jira/browse/CASSANDRASC-23
> Project: Sidecar for Apache Cassandra
> Issue Type: Improvement
> Components: Configuration
>Reporter: Jon Haddad
>Assignee: Jon Haddad
>Priority: Normal
>
> The first sidecar release will be for Cassandra 4.0, but one of the project
> goals is to be able to handle multiple versions. This will be especially
> important in mixed version clusters, or even mixed version 1:N sidecar to C*
> nodes (see CASSANDRASC-17).
> This JIRA is to lay the foundational work for supporting multiple Cassandra
> versions.
> Each Cassandra version (for example \{{cassandra40}}, \{{cassandra50}}, will
> be in a separate Gradle subproject, implementing a common interface defined
> in a \{{common}} subproject. A common cassandra integration testing
> framework will be able to test every version to ensure each version adheres
> to the expected interface.
> Each module will define the minimum compatibility it supports, so if someone
> (for example) wants to implement a Cassandra30 module for their own cluster,
> they will have the ability to do so.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRASC-23) Set up structure for handling multiple Cassandra versions
[
https://issues.apache.org/jira/browse/CASSANDRASC-23?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dinesh Joshi updated CASSANDRASC-23:
Status: In Progress (was: Patch Available)
> Set up structure for handling multiple Cassandra versions
> -
>
> Key: CASSANDRASC-23
> URL: https://issues.apache.org/jira/browse/CASSANDRASC-23
> Project: Sidecar for Apache Cassandra
> Issue Type: Improvement
> Components: Configuration
>Reporter: Jon Haddad
>Assignee: Jon Haddad
>Priority: Normal
>
> The first sidecar release will be for Cassandra 4.0, but one of the project
> goals is to be able to handle multiple versions. This will be especially
> important in mixed version clusters, or even mixed version 1:N sidecar to C*
> nodes (see CASSANDRASC-17).
> This JIRA is to lay the foundational work for supporting multiple Cassandra
> versions.
> Each Cassandra version (for example \{{cassandra40}}, \{{cassandra50}}, will
> be in a separate Gradle subproject, implementing a common interface defined
> in a \{{common}} subproject. A common cassandra integration testing
> framework will be able to test every version to ensure each version adheres
> to the expected interface.
> Each module will define the minimum compatibility it supports, so if someone
> (for example) wants to implement a Cassandra30 module for their own cluster,
> they will have the ability to do so.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15988) Add nodetool getfullquerylog
[ https://issues.apache.org/jira/browse/CASSANDRA-15988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17166013#comment-17166013 ] Ekaterina Dimitrova commented on CASSANDRA-15988: - Great! Looks like it is decided! Thanks both for looking into that! > Add nodetool getfullquerylog > - > > Key: CASSANDRA-15988 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15988 > Project: Cassandra > Issue Type: Improvement > Components: Tool/fql >Reporter: Ekaterina Dimitrova >Priority: Normal > Fix For: 4.0-rc > > > This ticket is raised based on CASSANDRA-15791 and valuable feedback provided > by [~jshook]. > There are two outstanding questions: > * forming the exact shape of such a command and how it can benefit the > users; to be discussed in detail in this ticket > * Is this a thing we as a project can add to 4.0 beta or it should be > considered in 4.1 for example > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15299) CASSANDRA-13304 follow-up: improve checksumming and compression in protocol v5-beta
[
https://issues.apache.org/jira/browse/CASSANDRA-15299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17165987#comment-17165987
]
Olivier Michallat commented on CASSANDRA-15299:
---
I've noticed an issue when the client starts protocol negotiation with an
unsupported version. For example at some point in the future we might have a
newer, v6-enabled driver trying to negotiate with a now legacy Cassandra 4.0.0.
This causes {{Frame.Decoder}} to throw a {{ProtocolException}} while decoding
the first OPTIONS frame. The exception bubbles up the "initial" pipeline to
reach {{PreV5Handlers.ExceptionHandler}}, which returns an error response:
{code}
ErrorMessage errorMessage = ErrorMessage.fromException(cause, handler);
...
ChannelFuture future = ctx.writeAndFlush(errorMessage);
{code}
But looking at {{PipelineConfigurator.configureInitialPipeline}}, the next
outbound handler is {{Frame.Encoder}}. It processes frames, not messages.
{{PreV5Handlers.ExceptionHandler}} needs to wrap the response. I'm not sure
which protocol version to use, but if I'm reading things correctly I think
trunk uses {{ProtocolVersion.CURRENT}} in this situation:
{code}
ChannelFuture future =
ctx.writeAndFlush(errorMessage.encode(ProtocolVersion.CURRENT));
{code}
With this change the negotiation works as expected.
> CASSANDRA-13304 follow-up: improve checksumming and compression in protocol
> v5-beta
> ---
>
> Key: CASSANDRA-15299
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15299
> Project: Cassandra
> Issue Type: Improvement
> Components: Messaging/Client
>Reporter: Aleksey Yeschenko
>Assignee: Alex Petrov
>Priority: Normal
> Labels: protocolv5
> Fix For: 4.0-alpha
>
>
> CASSANDRA-13304 made an important improvement to our native protocol: it
> introduced checksumming/CRC32 to request and response bodies. It’s an
> important step forward, but it doesn’t cover the entire stream. In
> particular, the message header is not covered by a checksum or a crc, which
> poses a correctness issue if, for example, {{streamId}} gets corrupted.
> Additionally, we aren’t quite using CRC32 correctly, in two ways:
> 1. We are calculating the CRC32 of the *decompressed* value instead of
> computing the CRC32 on the bytes written on the wire - losing the properties
> of the CRC32. In some cases, due to this sequencing, attempting to decompress
> a corrupt stream can cause a segfault by LZ4.
> 2. When using CRC32, the CRC32 value is written in the incorrect byte order,
> also losing some of the protections.
> See https://users.ece.cmu.edu/~koopman/pubs/KoopmanCRCWebinar9May2012.pdf for
> explanation for the two points above.
> Separately, there are some long-standing issues with the protocol - since
> *way* before CASSANDRA-13304. Importantly, both checksumming and compression
> operate on individual message bodies rather than frames of multiple complete
> messages. In reality, this has several important additional downsides. To
> name a couple:
> # For compression, we are getting poor compression ratios for smaller
> messages - when operating on tiny sequences of bytes. In reality, for most
> small requests and responses we are discarding the compressed value as it’d
> be smaller than the uncompressed one - incurring both redundant allocations
> and compressions.
> # For checksumming and CRC32 we pay a high overhead price for small messages.
> 4 bytes extra is *a lot* for an empty write response, for example.
> To address the correctness issue of {{streamId}} not being covered by the
> checksum/CRC32 and the inefficiency in compression and checksumming/CRC32, we
> should switch to a framing protocol with multiple messages in a single frame.
> I suggest we reuse the framing protocol recently implemented for internode
> messaging in CASSANDRA-15066 to the extent that its logic can be borrowed,
> and that we do it before native protocol v5 graduates from beta. See
> https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/net/FrameDecoderCrc.java
> and
> https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/net/FrameDecoderLZ4.java.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15988) Add nodetool getfullquerylog
[ https://issues.apache.org/jira/browse/CASSANDRA-15988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Capwell updated CASSANDRA-15988: -- Fix Version/s: 4.0-rc > Add nodetool getfullquerylog > - > > Key: CASSANDRA-15988 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15988 > Project: Cassandra > Issue Type: Improvement > Components: Tool/fql >Reporter: Ekaterina Dimitrova >Priority: Normal > Fix For: 4.0-rc > > > This ticket is raised based on CASSANDRA-15791 and valuable feedback provided > by [~jshook]. > There are two outstanding questions: > * forming the exact shape of such a command and how it can benefit the > users; to be discussed in detail in this ticket > * Is this a thing we as a project can add to 4.0 beta or it should be > considered in 4.1 for example > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15988) Add nodetool getfullquerylog
[ https://issues.apache.org/jira/browse/CASSANDRA-15988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Capwell updated CASSANDRA-15988: -- Change Category: Operability Complexity: Low Hanging Fruit Component/s: Tool/fql Status: Open (was: Triage Needed) > Add nodetool getfullquerylog > - > > Key: CASSANDRA-15988 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15988 > Project: Cassandra > Issue Type: Improvement > Components: Tool/fql >Reporter: Ekaterina Dimitrova >Priority: Normal > > This ticket is raised based on CASSANDRA-15791 and valuable feedback provided > by [~jshook]. > There are two outstanding questions: > * forming the exact shape of such a command and how it can benefit the > users; to be discussed in detail in this ticket > * Is this a thing we as a project can add to 4.0 beta or it should be > considered in 4.1 for example > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15988) Add nodetool getfullquerylog
[ https://issues.apache.org/jira/browse/CASSANDRA-15988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17165974#comment-17165974 ] David Capwell commented on CASSANDRA-15988: --- bq. I think that it is reasonable to show the user whether FQL is enabled or not as the first item. we have enable/disable/reset but don't have an "get" version; agree this should be added. bq. In terms of whether it goes into 4.0 or 4.1, I think it is obviously missing functionality it does not break any API, it does not increase the testing scope, and comes as low risk; personally I am in favor of 4.0. bq. Not being able to query the state of the service without changing it is a problem. yeah... I hear too much "just take a heap dump"... visibility really should be part of the features requirements bq. Consider a scenario where multiple users are managing a system together and need to double check the state of things before proceeding to the next step in their process A lot of automation does this as well, this isn't just for manual users but also benefits automation. > Add nodetool getfullquerylog > - > > Key: CASSANDRA-15988 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15988 > Project: Cassandra > Issue Type: Improvement >Reporter: Ekaterina Dimitrova >Priority: Normal > > This ticket is raised based on CASSANDRA-15791 and valuable feedback provided > by [~jshook]. > There are two outstanding questions: > * forming the exact shape of such a command and how it can benefit the > users; to be discussed in detail in this ticket > * Is this a thing we as a project can add to 4.0 beta or it should be > considered in 4.1 for example > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15938) Fix support for adding UDT fields to clustering keys
[ https://issues.apache.org/jira/browse/CASSANDRA-15938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17165958#comment-17165958 ] Caleb Rackliffe commented on CASSANDRA-15938: - [~ifesdjeen] Just to make sure I'm not missing anything, aside from an extra newline, the 3.0 -> trunk patches are identical? > Fix support for adding UDT fields to clustering keys > > > Key: CASSANDRA-15938 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15938 > Project: Cassandra > Issue Type: Bug > Components: Feature/UDT >Reporter: Alex Petrov >Assignee: Alex Petrov >Priority: Normal > Fix For: 3.0.x, 3.11.x, 4.0-beta > > > Adding UDT fields to clustering keys is broken in all versions, however > slightly differently. > In 4.0, there will be a brief moment while schema changes are propagated > during which we won’t be able to decode and compare byte sequences. > Unfortunately, it is unclear what we should do in such cases, since we can’t > just come up with a comparator, and we can’t ignore non-null trailing values, > since this will lead to cases where compare for tuples `a;1` and `a;2` would > return 0, effectively making them equal, and we don’t know how to compare > unknown trailing values. Probably we should reject such query since we can’t > sort correctly, but we should make the error message more descriptive than > just "Index 1 out of bounds for length 1”. The only problem is that we get > this exception only on flush right now, so data already propagates to the > node by that time. > In 3.0, the problem is a bit worse than that, since in 3.0 we do not ignore > trailing nulls, so some of the values, written before `ALTER TYPE .. ADD` > become inaccessible. Both old values, and the new ones should always be > accessible. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15989) Provide easy copypasta config formatting for nodetool get commands
[
https://issues.apache.org/jira/browse/CASSANDRA-15989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Shook updated CASSANDRA-15989:
---
Description:
Allow all nodetool commands which print out the state of the node or cluster to
do so in a way that makes it easy to re-use or paste on other nodes or config
files.
For example, the command getcompactionthroughput formats its output like this:
{noformat}
[jshook@cass4 bin]$ ./nodetool getcompactionthroughput
Current compaction throughput: 64 MB/s
{noformat}
But with an --as-yaml option, it could do this instead:
{noformat}
[jshook@cass4 bin]$ ./nodetool getcompactionthroughput --as-yaml
compaction_throughput_mb_per_sec: 64{noformat}
and with an --as-cli option, it could do this:
{noformat}
[jshook@cass4 bin]$ ./nodetool getcompactionthroughput --as-cli
./nodetool setcompactionthroughput 64{noformat}
Any other nodetool standard options should simply be carried along to the
--as-cli form, with the exception of -pw.
Any -pw options should be elided with a warning in comments, but -pwf options
should be allowed. This would allow users using -pw to append a password at
their discretion, but would allow -pwf to work as usual.
In the absence of either of the options above (--as-yaml or --as-cli) the
formatting should not be changed to avoid breaking extant tool integrations.
was:
Allow all nodetool commands which print out the state of the node or cluster to
do so in a way that makes it easy to re-use or paste on other nodes or config
files.
For example, the command getcompactionthroughput formats its output like this:
{noformat}
[jshook@cass4 bin]$ ./nodetool getcompactionthroughput
Current compaction throughput: 64 MB/s
{noformat}
But with an --as-yaml option, it could do this instead:
{noformat}
[jshook@cass4 bin]$ ./nodetool getcompactionthroughput --as-yaml
compaction_throughput_mb_per_sec: 64{noformat}
and with an --as-cli option, it could do this:
{noformat}
[jshook@cass4 bin]$ ./nodetool getcompactionthroughput --as-cli
./nodetool setcompactionthroughput 64{noformat}
Any other nodetool standard options should simply be carried along to the
--as-cli form, with the exception of -pw.
Any -pw options should be elided with a warning in comments, but -pwf options
should be allowed. This would allow users using -pw to append a password at
their discretion, but would allow -pwf to work as usual.
> Provide easy copypasta config formatting for nodetool get commands
> --
>
> Key: CASSANDRA-15989
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15989
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Jonathan Shook
>Priority: Normal
>
> Allow all nodetool commands which print out the state of the node or cluster
> to do so in a way that makes it easy to re-use or paste on other nodes or
> config files.
> For example, the command getcompactionthroughput formats its output like this:
> {noformat}
> [jshook@cass4 bin]$ ./nodetool getcompactionthroughput
> Current compaction throughput: 64 MB/s
> {noformat}
> But with an --as-yaml option, it could do this instead:
> {noformat}
> [jshook@cass4 bin]$ ./nodetool getcompactionthroughput --as-yaml
> compaction_throughput_mb_per_sec: 64{noformat}
> and with an --as-cli option, it could do this:
> {noformat}
> [jshook@cass4 bin]$ ./nodetool getcompactionthroughput --as-cli
> ./nodetool setcompactionthroughput 64{noformat}
> Any other nodetool standard options should simply be carried along to the
> --as-cli form, with the exception of -pw.
> Any -pw options should be elided with a warning in comments, but -pwf options
> should be allowed. This would allow users using -pw to append a password at
> their discretion, but would allow -pwf to work as usual.
> In the absence of either of the options above (--as-yaml or --as-cli) the
> formatting should not be changed to avoid breaking extant tool integrations.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-15989) Provide easy copypasta config formatting for nodetool get commands
Jonathan Shook created CASSANDRA-15989:
--
Summary: Provide easy copypasta config formatting for nodetool get
commands
Key: CASSANDRA-15989
URL: https://issues.apache.org/jira/browse/CASSANDRA-15989
Project: Cassandra
Issue Type: Improvement
Reporter: Jonathan Shook
Allow all nodetool commands which print out the state of the node or cluster to
do so in a way that makes it easy to re-use or paste on other nodes or config
files.
For example, the command getcompactionthroughput formats its output like this:
{noformat}
[jshook@cass4 bin]$ ./nodetool getcompactionthroughput
Current compaction throughput: 64 MB/s
{noformat}
But with an --as-yaml option, it could do this instead:
{noformat}
[jshook@cass4 bin]$ ./nodetool getcompactionthroughput --as-yaml
compaction_throughput_mb_per_sec: 64{noformat}
and with an --as-cli option, it could do this:
{noformat}
[jshook@cass4 bin]$ ./nodetool getcompactionthroughput --as-cli
./nodetool setcompactionthroughput 64{noformat}
Any other nodetool standard options should simply be carried along to the
--as-cli form, with the exception of -pw.
Any -pw options should be elided with a warning in comments, but -pwf options
should be allowed. This would allow users using -pw to append a password at
their discretion, but would allow -pwf to work as usual.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15946) NPE when sending REQUEST_RSP from 3.0 to 4.0 in in-jvm dtests
[
https://issues.apache.org/jira/browse/CASSANDRA-15946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jordan West updated CASSANDRA-15946:
Bug Category: Parent values: Code(13163)Level 1 values: Bug - Unclear
Impact(13164)
Complexity: Normal
Discovered By: Unit Test
Severity: Normal
Status: Open (was: Triage Needed)
> NPE when sending REQUEST_RSP from 3.0 to 4.0 in in-jvm dtests
> -
>
> Key: CASSANDRA-15946
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15946
> Project: Cassandra
> Issue Type: Bug
> Components: Test/dtest
>Reporter: Jacek Lewandowski
>Assignee: Jacek Lewandowski
>Priority: Normal
>
> There is a communication problem when testing upgrades using in-JVM dtest
> between Cassandra 3 and 4.
> In a method {{registerInboundFilter}} of {{Instance}}, we get a message which
> was just received and we prepare it for filtering as part of which, we
> serialize the payload again. This is fine when dealing with incoming
> Cassandra 4 message, because we can serialize it. However when we get the
> Cassandra 3 message, which uses a different protocol, and we get something
> like {{REQUEST_RSP}}, we can surely deserialize it through some special
> deserialization path, but we cannot serialize the payload for it as there is
> no serializer defined for {{REQUEST_RSP}} - no wonder, why would Cassandra
> 4.0 need to be able to serialize Cassandra 3.0 payloads?
> {code}
> java.lang.NullPointerException: null
> at
> org.apache.cassandra.net.Message$Serializer.serializePost40(Message.java:760)
> at
> org.apache.cassandra.net.Message$Serializer.serialize(Message.java:618)
> at
> org.apache.cassandra.distributed.impl.Instance.serializeMessage(Instance.java:267)
> at
> org.apache.cassandra.distributed.impl.Instance.lambda$registerInboundFilter$4(Instance.java:234)
> at
> org.apache.cassandra.net.InboundSink$Filtered.accept(InboundSink.java:62)
> at
> org.apache.cassandra.net.InboundSink$Filtered.accept(InboundSink.java:49)
> at org.apache.cassandra.net.InboundSink.accept(InboundSink.java:93)
> at
> org.apache.cassandra.distributed.impl.Instance.lambda$null$6(Instance.java:305)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:165)
> at
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:137)
> at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:119)
> at
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15907) Operational Improvements & Hardening for Replica Filtering Protection
[
https://issues.apache.org/jira/browse/CASSANDRA-15907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17165936#comment-17165936
]
Jordan West commented on CASSANDRA-15907:
-
Apologies for the delay on the review. Left some minor comments / nits on the
PR. Overall, these changes LGTM (+1). Thanks for the additional work on this!
> Operational Improvements & Hardening for Replica Filtering Protection
> -
>
> Key: CASSANDRA-15907
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15907
> Project: Cassandra
> Issue Type: Improvement
> Components: Consistency/Coordination, Feature/2i Index
>Reporter: Caleb Rackliffe
>Assignee: Caleb Rackliffe
>Priority: Normal
> Labels: 2i, memory
> Fix For: 3.0.x, 3.11.x, 4.0-beta
>
> Time Spent: 8h
> Remaining Estimate: 0h
>
> CASSANDRA-8272 uses additional space on the heap to ensure correctness for 2i
> and filtering queries at consistency levels above ONE/LOCAL_ONE. There are a
> few things we should follow up on, however, to make life a bit easier for
> operators and generally de-risk usage:
> (Note: Line numbers are based on {{trunk}} as of
> {{3cfe3c9f0dcf8ca8b25ad111800a21725bf152cb}}.)
> *Minor Optimizations*
> * {{ReplicaFilteringProtection:114}} - Given we size them up-front, we may be
> able to use simple arrays instead of lists for {{rowsToFetch}} and
> {{originalPartitions}}. Alternatively (or also), we may be able to null out
> references in these two collections more aggressively. (ex. Using
> {{ArrayList#set()}} instead of {{get()}} in {{queryProtectedPartitions()}},
> assuming we pass {{toFetch}} as an argument to {{querySourceOnKey()}}.)
> * {{ReplicaFilteringProtection:323}} - We may be able to use
> {{EncodingStats.merge()}} and remove the custom {{stats()}} method.
> * {{DataResolver:111 & 228}} - Cache an instance of
> {{UnaryOperator#identity()}} instead of creating one on the fly.
> * {{ReplicaFilteringProtection:217}} - We may be able to scatter/gather
> rather than serially querying every row that needs to be completed. This
> isn't a clear win perhaps, given it targets the latency of single queries and
> adds some complexity. (Certainly a decent candidate to kick even out of this
> issue.)
> *Documentation and Intelligibility*
> * There are a few places (CHANGES.txt, tracing output in
> {{ReplicaFilteringProtection}}, etc.) where we mention "replica-side
> filtering protection" (which makes it seem like the coordinator doesn't
> filter) rather than "replica filtering protection" (which sounds more like
> what we actually do, which is protect ourselves against incorrect replica
> filtering results). It's a minor fix, but would avoid confusion.
> * The method call chain in {{DataResolver}} might be a bit simpler if we put
> the {{repairedDataTracker}} in {{ResolveContext}}.
> *Testing*
> * I want to bite the bullet and get some basic tests for RFP (including any
> guardrails we might add here) onto the in-JVM dtest framework.
> *Guardrails*
> * As it stands, we don't have a way to enforce an upper bound on the memory
> usage of {{ReplicaFilteringProtection}} which caches row responses from the
> first round of requests. (Remember, these are later used to merged with the
> second round of results to complete the data for filtering.) Operators will
> likely need a way to protect themselves, i.e. simply fail queries if they hit
> a particular threshold rather than GC nodes into oblivion. (Having control
> over limits and page sizes doesn't quite get us there, because stale results
> _expand_ the number of incomplete results we must cache.) The fun question is
> how we do this, with the primary axes being scope (per-query, global, etc.)
> and granularity (per-partition, per-row, per-cell, actual heap usage, etc.).
> My starting disposition on the right trade-off between
> performance/complexity and accuracy is having something along the lines of
> cached rows per query. Prior art suggests this probably makes sense alongside
> things like {{tombstone_failure_threshold}} in {{cassandra.yaml}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15988) Add nodetool getfullquerylog
[ https://issues.apache.org/jira/browse/CASSANDRA-15988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17165937#comment-17165937 ] Jonathan Shook commented on CASSANDRA-15988: My take on this: I think that it is reasonable to show the user whether FQL is enabled or not as the first item. Additionally, the configuration of FQL should be dumped to stdout in the same formatting convention of other nodetool get... commands. In terms of whether it goes into 4.0 or 4.1, I think it is obviously missing functionality. Not being able to query the state of the service without changing it is a problem. Consider a scenario where multiple users are managing a system together and need to double check the state of things before proceeding to the next step in their process. As manual as this sounds, many teams still do this type of ops work and need visibility to the operational state of the system. > Add nodetool getfullquerylog > - > > Key: CASSANDRA-15988 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15988 > Project: Cassandra > Issue Type: Improvement >Reporter: Ekaterina Dimitrova >Priority: Normal > > This ticket is raised based on CASSANDRA-15791 and valuable feedback provided > by [~jshook]. > There are two outstanding questions: > * forming the exact shape of such a command and how it can benefit the > users; to be discussed in detail in this ticket > * Is this a thing we as a project can add to 4.0 beta or it should be > considered in 4.1 for example > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-15971) full query log needs improvement
[
https://issues.apache.org/jira/browse/CASSANDRA-15971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17165882#comment-17165882
]
Ekaterina Dimitrova edited comment on CASSANDRA-15971 at 7/27/20, 7:22 PM:
---
I just spent some time to go through the docs and check the points mentioned.
To summarize:
* Improvement of the file format and explicitly saying that we need to
implement two steps as follow would be good thing I think: 1) Update
cassandra.yaml 2) enable FQL logging through nodetool
*
{quote}(maybe)
{noformat}
tools/bin/fqltool help{noformat}
should print out help for all fqltool commands rather than simply repeating the
default The most commonly used fqltool commands are..
{quote}
I saw that when we run
{code:java}
tools/bin/fqltool help{code}
there is a line at the bottom:
{code:java}
See 'fqltool help ' for more information on a specific command.{code}
My personal opinion is that the help listing will be probably too long listing
the full detailed information for all fqltool commands and pointing to the way
to check on specific command is a good compromise.
* Adding such an additional command probably deserves a separate improvement
ticket? (Also, from the perspective that the doc part might be handled by the
people dedicated to the project documentation, and the command part is
engineering work.)
bq. nodetool missing getfullquerylog makes it difficult to verify current
fullquerylog state without changing it. The conventions for nodetool commands
should be followed to avoid confusing users.
was (Author: e.dimitrova):
I just spent some time to go through the docs and check the points mentioned.
To summarize:
* Improvement of the file format and explicitly saying that we need to
implement two steps as follow would be good thing I think: 1) Update
cassandra.yaml 2) enable FQL logging through nodetool
*
{quote}(maybe)
{noformat}
tools/bin/fqltool help{noformat}
should print out help for all fqltool commands rather than simply repeating the
default The most commonly used fqltool commands are..
{quote}
I saw that when we run
{code:java}
tools/bin/fqltool help{code}
there is a line at the bottom:
{code:java}
See 'fqltool help ' for more information on a specific command.{code}
My personal opinion is that the help listing will be probably too long listing
the full detailed information for all fqltool commands and pointing to the way
to check on specific command is a good compromise.
* Adding such an additional command probably deserves a separate improvement
ticket? (Also, from the perspective that the doc part might be handled by the
people dedicated to the project documentation, and the command part is
engineering work.)
nodetool missing getfullquerylog makes it difficult to verify current
fullquerylog state without changing it. The conventions for nodetool commands
should be followed to avoid confusing users.
> full query log needs improvement
>
>
> Key: CASSANDRA-15971
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15971
> Project: Cassandra
> Issue Type: Improvement
> Components: Tool/fql
>Reporter: Jonathan Shook
>Priority: Normal
> Attachments: st1.txt
>
>
> When trying out full query logging as a possible integration for nosqlbench
> usage, I ran across many issues which would make it painful for users. Since
> there were several, they will be added to a single issue for now. This issue
> can be broken up if needed.
>
> FQL doesn't work on my system, even though it says it is logging queries.
> With the following configuration in cassandra.yaml:
>
> {noformat}
> full_query_logging_options:
> log_dir: /REDACTED/fullquerylogs
> roll_cycle: HOURLY
> block: true
> max_queue_weight: 268435456 # 256 MiB
> max_log_size: 17179869184 # 16 GiB
> ## archive command is "/path/to/script.sh %path" where %path is replaced
> with the file being rolled:
> # archive_command:
> # max_archive_retries: 10
> {noformat}
> which appears to be the minimal configuration needed to enable fql, only a
> single file `directory-listing.cq4t` is created, which is a 64K sized file of
> zeroes.
>
>
> Calling bin/nodetool enablefullquerylog throws an error initially.
> [jshook@cass4 bin]$ ./nodetool enablefullquerylog
>
> {noformat}
> error: sun.nio.ch.FileChannelImpl.map0(int,long,long)
> -- StackTrace --
> java.lang.NoSuchMethodException:
> sun.nio.ch.FileChannelImpl.map0(int,long,long)
> at java.base/java.lang.Class.getDeclaredMethod(Class.java:2553)
> at net.openhft.chronicle.core.OS.lambda$static$0(OS.java:51)
> at
> net.openhft.chronicle.core.ClassLocal.computeValue(ClassLocal.java:53){noformat}
> (full stack trace attached to this ticket)
>
> Subsequent calls
[jira] [Commented] (CASSANDRA-15984) thrift_hsha_test.TestThriftHSHA test_closing_connections is broken on 3.0 and 3.11
[
https://issues.apache.org/jira/browse/CASSANDRA-15984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17165924#comment-17165924
]
David Capwell commented on CASSANDRA-15984:
---
build results for 3.11
https://app.circleci.com/pipelines/github/dcapwell/cassandra/353/workflows/0f7a1cf7-62d5-45f5-af1b-b6ad5b34ff64
rewrote the history to be
{code}
commit 0251e05c8c432f530829dbc077f74eca458a459b (HEAD -> ci/cassandra-3.11,
origin/ci/cassandra-3.11)
Author: David Capwell
Date: Mon Jul 27 11:45:47 2020 -0700
higher
commit 3f92ff78547fa3cb083a42d2c5c222afcef0e61d
Author: Mick Semb Wever
Date: Sat Jul 25 00:32:42 2020 +0200
Increment version to 3.11.8
{code}
this is the commit before
{code}
commit 25fd7bd84f1931d2a44e90e629f794c4cd11aa46 (upstream/cassandra-2.2,
cassandra-2.2)
Author: David Capwell
Date: Fri Jul 24 18:16:37 2020 -0700
Add support in jvm dtest to test thrift
patch by David Capwell; reviewed by Alex Petrov,Jon Meredith for
CASSANDRA-15967
{code}
So here we see this was failing before CASSANDRA-15967.
> thrift_hsha_test.TestThriftHSHA test_closing_connections is broken on 3.0 and
> 3.11
> --
>
> Key: CASSANDRA-15984
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15984
> Project: Cassandra
> Issue Type: Bug
> Components: Test/dtest
>Reporter: David Capwell
>Priority: Normal
> Fix For: 3.0.x, 3.11.x
>
>
> This test seems to have been broken on 3.0 for a while now; I ran Circle CI
> with HIGHER configs from ab6a87bf60174d9a6e7cd727702da3004c0dbeeb (from Jul 6
> 18:05:18 2020) all the way to HEAD ebf9c74c4ea8aefb1262458664571fdb52b76102
> (from Jul 24 18:47:39 2020).
> Interestingly, when I run the test locally against latest 3.0 it passes.
> This is not a flaky test, as it fails on no-vnode and vnode for every attempt
> (tried 9 times)
> {code}
> if rc != 0:
> > raise ToolError(cmd_args, rc, out, err)
> E ccmlib.node.ToolError: Subprocess ['nodetool', '-h', 'localhost',
> '-p', '7100', 'enablethrift'] exited with non-zero status; exit status: 2;
> E stderr: error: Could not create ServerSocket on address
> /127.0.0.1:9160.
> E -- StackTrace --
> E org.apache.thrift.transport.TTransportException: Could not create
> ServerSocket on address /127.0.0.1:9160.
> E at
> org.apache.thrift.transport.TNonblockingServerSocket.(TNonblockingServerSocket.java:96)
> E at
> org.apache.thrift.transport.TNonblockingServerSocket.(TNonblockingServerSocket.java:79)
> E at
> org.apache.thrift.transport.TNonblockingServerSocket.(TNonblockingServerSocket.java:75)
> E at
> org.apache.cassandra.thrift.TCustomNonblockingServerSocket.(TCustomNonblockingServerSocket.java:39)
> E at
> org.apache.cassandra.thrift.THsHaDisruptorServer$Factory.buildTServer(THsHaDisruptorServer.java:80)
> E at
> org.apache.cassandra.thrift.TServerCustomFactory.buildTServer(TServerCustomFactory.java:55)
> E at
> org.apache.cassandra.thrift.ThriftServer$ThriftServerThread.(ThriftServer.java:128)
> E at
> org.apache.cassandra.thrift.ThriftServer.start(ThriftServer.java:55)
> E at
> org.apache.cassandra.service.StorageService.startRPCServer(StorageService.java:386)
> E at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> E at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> E at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> E at java.lang.reflect.Method.invoke(Method.java:498)
> E at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:71)
> E at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> E at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> E at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> E at java.lang.reflect.Method.invoke(Method.java:498)
> E at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:275)
> E at
> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112)
> E at
> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46)
> E at
> com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)
> E at
> com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
> E at
> com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252)
> E at
>
[jira] [Comment Edited] (CASSANDRA-15971) full query log needs improvement
[
https://issues.apache.org/jira/browse/CASSANDRA-15971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17165882#comment-17165882
]
Ekaterina Dimitrova edited comment on CASSANDRA-15971 at 7/27/20, 7:21 PM:
---
I just spent some time to go through the docs and check the points mentioned.
To summarize:
* Improvement of the file format and explicitly saying that we need to
implement two steps as follow would be good thing I think: 1) Update
cassandra.yaml 2) enable FQL logging through nodetool
*
{quote}(maybe)
{noformat}
tools/bin/fqltool help{noformat}
should print out help for all fqltool commands rather than simply repeating the
default The most commonly used fqltool commands are..
{quote}
I saw that when we run
{code:java}
tools/bin/fqltool help{code}
there is a line at the bottom:
{code:java}
See 'fqltool help ' for more information on a specific command.{code}
My personal opinion is that the help listing will be probably too long listing
the full detailed information for all fqltool commands and pointing to the way
to check on specific command is a good compromise.
* Adding such an additional command probably deserves a separate improvement
ticket? (Also, from the perspective that the doc part might be handled by the
people dedicated to the project documentation, and the command part is
engineering work.)
nodetool missing getfullquerylog makes it difficult to verify current
fullquerylog state without changing it. The conventions for nodetool commands
should be followed to avoid confusing users.
was (Author: e.dimitrova):
I just spent some time to go through the docs and check the points mentioned.
To summarize:
* Improvement of the file format and explicitly saying that we need to
implement two steps as follow would be good thing I think: 1) Update
cassandra.yaml 2) enable FQL logging through nodetool
*
{quote}(maybe)
{noformat}
tools/bin/fqltool help{noformat}
should print out help for all fqltool commands rather than simply repeating the
default The most commonly used fqltool commands are..
{quote}
I saw that when we run tools/bin/fqltool help there is a line at the bottom:
{code:java}
See 'fqltool help ' for more information on a specific command.{code}
My personal opinion is that the help listing will be probably too long listing
the full detailed information for all fqltool commands and pointing to the way
to check on specific command is a good compromise.
* Adding such an additional command probably deserves a separate improvement
ticket? (Also, from the perspective that the doc part might be handled by the
people dedicated to the project documentation, and the command part is
engineering work.)
{quote}{quote} nodetool missing getfullquerylog makes it difficult to verify
current fullquerylog state without changing it. The conventions for nodetool
commands should be followed to avoid confusing users.
{quote}{quote}
> full query log needs improvement
>
>
> Key: CASSANDRA-15971
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15971
> Project: Cassandra
> Issue Type: Improvement
> Components: Tool/fql
>Reporter: Jonathan Shook
>Priority: Normal
> Attachments: st1.txt
>
>
> When trying out full query logging as a possible integration for nosqlbench
> usage, I ran across many issues which would make it painful for users. Since
> there were several, they will be added to a single issue for now. This issue
> can be broken up if needed.
>
> FQL doesn't work on my system, even though it says it is logging queries.
> With the following configuration in cassandra.yaml:
>
> {noformat}
> full_query_logging_options:
> log_dir: /REDACTED/fullquerylogs
> roll_cycle: HOURLY
> block: true
> max_queue_weight: 268435456 # 256 MiB
> max_log_size: 17179869184 # 16 GiB
> ## archive command is "/path/to/script.sh %path" where %path is replaced
> with the file being rolled:
> # archive_command:
> # max_archive_retries: 10
> {noformat}
> which appears to be the minimal configuration needed to enable fql, only a
> single file `directory-listing.cq4t` is created, which is a 64K sized file of
> zeroes.
>
>
> Calling bin/nodetool enablefullquerylog throws an error initially.
> [jshook@cass4 bin]$ ./nodetool enablefullquerylog
>
> {noformat}
> error: sun.nio.ch.FileChannelImpl.map0(int,long,long)
> -- StackTrace --
> java.lang.NoSuchMethodException:
> sun.nio.ch.FileChannelImpl.map0(int,long,long)
> at java.base/java.lang.Class.getDeclaredMethod(Class.java:2553)
> at net.openhft.chronicle.core.OS.lambda$static$0(OS.java:51)
> at
> net.openhft.chronicle.core.ClassLocal.computeValue(ClassLocal.java:53){noformat}
> (full stack trace attached to this ticket)
>
>
[jira] [Comment Edited] (CASSANDRA-15971) full query log needs improvement
[
https://issues.apache.org/jira/browse/CASSANDRA-15971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17165882#comment-17165882
]
Ekaterina Dimitrova edited comment on CASSANDRA-15971 at 7/27/20, 7:20 PM:
---
I just spent some time to go through the docs and check the points mentioned.
To summarize:
* Improvement of the file format and explicitly saying that we need to
implement two steps as follow would be good thing I think: 1) Update
cassandra.yaml 2) enable FQL logging through nodetool
*
{quote}(maybe)
{noformat}
tools/bin/fqltool help{noformat}
should print out help for all fqltool commands rather than simply repeating the
default The most commonly used fqltool commands are..
{quote}
I saw that when we run tools/bin/fqltool help there is a line at the bottom:
{code:java}
See 'fqltool help ' for more information on a specific command.{code}
My personal opinion is that the help listing will be probably too long listing
the full detailed information for all fqltool commands and pointing to the way
to check on specific command is a good compromise.
* Adding such an additional command probably deserves a separate improvement
ticket? (Also, from the perspective that the doc part might be handled by the
people dedicated to the project documentation, and the command part is
engineering work.)
{quote}{quote} nodetool missing getfullquerylog makes it difficult to verify
current fullquerylog state without changing it. The conventions for nodetool
commands should be followed to avoid confusing users.
{quote}{quote}
was (Author: e.dimitrova):
I just spend some time to go through the docs and check the points mentioned.
To summarize:
* Improvement of the file format and explicitly saying that we need to
implement two steps as follow would be good thing I think: 1) Update
cassandra.yaml 2) enable FQL logging through nodetool
*
{quote}(maybe)
{noformat}
tools/bin/fqltool help{noformat}
should print out help for all fqltool commands rather than simply repeating the
default The most commonly used fqltool commands are..
{quote}
I saw that when we run tools/bin/fqltool help there is a line at the bottom:
{code:java}
See 'fqltool help ' for more information on a specific command.{code}
My personal opinion is that the help listing will be probably too long listing
the full detailed information for all fqltool commands and pointing to the way
to check on specific command is a good compromise.
* Adding such an additional command probably deserves a separate improvement
ticket? (Also, from the perspective that the doc part might be handled by the
people dedicated to the project documentation, and the command part is
engineering work.)
{quote}bq. nodetool missing getfullquerylog makes it difficult to verify
current fullquerylog state without changing it. The conventions for nodetool
commands should be followed to avoid confusing users.{quote}
> full query log needs improvement
>
>
> Key: CASSANDRA-15971
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15971
> Project: Cassandra
> Issue Type: Improvement
> Components: Tool/fql
>Reporter: Jonathan Shook
>Priority: Normal
> Attachments: st1.txt
>
>
> When trying out full query logging as a possible integration for nosqlbench
> usage, I ran across many issues which would make it painful for users. Since
> there were several, they will be added to a single issue for now. This issue
> can be broken up if needed.
>
> FQL doesn't work on my system, even though it says it is logging queries.
> With the following configuration in cassandra.yaml:
>
> {noformat}
> full_query_logging_options:
> log_dir: /REDACTED/fullquerylogs
> roll_cycle: HOURLY
> block: true
> max_queue_weight: 268435456 # 256 MiB
> max_log_size: 17179869184 # 16 GiB
> ## archive command is "/path/to/script.sh %path" where %path is replaced
> with the file being rolled:
> # archive_command:
> # max_archive_retries: 10
> {noformat}
> which appears to be the minimal configuration needed to enable fql, only a
> single file `directory-listing.cq4t` is created, which is a 64K sized file of
> zeroes.
>
>
> Calling bin/nodetool enablefullquerylog throws an error initially.
> [jshook@cass4 bin]$ ./nodetool enablefullquerylog
>
> {noformat}
> error: sun.nio.ch.FileChannelImpl.map0(int,long,long)
> -- StackTrace --
> java.lang.NoSuchMethodException:
> sun.nio.ch.FileChannelImpl.map0(int,long,long)
> at java.base/java.lang.Class.getDeclaredMethod(Class.java:2553)
> at net.openhft.chronicle.core.OS.lambda$static$0(OS.java:51)
> at
> net.openhft.chronicle.core.ClassLocal.computeValue(ClassLocal.java:53){noformat}
> (full stack trace attached to this ticket)
>
> Subsequent
[jira] [Updated] (CASSANDRA-15982) how_to_commit.rst missing dash in git commands and added recommendation to compile before commit
[ https://issues.apache.org/jira/browse/CASSANDRA-15982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Capwell updated CASSANDRA-15982: -- Reviewers: Brandon Williams, David Capwell Status: Review In Progress (was: Patch Available) > how_to_commit.rst missing dash in git commands and added recommendation to > compile before commit > > > Key: CASSANDRA-15982 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15982 > Project: Cassandra > Issue Type: Bug > Components: Documentation/Website >Reporter: David Capwell >Assignee: David Capwell >Priority: Normal > Fix For: 4.0-beta > > > There are two typos in how_to_commit.rst: -amend -> --amend and -atomic -> > --atomic. > Also, the docs don't recommend building before committing, would be best to > encourage this. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15982) how_to_commit.rst missing dash in git commands and added recommendation to compile before commit
[ https://issues.apache.org/jira/browse/CASSANDRA-15982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Capwell updated CASSANDRA-15982: -- Fix Version/s: (was: 4.0-beta) 4.0-beta2 Since Version: 4.0-beta1 Source Control Link: http://github.com/apache/cassandra/commits/a2c3d5a4ce810d02085ace97bee21e187e67aa7e Resolution: Fixed Status: Resolved (was: Ready to Commit) > how_to_commit.rst missing dash in git commands and added recommendation to > compile before commit > > > Key: CASSANDRA-15982 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15982 > Project: Cassandra > Issue Type: Bug > Components: Documentation/Website >Reporter: David Capwell >Assignee: David Capwell >Priority: Normal > Fix For: 4.0-beta2 > > > There are two typos in how_to_commit.rst: -amend -> --amend and -atomic -> > --atomic. > Also, the docs don't recommend building before committing, would be best to > encourage this. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15982) how_to_commit.rst missing dash in git commands and added recommendation to compile before commit
[ https://issues.apache.org/jira/browse/CASSANDRA-15982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Capwell updated CASSANDRA-15982: -- Status: Ready to Commit (was: Review In Progress) > how_to_commit.rst missing dash in git commands and added recommendation to > compile before commit > > > Key: CASSANDRA-15982 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15982 > Project: Cassandra > Issue Type: Bug > Components: Documentation/Website >Reporter: David Capwell >Assignee: David Capwell >Priority: Normal > Fix For: 4.0-beta > > > There are two typos in how_to_commit.rst: -amend -> --amend and -atomic -> > --atomic. > Also, the docs don't recommend building before committing, would be best to > encourage this. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] branch trunk updated: how_to_commit.rst missing dash in git commands, added recommendation to compile before commit, and documented that a merge -s ours is needed for older branches even i
This is an automated email from the ASF dual-hosted git repository. dcapwell pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/cassandra.git The following commit(s) were added to refs/heads/trunk by this push: new a2c3d5a how_to_commit.rst missing dash in git commands, added recommendation to compile before commit, and documented that a merge -s ours is needed for older branches even if they do not impact trunk a2c3d5a is described below commit a2c3d5a4ce810d02085ace97bee21e187e67aa7e Author: David Capwell AuthorDate: Mon Jul 27 11:32:57 2020 -0700 how_to_commit.rst missing dash in git commands, added recommendation to compile before commit, and documented that a merge -s ours is needed for older branches even if they do not impact trunk patch by David Capwell; reviewed by Brandon Williams for CASSANDRA-15982 --- doc/source/development/how_to_commit.rst | 37 ++-- 1 file changed, 30 insertions(+), 7 deletions(-) diff --git a/doc/source/development/how_to_commit.rst b/doc/source/development/how_to_commit.rst index dff3983..528ac40 100644 --- a/doc/source/development/how_to_commit.rst +++ b/doc/source/development/how_to_commit.rst @@ -27,46 +27,69 @@ Hypothetical CASSANDRA-12345 ticket is a cassandra-3.0 based bug fix that requir On cassandra-3.0: #. ``git am -3 12345-3.0.patch`` (if we have a problem b/c of CHANGES.txt not merging anymore, we fix it ourselves, in place) + #. ``ant realclean && ant jar build-test`` (rebuild to make sure code compiles) On cassandra-3.3: #. ``git merge cassandra-3.0 -s ours`` #. ``git apply -3 12345-3.3.patch`` (likely to have an issue with CHANGES.txt here: fix it ourselves, then git add CHANGES.txt) - #. ``git commit -amend`` + #. ``ant realclean && ant jar build-test`` (rebuild to make sure code compiles) + #. ``git commit --amend`` On trunk: #. ``git merge cassandra-3.3 -s ours`` #. ``git apply -3 12345-trunk.patch`` (likely to have an issue with CHANGES.txt here: fix it ourselves, then git add CHANGES.txt) - #. ``git commit -amend`` + #. ``ant realclean && ant jar build-test`` (rebuild to make sure code compiles) + #. ``git commit --amend`` On any branch: - #. ``git push origin cassandra-3.0 cassandra-3.3 trunk -atomic`` + #. ``git push origin cassandra-3.0 cassandra-3.3 trunk --atomic`` Same scenario, but a branch-based contribution: On cassandra-3.0: #. ``git cherry-pick `` (if we have a problem b/c of CHANGES.txt not merging anymore, we fix it ourselves, in place) + #. ``ant realclean && ant jar build-test`` (rebuild to make sure code compiles) On cassandra-3.3: #. ``git merge cassandra-3.0 -s ours`` #. ``git format-patch -1 `` #. ``git apply -3 .patch`` (likely to have an issue with CHANGES.txt here: fix it ourselves, then git add CHANGES.txt) - #. ``git commit -amend`` + #. ``ant realclean && ant jar build-test`` (rebuild to make sure code compiles) + #. ``git commit --amend`` On trunk: #. ``git merge cassandra-3.3 -s ours`` #. ``git format-patch -1 `` #. ``git apply -3 .patch`` (likely to have an issue with CHANGES.txt here: fix it ourselves, then git add CHANGES.txt) - #. ``git commit -amend`` + #. ``ant realclean && ant jar build-test`` (rebuild to make sure code compiles) + #. ``git commit --amend`` On any branch: - #. ``git push origin cassandra-3.0 cassandra-3.3 trunk -atomic`` + #. ``git push origin cassandra-3.0 cassandra-3.3 trunk --atomic`` + +If the patch is for an older branch, and doesn't impact later branches (such as trunk), we still need to merge up. + +On cassandra-3.0: + #. ``git cherry-pick `` (if we have a problem b/c of CHANGES.txt not merging anymore, we fix it ourselves, in place) + #. ``ant realclean && ant jar build-test`` (rebuild to make sure code compiles) + +On cassandra-3.3: + #. ``git merge cassandra-3.0 -s ours`` + #. ``ant realclean && ant jar build-test`` (rebuild to make sure code compiles) + +On trunk: + #. ``git merge cassandra-3.3 -s ours`` + #. ``ant realclean && ant jar build-test`` (rebuild to make sure code compiles) + +On any branch: + #. ``git push origin cassandra-3.0 cassandra-3.3 trunk --atomic`` .. tip:: Notes on git flags: ``-3`` flag to am and apply will instruct git to perform a 3-way merge for you. If a conflict is detected, you can either resolve it manually or invoke git mergetool - for both am and apply. - ``-atomic`` flag to git push does the obvious thing: pushes all or nothing. Without the flag, the command is equivalent to running git push once per each branch. This is nifty in case a race condition happens - you won’t push half the branches, blocking other committers’ progress while you are resolving the issue. + ``--atomic`` flag to git push does the obvious thing: pushes all or nothing. Without the flag, the command is equivalent to running git
[jira] [Commented] (CASSANDRA-15971) full query log needs improvement
[
https://issues.apache.org/jira/browse/CASSANDRA-15971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17165900#comment-17165900
]
Ekaterina Dimitrova commented on CASSANDRA-15971:
-
I created CASSANDRA-15988 to accommodate the last point. Even if it is not
immediately implemented it will be on our to do list.
> full query log needs improvement
>
>
> Key: CASSANDRA-15971
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15971
> Project: Cassandra
> Issue Type: Improvement
> Components: Tool/fql
>Reporter: Jonathan Shook
>Priority: Normal
> Attachments: st1.txt
>
>
> When trying out full query logging as a possible integration for nosqlbench
> usage, I ran across many issues which would make it painful for users. Since
> there were several, they will be added to a single issue for now. This issue
> can be broken up if needed.
>
> FQL doesn't work on my system, even though it says it is logging queries.
> With the following configuration in cassandra.yaml:
>
> {noformat}
> full_query_logging_options:
> log_dir: /REDACTED/fullquerylogs
> roll_cycle: HOURLY
> block: true
> max_queue_weight: 268435456 # 256 MiB
> max_log_size: 17179869184 # 16 GiB
> ## archive command is "/path/to/script.sh %path" where %path is replaced
> with the file being rolled:
> # archive_command:
> # max_archive_retries: 10
> {noformat}
> which appears to be the minimal configuration needed to enable fql, only a
> single file `directory-listing.cq4t` is created, which is a 64K sized file of
> zeroes.
>
>
> Calling bin/nodetool enablefullquerylog throws an error initially.
> [jshook@cass4 bin]$ ./nodetool enablefullquerylog
>
> {noformat}
> error: sun.nio.ch.FileChannelImpl.map0(int,long,long)
> -- StackTrace --
> java.lang.NoSuchMethodException:
> sun.nio.ch.FileChannelImpl.map0(int,long,long)
> at java.base/java.lang.Class.getDeclaredMethod(Class.java:2553)
> at net.openhft.chronicle.core.OS.lambda$static$0(OS.java:51)
> at
> net.openhft.chronicle.core.ClassLocal.computeValue(ClassLocal.java:53){noformat}
> (full stack trace attached to this ticket)
>
> Subsequent calls produce normal output:
>
> {noformat}
> [jshook@cass4 c4b1]$ bin/nodetool enablefullquerylog
> nodetool: Already logging to /home/jshook/c4b1/data/fullquerylogs
> See 'nodetool help' or 'nodetool help '.{noformat}
>
>
> nodetool missing getfullquerylog makes it difficult to verify current
> fullquerylog state without changing it. The conventions for nodetool commands
> should be followed to avoid confusing users.
>
> (maybe)
> {noformat}
> tools/bin/fqltool help{noformat}
> should print out help for all fqltool commands rather than simply repeating
> the default The most commonly used fqltool commands are..
>
> [https://cassandra.apache.org/doc/latest/new/fqllogging.html] is
> malformatted, mixing the appearance of configuration with comments, which is
> confusing at best.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15984) thrift_hsha_test.TestThriftHSHA test_closing_connections is broken on 3.0 and 3.11
[
https://issues.apache.org/jira/browse/CASSANDRA-15984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17165894#comment-17165894
]
David Capwell commented on CASSANDRA-15984:
---
I took the test list from the job and reran locally; test passes just fine.
I ran the following test list
{code}
auth_test.py::TestAuth::test_login
bootstrap_test.py::TestBootstrap::test_bootstrap_waits_for_streaming_to_finish
consistency_test.py::TestConsistency::test_12872
cqlsh_tests/test_cqlsh.py::TestCqlshSmoke::test_batch
disk_balance_test.py::TestDiskBalance::test_disk_balance_after_joining_ring_stcs
materialized_views_test.py::TestMaterializedViews::test_ttl
pushed_notifications_test.py::TestPushedNotifications::test_schema_changes
replace_address_test.py::TestReplaceAddress::test_insert_data_during_replace_different_address
schema_metadata_test.py::TestSchemaMetadata::test_static_column
thrift_hsha_test.py::TestThriftHSHA::test_closing_connections
thrift_test.py::TestMutations::test_super_subcolumn_limit
user_types_test.py::TestUserTypes::test_type_secondary_indexing
{code}
In this run I had consistency_test.TestConsistency#test_12872 fail, but here is
the results for this test
{code}
7:52:16,991 ccm DEBUG Log-watching thread starting.
{code}
> thrift_hsha_test.TestThriftHSHA test_closing_connections is broken on 3.0 and
> 3.11
> --
>
> Key: CASSANDRA-15984
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15984
> Project: Cassandra
> Issue Type: Bug
> Components: Test/dtest
>Reporter: David Capwell
>Priority: Normal
> Fix For: 3.0.x, 3.11.x
>
>
> This test seems to have been broken on 3.0 for a while now; I ran Circle CI
> with HIGHER configs from ab6a87bf60174d9a6e7cd727702da3004c0dbeeb (from Jul 6
> 18:05:18 2020) all the way to HEAD ebf9c74c4ea8aefb1262458664571fdb52b76102
> (from Jul 24 18:47:39 2020).
> Interestingly, when I run the test locally against latest 3.0 it passes.
> This is not a flaky test, as it fails on no-vnode and vnode for every attempt
> (tried 9 times)
> {code}
> if rc != 0:
> > raise ToolError(cmd_args, rc, out, err)
> E ccmlib.node.ToolError: Subprocess ['nodetool', '-h', 'localhost',
> '-p', '7100', 'enablethrift'] exited with non-zero status; exit status: 2;
> E stderr: error: Could not create ServerSocket on address
> /127.0.0.1:9160.
> E -- StackTrace --
> E org.apache.thrift.transport.TTransportException: Could not create
> ServerSocket on address /127.0.0.1:9160.
> E at
> org.apache.thrift.transport.TNonblockingServerSocket.(TNonblockingServerSocket.java:96)
> E at
> org.apache.thrift.transport.TNonblockingServerSocket.(TNonblockingServerSocket.java:79)
> E at
> org.apache.thrift.transport.TNonblockingServerSocket.(TNonblockingServerSocket.java:75)
> E at
> org.apache.cassandra.thrift.TCustomNonblockingServerSocket.(TCustomNonblockingServerSocket.java:39)
> E at
> org.apache.cassandra.thrift.THsHaDisruptorServer$Factory.buildTServer(THsHaDisruptorServer.java:80)
> E at
> org.apache.cassandra.thrift.TServerCustomFactory.buildTServer(TServerCustomFactory.java:55)
> E at
> org.apache.cassandra.thrift.ThriftServer$ThriftServerThread.(ThriftServer.java:128)
> E at
> org.apache.cassandra.thrift.ThriftServer.start(ThriftServer.java:55)
> E at
> org.apache.cassandra.service.StorageService.startRPCServer(StorageService.java:386)
> E at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> E at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> E at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> E at java.lang.reflect.Method.invoke(Method.java:498)
> E at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:71)
> E at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> E at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> E at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> E at java.lang.reflect.Method.invoke(Method.java:498)
> E at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:275)
> E at
> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112)
> E at
> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46)
> E at
> com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)
> E at
>
[jira] [Commented] (CASSANDRA-15967) Add support in jvm dtest to test thrift
[
https://issues.apache.org/jira/browse/CASSANDRA-15967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17165890#comment-17165890
]
David Capwell commented on CASSANDRA-15967:
---
Adding the following to make this more clear
{code}
$ git diff
diff --git a/doc/source/development/how_to_commit.rst
b/doc/source/development/how_to_commit.rst
index 8b59049876..528ac40ced 100644
--- a/doc/source/development/how_to_commit.rst
+++ b/doc/source/development/how_to_commit.rst
@@ -67,6 +67,23 @@ On trunk:
On any branch:
#. ``git push origin cassandra-3.0 cassandra-3.3 trunk --atomic``
+If the patch is for an older branch, and doesn't impact later branches (such
as trunk), we still need to merge up.
+
+On cassandra-3.0:
+ #. ``git cherry-pick `` (if we have a problem b/c of
CHANGES.txt not merging anymore, we fix it ourselves, in place)
+ #. ``ant realclean && ant jar build-test`` (rebuild to make sure code
compiles)
+
+On cassandra-3.3:
+ #. ``git merge cassandra-3.0 -s ours``
+ #. ``ant realclean && ant jar build-test`` (rebuild to make sure code
compiles)
+
+On trunk:
+ #. ``git merge cassandra-3.3 -s ours``
+ #. ``ant realclean && ant jar build-test`` (rebuild to make sure code
compiles)
+
+On any branch:
+ #. ``git push origin cassandra-3.0 cassandra-3.3 trunk --atomic``
+
{code}
> Add support in jvm dtest to test thrift
> ---
>
> Key: CASSANDRA-15967
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15967
> Project: Cassandra
> Issue Type: Improvement
> Components: Test/dtest
>Reporter: David Capwell
>Assignee: David Capwell
>Priority: Normal
> Fix For: 3.0.22, 3.11.8, 2.2.18
>
>
> In 2.2 and 3.x thrift is supported and can cause problems for the non thrift
> code. To make sure we can test these interactions, need to add support for
> thrift in jvm dtest.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-15988) Add nodetool getfullquerylog
Ekaterina Dimitrova created CASSANDRA-15988: --- Summary: Add nodetool getfullquerylog Key: CASSANDRA-15988 URL: https://issues.apache.org/jira/browse/CASSANDRA-15988 Project: Cassandra Issue Type: Improvement Reporter: Ekaterina Dimitrova This ticket is raised based on CASSANDRA-15791 and valuable feedback provided by [~jshook]. There are two outstanding questions: * forming the exact shape of such a command and how it can benefit the users; to be discussed in detail in this ticket * Is this a thing we as a project can add to 4.0 beta or it should be considered in 4.1 for example -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15971) full query log needs improvement
[
https://issues.apache.org/jira/browse/CASSANDRA-15971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17165882#comment-17165882
]
Ekaterina Dimitrova commented on CASSANDRA-15971:
-
I just spend some time to go through the docs and check the points mentioned.
To summarize:
* Improvement of the file format and explicitly saying that we need to
implement two steps as follow would be good thing I think: 1) Update
cassandra.yaml 2) enable FQL logging through nodetool
*
{quote}(maybe)
{noformat}
tools/bin/fqltool help{noformat}
should print out help for all fqltool commands rather than simply repeating the
default The most commonly used fqltool commands are..
{quote}
I saw that when we run tools/bin/fqltool help there is a line at the bottom:
{code:java}
See 'fqltool help ' for more information on a specific command.{code}
My personal opinion is that the help listing will be probably too long listing
the full detailed information for all fqltool commands and pointing to the way
to check on specific command is a good compromise.
* Adding such an additional command probably deserves a separate improvement
ticket? (Also, from the perspective that the doc part might be handled by the
people dedicated to the project documentation, and the command part is
engineering work.)
{quote}bq. nodetool missing getfullquerylog makes it difficult to verify
current fullquerylog state without changing it. The conventions for nodetool
commands should be followed to avoid confusing users.{quote}
> full query log needs improvement
>
>
> Key: CASSANDRA-15971
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15971
> Project: Cassandra
> Issue Type: Improvement
> Components: Tool/fql
>Reporter: Jonathan Shook
>Priority: Normal
> Attachments: st1.txt
>
>
> When trying out full query logging as a possible integration for nosqlbench
> usage, I ran across many issues which would make it painful for users. Since
> there were several, they will be added to a single issue for now. This issue
> can be broken up if needed.
>
> FQL doesn't work on my system, even though it says it is logging queries.
> With the following configuration in cassandra.yaml:
>
> {noformat}
> full_query_logging_options:
> log_dir: /REDACTED/fullquerylogs
> roll_cycle: HOURLY
> block: true
> max_queue_weight: 268435456 # 256 MiB
> max_log_size: 17179869184 # 16 GiB
> ## archive command is "/path/to/script.sh %path" where %path is replaced
> with the file being rolled:
> # archive_command:
> # max_archive_retries: 10
> {noformat}
> which appears to be the minimal configuration needed to enable fql, only a
> single file `directory-listing.cq4t` is created, which is a 64K sized file of
> zeroes.
>
>
> Calling bin/nodetool enablefullquerylog throws an error initially.
> [jshook@cass4 bin]$ ./nodetool enablefullquerylog
>
> {noformat}
> error: sun.nio.ch.FileChannelImpl.map0(int,long,long)
> -- StackTrace --
> java.lang.NoSuchMethodException:
> sun.nio.ch.FileChannelImpl.map0(int,long,long)
> at java.base/java.lang.Class.getDeclaredMethod(Class.java:2553)
> at net.openhft.chronicle.core.OS.lambda$static$0(OS.java:51)
> at
> net.openhft.chronicle.core.ClassLocal.computeValue(ClassLocal.java:53){noformat}
> (full stack trace attached to this ticket)
>
> Subsequent calls produce normal output:
>
> {noformat}
> [jshook@cass4 c4b1]$ bin/nodetool enablefullquerylog
> nodetool: Already logging to /home/jshook/c4b1/data/fullquerylogs
> See 'nodetool help' or 'nodetool help '.{noformat}
>
>
> nodetool missing getfullquerylog makes it difficult to verify current
> fullquerylog state without changing it. The conventions for nodetool commands
> should be followed to avoid confusing users.
>
> (maybe)
> {noformat}
> tools/bin/fqltool help{noformat}
> should print out help for all fqltool commands rather than simply repeating
> the default The most commonly used fqltool commands are..
>
> [https://cassandra.apache.org/doc/latest/new/fqllogging.html] is
> malformatted, mixing the appearance of configuration with comments, which is
> confusing at best.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15583) 4.0 quality testing: Tooling, Bundled and First Party
[ https://issues.apache.org/jira/browse/CASSANDRA-15583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17165859#comment-17165859 ] David Capwell commented on CASSANDRA-15583: --- thanks > 4.0 quality testing: Tooling, Bundled and First Party > - > > Key: CASSANDRA-15583 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15583 > Project: Cassandra > Issue Type: Task > Components: Test/dtest, Test/unit >Reporter: Josh McKenzie >Assignee: Berenguer Blasi >Priority: Normal > Fix For: 4.0-beta > > > Reference [doc from > NGCC|https://docs.google.com/document/d/1uhUOp7wpE9ZXNDgxoCZHejHt5SO4Qw1dArZqqsJccyQ/edit#] > for context. > *Shepherd: Sam Tunnicliffe* > Test plans should cover bundled first-party tooling and CLIs such as > nodetool, cqlsh, and new tools supporting full query and audit logging > (CASSANDRA-13983, CASSANDRA-12151). > ||Tool||UX test||UT coverage||dtest coverage||Comments|| > |Nodetool|(x)|(x)|(!)|By no means all the sub commands are tested. Dtest also > test nodetool as a side effect| > |Cqlsh|(x)|(x)|(!)| | > |Cassandra-stress|(x)|(x)|(x)| Some UT| > |debug-cql|(x)|(x)|(x)| | > |fqltool|(x)|(/)|(!)| | > |auditlogviewer|(x)|(!)|(!)| | > |*Sstable utilities*| | | | | > |sstabledump|(x)|(x)|(!)| | > |sstableexpiredblockers|(x)|(x)|(!)| | > |sstablelevelreset|(x)|(x)|(!)| | > |sstableloader|(x)|(x)|(!)| | > |sstablemetadata|(x)|(x)|(x)|Ran in dtests, no dedicated test| > |sstableofflinerelevel|(x)|(x)|(!)| | > |sstablerepairedset|(x)|(x)|(x)|Ran in dtests, no dedicated test| > |sstablescrub|(x)|(x)|(!)| | > |sstablesplit|(x)|(x)|(!)| | > |sstableupgrade|(x)|(x)|(!)| | > |sstableutil|(x)|(x)|(!)| | > |sstableverify|(x)|(x)|(!)| | -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15981) jvm-dtests crash on java 11
[
https://issues.apache.org/jira/browse/CASSANDRA-15981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17165857#comment-17165857
]
David Capwell commented on CASSANDRA-15981:
---
nah, just a JVM bug with CMS and class unloading
{code}
# SIGSEGV (0xb) at pc=0x7f240160125f, pid=1252, tid=1259
...
# V [libjvm.so+0x48225f] MarkAndPushClosure::do_oop(oopDesc**)+0xaf
{code}
doesn't impact other GCs and doesn't fail with -XX:-CMSClassUnloadingEnabled
> jvm-dtests crash on java 11
> ---
>
> Key: CASSANDRA-15981
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15981
> Project: Cassandra
> Issue Type: Bug
> Components: Build
>Reporter: David Capwell
>Priority: Normal
> Fix For: 4.0-beta
>
>
> There is a race condition bug with CMS and class unloading which cause the
> JVM to crash. Since jvm-dtests rely on class loaders and unloading, this
> causes sporadic JVM crashes that look like the following in CI logs
> {code}
> junit.framework.AssertionFailedError: Forked Java VM exited abnormally.
> Please note the time in the report does not reflect the time until the VM
> exit.
> at jdk.internal.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.util.Vector.forEach(Vector.java:1387)
> at jdk.internal.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at jdk.internal.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.util.Vector.forEach(Vector.java:1387)
> at jdk.internal.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.Thread.run(Thread.java:834)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15967) Add support in jvm dtest to test thrift
[ https://issues.apache.org/jira/browse/CASSANDRA-15967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17165824#comment-17165824 ] David Capwell commented on CASSANDRA-15967: --- Thanks, didn't know that; will upgrade the doc https://cassandra.apache.org/doc/latest/development/how_to_commit.html > Add support in jvm dtest to test thrift > --- > > Key: CASSANDRA-15967 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15967 > Project: Cassandra > Issue Type: Improvement > Components: Test/dtest >Reporter: David Capwell >Assignee: David Capwell >Priority: Normal > Fix For: 3.0.22, 3.11.8, 2.2.18 > > > In 2.2 and 3.x thrift is supported and can cause problems for the non thrift > code. To make sure we can test these interactions, need to add support for > thrift in jvm dtest. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-15987) Failure in ViewTest#testNullInClusteringColumns on FreeBSD
Angelo Polo created CASSANDRA-15987:
---
Summary: Failure in ViewTest#testNullInClusteringColumns on FreeBSD
Key: CASSANDRA-15987
URL: https://issues.apache.org/jira/browse/CASSANDRA-15987
Project: Cassandra
Issue Type: Bug
Components: Test/unit
Reporter: Angelo Polo
Failure occurs on FreeBSD 12.1-RELEASE-p5 amd64, OpenJDK 64-Bit Server VM
(build 11.0.8+10-1, mixed mode), 4.0-beta1.
{noformat}
[junit-timeout] Testsuite: org.apache.cassandra.cql3.ViewTest Tests run: 41,
Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 110.152 sec
[junit-timeout]
[junit-timeout] Testcase:
testNullInClusteringColumns(org.apache.cassandra.cql3.ViewTest): FAILED
[junit-timeout] Invalid value for row 0 column 2 (v1 of type varchar), expected
(-1 bytes) but got <'foo'> (3 bytes) (using protocol version 4/v4)
[junit-timeout] junit.framework.AssertionFailedError: Invalid value for row 0
column 2 (v1 of type varchar), expected (-1 bytes) but got <'foo'> (3
bytes) (using protocol version 4/v4)
[junit-timeout] at
org.apache.cassandra.cql3.CQLTester.assertRowsNet(CQLTester.java:1034)
[junit-timeout] at
org.apache.cassandra.cql3.ViewTest.testNullInClusteringColumns(ViewTest.java:1262)
[junit-timeout] at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[junit-timeout] at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[junit-timeout] at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[junit-timeout]
[junit-timeout]
[junit-timeout] Test org.apache.cassandra.cql3.ViewTest FAILED{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15986) Repair tests flakiness
[ https://issues.apache.org/jira/browse/CASSANDRA-15986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17165805#comment-17165805 ] Berenguer Blasi commented on CASSANDRA-15986: - [~e.dimitrova] reported she can repro locally and the test will still fail but differently. Unassigning so sbdy that can repro can pick it up. > Repair tests flakiness > -- > > Key: CASSANDRA-15986 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15986 > Project: Cassandra > Issue Type: Task > Components: Test/dtest >Reporter: Berenguer Blasi >Assignee: Berenguer Blasi >Priority: Normal > Fix For: 4.0-beta > > > Repair tests come up in test failure reports every now and then. I have tried > to repro the > [latest|https://ci-cassandra.apache.org/job/Cassandra-trunk/241/testReport/junit/dtest-novnode.repair_tests.repair_test/TestRepair/test_simple_sequential_repair/] > locally 100 times with no luck. > Still from experience from fixing other flaky tests I have some intuition > where the problems may lie. The proposed fix should add no harm if merged. We > can reopen the ticket if repair tests keep failing. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15986) Repair tests flakiness
[ https://issues.apache.org/jira/browse/CASSANDRA-15986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Berenguer Blasi updated CASSANDRA-15986: Status: Open (was: Patch Available) > Repair tests flakiness > -- > > Key: CASSANDRA-15986 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15986 > Project: Cassandra > Issue Type: Task > Components: Test/dtest >Reporter: Berenguer Blasi >Assignee: Berenguer Blasi >Priority: Normal > Fix For: 4.0-beta > > > Repair tests come up in test failure reports every now and then. I have tried > to repro the > [latest|https://ci-cassandra.apache.org/job/Cassandra-trunk/241/testReport/junit/dtest-novnode.repair_tests.repair_test/TestRepair/test_simple_sequential_repair/] > locally 100 times with no luck. > Still from experience from fixing other flaky tests I have some intuition > where the problems may lie. The proposed fix should add no harm if merged. We > can reopen the ticket if repair tests keep failing. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15933) Forbid adding new fields to UDTs used in partition key columns
[ https://issues.apache.org/jira/browse/CASSANDRA-15933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aleksey Yeschenko updated CASSANDRA-15933: -- Fix Version/s: (was: 3.11.x) (was: 4.0) (was: 3.0.x) 4.0-beta2 3.11.8 3.0.21 Since Version: 3.0.0 Source Control Link: [dc725bc01a69893a59db27312f6eb2bda9aa0ac9|https://github.com/apache/cassandra/commit/dc725bc01a69893a59db27312f6eb2bda9aa0ac9] Resolution: Fixed Status: Resolved (was: Ready to Commit) > Forbid adding new fields to UDTs used in partition key columns > -- > > Key: CASSANDRA-15933 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15933 > Project: Cassandra > Issue Type: Bug > Components: Cluster/Schema >Reporter: Aleksey Yeschenko >Assignee: Aleksey Yeschenko >Priority: Normal > Fix For: 3.0.21, 3.11.8, 4.0-beta2 > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15933) Forbid adding new fields to UDTs used in partition key columns
[ https://issues.apache.org/jira/browse/CASSANDRA-15933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17165796#comment-17165796 ] Aleksey Yeschenko commented on CASSANDRA-15933: --- Thanks for review, committed as [dc725bc01a69893a59db27312f6eb2bda9aa0ac9|https://github.com/apache/cassandra/commit/dc725bc01a69893a59db27312f6eb2bda9aa0ac9] to 3.0 and merged up into 3.11 and trunk. > Forbid adding new fields to UDTs used in partition key columns > -- > > Key: CASSANDRA-15933 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15933 > Project: Cassandra > Issue Type: Bug > Components: Cluster/Schema >Reporter: Aleksey Yeschenko >Assignee: Aleksey Yeschenko >Priority: Normal > Fix For: 4.0, 3.0.x, 3.11.x > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] branch trunk updated (a821214 -> 9e61249)
This is an automated email from the ASF dual-hosted git repository. aleksey pushed a change to branch trunk in repository https://gitbox.apache.org/repos/asf/cassandra.git. from a821214 CASSANDRA-15942 Improve tooling testing framework add 25fd7bd Add support in jvm dtest to test thrift add ebf9c74 Merge branch 'cassandra-2.2' into cassandra-3.0 add 8c61214 Merge branch 'cassandra-3.0' into cassandra-3.11 add dc725bc Forbid altering UDTs used in partition keys add c61f390 Merge branch 'cassandra-3.0' into cassandra-3.11 new 9e61249 Merge branch 'cassandra-3.11' into trunk The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: CHANGES.txt| 1 + .../cql3/statements/schema/AlterTypeStatement.java | 22 ++ .../cql3/validation/operations/AlterTest.java | 20 3 files changed, 43 insertions(+) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] 01/01: Merge branch 'cassandra-3.11' into trunk
This is an automated email from the ASF dual-hosted git repository.
aleksey pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra.git
commit 9e61249764a3a7e08abded87602b118e2d4e4681
Merge: a821214 c61f390
Author: Aleksey Yeshchenko
AuthorDate: Mon Jul 27 16:20:02 2020 +0100
Merge branch 'cassandra-3.11' into trunk
CHANGES.txt| 1 +
.../cql3/statements/schema/AlterTypeStatement.java | 22 ++
.../cql3/validation/operations/AlterTest.java | 20
3 files changed, 43 insertions(+)
diff --cc CHANGES.txt
index a36f379,812e020..1acd5a2
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,10 -1,7 +1,11 @@@
-3.11.8
+4.0-beta2
++ * Forbid altering UDTs used in partition keys (CASSANDRA-15933)
+ * Fix version parsing logic when upgrading from 3.0 (CASSANDRA-15973)
+ * Optimize NoSpamLogger use in hot paths (CASSANDRA-15766)
+ * Verify sstable components on startup (CASSANDRA-15945)
+Merged from 3.11:
* Frozen RawTuple is not annotated with frozen in the toString method
(CASSANDRA-15857)
Merged from 3.0:
- * Forbid altering UDTs used in partition keys (CASSANDRA-15933)
* Fix empty/null json string representation (CASSANDRA-15896)
Merged from 2.2:
* Fix CQL parsing of collections when the column type is reversed
(CASSANDRA-15814)
diff --cc
src/java/org/apache/cassandra/cql3/statements/schema/AlterTypeStatement.java
index f883038,000..9228f34
mode 100644,00..100644
---
a/src/java/org/apache/cassandra/cql3/statements/schema/AlterTypeStatement.java
+++
b/src/java/org/apache/cassandra/cql3/statements/schema/AlterTypeStatement.java
@@@ -1,234 -1,0 +1,256 @@@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.cassandra.cql3.statements.schema;
+
+import java.util.ArrayList;
++import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.cassandra.audit.AuditLogContext;
+import org.apache.cassandra.audit.AuditLogEntryType;
+import org.apache.cassandra.auth.Permission;
+import org.apache.cassandra.cql3.*;
+import org.apache.cassandra.db.marshal.AbstractType;
+import org.apache.cassandra.db.marshal.UserType;
+import org.apache.cassandra.schema.KeyspaceMetadata;
+import org.apache.cassandra.schema.Keyspaces;
++import org.apache.cassandra.schema.TableMetadata;
+import org.apache.cassandra.service.ClientState;
+import org.apache.cassandra.transport.Event.SchemaChange;
+import org.apache.cassandra.transport.Event.SchemaChange.Change;
+import org.apache.cassandra.transport.Event.SchemaChange.Target;
+
++import static com.google.common.collect.Iterables.any;
++import static com.google.common.collect.Iterables.filter;
++import static com.google.common.collect.Iterables.transform;
+import static java.lang.String.join;
+import static java.util.function.Predicate.isEqual;
+import static java.util.stream.Collectors.toList;
+
+import static org.apache.cassandra.utils.ByteBufferUtil.bytes;
+
+public abstract class AlterTypeStatement extends AlterSchemaStatement
+{
+protected final String typeName;
+
+public AlterTypeStatement(String keyspaceName, String typeName)
+{
+super(keyspaceName);
+this.typeName = typeName;
+}
+
+public void authorize(ClientState client)
+{
+client.ensureKeyspacePermission(keyspaceName, Permission.ALTER);
+}
+
+SchemaChange schemaChangeEvent(Keyspaces.KeyspacesDiff diff)
+{
+return new SchemaChange(Change.UPDATED, Target.TYPE, keyspaceName,
typeName);
+}
+
+public Keyspaces apply(Keyspaces schema)
+{
+KeyspaceMetadata keyspace = schema.getNullable(keyspaceName);
+
+UserType type = null == keyspace
+ ? null
+ : keyspace.types.getNullable(bytes(typeName));
+
+if (null == type)
+throw ire("Type %s.%s doesn't exist", keyspaceName, typeName);
+
+return
schema.withAddedOrUpdated(keyspace.withUpdatedUserType(apply(keyspace, type)));
+}
+
+abstract UserType apply(KeyspaceMetadata
[jira] [Commented] (CASSANDRA-15967) Add support in jvm dtest to test thrift
[ https://issues.apache.org/jira/browse/CASSANDRA-15967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17165788#comment-17165788 ] Aleksey Yeschenko commented on CASSANDRA-15967: --- Looks like you guys haven't merged it into trunk (with -s ours). Taking care of it, just remember to always propagate the change all the way to trunk, even as an empty merge commit. > Add support in jvm dtest to test thrift > --- > > Key: CASSANDRA-15967 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15967 > Project: Cassandra > Issue Type: Improvement > Components: Test/dtest >Reporter: David Capwell >Assignee: David Capwell >Priority: Normal > Fix For: 3.0.22, 3.11.8, 2.2.18 > > > In 2.2 and 3.x thrift is supported and can cause problems for the non thrift > code. To make sure we can test these interactions, need to add support for > thrift in jvm dtest. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] 01/01: Merge branch 'cassandra-3.0' into cassandra-3.11
This is an automated email from the ASF dual-hosted git repository.
aleksey pushed a commit to branch cassandra-3.11
in repository https://gitbox.apache.org/repos/asf/cassandra.git
commit c61f3902caee4a2fbff4adff6ec7ff58c09a5d09
Merge: 8c61214 dc725bc
Author: Aleksey Yeshchenko
AuthorDate: Mon Jul 27 16:11:50 2020 +0100
Merge branch 'cassandra-3.0' into cassandra-3.11
CHANGES.txt| 1 +
.../cql3/statements/AlterTypeStatement.java| 23 ++
.../cql3/validation/operations/AlterTest.java | 19 ++
3 files changed, 43 insertions(+)
diff --cc CHANGES.txt
index 5f08069,a42de51..812e020
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,6 -1,5 +1,7 @@@
-3.0.22:
+3.11.8
+ * Frozen RawTuple is not annotated with frozen in the toString method
(CASSANDRA-15857)
+Merged from 3.0:
+ * Forbid altering UDTs used in partition keys (CASSANDRA-15933)
* Fix empty/null json string representation (CASSANDRA-15896)
Merged from 2.2:
* Fix CQL parsing of collections when the column type is reversed
(CASSANDRA-15814)
diff --cc src/java/org/apache/cassandra/cql3/statements/AlterTypeStatement.java
index 71d19fa,0f506f7..d0e2f33
--- a/src/java/org/apache/cassandra/cql3/statements/AlterTypeStatement.java
+++ b/src/java/org/apache/cassandra/cql3/statements/AlterTypeStatement.java
@@@ -266,8 -288,17 +280,17 @@@ public abstract class AlterTypeStatemen
newTypes.addAll(toUpdate.fieldTypes());
newTypes.add(addType);
-return new UserType(toUpdate.keyspace, toUpdate.name, newNames,
newTypes);
+return new UserType(toUpdate.keyspace, toUpdate.name, newNames,
newTypes, toUpdate.isMultiCell());
}
+
+ private static Collection
findTablesReferencingTypeInPartitionKey(KeyspaceMetadata keyspace, String
userTypeName)
+ {
+ Collection tables = new ArrayList<>();
+ filter(keyspace.tablesAndViews(),
+table -> any(table.partitionKeyColumns(), column ->
column.type.referencesUserType(userTypeName)))
+ .forEach(tables::add);
+ return tables;
+ }
}
private static class Renames extends AlterTypeStatement
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] branch cassandra-3.11 updated (8c61214 -> c61f390)
This is an automated email from the ASF dual-hosted git repository. aleksey pushed a change to branch cassandra-3.11 in repository https://gitbox.apache.org/repos/asf/cassandra.git. from 8c61214 Merge branch 'cassandra-3.0' into cassandra-3.11 add dc725bc Forbid altering UDTs used in partition keys new c61f390 Merge branch 'cassandra-3.0' into cassandra-3.11 The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: CHANGES.txt| 1 + .../cql3/statements/AlterTypeStatement.java| 23 ++ .../cql3/validation/operations/AlterTest.java | 19 ++ 3 files changed, 43 insertions(+) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] branch cassandra-3.0 updated (ebf9c74 -> dc725bc)
This is an automated email from the ASF dual-hosted git repository. aleksey pushed a change to branch cassandra-3.0 in repository https://gitbox.apache.org/repos/asf/cassandra.git. from ebf9c74 Merge branch 'cassandra-2.2' into cassandra-3.0 add dc725bc Forbid altering UDTs used in partition keys No new revisions were added by this update. Summary of changes: CHANGES.txt| 1 + .../cql3/statements/AlterTypeStatement.java| 23 ++ .../cql3/validation/operations/AlterTest.java | 19 ++ 3 files changed, 43 insertions(+) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15986) Repair tests flakiness
[ https://issues.apache.org/jira/browse/CASSANDRA-15986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Berenguer Blasi updated CASSANDRA-15986: Description: Repair tests come up in test failure reports every now and then. I have tried to repro the [latest|https://ci-cassandra.apache.org/job/Cassandra-trunk/241/testReport/junit/dtest-novnode.repair_tests.repair_test/TestRepair/test_simple_sequential_repair/] locally 100 times with no luck. Still from experience from fixing other flaky tests I have some intuition where the problems may lie. The proposed fix should add no harm if merged. We can reopen the ticket if repair tests keep failing. was: Repair tests come up in test failure reports every now and then. I have tried to repro the [latest|https://ci-cassandra.apache.org/job/Cassandra-trunk/241/testReport/junit/dtest-novnode.repair_tests.repair_test/TestRepair/test_simple_sequential_repair/] locally 100 times with no luck. Still from experience from fixing other flaky tests I have some intuition where the problems may lie. The proposed fix should add no harm if merged. > Repair tests flakiness > -- > > Key: CASSANDRA-15986 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15986 > Project: Cassandra > Issue Type: Task > Components: Test/dtest >Reporter: Berenguer Blasi >Assignee: Berenguer Blasi >Priority: Normal > Fix For: 4.0-beta > > > Repair tests come up in test failure reports every now and then. I have tried > to repro the > [latest|https://ci-cassandra.apache.org/job/Cassandra-trunk/241/testReport/junit/dtest-novnode.repair_tests.repair_test/TestRepair/test_simple_sequential_repair/] > locally 100 times with no luck. > Still from experience from fixing other flaky tests I have some intuition > where the problems may lie. The proposed fix should add no harm if merged. We > can reopen the ticket if repair tests keep failing. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15986) Repair tests flakiness
[ https://issues.apache.org/jira/browse/CASSANDRA-15986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Berenguer Blasi updated CASSANDRA-15986: Test and Documentation Plan: See PR Status: Patch Available (was: In Progress) > Repair tests flakiness > -- > > Key: CASSANDRA-15986 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15986 > Project: Cassandra > Issue Type: Task > Components: Test/dtest >Reporter: Berenguer Blasi >Assignee: Berenguer Blasi >Priority: Normal > Fix For: 4.0-beta > > > Repair tests come up in test failure reports every now and then. I have tried > to repro the > [latest|https://ci-cassandra.apache.org/job/Cassandra-trunk/241/testReport/junit/dtest-novnode.repair_tests.repair_test/TestRepair/test_simple_sequential_repair/] > locally 100 times with no luck. > Still from experience from fixing other flaky tests I have some intuition > where the problems may lie. The proposed fix should add no harm if merged. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15986) Repair tests flakiness
[ https://issues.apache.org/jira/browse/CASSANDRA-15986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Berenguer Blasi updated CASSANDRA-15986: Change Category: Quality Assurance Complexity: Normal Fix Version/s: 4.0-beta Status: Open (was: Triage Needed) > Repair tests flakiness > -- > > Key: CASSANDRA-15986 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15986 > Project: Cassandra > Issue Type: Task > Components: Test/dtest >Reporter: Berenguer Blasi >Assignee: Berenguer Blasi >Priority: Normal > Fix For: 4.0-beta > > > Repair tests come up in test failure reports every now and then. I have tried > to repro the > [latest|https://ci-cassandra.apache.org/job/Cassandra-trunk/241/testReport/junit/dtest-novnode.repair_tests.repair_test/TestRepair/test_simple_sequential_repair/] > locally 100 times with no luck. > Still from experience from fixing other flaky tests I have some intuition > where the problems may lie. The proposed fix should add no harm if merged. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-15986) Repair tests flakiness
Berenguer Blasi created CASSANDRA-15986: --- Summary: Repair tests flakiness Key: CASSANDRA-15986 URL: https://issues.apache.org/jira/browse/CASSANDRA-15986 Project: Cassandra Issue Type: Task Components: Test/dtest Reporter: Berenguer Blasi Assignee: Berenguer Blasi Repair tests come up in test failure reports every now and then. I have tried to repro the [latest|https://ci-cassandra.apache.org/job/Cassandra-trunk/241/testReport/junit/dtest-novnode.repair_tests.repair_test/TestRepair/test_simple_sequential_repair/] locally 100 times with no luck. Still from experience from fixing other flaky tests I have some intuition where the problems may lie. The proposed fix should add no harm if merged. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14877) StreamCoordinator "leaks" threads
[ https://issues.apache.org/jira/browse/CASSANDRA-14877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17165730#comment-17165730 ] Ekaterina Dimitrova commented on CASSANDRA-14877: - Hey [~maxtomassi], Any update on this ticket? > StreamCoordinator "leaks" threads > - > > Key: CASSANDRA-14877 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14877 > Project: Cassandra > Issue Type: Bug > Components: Legacy/Streaming and Messaging >Reporter: Massimiliano Tomassi >Assignee: Massimiliano Tomassi >Priority: Low > Fix For: 2.1.x, 2.2.x, 3.0.x, 3.11.x, 4.x > > > Since Cassandra 2.1, streaming sessions are started by running a > StreamSessionConnector task for each session in a dedicated executor (a > static field of StreamCoordinator). > That executor is initialized with > DebuggableThreadPoolExecutor.createWithFixedPoolSize, which means that once > created (up to the given limit of the number of logical cores), its threads > are kept alive for Integer.MAX_VALUE seconds. > This practically means that once a node needs to establish streaming sessions > to n other nodes, it will create Math.min(n, numLogicalCores) > StreamConnectionEstablisher threads that will stay parked forever after > initializing (not completing) the session. > It seems preferable to replace > DebuggableThreadPoolExecutor.createWithFixedPoolSize with > DebuggableThreadPoolExecutor.createWithMaximumPoolSize which allows providing > a saner keep-alive period (e.g. a minute). > That's also what createWithFixedPoolSize's Javadoc recommends: If (most) > threads are expected to be idle most of the time, prefer createWithMaxSize() > instead. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14906) FastByteOperations.UnsafeOperations fails to handle read only byte buffers correctly
[ https://issues.apache.org/jira/browse/CASSANDRA-14906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17165725#comment-17165725 ] Ekaterina Dimitrova commented on CASSANDRA-14906: - [~Ge] is there any update/plan on this one? > FastByteOperations.UnsafeOperations fails to handle read only byte buffers > correctly > > > Key: CASSANDRA-14906 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14906 > Project: Cassandra > Issue Type: Bug > Components: Legacy/Core >Reporter: Aleksandr Sorokoumov >Assignee: Aleksandr Sorokoumov >Priority: Low > Fix For: 4.x > > Attachments: 14906-dtest.png, 14906-testall.png > > > If a buffer is read only it reports having no array, though it may well be > backed by an array. Code in UnsafeOperation works under the impression that a > buffer for which hasArray() == false is necessarily a direct buffer. Which is > not the case. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-15942) 15583 - Tooling testing foundation
[ https://issues.apache.org/jira/browse/CASSANDRA-15942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17165708#comment-17165708 ] Berenguer Blasi edited comment on CASSANDRA-15942 at 7/27/20, 1:37 PM: --- Whoa, you rock Sir! :-) Edit CASSANDRA-15956 is super small and if it could get merged as well that would be the cherry on top, as it adds a few improvements to be able to cover for all cases of 15583 sub tickets. was (Author: bereng): Whoa, you rock Sir! :-) > 15583 - Tooling testing foundation > -- > > Key: CASSANDRA-15942 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15942 > Project: Cassandra > Issue Type: Improvement > Components: Test/unit >Reporter: Berenguer Blasi >Assignee: Berenguer Blasi >Priority: Normal > Fix For: 4.0-beta > > > Tooling testing is not as thorough as we'd like it to be. The purpose of this > ticket, under the umbrella of CASSANDRA-15583, is to provide the foundations > to make it easy for devs, migrate the few existing tests as a POC and put > that forward as a proposal to build on top of. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15942) 15583 - Tooling testing foundation
[ https://issues.apache.org/jira/browse/CASSANDRA-15942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17165708#comment-17165708 ] Berenguer Blasi commented on CASSANDRA-15942: - Whoa, you rock Sir! :-) > 15583 - Tooling testing foundation > -- > > Key: CASSANDRA-15942 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15942 > Project: Cassandra > Issue Type: Improvement > Components: Test/unit >Reporter: Berenguer Blasi >Assignee: Berenguer Blasi >Priority: Normal > Fix For: 4.0-beta > > > Tooling testing is not as thorough as we'd like it to be. The purpose of this > ticket, under the umbrella of CASSANDRA-15583, is to provide the foundations > to make it easy for devs, migrate the few existing tests as a POC and put > that forward as a proposal to build on top of. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15942) 15583 - Tooling testing foundation
[ https://issues.apache.org/jira/browse/CASSANDRA-15942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17165702#comment-17165702 ] Sam Tunnicliffe commented on CASSANDRA-15942: - [~Bereng], I've merged this to trunk so it's available for subtasks of CASSANDRA-15583 > 15583 - Tooling testing foundation > -- > > Key: CASSANDRA-15942 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15942 > Project: Cassandra > Issue Type: Improvement > Components: Test/unit >Reporter: Berenguer Blasi >Assignee: Berenguer Blasi >Priority: Normal > Fix For: 4.0-beta > > > Tooling testing is not as thorough as we'd like it to be. The purpose of this > ticket, under the umbrella of CASSANDRA-15583, is to provide the foundations > to make it easy for devs, migrate the few existing tests as a POC and put > that forward as a proposal to build on top of. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15942) 15583 - Tooling testing foundation
[ https://issues.apache.org/jira/browse/CASSANDRA-15942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam Tunnicliffe updated CASSANDRA-15942: Source Control Link: https://github.com/apache/cassandra/commit/a8212149aba7a85139de9758988f6e01f7bc8d91 (was: See PR) > 15583 - Tooling testing foundation > -- > > Key: CASSANDRA-15942 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15942 > Project: Cassandra > Issue Type: Improvement > Components: Test/unit >Reporter: Berenguer Blasi >Assignee: Berenguer Blasi >Priority: Normal > Fix For: 4.0-beta > > > Tooling testing is not as thorough as we'd like it to be. The purpose of this > ticket, under the umbrella of CASSANDRA-15583, is to provide the foundations > to make it easy for devs, migrate the few existing tests as a POC and put > that forward as a proposal to build on top of. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15942) 15583 - Tooling testing foundation
[ https://issues.apache.org/jira/browse/CASSANDRA-15942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam Tunnicliffe updated CASSANDRA-15942: Status: Resolved (was: Open) > 15583 - Tooling testing foundation > -- > > Key: CASSANDRA-15942 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15942 > Project: Cassandra > Issue Type: Improvement > Components: Test/unit >Reporter: Berenguer Blasi >Assignee: Berenguer Blasi >Priority: Normal > Fix For: 4.0-beta > > > Tooling testing is not as thorough as we'd like it to be. The purpose of this > ticket, under the umbrella of CASSANDRA-15583, is to provide the foundations > to make it easy for devs, migrate the few existing tests as a POC and put > that forward as a proposal to build on top of. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15942) 15583 - Tooling testing foundation
[ https://issues.apache.org/jira/browse/CASSANDRA-15942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam Tunnicliffe updated CASSANDRA-15942: Status: Open (was: Resolved) > 15583 - Tooling testing foundation > -- > > Key: CASSANDRA-15942 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15942 > Project: Cassandra > Issue Type: Improvement > Components: Test/unit >Reporter: Berenguer Blasi >Assignee: Berenguer Blasi >Priority: Normal > Fix For: 4.0-beta > > > Tooling testing is not as thorough as we'd like it to be. The purpose of this > ticket, under the umbrella of CASSANDRA-15583, is to provide the foundations > to make it easy for devs, migrate the few existing tests as a POC and put > that forward as a proposal to build on top of. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] branch trunk updated: CASSANDRA-15942 Improve tooling testing framework
This is an automated email from the ASF dual-hosted git repository.
samt pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra.git
The following commit(s) were added to refs/heads/trunk by this push:
new a821214 CASSANDRA-15942 Improve tooling testing framework
a821214 is described below
commit a8212149aba7a85139de9758988f6e01f7bc8d91
Author: Bereng
AuthorDate: Tue Jul 14 16:44:58 2020 +0200
CASSANDRA-15942 Improve tooling testing framework
Patch by Berenguer Blasi; reviewed by Robert Stupp
and Sam Tunnicliffe for CASSANDRA-15942
---
.../config/DatabaseDescriptorRefTest.java | 2 +-
test/unit/org/apache/cassandra/cql3/CQLTester.java | 66 +++-
.../org/apache/cassandra/tools/BulkLoaderTest.java | 28 +-
.../apache/cassandra/tools/ClearSnapshotTest.java | 91 ++---
.../cassandra/tools/CompactionStressTest.java | 41 +-
.../org/apache/cassandra/tools/GetVersionTest.java | 6 +-
.../apache/cassandra/tools/LoaderOptionsTest.java | 2 +-
.../{ToolsTester.java => OfflineToolUtils.java}| 66 +---
.../tools/SSTableExpiredBlockersTest.java | 10 +-
.../apache/cassandra/tools/SSTableExportTest.java | 11 +-
.../cassandra/tools/SSTableLevelResetterTest.java | 11 +-
.../cassandra/tools/SSTableMetadataViewerTest.java | 10 +-
.../cassandra/tools/SSTableOfflineRelevelTest.java | 10 +-
.../tools/SSTableRepairedAtSetterTest.java | 11 +-
.../cassandra/tools/StandaloneSSTableUtilTest.java | 11 +-
.../cassandra/tools/StandaloneScrubberTest.java| 11 +-
.../cassandra/tools/StandaloneSplitterTest.java| 8 +-
.../cassandra/tools/StandaloneUpgraderTest.java| 11 +-
.../cassandra/tools/StandaloneVerifierTest.java| 11 +-
.../org/apache/cassandra/tools/ToolRunner.java | 437 +
20 files changed, 676 insertions(+), 178 deletions(-)
diff --git
a/test/unit/org/apache/cassandra/config/DatabaseDescriptorRefTest.java
b/test/unit/org/apache/cassandra/config/DatabaseDescriptorRefTest.java
index 576862b..8f731e2 100644
--- a/test/unit/org/apache/cassandra/config/DatabaseDescriptorRefTest.java
+++ b/test/unit/org/apache/cassandra/config/DatabaseDescriptorRefTest.java
@@ -49,7 +49,7 @@ import static org.junit.Assert.fail;
* unexpected threads.
*
* {@link DatabaseDescriptor#toolInitialization()} is tested via unit tests
extending
- * {@link org.apache.cassandra.tools.ToolsTester}.
+ * {@link org.apache.cassandra.tools.OfflineToolUtils}.
*/
public class DatabaseDescriptorRefTest
{
diff --git a/test/unit/org/apache/cassandra/cql3/CQLTester.java
b/test/unit/org/apache/cassandra/cql3/CQLTester.java
index 619fdad..0455f97 100644
--- a/test/unit/org/apache/cassandra/cql3/CQLTester.java
+++ b/test/unit/org/apache/cassandra/cql3/CQLTester.java
@@ -22,9 +22,12 @@ import java.io.IOException;
import java.math.BigDecimal;
import java.math.BigInteger;
import java.net.InetAddress;
+import java.net.InetSocketAddress;
+import java.net.MalformedURLException;
import java.net.ServerSocket;
import java.net.UnknownHostException;
import java.nio.ByteBuffer;
+import java.rmi.server.RMISocketFactory;
import java.util.*;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.ExecutionException;
@@ -34,6 +37,12 @@ import java.util.regex.Matcher;
import java.util.regex.Pattern;
import java.util.stream.Collectors;
+import javax.management.MBeanServerConnection;
+import javax.management.remote.JMXConnector;
+import javax.management.remote.JMXConnectorFactory;
+import javax.management.remote.JMXConnectorServer;
+import javax.management.remote.JMXServiceURL;
+
import com.google.common.base.Objects;
import com.google.common.base.Strings;
import com.google.common.collect.ImmutableSet;
@@ -80,6 +89,7 @@ import org.apache.cassandra.transport.ProtocolVersion;
import org.apache.cassandra.transport.messages.ResultMessage;
import org.apache.cassandra.utils.ByteBufferUtil;
import org.apache.cassandra.utils.FBUtilities;
+import org.apache.cassandra.utils.JMXServerUtils;
import org.apache.cassandra.security.ThreadAwareSecurityManager;
import static junit.framework.Assert.assertNotNull;
@@ -103,6 +113,11 @@ public abstract class CQLTester
public static final String RACK1 = "rack1";
private static org.apache.cassandra.transport.Server server;
+private static JMXConnectorServer jmxServer;
+protected static String jmxHost;
+protected static int jmxPort;
+protected static MBeanServerConnection jmxConnection;
+
protected static final int nativePort;
protected static final InetAddress nativeAddr;
protected static final Set remoteAddrs = new
HashSet<>();
@@ -240,6 +255,43 @@ public abstract class CQLTester
AuditLogManager.instance.initialize();
isServerPrepared = true;
}
+
+/**
+ * Starts the JMX server. It's safe to call this method multiple times.
+ */
+public static void
[jira] [Updated] (CASSANDRA-15983) DOC - Fix incorrect CREATE SCHEMA command in README.asc
[
https://issues.apache.org/jira/browse/CASSANDRA-15983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Erick Ramirez updated CASSANDRA-15983:
--
Status: Patch Available (was: In Progress)
On [~mck]'s advice, resubmitting with separate PRs for all supported versions:
* trunk PR - https://github.com/apache/cassandra/pull/693
* 2.1 PR - https://github.com/apache/cassandra/pull/694
* 2.2 PR - https://github.com/apache/cassandra/pull/695
* 3.0 PR - https://github.com/apache/cassandra/pull/696
* 3.11 PR - https://github.com/apache/cassandra/pull/697
Patch - [^CASSANDRA-15983.txt]
> DOC - Fix incorrect CREATE SCHEMA command in README.asc
>
>
> Key: CASSANDRA-15983
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15983
> Project: Cassandra
> Issue Type: Task
> Components: Documentation/Website
>Reporter: Erick Ramirez
>Assignee: Erick Ramirez
>Priority: Normal
> Fix For: 4.0, 2.1.x, 2.2.x, 3.0.x, 3.11.x
>
> Attachments: CASSANDRA-15983.txt
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> It looks like no one has reviewed the contents of
> [{{README.asc}}|https://github.com/apache/cassandra/blob/cassandra-3.11.7/README.asc].
> Instead of the {{CREATE KEYSPACE}} command, the example CQL reads:
> {noformat}
> cqlsh> CREATE SCHEMA schema1
>WITH replication = { 'class' : 'SimpleStrategy', 'replication_factor'
> : 1 };
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-15925) Jenkins pipeline can copy wrong test report artefacts from stage builds
[ https://issues.apache.org/jira/browse/CASSANDRA-15925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17165594#comment-17165594 ] Michael Semb Wever edited comment on CASSANDRA-15925 at 7/27/20, 11:58 AM: --- bq. org.apache.cassandra.transport.frame.checksum.ChecksummingTransformerTest.corruptionCausesFailure is missing from the top #220 failures report and timestamps and out from testCapNoWarnExpirationOverflowPolicy failure doesn't match that of the matching #220 top failures report The *test output* copied in the pipeline are actually from the test-cdc stage job. This is evident [here|https://ci-cassandra.apache.org/job/Cassandra-devbranch/220/testReport/org.apache.cassandra.cql3.validation.operations/TTLTest/testCapNoWarnExpirationOverflowPolicy/] compared to [here|https://ci-cassandra.apache.org/view/patches/job/Cassandra-devbranch-test-cdc/146/testReport/org.apache.cassandra.cql3.validation.operations/TTLTest/testCapNoWarnExpirationOverflowPolicyDefaultTTL_cdc/] The test names are also not getting passed through and honoured correctly… Nor are tests always present, for example [here|https://ci-cassandra.apache.org/job/Cassandra-devbranch/220/testReport/junit/org.apache.cassandra.transport.frame.checksum/ChecksummingTransformerTest/]. So… the problem here is no longer about which artefacts are getting copied, but how they get aggregated together (which is a custom process in the `Summary` stage). was (Author: michaelsembwever): bq. org.apache.cassandra.transport.frame.checksum.ChecksummingTransformerTest.corruptionCausesFailure is missing from the top #220 failures report and timestamps and out from testCapNoWarnExpirationOverflowPolicy failure doesn't match that of the matching #220 top failures report The *test output* copied in the pipeline are actually from the test-cdc stage job. This is evident [here|https://ci-cassandra.apache.org/job/Cassandra-devbranch/220/testReport/org.apache.cassandra.cql3.validation.operations/TTLTest/testCapNoWarnExpirationOverflowPolicy/] compared to [here|https://ci-cassandra.apache.org/view/patches/job/Cassandra-devbranch-test-cdc/146/testReport/org.apache.cassandra.cql3.validation.operations/TTLTest/testCapNoWarnExpirationOverflowPolicyDefaultTTL_cdc/] The test names are also not getting passed through and honoured correctly… Nor are tests always present, for example [here|https://ci-cassandra.apache.org/job/Cassandra-devbranch/220/testReport/junit/org.apache.cassandra.transport.frame.checksum/ChecksummingTransformerTest/]. > Jenkins pipeline can copy wrong test report artefacts from stage builds > --- > > Key: CASSANDRA-15925 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15925 > Project: Cassandra > Issue Type: Bug > Components: CI >Reporter: Michael Semb Wever >Assignee: Michael Semb Wever >Priority: Normal > Fix For: 4.0-rc > > > Spotted in > https://ci-cassandra.apache.org/view/patches/job/Cassandra-devbranch/196/console > Looks like copyArtifact will need to be specific to a build. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15983) DOC - Fix incorrect CREATE SCHEMA command in README.asc
[
https://issues.apache.org/jira/browse/CASSANDRA-15983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Erick Ramirez updated CASSANDRA-15983:
--
Status: In Progress (was: Patch Available)
> DOC - Fix incorrect CREATE SCHEMA command in README.asc
>
>
> Key: CASSANDRA-15983
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15983
> Project: Cassandra
> Issue Type: Task
> Components: Documentation/Website
>Reporter: Erick Ramirez
>Assignee: Erick Ramirez
>Priority: Normal
> Fix For: 4.0, 2.1.x, 2.2.x, 3.0.x, 3.11.x
>
> Attachments: CASSANDRA-15983.txt
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> It looks like no one has reviewed the contents of
> [{{README.asc}}|https://github.com/apache/cassandra/blob/cassandra-3.11.7/README.asc].
> Instead of the {{CREATE KEYSPACE}} command, the example CQL reads:
> {noformat}
> cqlsh> CREATE SCHEMA schema1
>WITH replication = { 'class' : 'SimpleStrategy', 'replication_factor'
> : 1 };
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15933) Forbid adding new fields to UDTs used in partition key columns
[ https://issues.apache.org/jira/browse/CASSANDRA-15933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam Tunnicliffe updated CASSANDRA-15933: Reviewers: Alex Petrov, Sam Tunnicliffe, Sam Tunnicliffe (was: Alex Petrov, Sam Tunnicliffe) Alex Petrov, Sam Tunnicliffe, Sam Tunnicliffe (was: Alex Petrov, Sam Tunnicliffe) Status: Review In Progress (was: Patch Available) > Forbid adding new fields to UDTs used in partition key columns > -- > > Key: CASSANDRA-15933 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15933 > Project: Cassandra > Issue Type: Bug > Components: Cluster/Schema >Reporter: Aleksey Yeschenko >Assignee: Aleksey Yeschenko >Priority: Normal > Fix For: 4.0, 3.0.x, 3.11.x > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15933) Forbid adding new fields to UDTs used in partition key columns
[ https://issues.apache.org/jira/browse/CASSANDRA-15933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam Tunnicliffe updated CASSANDRA-15933: Status: Ready to Commit (was: Review In Progress) +1 > Forbid adding new fields to UDTs used in partition key columns > -- > > Key: CASSANDRA-15933 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15933 > Project: Cassandra > Issue Type: Bug > Components: Cluster/Schema >Reporter: Aleksey Yeschenko >Assignee: Aleksey Yeschenko >Priority: Normal > Fix For: 4.0, 3.0.x, 3.11.x > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15953) Support fetching all user tables to compare in Cassandra-diff
[
https://issues.apache.org/jira/browse/CASSANDRA-15953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sam Tunnicliffe updated CASSANDRA-15953:
Fix Version/s: 4.0-beta
Source Control Link:
https://github.com/apache/cassandra-diff/commit/2267933c3ac009808271e1831cf56258c99cf2d3
Resolution: Fixed
Status: Resolved (was: Ready to Commit)
LGTM, thanks. Committed with a tiny tweak to {{README.md}} to reflect the
changes to automatic filtering of system keyspaces.
> Support fetching all user tables to compare in Cassandra-diff
> -
>
> Key: CASSANDRA-15953
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15953
> Project: Cassandra
> Issue Type: Improvement
> Components: Tool/diff
>Reporter: Yifan Cai
>Assignee: Yifan Cai
>Priority: Normal
> Labels: pull-request-available
> Fix For: 4.0-beta
>
>
> The spark diff job may fail to launch with kernel error "E2BIG: Argument list
> too long", when passing a large list of keyspace table list to compare.
> Proposing a mode to fetch all user tables from the clusters to be compared.
> When the mode is on, the spark job ignores the parameter "keyspace_tables".
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra-diff] branch master updated: Support auto discover user tables for comparison
This is an automated email from the ASF dual-hosted git repository.
samt pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/cassandra-diff.git
The following commit(s) were added to refs/heads/master by this push:
new 2267933 Support auto discover user tables for comparison
2267933 is described below
commit 2267933c3ac009808271e1831cf56258c99cf2d3
Author: Yifan Cai
AuthorDate: Thu Jul 16 20:40:40 2020 -0700
Support auto discover user tables for comparison
Patch by Yifan Cai; reviewed by Sam Tunnicliffe for CASSANDRA-15953
closes #10
---
README.md | 7 ++
.../apache/cassandra/diff/api/DiffAPIServer.java | 3 +-
.../apache/cassandra/diff/JobConfiguration.java| 35 -
.../cassandra/diff/YamlJobConfiguration.java | 16 ++--
.../cassandra/diff/YamlJobConfigurationTest.java | 23 +-
...est_load_config_all_keyspaces_filtered_out.yaml | 57 +++
.../test_load_config_no_keyspace_tables.yaml | 45
spark-job/localconfig-auto-discover.yaml | 45
.../java/org/apache/cassandra/diff/DiffJob.java| 38 +++---
.../java/org/apache/cassandra/diff/Schema.java | 76 +++
.../diff/AbstractMockJobConfiguration.java | 79
.../java/org/apache/cassandra/diff/SchemaTest.java | 85 ++
12 files changed, 489 insertions(+), 20 deletions(-)
diff --git a/README.md b/README.md
index c17ef59..7793392 100644
--- a/README.md
+++ b/README.md
@@ -3,6 +3,11 @@
## Configuration
See `spark-job/localconfig.yaml` for an example config.
+See `spark-job/localconfig-multi-keyspaces.yaml` for an example config that
compares tables under multiple keyspaces.
+
+See `spark-job/localconfig-auto-discover.yaml` for an example config that auto
discovers all user tables to compare.
+The auto discover mode excludes all system keyspaces and any keyspaces defined
at `disallowed_keyspaces` in the yaml file.
+
## Custom cluster providers
To make it easy to run in any environment the cluster providers are pluggable
- there are two interfaces to implement.
First, the `ClusterProvider` interface is used to create a connection to the
clusters, and it is configured using
@@ -50,6 +55,8 @@ $ docker exec cas-tgt cassandra-stress write n=1k -schema
keyspace="keyspace2"
$ spark-submit --verbose --files ./spark-job/localconfig.yaml --class
org.apache.cassandra.diff.DiffJob
spark-uberjar/target/spark-uberjar-0.2-SNAPSHOT.jar localconfig.yaml
# If rows are created in "keyspace2", you can run pick up the
localconfig-multi-keyspaces.yaml to compare data across multiple keyspaces! See
the command below.
# $ spark-submit --verbose --files
./spark-job/localconfig-multi-keyspaces.yaml --class
org.apache.cassandra.diff.DiffJob
spark-uberjar/target/spark-uberjar-0.2-SNAPSHOT.jar
localconfig-multi-keyspaces.yaml
+# To use the auto discover mode, you can run the job with
localconfig-auto-discover.yaml, which has the keyspace_tables field removed.
+# $ spark-submit --verbose --files ./spark-job/localconfig-auto-discover.yaml
--class org.apache.cassandra.diff.DiffJob
spark-uberjar/target/spark-uberjar-0.2-SNAPSHOT.jar
localconfig-auto-discover.yaml
# ... logs
INFO DiffJob:124 - FINISHED: {standard1=Matched Partitions - 1000, Mismatched
Partitions - 0, Partition Errors - 0, Partitions Only In Source - 0, Partitions
Only In Target - 0, Skipped Partitions - 0, Matched Rows - 1000, Matched Values
- 6000, Mismatched Values - 0 }
## start api-server:
diff --git
a/api-server/src/main/java/org/apache/cassandra/diff/api/DiffAPIServer.java
b/api-server/src/main/java/org/apache/cassandra/diff/api/DiffAPIServer.java
index 6f14128..533fa22 100644
--- a/api-server/src/main/java/org/apache/cassandra/diff/api/DiffAPIServer.java
+++ b/api-server/src/main/java/org/apache/cassandra/diff/api/DiffAPIServer.java
@@ -19,6 +19,7 @@
package org.apache.cassandra.diff.api;
+import java.io.FileInputStream;
import java.io.IOException;
import com.google.common.collect.Lists;
@@ -35,7 +36,7 @@ public class DiffAPIServer {
String filename = args[0];
JAXRSServerFactoryBean factoryBean = new JAXRSServerFactoryBean();
-DiffJobsResource diffResource = new
DiffJobsResource(YamlJobConfiguration.load(filename));
+DiffJobsResource diffResource = new
DiffJobsResource(YamlJobConfiguration.load(new FileInputStream(filename)));
factoryBean.setResourceProviders(Lists.newArrayList(new
SingletonResourceProvider(diffResource),
new
SingletonResourceProvider(new HealthResource(;
factoryBean.setAddress("http://localhost:8089/;);
diff --git
a/common/src/main/java/org/apache/cassandra/diff/JobConfiguration.java
b/common/src/main/java/org/apache/cassandra/diff/JobConfiguration.java
index 2d6cb51..cf12ea3 100644
[jira] [Updated] (CASSANDRA-15953) Support fetching all user tables to compare in Cassandra-diff
[ https://issues.apache.org/jira/browse/CASSANDRA-15953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam Tunnicliffe updated CASSANDRA-15953: Status: Ready to Commit (was: Changes Suggested) > Support fetching all user tables to compare in Cassandra-diff > - > > Key: CASSANDRA-15953 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15953 > Project: Cassandra > Issue Type: Improvement > Components: Tool/diff >Reporter: Yifan Cai >Assignee: Yifan Cai >Priority: Normal > Labels: pull-request-available > > The spark diff job may fail to launch with kernel error "E2BIG: Argument list > too long", when passing a large list of keyspace table list to compare. > Proposing a mode to fetch all user tables from the clusters to be compared. > When the mode is on, the spark job ignores the parameter "keyspace_tables". -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra-diff] branch master updated: Support auto discover user tables for comparison
This is an automated email from the ASF dual-hosted git repository.
samt pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/cassandra-diff.git
The following commit(s) were added to refs/heads/master by this push:
new 2267933 Support auto discover user tables for comparison
2267933 is described below
commit 2267933c3ac009808271e1831cf56258c99cf2d3
Author: Yifan Cai
AuthorDate: Thu Jul 16 20:40:40 2020 -0700
Support auto discover user tables for comparison
Patch by Yifan Cai; reviewed by Sam Tunnicliffe for CASSANDRA-15953
closes #10
---
README.md | 7 ++
.../apache/cassandra/diff/api/DiffAPIServer.java | 3 +-
.../apache/cassandra/diff/JobConfiguration.java| 35 -
.../cassandra/diff/YamlJobConfiguration.java | 16 ++--
.../cassandra/diff/YamlJobConfigurationTest.java | 23 +-
...est_load_config_all_keyspaces_filtered_out.yaml | 57 +++
.../test_load_config_no_keyspace_tables.yaml | 45
spark-job/localconfig-auto-discover.yaml | 45
.../java/org/apache/cassandra/diff/DiffJob.java| 38 +++---
.../java/org/apache/cassandra/diff/Schema.java | 76 +++
.../diff/AbstractMockJobConfiguration.java | 79
.../java/org/apache/cassandra/diff/SchemaTest.java | 85 ++
12 files changed, 489 insertions(+), 20 deletions(-)
diff --git a/README.md b/README.md
index c17ef59..7793392 100644
--- a/README.md
+++ b/README.md
@@ -3,6 +3,11 @@
## Configuration
See `spark-job/localconfig.yaml` for an example config.
+See `spark-job/localconfig-multi-keyspaces.yaml` for an example config that
compares tables under multiple keyspaces.
+
+See `spark-job/localconfig-auto-discover.yaml` for an example config that auto
discovers all user tables to compare.
+The auto discover mode excludes all system keyspaces and any keyspaces defined
at `disallowed_keyspaces` in the yaml file.
+
## Custom cluster providers
To make it easy to run in any environment the cluster providers are pluggable
- there are two interfaces to implement.
First, the `ClusterProvider` interface is used to create a connection to the
clusters, and it is configured using
@@ -50,6 +55,8 @@ $ docker exec cas-tgt cassandra-stress write n=1k -schema
keyspace="keyspace2"
$ spark-submit --verbose --files ./spark-job/localconfig.yaml --class
org.apache.cassandra.diff.DiffJob
spark-uberjar/target/spark-uberjar-0.2-SNAPSHOT.jar localconfig.yaml
# If rows are created in "keyspace2", you can run pick up the
localconfig-multi-keyspaces.yaml to compare data across multiple keyspaces! See
the command below.
# $ spark-submit --verbose --files
./spark-job/localconfig-multi-keyspaces.yaml --class
org.apache.cassandra.diff.DiffJob
spark-uberjar/target/spark-uberjar-0.2-SNAPSHOT.jar
localconfig-multi-keyspaces.yaml
+# To use the auto discover mode, you can run the job with
localconfig-auto-discover.yaml, which has the keyspace_tables field removed.
+# $ spark-submit --verbose --files ./spark-job/localconfig-auto-discover.yaml
--class org.apache.cassandra.diff.DiffJob
spark-uberjar/target/spark-uberjar-0.2-SNAPSHOT.jar
localconfig-auto-discover.yaml
# ... logs
INFO DiffJob:124 - FINISHED: {standard1=Matched Partitions - 1000, Mismatched
Partitions - 0, Partition Errors - 0, Partitions Only In Source - 0, Partitions
Only In Target - 0, Skipped Partitions - 0, Matched Rows - 1000, Matched Values
- 6000, Mismatched Values - 0 }
## start api-server:
diff --git
a/api-server/src/main/java/org/apache/cassandra/diff/api/DiffAPIServer.java
b/api-server/src/main/java/org/apache/cassandra/diff/api/DiffAPIServer.java
index 6f14128..533fa22 100644
--- a/api-server/src/main/java/org/apache/cassandra/diff/api/DiffAPIServer.java
+++ b/api-server/src/main/java/org/apache/cassandra/diff/api/DiffAPIServer.java
@@ -19,6 +19,7 @@
package org.apache.cassandra.diff.api;
+import java.io.FileInputStream;
import java.io.IOException;
import com.google.common.collect.Lists;
@@ -35,7 +36,7 @@ public class DiffAPIServer {
String filename = args[0];
JAXRSServerFactoryBean factoryBean = new JAXRSServerFactoryBean();
-DiffJobsResource diffResource = new
DiffJobsResource(YamlJobConfiguration.load(filename));
+DiffJobsResource diffResource = new
DiffJobsResource(YamlJobConfiguration.load(new FileInputStream(filename)));
factoryBean.setResourceProviders(Lists.newArrayList(new
SingletonResourceProvider(diffResource),
new
SingletonResourceProvider(new HealthResource(;
factoryBean.setAddress("http://localhost:8089/;);
diff --git
a/common/src/main/java/org/apache/cassandra/diff/JobConfiguration.java
b/common/src/main/java/org/apache/cassandra/diff/JobConfiguration.java
index 2d6cb51..cf12ea3 100644
[jira] [Commented] (CASSANDRA-15158) Wait for schema agreement rather than in flight schema requests when bootstrapping
[
https://issues.apache.org/jira/browse/CASSANDRA-15158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17165619#comment-17165619
]
Stefan Miklosovic commented on CASSANDRA-15158:
---
In general looks good to me in spite of getting lost a bit on the actual schema
migration response:
{code:java}
Future response(Collection mutations)
{
synchronized (info)
{
if (shouldApplySchemaFrom(endpoint, info))
mergeSchemaFrom(endpoint, mutations);
return pullComplete(endpoint, info, true); /// why?
}
}
{code}
I am not completely sure why are we pulling again here. I would rewrite the
whole solution in a such way that this Callable just does one thing on a
successful response (merging of a schema) and the actual "retry" would be
handled from outside. The reader has to make quite a mental exercise to
visualise that this callback might actually call another callback in it until
some "version" is completed etc ... At least for me, it was quite tedious to
track.
I understand the motivation behind that but callback should just do one task
and thats it, it shouldnt be responsible for potentially scheduling another
callback recursively until some conditions on some signals or what have you are
met ... just my 2 cents.
There is also this:
{code:java}
synchronized Future reportEndpointVersion(InetAddress endpoint, UUID
version)
{
UUID current = endpointVersions.get(endpoint);
if (current != null && current.equals(version))
return FINISHED_FUTURE;
VersionInfo info = versionInfo.computeIfAbsent(version, VersionInfo::new);
if (isLocalVersion(version))
info.markReceived();
info.endpoints.add(endpoint);
info.requestQueue.add(endpoint);
endpointVersions.put(endpoint, version);
removeEndpointFromVersion(endpoint, current); / why?
return maybePullSchema(info);
}
{code}
TBH that is quite counterintuitive too, maybe renaming of that method would
help.
The test has failed for me (repeatedly):
{code:java}
java.lang.AssertionError: java.lang.AssertionError:
Expected :2
Actual :0
at org.junit.Assert.fail(Assert.java:92) at
org.junit.Assert.failNotEquals(Assert.java:689) at
org.junit.Assert.assertEquals(Assert.java:127) at
org.junit.Assert.assertEquals(Assert.java:514) at
org.junit.Assert.assertEquals(Assert.java:498) at
org.apache.cassandra.service.MigrationCoordinatorTest.testWeKeepSendingRequests(MigrationCoordinatorTest.java:278)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:44)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:180) at
org.junit.runners.ParentRunner.access$000(ParentRunner.java:41) at
org.junit.runners.ParentRunner$1.evaluate(ParentRunner.java:173) at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
at org.junit.runners.ParentRunner.run(ParentRunner.java:220) at
org.junit.runner.JUnitCore.run(JUnitCore.java:159) at
com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
at
com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:33)
at
com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:230)
at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:58)
{code}
> Wait for schema agreement rather than in flight schema requests when
> bootstrapping
> --
>
> Key: CASSANDRA-15158
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15158
> Project: Cassandra
> Issue Type: Bug
> Components: Cluster/Gossip, Cluster/Schema
>Reporter: Vincent White
>Assignee: Ben Bromhead
>Priority: Normal
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Currently when a node is bootstrapping we use a set
[jira] [Comment Edited] (CASSANDRA-15925) Jenkins pipeline can copy wrong test report artefacts from stage builds
[ https://issues.apache.org/jira/browse/CASSANDRA-15925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17165594#comment-17165594 ] Michael Semb Wever edited comment on CASSANDRA-15925 at 7/27/20, 10:38 AM: --- bq. org.apache.cassandra.transport.frame.checksum.ChecksummingTransformerTest.corruptionCausesFailure is missing from the top #220 failures report and timestamps and out from testCapNoWarnExpirationOverflowPolicy failure doesn't match that of the matching #220 top failures report The *test output* copied in the pipeline are actually from the test-cdc stage job. This is evident [here|https://ci-cassandra.apache.org/job/Cassandra-devbranch/220/testReport/org.apache.cassandra.cql3.validation.operations/TTLTest/testCapNoWarnExpirationOverflowPolicy/] compared to [here|https://ci-cassandra.apache.org/view/patches/job/Cassandra-devbranch-test-cdc/146/testReport/org.apache.cassandra.cql3.validation.operations/TTLTest/testCapNoWarnExpirationOverflowPolicyDefaultTTL_cdc/] The test names are also not getting passed through and honoured correctly… Nor are tests always present, for example [here|https://ci-cassandra.apache.org/job/Cassandra-devbranch/220/testReport/junit/org.apache.cassandra.transport.frame.checksum/ChecksummingTransformerTest/]. was (Author: michaelsembwever): bq. org.apache.cassandra.transport.frame.checksum.ChecksummingTransformerTest.corruptionCausesFailure is missing from the top #220 failures report and timestamps and out from testCapNoWarnExpirationOverflowPolicy failure doesn't match that of the matching #220 top failures report The *test output* copied in the pipeline are actually from the test-cdc stage job. This is evident [here|https://ci-cassandra.apache.org/job/Cassandra-devbranch/220/testReport/org.apache.cassandra.cql3.validation.operations/TTLTest/testCapNoWarnExpirationOverflowPolicy/] compared to [here|https://ci-cassandra.apache.org/view/patches/job/Cassandra-devbranch-test-cdc/146/testReport/org.apache.cassandra.cql3.validation.operations/TTLTest/testCapNoWarnExpirationOverflowPolicyDefaultTTL_cdc/] The test names are also not getting passed through and honoured correctly… > Jenkins pipeline can copy wrong test report artefacts from stage builds > --- > > Key: CASSANDRA-15925 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15925 > Project: Cassandra > Issue Type: Bug > Components: CI >Reporter: Michael Semb Wever >Assignee: Michael Semb Wever >Priority: Normal > Fix For: 4.0-rc > > > Spotted in > https://ci-cassandra.apache.org/view/patches/job/Cassandra-devbranch/196/console > Looks like copyArtifact will need to be specific to a build. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-15925) Jenkins pipeline can copy wrong test report artefacts from stage builds
[ https://issues.apache.org/jira/browse/CASSANDRA-15925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17165594#comment-17165594 ] Michael Semb Wever edited comment on CASSANDRA-15925 at 7/27/20, 10:36 AM: --- bq. org.apache.cassandra.transport.frame.checksum.ChecksummingTransformerTest.corruptionCausesFailure is missing from the top #220 failures report and timestamps and out from testCapNoWarnExpirationOverflowPolicy failure doesn't match that of the matching #220 top failures report The *test output* copied in the pipeline are actually from the test-cdc stage job. This is evident [here|https://ci-cassandra.apache.org/job/Cassandra-devbranch/220/testReport/org.apache.cassandra.cql3.validation.operations/TTLTest/testCapNoWarnExpirationOverflowPolicy/] compared to [here|https://ci-cassandra.apache.org/view/patches/job/Cassandra-devbranch-test-cdc/146/testReport/org.apache.cassandra.cql3.validation.operations/TTLTest/testCapNoWarnExpirationOverflowPolicyDefaultTTL_cdc/] The test names are also not getting passed through and honoured correctly… was (Author: michaelsembwever): bq. org.apache.cassandra.transport.frame.checksum.ChecksummingTransformerTest.corruptionCausesFailure is missing from the top #220 failures report and timestamps and out from testCapNoWarnExpirationOverflowPolicy failure doesn't match that of the matching #220 top failures report Results copied in the pipeline are actually from the test-cdc stage job. This is evident [here|https://ci-cassandra.apache.org/job/Cassandra-devbranch/220/testReport/org.apache.cassandra.cql3.validation.operations/TTLTest/testCapNoWarnExpirationOverflowPolicy/]. This is a bit outside of the scope of this ticket, but I will investigate why test results between jobs are getting clobbered like this when the dtests are not… > Jenkins pipeline can copy wrong test report artefacts from stage builds > --- > > Key: CASSANDRA-15925 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15925 > Project: Cassandra > Issue Type: Bug > Components: CI >Reporter: Michael Semb Wever >Assignee: Michael Semb Wever >Priority: Normal > Fix For: 4.0-rc > > > Spotted in > https://ci-cassandra.apache.org/view/patches/job/Cassandra-devbranch/196/console > Looks like copyArtifact will need to be specific to a build. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15925) Jenkins pipeline can copy wrong test report artefacts from stage builds
[ https://issues.apache.org/jira/browse/CASSANDRA-15925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17165594#comment-17165594 ] Michael Semb Wever commented on CASSANDRA-15925: bq. org.apache.cassandra.transport.frame.checksum.ChecksummingTransformerTest.corruptionCausesFailure is missing from the top #220 failures report and timestamps and out from testCapNoWarnExpirationOverflowPolicy failure doesn't match that of the matching #220 top failures report Results copied in the pipeline are actually from the test-cdc stage job. This is evident [here|https://ci-cassandra.apache.org/job/Cassandra-devbranch/220/testReport/org.apache.cassandra.cql3.validation.operations/TTLTest/testCapNoWarnExpirationOverflowPolicy/]. This is a bit outside of the scope of this ticket, but I will investigate why test results between jobs are getting clobbered like this when the dtests are not… > Jenkins pipeline can copy wrong test report artefacts from stage builds > --- > > Key: CASSANDRA-15925 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15925 > Project: Cassandra > Issue Type: Bug > Components: CI >Reporter: Michael Semb Wever >Assignee: Michael Semb Wever >Priority: Normal > Fix For: 4.0-rc > > > Spotted in > https://ci-cassandra.apache.org/view/patches/job/Cassandra-devbranch/196/console > Looks like copyArtifact will need to be specific to a build. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15925) Jenkins pipeline can copy wrong test report artefacts from stage builds
[ https://issues.apache.org/jira/browse/CASSANDRA-15925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Berenguer Blasi updated CASSANDRA-15925: Reviewers: Berenguer Blasi > Jenkins pipeline can copy wrong test report artefacts from stage builds > --- > > Key: CASSANDRA-15925 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15925 > Project: Cassandra > Issue Type: Bug > Components: CI >Reporter: Michael Semb Wever >Assignee: Michael Semb Wever >Priority: Normal > Fix For: 4.0-rc > > > Spotted in > https://ci-cassandra.apache.org/view/patches/job/Cassandra-devbranch/196/console > Looks like copyArtifact will need to be specific to a build. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-15925) Jenkins pipeline can copy wrong test report artefacts from stage builds
[ https://issues.apache.org/jira/browse/CASSANDRA-15925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17165590#comment-17165590 ] Berenguer Blasi edited comment on CASSANDRA-15925 at 7/27/20, 10:10 AM: [~mck] I was looking into this and saw some problems. Let me know if I am holding the wrong end of the stick though: - Take run [#220|https://ci-cassandra.apache.org/job/Cassandra-devbranch/220/] reporting 3 failures - On it's console out you can see it runs [Cassandra-devbranch-test #198|https://ci-cassandra.apache.org/job/Cassandra-devbranch/220/console] with [2 failures|https://ci-cassandra.apache.org/job/Cassandra-devbranch-test/198/] - org.apache.cassandra.transport.frame.checksum.ChecksummingTransformerTest.corruptionCausesFailure is missing from the top #220 failures report and timestamps and [out|https://ci-cassandra.apache.org/job/Cassandra-devbranch-test/198/testReport/junit/org.apache.cassandra.cql3.validation.operations/TTLTest/testCapNoWarnExpirationOverflowPolicy/] from testCapNoWarnExpirationOverflowPolicy failure doesn't match that of the matching #220 top failures [report|https://ci-cassandra.apache.org/job/Cassandra-devbranch/220/testReport/junit/org.apache.cassandra.cql3.validation.operations/TTLTest/testCapNoWarnExpirationOverflowPolicy/] If out, timestamps and failing tests list don't match I'd bet there is some jenkins funnies going on somewhere? was (Author: bereng): [~mck] I was looking into this and saw some problems. Let me know if I am holding the wrong end of the stick though: - Take run [#220|https://ci-cassandra.apache.org/job/Cassandra-devbranch/220/] reporting 3 failures - On it's console out you can see it runs [Cassandra-devbranch-test #198|https://ci-cassandra.apache.org/job/Cassandra-devbranch/220/console] with [2 failures|https://ci-cassandra.apache.org/job/Cassandra-devbranch-test/198/] - org.apache.cassandra.transport.frame.checksum.ChecksummingTransformerTest.corruptionCausesFailure is missing from the top #220 failures repot and timestamps and [out|https://ci-cassandra.apache.org/job/Cassandra-devbranch-test/198/testReport/junit/org.apache.cassandra.cql3.validation.operations/TTLTest/testCapNoWarnExpirationOverflowPolicy/] from that failure doesn't match that of the matching #220 top failures [report|https://ci-cassandra.apache.org/job/Cassandra-devbranch/220/testReport/junit/org.apache.cassandra.cql3.validation.operations/TTLTest/testCapNoWarnExpirationOverflowPolicy/] If out, timestamps and failing tests list don't match I'd bet there is some jenkins funnies going on somewhere? > Jenkins pipeline can copy wrong test report artefacts from stage builds > --- > > Key: CASSANDRA-15925 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15925 > Project: Cassandra > Issue Type: Bug > Components: CI >Reporter: Michael Semb Wever >Assignee: Michael Semb Wever >Priority: Normal > Fix For: 4.0-rc > > > Spotted in > https://ci-cassandra.apache.org/view/patches/job/Cassandra-devbranch/196/console > Looks like copyArtifact will need to be specific to a build. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15925) Jenkins pipeline can copy wrong test report artefacts from stage builds
[ https://issues.apache.org/jira/browse/CASSANDRA-15925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17165590#comment-17165590 ] Berenguer Blasi commented on CASSANDRA-15925: - [~mck] I was looking into this and saw some problems. Let me know if I am holding the wrong end of the stick though: - Take run [#220|https://ci-cassandra.apache.org/job/Cassandra-devbranch/220/] reporting 3 failures - On it's console out you can see it runs [Cassandra-devbranch-test #198|https://ci-cassandra.apache.org/job/Cassandra-devbranch/220/console] with [2 failures|https://ci-cassandra.apache.org/job/Cassandra-devbranch-test/198/] - org.apache.cassandra.transport.frame.checksum.ChecksummingTransformerTest.corruptionCausesFailure is missing from the top #220 failures repot and timestamps and [out|https://ci-cassandra.apache.org/job/Cassandra-devbranch-test/198/testReport/junit/org.apache.cassandra.cql3.validation.operations/TTLTest/testCapNoWarnExpirationOverflowPolicy/] from that failure doesn't match that of the matching #220 top failures [report|https://ci-cassandra.apache.org/job/Cassandra-devbranch/220/testReport/junit/org.apache.cassandra.cql3.validation.operations/TTLTest/testCapNoWarnExpirationOverflowPolicy/] If out, timestamps and failing tests list don't match I'd bet there is some jenkins funnies going on somewhere? > Jenkins pipeline can copy wrong test report artefacts from stage builds > --- > > Key: CASSANDRA-15925 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15925 > Project: Cassandra > Issue Type: Bug > Components: CI >Reporter: Michael Semb Wever >Assignee: Michael Semb Wever >Priority: Normal > Fix For: 4.0-rc > > > Spotted in > https://ci-cassandra.apache.org/view/patches/job/Cassandra-devbranch/196/console > Looks like copyArtifact will need to be specific to a build. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15983) DOC - Fix incorrect CREATE SCHEMA command in README.asc
[
https://issues.apache.org/jira/browse/CASSANDRA-15983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Semb Wever updated CASSANDRA-15983:
---
Fix Version/s: 3.11.x
3.0.x
2.2.x
2.1.x
4.0
> DOC - Fix incorrect CREATE SCHEMA command in README.asc
>
>
> Key: CASSANDRA-15983
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15983
> Project: Cassandra
> Issue Type: Task
> Components: Documentation/Website
>Reporter: Erick Ramirez
>Assignee: Erick Ramirez
>Priority: Normal
> Fix For: 4.0, 2.1.x, 2.2.x, 3.0.x, 3.11.x
>
> Attachments: CASSANDRA-15983.txt
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> It looks like no one has reviewed the contents of
> [{{README.asc}}|https://github.com/apache/cassandra/blob/cassandra-3.11.7/README.asc].
> Instead of the {{CREATE KEYSPACE}} command, the example CQL reads:
> {noformat}
> cqlsh> CREATE SCHEMA schema1
>WITH replication = { 'class' : 'SimpleStrategy', 'replication_factor'
> : 1 };
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15925) Jenkins pipeline can copy wrong test report artefacts from stage builds
[ https://issues.apache.org/jira/browse/CASSANDRA-15925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Semb Wever updated CASSANDRA-15925: --- Test and Documentation Plan: local jenkins testing Status: Patch Available (was: In Progress) > Jenkins pipeline can copy wrong test report artefacts from stage builds > --- > > Key: CASSANDRA-15925 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15925 > Project: Cassandra > Issue Type: Bug > Components: CI >Reporter: Michael Semb Wever >Assignee: Michael Semb Wever >Priority: Normal > Fix For: 4.0-rc > > > Spotted in > https://ci-cassandra.apache.org/view/patches/job/Cassandra-devbranch/196/console > Looks like copyArtifact will need to be specific to a build. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15981) jvm-dtests crash on java 11
[
https://issues.apache.org/jira/browse/CASSANDRA-15981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17165522#comment-17165522
]
Alex Petrov commented on CASSANDRA-15981:
-
Does this mean we keep allow some runnables to continue running already after
class loader is closed?
> jvm-dtests crash on java 11
> ---
>
> Key: CASSANDRA-15981
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15981
> Project: Cassandra
> Issue Type: Bug
> Components: Build
>Reporter: David Capwell
>Priority: Normal
> Fix For: 4.0-beta
>
>
> There is a race condition bug with CMS and class unloading which cause the
> JVM to crash. Since jvm-dtests rely on class loaders and unloading, this
> causes sporadic JVM crashes that look like the following in CI logs
> {code}
> junit.framework.AssertionFailedError: Forked Java VM exited abnormally.
> Please note the time in the report does not reflect the time until the VM
> exit.
> at jdk.internal.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.util.Vector.forEach(Vector.java:1387)
> at jdk.internal.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at jdk.internal.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.util.Vector.forEach(Vector.java:1387)
> at jdk.internal.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.Thread.run(Thread.java:834)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15583) 4.0 quality testing: Tooling, Bundled and First Party
[ https://issues.apache.org/jira/browse/CASSANDRA-15583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17165521#comment-17165521 ] Berenguer Blasi commented on CASSANDRA-15583: - Apologies :) UT=Unit Tests UX='User interface' tests such as making sure mandatory params are indeed mandatory, 'help' produces an actual help, providing wrong params renders a human readable error, etc > 4.0 quality testing: Tooling, Bundled and First Party > - > > Key: CASSANDRA-15583 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15583 > Project: Cassandra > Issue Type: Task > Components: Test/dtest, Test/unit >Reporter: Josh McKenzie >Assignee: Berenguer Blasi >Priority: Normal > Fix For: 4.0-beta > > > Reference [doc from > NGCC|https://docs.google.com/document/d/1uhUOp7wpE9ZXNDgxoCZHejHt5SO4Qw1dArZqqsJccyQ/edit#] > for context. > *Shepherd: Sam Tunnicliffe* > Test plans should cover bundled first-party tooling and CLIs such as > nodetool, cqlsh, and new tools supporting full query and audit logging > (CASSANDRA-13983, CASSANDRA-12151). > ||Tool||UX test||UT coverage||dtest coverage||Comments|| > |Nodetool|(x)|(x)|(!)|By no means all the sub commands are tested. Dtest also > test nodetool as a side effect| > |Cqlsh|(x)|(x)|(!)| | > |Cassandra-stress|(x)|(x)|(x)| Some UT| > |debug-cql|(x)|(x)|(x)| | > |fqltool|(x)|(/)|(!)| | > |auditlogviewer|(x)|(!)|(!)| | > |*Sstable utilities*| | | | | > |sstabledump|(x)|(x)|(!)| | > |sstableexpiredblockers|(x)|(x)|(!)| | > |sstablelevelreset|(x)|(x)|(!)| | > |sstableloader|(x)|(x)|(!)| | > |sstablemetadata|(x)|(x)|(x)|Ran in dtests, no dedicated test| > |sstableofflinerelevel|(x)|(x)|(!)| | > |sstablerepairedset|(x)|(x)|(x)|Ran in dtests, no dedicated test| > |sstablescrub|(x)|(x)|(!)| | > |sstablesplit|(x)|(x)|(!)| | > |sstableupgrade|(x)|(x)|(!)| | > |sstableutil|(x)|(x)|(!)| | > |sstableverify|(x)|(x)|(!)| | -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra-builds] branch master updated: In Jenkins always use `git clone --depth 1 --single-branch …`
This is an automated email from the ASF dual-hosted git repository.
mck pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/cassandra-builds.git
The following commit(s) were added to refs/heads/master by this push:
new eb58d1b In Jenkins always use `git clone --depth 1 --single-branch …`
eb58d1b is described below
commit eb58d1b98c03c1c4344703d0d258b9211b1ddf29
Author: mck
AuthorDate: Sat Jul 25 18:39:51 2020 +0200
In Jenkins always use `git clone --depth 1 --single-branch …`
---
docker/jenkins/dtest.sh | 8
docker/jenkins/jenkinscommand.sh | 4 ++--
jenkins-dsl/cassandra_job_dsl_seed.groovy | 14 +++---
jenkins-dsl/cassandra_pipeline.groovy | 6 +++---
4 files changed, 16 insertions(+), 16 deletions(-)
diff --git a/docker/jenkins/dtest.sh b/docker/jenkins/dtest.sh
index 20915b6..47ec642 100644
--- a/docker/jenkins/dtest.sh
+++ b/docker/jenkins/dtest.sh
@@ -4,9 +4,9 @@ export LANG=en_US.UTF-8
export LC_CTYPE=en_US.UTF-8
export PYTHONIOENCODING=utf-8
export PYTHONUNBUFFERED=true
-echo "dtest.sh: running: git clone --depth=1 --branch=$BRANCH
https://github.com/$REPO/cassandra.git;
-git clone --depth=1 --branch=$BRANCH https://github.com/$REPO/cassandra.git
+echo "dtest.sh: running: git clone --depth 1 --single-branch --branch=$BRANCH
https://github.com/$REPO/cassandra.git;
+git clone --depth 1 --single-branch --branch=$BRANCH
https://github.com/$REPO/cassandra.git
cd cassandra
-echo git clone --branch=$DTEST_BRANCH $DTEST_REPO
-git clone --branch=$DTEST_BRANCH $DTEST_REPO
+echo git clone --depth 1 --single-branch --branch=$DTEST_BRANCH $DTEST_REPO
+git clone --depth 1 --single-branch --branch=$DTEST_BRANCH $DTEST_REPO
../cassandra-builds/build-scripts/cassandra-dtest-pytest.sh $1
diff --git a/docker/jenkins/jenkinscommand.sh b/docker/jenkins/jenkinscommand.sh
index ea6d764..bc2b835 100644
--- a/docker/jenkins/jenkinscommand.sh
+++ b/docker/jenkins/jenkinscommand.sh
@@ -14,8 +14,8 @@ DTEST_REPO=$3
DTEST_BRANCH=$4
EOF
-echo "jenkinscommand.sh: running: git clone --branch $BUILDSBRANCH
$BUILDSREPO; sh ./cassandra-builds/docker/jenkins/dtest.sh $TARGET"
-ID=$(docker run --env-file env.list -dt $DOCKER_IMAGE dumb-init bash -ilc "git
clone --branch $BUILDSBRANCH $BUILDSREPO; sh
./cassandra-builds/docker/jenkins/dtest.sh $TARGET")
+echo "jenkinscommand.sh: running: git clone --single-branch --depth 1 --branch
$BUILDSBRANCH $BUILDSREPO; sh ./cassandra-builds/docker/jenkins/dtest.sh
$TARGET"
+ID=$(docker run --env-file env.list -dt $DOCKER_IMAGE dumb-init bash -ilc "git
clone --single-branch --depth 1 --branch $BUILDSBRANCH $BUILDSREPO; sh
./cassandra-builds/docker/jenkins/dtest.sh $TARGET")
# use docker attach instead of docker wait to get output
docker attach --no-stdin $ID
diff --git a/jenkins-dsl/cassandra_job_dsl_seed.groovy
b/jenkins-dsl/cassandra_job_dsl_seed.groovy
index 254e5e0..39bc2b1 100644
--- a/jenkins-dsl/cassandra_job_dsl_seed.groovy
+++ b/jenkins-dsl/cassandra_job_dsl_seed.groovy
@@ -105,7 +105,7 @@ matrixJob('Cassandra-template-artifacts') {
}
steps {
buildDescription('', buildDescStr)
-shell("git clean -xdff ; git clone -b ${buildsBranch} ${buildsRepo}")
+shell("git clean -xdff ; git clone --depth 1 --single-branch -b
${buildsBranch} ${buildsRepo}")
}
publishers {
archiveArtifacts('build/apache-cassandra-*.tar.gz,
build/apache-cassandra-*.jar, build/apache-cassandra-*.pom,
build/cassandra*.deb, build/cassandra*.rpm,
build/**/eclipse_compiler_checks.txt')
@@ -186,7 +186,7 @@ job('Cassandra-template-test') {
}
steps {
buildDescription('', buildDescStr)
-shell("git clean -xdff ; git clone -b ${buildsBranch} ${buildsRepo}")
+shell("git clean -xdff ; git clone --depth 1 --single-branch -b
${buildsBranch} ${buildsRepo}")
}
publishers {
archiveArtifacts {
@@ -249,7 +249,7 @@ job('Cassandra-template-dtest') {
}
steps {
buildDescription('', buildDescStr)
-shell("git clean -xdff ; git clone -b ${buildsBranch} ${buildsRepo} ;
git clone ${dtestRepo}")
+shell("git clean -xdff ; git clone --depth 1 --single-branch -b
${buildsBranch} ${buildsRepo} ; git clone --depth 1 --single-branch
${dtestRepo}")
}
publishers {
archiveArtifacts {
@@ -319,7 +319,7 @@ matrixJob('Cassandra-template-cqlsh-tests') {
}
steps {
buildDescription('', buildDescStr)
-shell("git clean -xdff ; git clone ${dtestRepo}")
+shell("git clean -xdff ; git clone --depth 1 --single-branch
${dtestRepo}")
}
publishers {
archiveArtifacts {
@@ -536,7 +536,7 @@ matrixJob('Cassandra-devbranch-artifacts') {
}
steps {
buildDescription('', buildDescStr)
-shell("git clean -xdff ; git clone -b ${buildsBranch} ${buildsRepo}")
+shell("git clean -xdff ; git clone --depth 1
