[jira] [Comment Edited] (CASSANDRA-15234) Standardise config and JVM parameters
[
https://issues.apache.org/jira/browse/CASSANDRA-15234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17125553#comment-17125553
]
Ekaterina Dimitrova edited comment on CASSANDRA-15234 at 6/4/20, 5:47 AM:
--
[ [Part 2
|https://github.com/ekaterinadimitrova2/cassandra/tree/CASSANDRA-15234-3] ] [
[DTests
|https://github.com/ekaterinadimitrova2/cassandra-dtest/tree/CASSANDRA-15234-3]
] [ [CCM |https://github.com/ekaterinadimitrova2/ccm/tree/CASSANDRA-15234-3] ]
[ [CI run
|https://app.circleci.com/pipelines/github/ekaterinadimitrova2/cassandra/187/workflows/615cdebf-1d49-47ab-93a4-efad25a247b1]
]
- Unit tests updated
- Getters and Setters names in the DatabaseDescriptor were updated to follow
the new names of the parameters updated
- The conversion of some of the parameters values which is happening in their
getters is kept as it is needed in runtime. Some parameters are used with
different units in the different areas of the codebase. So I left this part
untouched.
- New class created, ParseAndConvertUnits - it pulls all the functionality
needed to cover the addition of units suffixes to the Memory, Duration, and
Rate parameters in the cassandra.yaml
- the cassandra.yaml file in this branch contains currently the new parameters
with new names and unit suffixes but the arrangement of the sections is the old
one as this was causing me issues every time I rebase/commit. If we confirm the
format, this will be the last step to switch to the new format
- Added some additional parsing and supporting conversion functions for the
double duration parameters.
- The 2 failing cqlsh tests were fixed locally and tested. It was formatting
issue.
-1 unit test failed - it doesn't look a related issue but will check the config
loaded to ensure that tomorrow
- 8 failing dtest, it doesn't look related but I am gonna compare tomorrow
morning
- Current version of the code needs a final rebase/merge of the last 6 commits
on trunk. Will do tomorrow after I recheck the CI and current version
- Have to return the default requirements.txt and config.yml files. The current
versions were required only for testing purposes with CircleCI
*WARNING:* we should be careful with the dtests and ccm. One missed check for
old/new parameter and default value of the respective parameter from Config
will be loaded. This might not lead to failing test and thus silently cover the
miss.
*NOTE:* Custom Types, as they were suggested, don't really work with SnakeYAML,
unfortunately. SnakeYAML supports nothing more than mapping between the yaml
and the Config class.
* _So something like that: Memory max_value = new Memory("5ms") wouldn't work.
If there is no value in the yaml, the config default max_value will be parsed
properly but if there is value assigned to max_value at the yaml, it won't be
assigned properly in Config. Classes are used for nested parameters. Also,
currently there won't be a big benefit of the custom types as the different
parameters validations in the DatabaseDescriptor are taking different
approaches, there is no general framework/approach per group of parameters
(Memory, Duration, Rate)._
_-_
The last part was updated as per the recommendations. CI ran successfully. Need
to rebase and recheck nothing that might need the attention of this patch has
changed in the meanwhile. Three weeks passed since I took care of this one.
[ [Part3
|https://github.com/ekaterinadimitrova2/cassandra/tree/CASSANDRA-15234-5] ] [
[CI JAVA8
|https://app.circleci.com/pipelines/github/ekaterinadimitrova2/cassandra/146/workflows/fae5da98-acf5-4824-bf61-3ffab3d86cd6]
] [ [CI Java11
|https://app.circleci.com/pipelines/github/ekaterinadimitrova2/cassandra/146/workflows/21b088c4-8984-4b20-8302-2ff8adaf55f7]
]
was (Author: e.dimitrova):
[ [Part 2
|https://github.com/ekaterinadimitrova2/cassandra/tree/CASSANDRA-15234-3] ] [
[DTests
|https://github.com/ekaterinadimitrova2/cassandra-dtest/tree/CASSANDRA-15234-3]
] [ [CCM |https://github.com/ekaterinadimitrova2/ccm/tree/CASSANDRA-15234-3] ]
[ [CI run
|https://app.circleci.com/pipelines/github/ekaterinadimitrova2/cassandra/187/workflows/615cdebf-1d49-47ab-93a4-efad25a247b1]
]
- Unit tests updated
- Getters and Setters names in the DatabaseDescriptor were updated to follow
the new names of the parameters updated
- The conversion of some of the parameters values which is happening in their
getters is kept as it is needed in runtime. Some parameters are used with
different units in the different areas of the codebase. So I left this part
untouched.
- New class created, ParseAndConvertUnits - it pulls all the functionality
needed to cover the addition of units suffixes to the Memory, Duration, and
Rate parameters in the cassandra.yaml
- the cassandra.yaml file in this branch contains currently the new parameters
with
[jira] [Comment Edited] (CASSANDRA-15234) Standardise config and JVM parameters
[
https://issues.apache.org/jira/browse/CASSANDRA-15234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17125553#comment-17125553
]
Ekaterina Dimitrova edited comment on CASSANDRA-15234 at 6/4/20, 5:43 AM:
--
[ [Part 2
|https://github.com/ekaterinadimitrova2/cassandra/tree/CASSANDRA-15234-3] ] [
[DTests
|https://github.com/ekaterinadimitrova2/cassandra-dtest/tree/CASSANDRA-15234-3]
] [ [CCM |https://github.com/ekaterinadimitrova2/ccm/tree/CASSANDRA-15234-3] ]
[ [CI run
|https://app.circleci.com/pipelines/github/ekaterinadimitrova2/cassandra/187/workflows/615cdebf-1d49-47ab-93a4-efad25a247b1]
]
- Unit tests updated
- Getters and Setters names in the DatabaseDescriptor were updated to follow
the new names of the parameters updated
- The conversion of some of the parameters values which is happening in their
getters is kept as it is needed in runtime. Some parameters are used with
different units in the different areas of the codebase. So I left this part
untouched.
- New class created, ParseAndConvertUnits - it pulls all the functionality
needed to cover the addition of units suffixes to the Memory, Duration, and
Rate parameters in the cassandra.yaml
- the cassandra.yaml file in this branch contains currently the new parameters
with new names and unit suffixes but the arrangement of the sections is the old
one as this was causing me issues every time I rebase/commit. If we confirm the
format, this will be the last step to switch to the new format
- Added some additional parsing and supporting conversion functions for the
double duration parameters.
- The 2 failing cqlsh tests were fixed locally and tested. It was formatting
issue.
-1 unit test failed - it doesn't look a related issue but will check the config
loaded to ensure that tomorrow
- 8 failing dtest, it doesn't look related but I am gonna compare tomorrow
morning
- Current version of the code needs a final rebase/merge of the last 6 commits
on trunk. Will do tomorrow after I recheck the CI and current version
- Have to return the default requirements.txt and config.yml files. The current
versions were required only for testing purposes with CircleCI
*WARNING:* we should be careful with the dtests and ccm. One missed check for
old/new parameter and default value of the respective parameter from Config
will be loaded. This might not lead to failing test and thus silently cover the
miss.
*NOTE:* Custom Types, as they were suggested, don't really work with SnakeYAML,
unfortunately. SnakeYAML supports nothing more than mapping between the yaml
and the Config class.
* _So something like that: Memory max_value = new Memory("5ms") wouldn't work.
If there is no value in the yaml, the config default max_value will be parsed
properly but if there is value assigned to max_value at the yaml, it won't be
assigned properly in Config. Classes are used for nested parameters. Also,
currently there won't be a big benefit of the custom types as the different
parameters validations in the db descriptor are taking different approaches,
there is no general framework/approach._
_-_
The last part was updated as per the recommendations. CI ran successfully. Need
to rebase and recheck nothing that might need the attention of this patch has
changed in the meanwhile. Three weeks passed since I took care of this one.
[ [Part3
|https://github.com/ekaterinadimitrova2/cassandra/tree/CASSANDRA-15234-5] ] [
[CI JAVA8
|https://app.circleci.com/pipelines/github/ekaterinadimitrova2/cassandra/146/workflows/fae5da98-acf5-4824-bf61-3ffab3d86cd6]
] [ [CI Java11
|https://app.circleci.com/pipelines/github/ekaterinadimitrova2/cassandra/146/workflows/21b088c4-8984-4b20-8302-2ff8adaf55f7]
]
was (Author: e.dimitrova):
[ [Part 2
|https://github.com/ekaterinadimitrova2/cassandra/tree/CASSANDRA-15234-3] ] [
[DTests
|https://github.com/ekaterinadimitrova2/cassandra-dtest/tree/CASSANDRA-15234-3]
] [ [CCM |https://github.com/ekaterinadimitrova2/ccm/tree/CASSANDRA-15234-3] ]
[ [CI run
|https://app.circleci.com/pipelines/github/ekaterinadimitrova2/cassandra/187/workflows/615cdebf-1d49-47ab-93a4-efad25a247b1]
]
- Unit tests updated
- Getters and Setters names in the DatabaseDescriptor were updated to follow
the new names of the parameters updated
- The conversion of some of the parameters values which is happening in their
getters is kept as it is needed in runtime. Some parameters are used with
different units in the different areas of the codebase. So I left this part
untouched.
- New class created, ParseAndConvertUnits - it pulls all the functionality
needed to cover the addition of units suffixes to the Memory, Duration, and
Rate parameters in the cassandra.yaml
- the cassandra.yaml file in this branch contains currently the new parameters
with new names and unit suffixes but the arrangement of the
[jira] [Comment Edited] (CASSANDRA-15234) Standardise config and JVM parameters
[
https://issues.apache.org/jira/browse/CASSANDRA-15234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17125553#comment-17125553
]
Ekaterina Dimitrova edited comment on CASSANDRA-15234 at 6/4/20, 5:42 AM:
--
[ [Part 2
|https://github.com/ekaterinadimitrova2/cassandra/tree/CASSANDRA-15234-3] ] [
[DTests
|https://github.com/ekaterinadimitrova2/cassandra-dtest/tree/CASSANDRA-15234-3]
] [ [CCM |https://github.com/ekaterinadimitrova2/ccm/tree/CASSANDRA-15234-3] ]
[ [CI run
|https://app.circleci.com/pipelines/github/ekaterinadimitrova2/cassandra/187/workflows/615cdebf-1d49-47ab-93a4-efad25a247b1]
]
- Unit tests updated
- Getters and Setters names in the DatabaseDescriptor were updated to follow
the new names of the parameters updated
- The conversion of some of the parameters values which is happening in their
getters is kept as it is needed in runtime. Some parameters are used with
different units in the different areas of the codebase. So I left this part
untouched.
- New class created, ParseAndConvertUnits - it pulls all the functionality
needed to cover the addition of units suffixes to the Memory, Duration, and
Rate parameters in the cassandra.yaml
- the cassandra.yaml file in this branch contains currently the new parameters
with new names and unit suffixes but the arrangement of the sections is the old
one as this was causing me issues every time I rebase/commit. If we confirm the
format, this will be the last step to switch to the new format
- Added some additional parsing and supporting conversion functions for the
double duration parameters.
- The 2 failing cqlsh tests were fixed locally and tested. It was formatting
issue.
-1 unit test failed - it doesn't look a related issue but will check the config
loaded to ensure that tomorrow
- 8 failing dtest, it doesn't look related but I am gonna compare tomorrow
morning
- Current version of the code needs a final rebase/merge of the last 6 commits
on trunk. Will do tomorrow after I recheck the CI and current version
- Have to return the default requirements.txt and config.yml files. The current
versions were required only for testing purposes with CircleCI
*WARNING:* we should be careful with the dtests and ccm. One missed check for
old/new parameter and default value of the respective parameter from Config
will be loaded. This might not lead to failing test and thus silently cover the
miss.
*NOTE:* Custom Types, as they were suggested, don't really work with SnakeYAML,
unfortunately. SnakeYAML supports nothing more than mapping between the yaml
and the Config class.
* _So something like that: Memory max_value = new Memory("5ms") wouldn't work.
If there is no value in the yaml, the config default max_value will be parsed
properly but if there is value assigned to max_value at the yaml, it won't be
assigned properly in Config. Classes are used for nested parameters. Also,
currently there won't be a big benefit of the custom types as the different
parameters validations in the db descriptor are taking different approaches,
there is no general framework/approach._
_-_
The last part was updated as per the recommendations. CI ran successfully. Need
to rebase and recheck nothing that might need the attention of this patch has
changed in the meanwhile. Three weeks passed since I took care of this one.
[ [Part3
|https://github.com/ekaterinadimitrova2/cassandra/tree/CASSANDRA-15234-5] ] [
[CI JAVA8
|https://app.circleci.com/pipelines/github/ekaterinadimitrova2/cassandra/146/workflows/fae5da98-acf5-4824-bf61-3ffab3d86cd6]
] [ [CI Java11
|https://app.circleci.com/pipelines/github/ekaterinadimitrova2/cassandra/146/workflows/21b088c4-8984-4b20-8302-2ff8adaf55f7]
]
was (Author: e.dimitrova):
[ [Part 2
|https://github.com/ekaterinadimitrova2/cassandra/tree/CASSANDRA-15234-3] ] [
[DTests
|https://github.com/ekaterinadimitrova2/cassandra-dtest/tree/CASSANDRA-15234-3]
] [ [CCM |https://github.com/ekaterinadimitrova2/ccm/tree/CASSANDRA-15234-3] ]
[ [CI run
|https://app.circleci.com/pipelines/github/ekaterinadimitrova2/cassandra/187/workflows/615cdebf-1d49-47ab-93a4-efad25a247b1]
]
- Unit tests updated
- Getters and Setters names in the DatabaseDescriptor were updated to follow
the new names of the parameters updated
- The conversion of some of the parameters values which is happening in their
getters is kept as it is needed in runtime. Some parameters are used with
different units in the different areas of the codebase. So I left this part
untouched.
- New class created, ParseAndConvertUnits - it pulls all the functionality
needed to cover the addition of units suffixes to the Memory, Duration, and
Rate parameters in the cassandra.yaml
- the cassandra.yaml file in this branch contains currently the new parameters
with new names and unit suffixes but the arrangement of the
[jira] [Commented] (CASSANDRA-15234) Standardise config and JVM parameters
[
https://issues.apache.org/jira/browse/CASSANDRA-15234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17125553#comment-17125553
]
Ekaterina Dimitrova commented on CASSANDRA-15234:
-
[ [Part 2
|https://github.com/ekaterinadimitrova2/cassandra/tree/CASSANDRA-15234-3] ] [
[DTests
|https://github.com/ekaterinadimitrova2/cassandra-dtest/tree/CASSANDRA-15234-3]
] [ [CCM |https://github.com/ekaterinadimitrova2/ccm/tree/CASSANDRA-15234-3] ]
[ [CI run
|https://app.circleci.com/pipelines/github/ekaterinadimitrova2/cassandra/187/workflows/615cdebf-1d49-47ab-93a4-efad25a247b1]
]
- Unit tests updated
- Getters and Setters names in the DatabaseDescriptor were updated to follow
the new names of the parameters updated
- The conversion of some of the parameters values which is happening in their
getters is kept as it is needed in runtime. Some parameters are used with
different units in the different areas of the codebase. So I left this part
untouched.
- New class created, ParseAndConvertUnits - it pulls all the functionality
needed to cover the addition of units suffixes to the Memory, Duration, and
Rate parameters in the cassandra.yaml
- the cassandra.yaml file in this branch contains currently the new parameters
with new names and unit suffixes but the arrangement of the sections is the old
one as this was causing me issues every time I rebase/commit. If we confirm the
format, this will be the last step to switch to the new format
- Added some additional parsing and supporting conversion functions for the
double duration parameters.
- The 2 failing cqlsh tests were fixed locally and tested. It was formatting
issue.
-1 unit test failed - it doesn't look a related issue but will check the config
loaded to ensure that tomorrow
- 8 failing dtest, it doesn't look related but I am gonna compare tomorrow
morning
- Current version of the code needs a final rebase/merge of the last 6 commits
on trunk. Will do tomorrow after I recheck the CI and current version
- Have to return the default requirements.txt and config.yml files. The current
versions were required only for testing purposes with CircleCI
*WARNING:* we should be careful with the dtests and ccm. One missed check for
old/new parameter and default value of the respective parameter from Config
will be loaded. This might not lead to failing test and thus silently cover the
miss.
*NOTE:* Custom Types, as they were suggested, don't really work with SnakeYAML,
unfortunately. SnakeYAML supports nothing more than mapping between the yaml
and the Config class.
* _So something like that: Memory max_value = new Memory("5ms") wouldn't work.
If there is no value in the yaml, the config default max_value will be parsed
properly but if there is value assigned to max_value at the yaml, it won't be
assigned properly in Config. Classes are used for nested parameters. Also,
currently there won't be a big benefit of the custom types as the different
parameters validations in the db descriptor are taking different approaches,
there is no general framework/approach._
_-_
The last part was updated as per the recommendations. CI ran successfully. Need
to rebase and recheck nothing that might need the attention of this patch has
changed in the meanwhile. Three weeks passed since I took care of this one.
[ [Part3
|https://github.com/ekaterinadimitrova2/cassandra/tree/CASSANDRA-15234-5] ] [
[CI JAVA8
|https://app.circleci.com/pipelines/github/ekaterinadimitrova2/cassandra/146/workflows/fae5da98-acf5-4824-bf61-3ffab3d86cd6]]
[ [CI Java11
|https://app.circleci.com/pipelines/github/ekaterinadimitrova2/cassandra/146/workflows/21b088c4-8984-4b20-8302-2ff8adaf55f7]
]
> Standardise config and JVM parameters
> -
>
> Key: CASSANDRA-15234
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15234
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Config
>Reporter: Benedict Elliott Smith
>Assignee: Ekaterina Dimitrova
>Priority: Normal
> Fix For: 4.0-alpha
>
>
> We have a bunch of inconsistent names and config patterns in the codebase,
> both from the yams and JVM properties. It would be nice to standardise the
> naming (such as otc_ vs internode_) as well as the provision of values with
> units - while maintaining perpetual backwards compatibility with the old
> parameter names, of course.
> For temporal units, I would propose parsing strings with suffixes of:
> {{code}}
> u|micros(econds?)?
> ms|millis(econds?)?
> s(econds?)?
> m(inutes?)?
> h(ours?)?
> d(ays?)?
> mo(nths?)?
> {{code}}
> For rate units, I would propose parsing any of the standard {{B/s, KiB/s,
> MiB/s, GiB/s, TiB/s}}.
> Perhaps for avoiding ambiguity we could not accept bauds {{bs, Mbps}} or
> powers of 1000 such as
[jira] [Updated] (CASSANDRA-15850) Delay between Gossip settle and CQL port opening during the startup
[
https://issues.apache.org/jira/browse/CASSANDRA-15850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jai Bheemsen Rao Dhanwada updated CASSANDRA-15850:
--
Impacts: (was: None)
> Delay between Gossip settle and CQL port opening during the startup
> ---
>
> Key: CASSANDRA-15850
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15850
> Project: Cassandra
> Issue Type: Bug
>Reporter: Jai Bheemsen Rao Dhanwada
>Priority: Normal
>
> Hello,
> When I am bootstrapping/restarting a Cassandra Node, there is a delay between
> gossip settle and CQL port opening. Can someone please explain me where this
> delay is configured and can this be changed? I don't see any information in
> the logs
> In my case if you see there is a ~3 minutes delay and this increases if I
> increase the #of tables and #of nodes and DC.
> {code:java}
> INFO [main] 2020-05-31 23:51:07,554 Gossiper.java:1692 - Waiting for gossip
> to settle...
> INFO [main] 2020-05-31 23:51:15,555 Gossiper.java:1723 - No gossip backlog;
> proceeding
> INFO [main] 2020-05-31 23:54:06,867 NativeTransportService.java:70 - Netty
> using native Epoll event loop
> INFO [main] 2020-05-31 23:54:06,913 Server.java:155 - Using Netty Version:
> [netty-buffer=netty-buffer-4.0.44.Final.452812a,
> netty-codec=netty-codec-4.0.44.Final.452812a,
> netty-codec-haproxy=netty-codec-haproxy-4.0.44.Final.452812a,
> netty-codec-http=netty-codec-http-4.0.44.Final.452812a,
> netty-codec-socks=netty-codec-socks-4.0.44.Final.452812a,
> netty-common=netty-common-4.0.44.Final.452812a,
> netty-handler=netty-handler-4.0.44.Final.452812a,
> netty-tcnative=netty-tcnative-1.1.33.Fork26.142ecbb,
> netty-transport=netty-transport-4.0.44.Final.452812a,
> netty-transport-native-epoll=netty-transport-native-epoll-4.0.44.Final.452812a,
> netty-transport-rxtx=netty-transport-rxtx-4.0.44.Final.452812a,
> netty-transport-sctp=netty-transport-sctp-4.0.44.Final.452812a,
> netty-transport-udt=netty-transport-udt-4.0.44.Final.452812a]
> INFO [main] 2020-05-31 23:54:06,913 Server.java:156 - Starting listening for
> CQL clients on /x.x.x.x:9042 (encrypted)...
> {code}
> Also during this 3-10 minutes delay, I see
> {noformat}
> nodetool compactionstats
> {noformat}
> command is hung and never respond, until the CQL port is up and running.
> Can someone please help me understand the delay here?
> Cassandra Version: 3.11.3
> The issue can be easily reproducible with around 300 Tables and 100 nodes in
> a cluster.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-15850) Delay between Gossip settle and CQL port opening during the startup
Jai Bheemsen Rao Dhanwada created CASSANDRA-15850:
-
Summary: Delay between Gossip settle and CQL port opening during
the startup
Key: CASSANDRA-15850
URL: https://issues.apache.org/jira/browse/CASSANDRA-15850
Project: Cassandra
Issue Type: Bug
Reporter: Jai Bheemsen Rao Dhanwada
Hello,
When I am bootstrapping/restarting a Cassandra Node, there is a delay between
gossip settle and CQL port opening. Can someone please explain me where this
delay is configured and can this be changed? I don't see any information in the
logs
In my case if you see there is a ~3 minutes delay and this increases if I
increase the #of tables and #of nodes and DC.
{code:java}
INFO [main] 2020-05-31 23:51:07,554 Gossiper.java:1692 - Waiting for gossip to
settle...
INFO [main] 2020-05-31 23:51:15,555 Gossiper.java:1723 - No gossip backlog;
proceeding
INFO [main] 2020-05-31 23:54:06,867 NativeTransportService.java:70 - Netty
using native Epoll event loop
INFO [main] 2020-05-31 23:54:06,913 Server.java:155 - Using Netty Version:
[netty-buffer=netty-buffer-4.0.44.Final.452812a,
netty-codec=netty-codec-4.0.44.Final.452812a,
netty-codec-haproxy=netty-codec-haproxy-4.0.44.Final.452812a,
netty-codec-http=netty-codec-http-4.0.44.Final.452812a,
netty-codec-socks=netty-codec-socks-4.0.44.Final.452812a,
netty-common=netty-common-4.0.44.Final.452812a,
netty-handler=netty-handler-4.0.44.Final.452812a,
netty-tcnative=netty-tcnative-1.1.33.Fork26.142ecbb,
netty-transport=netty-transport-4.0.44.Final.452812a,
netty-transport-native-epoll=netty-transport-native-epoll-4.0.44.Final.452812a,
netty-transport-rxtx=netty-transport-rxtx-4.0.44.Final.452812a,
netty-transport-sctp=netty-transport-sctp-4.0.44.Final.452812a,
netty-transport-udt=netty-transport-udt-4.0.44.Final.452812a]
INFO [main] 2020-05-31 23:54:06,913 Server.java:156 - Starting listening for
CQL clients on /x.x.x.x:9042 (encrypted)...
{code}
Also during this 3-10 minutes delay, I see
{noformat}
nodetool compactionstats
{noformat}
command is hung and never respond, until the CQL port is up and running.
Can someone please help me understand the delay here?
Cassandra Version: 3.11.3
The issue can be easily reproducible with around 300 Tables and 100 nodes in a
cluster.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15733) jvm dtest builder should be provided to the factory and expose state
[ https://issues.apache.org/jira/browse/CASSANDRA-15733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17125393#comment-17125393 ] David Capwell commented on CASSANDRA-15733: --- [~ifesdjeen] poke =) > jvm dtest builder should be provided to the factory and expose state > > > Key: CASSANDRA-15733 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15733 > Project: Cassandra > Issue Type: Improvement > Components: Test/dtest >Reporter: David Capwell >Assignee: David Capwell >Priority: Normal > Labels: pull-request-available > Time Spent: 3h > Remaining Estimate: 0h > > Currently the builder is rather heavy and creates configs plus call the > factory with specific fields only, this isn’t that flexible and makes it > harder to have custom cluster definitions which require additional fields to > be defined. To solve this we should make the builder be sent to the factory > and expose the state so the factory can get all the fields it needs, the > factory should also be in charge of creating the configs -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15299) CASSANDRA-13304 follow-up: improve checksumming and compression in protocol v5-beta
[
https://issues.apache.org/jira/browse/CASSANDRA-15299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17125310#comment-17125310
]
Josh McKenzie commented on CASSANDRA-15299:
---
{quote}There's also still quite a bit of internal stuff to be done,
{quote}
Anything anyone can do to help on this? This is one of our last blockers before
cutting beta.
> CASSANDRA-13304 follow-up: improve checksumming and compression in protocol
> v5-beta
> ---
>
> Key: CASSANDRA-15299
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15299
> Project: Cassandra
> Issue Type: Improvement
> Components: Messaging/Client
>Reporter: Aleksey Yeschenko
>Assignee: Sam Tunnicliffe
>Priority: Normal
> Labels: protocolv5
> Fix For: 4.0-alpha
>
>
> CASSANDRA-13304 made an important improvement to our native protocol: it
> introduced checksumming/CRC32 to request and response bodies. It’s an
> important step forward, but it doesn’t cover the entire stream. In
> particular, the message header is not covered by a checksum or a crc, which
> poses a correctness issue if, for example, {{streamId}} gets corrupted.
> Additionally, we aren’t quite using CRC32 correctly, in two ways:
> 1. We are calculating the CRC32 of the *decompressed* value instead of
> computing the CRC32 on the bytes written on the wire - losing the properties
> of the CRC32. In some cases, due to this sequencing, attempting to decompress
> a corrupt stream can cause a segfault by LZ4.
> 2. When using CRC32, the CRC32 value is written in the incorrect byte order,
> also losing some of the protections.
> See https://users.ece.cmu.edu/~koopman/pubs/KoopmanCRCWebinar9May2012.pdf for
> explanation for the two points above.
> Separately, there are some long-standing issues with the protocol - since
> *way* before CASSANDRA-13304. Importantly, both checksumming and compression
> operate on individual message bodies rather than frames of multiple complete
> messages. In reality, this has several important additional downsides. To
> name a couple:
> # For compression, we are getting poor compression ratios for smaller
> messages - when operating on tiny sequences of bytes. In reality, for most
> small requests and responses we are discarding the compressed value as it’d
> be smaller than the uncompressed one - incurring both redundant allocations
> and compressions.
> # For checksumming and CRC32 we pay a high overhead price for small messages.
> 4 bytes extra is *a lot* for an empty write response, for example.
> To address the correctness issue of {{streamId}} not being covered by the
> checksum/CRC32 and the inefficiency in compression and checksumming/CRC32, we
> should switch to a framing protocol with multiple messages in a single frame.
> I suggest we reuse the framing protocol recently implemented for internode
> messaging in CASSANDRA-15066 to the extent that its logic can be borrowed,
> and that we do it before native protocol v5 graduates from beta. See
> https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/net/FrameDecoderCrc.java
> and
> https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/net/FrameDecoderLZ4.java.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14761) Rename speculative_retry to match additional_write_policy
[ https://issues.apache.org/jira/browse/CASSANDRA-14761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17125307#comment-17125307 ] Josh McKenzie commented on CASSANDRA-14761: --- Removing fixver for now; I assume it's a major only type change due to API breaking but don't want to presume since I'm not sure what the path forward is for an experimental feature. > Rename speculative_retry to match additional_write_policy > - > > Key: CASSANDRA-14761 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14761 > Project: Cassandra > Issue Type: Improvement > Components: Legacy/Core >Reporter: Ariel Weisberg >Priority: Normal > > It's not really speculative. This commit is where it was last named and shows > what to update > https://github.com/aweisberg/cassandra/commit/e1df8e977d942a1b0da7c2a7554149c781d0e6c3 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14761) Rename speculative_retry to match additional_write_policy
[ https://issues.apache.org/jira/browse/CASSANDRA-14761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh McKenzie updated CASSANDRA-14761: -- Fix Version/s: (was: 4.0-alpha) > Rename speculative_retry to match additional_write_policy > - > > Key: CASSANDRA-14761 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14761 > Project: Cassandra > Issue Type: Improvement > Components: Legacy/Core >Reporter: Ariel Weisberg >Priority: Normal > > It's not really speculative. This commit is where it was last named and shows > what to update > https://github.com/aweisberg/cassandra/commit/e1df8e977d942a1b0da7c2a7554149c781d0e6c3 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15792) test_speculative_data_request - read_repair_test.TestSpeculativeReadRepair
[ https://issues.apache.org/jira/browse/CASSANDRA-15792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17125305#comment-17125305 ] Josh McKenzie commented on CASSANDRA-15792: --- Thanks for the insight; agree on fixver. A little sad about the delay-based testing, though I assume converting to any kind of determinism would be significantly more invasive and artificial in the codebase so not worth pursuing. Perhaps commenting in the dtest the issue we ran into here so if we see it pop up subsequently (or annotating / elaborating in the failure state) pointing to this ticket could help in the future if this crops back up? Just a thought. > test_speculative_data_request - read_repair_test.TestSpeculativeReadRepair > -- > > Key: CASSANDRA-15792 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15792 > Project: Cassandra > Issue Type: Bug > Components: Test/dtest >Reporter: Ekaterina Dimitrova >Assignee: Gianluca Righetto >Priority: Normal > Fix For: 4.0-beta > > > Failing on the latest trunk here: > https://app.circleci.com/pipelines/github/ekaterinadimitrova2/cassandra/127/workflows/dfba669d-4a5c-4553-b6a2-85647d0d8d2b/jobs/668/tests > Failing once in 30 times as per Jenkins: > https://jenkins-cm4.apache.org/job/Cassandra-trunk-dtest/69/testReport/dtest.read_repair_test/TestSpeculativeReadRepair/test_speculative_data_request/ -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15792) test_speculative_data_request - read_repair_test.TestSpeculativeReadRepair
[ https://issues.apache.org/jira/browse/CASSANDRA-15792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh McKenzie updated CASSANDRA-15792: -- Fix Version/s: (was: 4.0-alpha) 4.0-beta > test_speculative_data_request - read_repair_test.TestSpeculativeReadRepair > -- > > Key: CASSANDRA-15792 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15792 > Project: Cassandra > Issue Type: Bug > Components: Test/dtest >Reporter: Ekaterina Dimitrova >Assignee: Gianluca Righetto >Priority: Normal > Fix For: 4.0-beta > > > Failing on the latest trunk here: > https://app.circleci.com/pipelines/github/ekaterinadimitrova2/cassandra/127/workflows/dfba669d-4a5c-4553-b6a2-85647d0d8d2b/jobs/668/tests > Failing once in 30 times as per Jenkins: > https://jenkins-cm4.apache.org/job/Cassandra-trunk-dtest/69/testReport/dtest.read_repair_test/TestSpeculativeReadRepair/test_speculative_data_request/ -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15685) flaky testWithMismatchingPending - org.apache.cassandra.distributed.test.PreviewRepairTest
[
https://issues.apache.org/jira/browse/CASSANDRA-15685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17125301#comment-17125301
]
Josh McKenzie commented on CASSANDRA-15685:
---
What do we think about moving fixver to 4.0-beta on this?
> flaky testWithMismatchingPending -
> org.apache.cassandra.distributed.test.PreviewRepairTest
> --
>
> Key: CASSANDRA-15685
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15685
> Project: Cassandra
> Issue Type: Bug
> Components: Test/dtest
>Reporter: Kevin Gallardo
>Assignee: Ekaterina Dimitrova
>Priority: Normal
> Labels: pull-request-available
> Fix For: 4.0-alpha
>
> Attachments: log-CASSANDRA-15685.txt, output
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Observed in:
> https://app.circleci.com/pipelines/github/newkek/cassandra/34/workflows/1c6b157d-13c3-48a9-85fb-9fe8c153256b/jobs/191/tests
> Failure:
> {noformat}
> testWithMismatchingPending -
> org.apache.cassandra.distributed.test.PreviewRepairTest
> junit.framework.AssertionFailedError
> at
> org.apache.cassandra.distributed.test.PreviewRepairTest.testWithMismatchingPending(PreviewRepairTest.java:97)
> {noformat}
> [Circle
> CI|https://circleci.com/gh/dcapwell/cassandra/tree/bug%2FCASSANDRA-15685]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15848) Fully purged static row causes NPE in repaired data tracking
[ https://issues.apache.org/jira/browse/CASSANDRA-15848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcus Eriksson updated CASSANDRA-15848: Reviewers: Marcus Eriksson > Fully purged static row causes NPE in repaired data tracking > > > Key: CASSANDRA-15848 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15848 > Project: Cassandra > Issue Type: Bug > Components: Legacy/Local Write-Read Paths >Reporter: Sam Tunnicliffe >Assignee: Sam Tunnicliffe >Priority: Normal > Fix For: 4.0-alpha > > > During repaired data tracking, if the result of applying the purge function > to a static row is null, an exception is thrown from RepairedDataInfo. This > will cause a read exception from the replica and could lead to unavailable > results if hit on multiple replicas. A workaround is to disable repaired data > tracking. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15848) Fully purged static row causes NPE in repaired data tracking
[ https://issues.apache.org/jira/browse/CASSANDRA-15848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam Tunnicliffe updated CASSANDRA-15848: Test and Documentation Plan: Added unit test. Existing unit/dtest/in-jvm dtests run. Status: Patch Available (was: Open) ||branch||utests||in-jvm dtests||dtests_with_vnodes||dtests_no_vnodes|| |[15848-trunk|https://github.com/beobal/cassandra/tree/15848-trunk]|[jdk8|https://circleci.com/gh/beobal/cassandra/1460], [jdk11|https://circleci.com/gh/beobal/cassandra/1461]|[jdk8|https://circleci.com/gh/beobal/cassandra/1459], [jdk11|https://circleci.com/gh/beobal/cassandra/1458]|[jdk8|https://circleci.com/gh/beobal/cassandra/1462], [jdk11|https://circleci.com/gh/beobal/cassandra/1465]|[jdk8|https://circleci.com/gh/beobal/cassandra/1463], [jdk11|https://circleci.com/gh/beobal/cassandra/1464]| > Fully purged static row causes NPE in repaired data tracking > > > Key: CASSANDRA-15848 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15848 > Project: Cassandra > Issue Type: Bug > Components: Legacy/Local Write-Read Paths >Reporter: Sam Tunnicliffe >Assignee: Sam Tunnicliffe >Priority: Normal > Fix For: 4.0-alpha > > > During repaired data tracking, if the result of applying the purge function > to a static row is null, an exception is thrown from RepairedDataInfo. This > will cause a read exception from the replica and could lead to unavailable > results if hit on multiple replicas. A workaround is to disable repaired data > tracking. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15822) DOC - Improve C* configuration docs
[ https://issues.apache.org/jira/browse/CASSANDRA-15822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon Haddad updated CASSANDRA-15822: --- Fix Version/s: 4.0-alpha5 Source Control Link: https://issues.apache.org/jira/browse/CASSANDRA-15822 Resolution: Fixed Status: Resolved (was: Ready to Commit) Made a couple minor tweaks, rebased and committed. Thanks! > DOC - Improve C* configuration docs > --- > > Key: CASSANDRA-15822 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15822 > Project: Cassandra > Issue Type: Improvement > Components: Documentation/Website >Reporter: Lorina Poland >Assignee: Lorina Poland >Priority: Normal > Fix For: 4.0-alpha5 > > > Two sections, Getting started > Configuring and Configuration, could use > improvement. > Adding information about some of the other config files besides > cassandra.yaml is one key goal. > > At the risk of contaminating one ticket with another project, I started > creating a separate glossary file, so that key terms in configuration can be > linked to a glossary that describes terms. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15822) DOC - Improve C* configuration docs
[ https://issues.apache.org/jira/browse/CASSANDRA-15822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon Haddad updated CASSANDRA-15822: --- Status: Ready to Commit (was: Review In Progress) > DOC - Improve C* configuration docs > --- > > Key: CASSANDRA-15822 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15822 > Project: Cassandra > Issue Type: Improvement > Components: Documentation/Website >Reporter: Lorina Poland >Assignee: Lorina Poland >Priority: Normal > > Two sections, Getting started > Configuring and Configuration, could use > improvement. > Adding information about some of the other config files besides > cassandra.yaml is one key goal. > > At the risk of contaminating one ticket with another project, I started > creating a separate glossary file, so that key terms in configuration can be > linked to a glossary that describes terms. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15822) DOC - Improve C* configuration docs
[ https://issues.apache.org/jira/browse/CASSANDRA-15822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon Haddad updated CASSANDRA-15822: --- Reviewers: Jon Haddad, Jon Haddad (was: Jon Haddad) Jon Haddad, Jon Haddad (was: Jon Haddad) Status: Review In Progress (was: Patch Available) > DOC - Improve C* configuration docs > --- > > Key: CASSANDRA-15822 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15822 > Project: Cassandra > Issue Type: Improvement > Components: Documentation/Website >Reporter: Lorina Poland >Assignee: Lorina Poland >Priority: Normal > > Two sections, Getting started > Configuring and Configuration, could use > improvement. > Adding information about some of the other config files besides > cassandra.yaml is one key goal. > > At the risk of contaminating one ticket with another project, I started > creating a separate glossary file, so that key terms in configuration can be > linked to a glossary that describes terms. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] branch trunk updated: Improving Cassandra configuration docs
This is an automated email from the ASF dual-hosted git repository. rustyrazorblade pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/cassandra.git The following commit(s) were added to refs/heads/trunk by this push: new 57a63f8 Improving Cassandra configuration docs 57a63f8 is described below commit 57a63f8551448d10d39f67d138842e0c81414526 Author: polandll AuthorDate: Fri May 8 17:24:32 2020 -0700 Improving Cassandra configuration docs * Update copyright date to 2020 * Add glossary * Rearranged rackdc for better flow * Add topologies properties file * Add commitlog_archiving file * Add logback.xml info * jvm.options files * Removed invalid reference to wiki from configuration file Patch by Lorina Poland; Reviewed by Jon Haddad for CASSANDRA-15822 --- conf/cassandra.yaml|5 +- doc/Dockerfile |2 +- doc/source/conf.py |2 +- doc/source/configuration/cass_cl_archive_file.rst | 46 + doc/source/configuration/cass_env_sh_file.rst | 132 ++ doc/source/configuration/cass_jvm_options_file.rst | 10 + doc/source/configuration/cass_logback_xml_file.rst | 157 ++ doc/source/configuration/cass_rackdc_file.rst | 67 + doc/source/configuration/cass_topo_file.rst| 48 + doc/source/configuration/cass_yaml_file.rst| 2074 doc/source/configuration/index.rst |8 +- doc/source/getting_started/configuring.rst | 59 +- doc/source/glossary.rst| 35 + 13 files changed, 2615 insertions(+), 30 deletions(-) diff --git a/conf/cassandra.yaml b/conf/cassandra.yaml index 6db9557..3a44a09 100644 --- a/conf/cassandra.yaml +++ b/conf/cassandra.yaml @@ -1,7 +1,7 @@ # Cassandra storage config YAML # NOTE: -# See http://wiki.apache.org/cassandra/StorageConfiguration for +# See https://cassandra.apache.org/doc/latest/configuration/ for # full explanations of configuration directives # /NOTE @@ -20,8 +20,6 @@ cluster_name: 'Test Cluster' # Specifying initial_token will override this setting on the node's initial start, # on subsequent starts, this setting will apply even if initial token is set. # -# If you already have a cluster with 1 token per node, and wish to migrate to -# multiple tokens per node, see http://wiki.apache.org/cassandra/Operations num_tokens: 256 # Triggers automatic allocation of num_tokens tokens for this node. The allocation @@ -47,7 +45,6 @@ num_tokens: 256 # that do not have vnodes enabled. # initial_token: -# See http://wiki.apache.org/cassandra/HintedHandoff # May either be "true" or "false" to enable globally hinted_handoff_enabled: true diff --git a/doc/Dockerfile b/doc/Dockerfile index 8999464..fcb4c41 100644 --- a/doc/Dockerfile +++ b/doc/Dockerfile @@ -13,7 +13,7 @@ RUN apt-get update && apt-get install -y software-properties-common RUN wget -qO - https://adoptopenjdk.jfrog.io/adoptopenjdk/api/gpg/key/public | apt-key add - \ && add-apt-repository --yes https://adoptopenjdk.jfrog.io/adoptopenjdk/deb/ \ && apt-get update \ -&& apt-get install -y adoptopenjdk-8-hotspot ant +&& apt-get install -y adoptopenjdk-11-hotspot ant RUN apt-get clean diff --git a/doc/source/conf.py b/doc/source/conf.py index 7143b23..48f87a8 100644 --- a/doc/source/conf.py +++ b/doc/source/conf.py @@ -75,7 +75,7 @@ master_doc = 'index' # General information about the project. project = u'Apache Cassandra' -copyright = u'2016, The Apache Cassandra team' +copyright = u'2020, The Apache Cassandra team' author = u'The Apache Cassandra team' # The version info for the project you're documenting, acts as replacement for diff --git a/doc/source/configuration/cass_cl_archive_file.rst b/doc/source/configuration/cass_cl_archive_file.rst new file mode 100644 index 000..fc14440 --- /dev/null +++ b/doc/source/configuration/cass_cl_archive_file.rst @@ -0,0 +1,46 @@ +.. _cassandra-cl-archive: + +commitlog-archiving.properties file + + +The ``commitlog-archiving.properties`` configuration file can optionally set commands that are executed when archiving or restoring a commitlog segment. + +=== +Options +=== + +``archive_command=`` +-- +One command can be inserted with %path and %name arguments. %path is the fully qualified path of the commitlog segment to archive. %name is the filename of the commitlog. STDOUT, STDIN, or multiple commands cannot be executed. If multiple commands are required, add a pointer to a script in this option. + +**Example:** archive_command=/bin/ln %path /backup/%name + +**Default value:** blank + +``restore_command=`` +-- +One command can be inserted with %from and %to arguments. %from is the fully qualified path to an archived commitlog segment
[jira] [Commented] (CASSANDRA-15700) Performance regression on internode messaging
[
https://issues.apache.org/jira/browse/CASSANDRA-15700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17125211#comment-17125211
]
Sergio Bossa commented on CASSANDRA-15700:
--
[~aleksey] thanks for reviewing: getting back to this after two months and
having to rebuild the whole thing in my mind, made me realize how complex this
is, and that I should have put a couple more comments, so apologies and kudos
for getting through it all alone :)
That said, to answer your question straight: the race you point out is
absolutely correct, and was unfortunately overlooked by myself, but there
should be an easy fix. Before going with that, I think I have to clarify first
why I introduced the new {{nextExpirationDeadline}} variable along
{{earliestExpiryTime}}. Simply put, if we only track time via
{{earliestExpiryTime}} during both {{add}} and {{prune}}, we risk getting into
a race where we accumulate an unbounded number of messages until the next
expiration, as shown in the following scenario:
1. {{add}} is called N times with {{earliestExpiryTime}} set as the minimum
time among the added messages; you can't adjust in this case by the current
time otherwise you would never expire (as the time would always shift).
2. {{prune}} is called and {{earliestExpiryTime}} is set as the minimum between
the minimum time among the pruned messages and the current expiry time.
3. This means that any messages arrived between the start and end of the
pruning whose expiry time was not the minimum, but still less than the minimum
expiry time among pruned messages, would be "ignored" and remain in the queue.
At this point you might say it really sounds like the race you discovered and
in a way it is :) Although in the above case the race window would be much
larger (the whole {{prune}}).
Anyway, as I said there should be an easy fix: the deadline should be updated
only if it's actually the minimum value (adjusted by current time), and I've
sent a new commit with such fix. I don't see a way to fix the same kind of
races by just keeping a single variable, but let me know if you find any.
> Performance regression on internode messaging
> -
>
> Key: CASSANDRA-15700
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15700
> Project: Cassandra
> Issue Type: Bug
> Components: Messaging/Internode
>Reporter: Sergio Bossa
>Assignee: Sergio Bossa
>Priority: Normal
> Labels: pull-request-available
> Fix For: 4.0-beta
>
> Attachments: Oss40patchedvsOss311.png, Oss40vsOss311.png, oss40.gc,
> oss40_nogc.tar.xz, oss40_system.log
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Me and [~jasonstack] have been investigating a performance regression
> affecting 4.0 during a 3 nodes, RF 3 write throughput test with a timeseries
> like workload, as shown in this plot, where blue is 3.11 and orange is 4.0:
> !Oss40vsOss311.png|width=389,height=214!
> It's been a bit of a long investigation, but two clues ended up standing out:
> 1) An abnormal number of expired messages on 4.0 (as shown in the attached
> system log), while 3.11 has almost none.
> 2) An abnormal GC activity (as shown in the attached gc log).
> Turns out the two are related, as the [on expired
> callback|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/net/OutboundConnection.java#L462]
> creates a huge amount of strings in the {{id()}} call. The next question is
> what causes all those message expirations; we thoroughly reviewed the
> internode messaging code and the only issue we could find so far is related
> to the "batch pruning" calls
> [here|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/net/OutboundMessageQueue.java#L81]
> and
> [here|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/net/OutboundMessageQueue.java#L188]:
> it _seems_ too much time is spent on those, causing the event loop to fall
> behind in processing the rest of the messages, which will end up being
> expired. This is supported by the analysis of the collapsed stacks (after
> fixing the GC issue):
> {noformat}
> (tprint (top-aggregated-calls oss40nogc "EventLoopDelivery:doRun" 5))
> org/apache/cassandra/net/OutboundConnection$EventLoopDelivery:doRun 3456
> org/apache/cassandra/net/OutboundMessageQueue:access$600 1621
> org/apache/cassandra/net/PrunableArrayQueue:prune 1621
> org/apache/cassandra/net/OutboundMessageQueue$WithLock:close 1621
> org/apache/cassandra/net/OutboundMessageQueue:pruneInternalQueueWithLock 1620
> {noformat}
> Those are the top 5 sampled calls from {{EventLoopDelivery#doRun()}} which
> spends half of its time pruning. But only a tiny portion of such pruning time
> is spent actually expiring:
> {noformat}
>
[jira] [Updated] (CASSANDRA-15849) Remove generated files from apache-cassandra-*-src.tar.gz artifacts
[ https://issues.apache.org/jira/browse/CASSANDRA-15849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Semb Wever updated CASSANDRA-15849: --- Change Category: Semantic Complexity: Low Hanging Fruit Fix Version/s: 4.0-beta Status: Open (was: Triage Needed) > Remove generated files from apache-cassandra-*-src.tar.gz artifacts > --- > > Key: CASSANDRA-15849 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15849 > Project: Cassandra > Issue Type: Task > Components: Build >Reporter: Michael Semb Wever >Assignee: Michael Semb Wever >Priority: Normal > Fix For: 4.0-beta > > > The apache-cassandra-*-src.tar.gz source artifacts contain a lot of generated > files. > These need to be excluded when putting together the source tarball. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14712) Add fqltool and auditlogviewer to rpm and deb packages
[ https://issues.apache.org/jira/browse/CASSANDRA-14712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Semb Wever updated CASSANDRA-14712: --- Fix Version/s: (was: 4.x) 4.0-alpha5 4.0 Since Version: 4.0-alpha1 Source Control Link: https://github.com/apache/cassandra/commit/ecf2c9ccd482c4b08355a288213326dc7a8cdcef Resolution: Fixed Status: Resolved (was: Ready to Commit) Committed as [ecf2c9ccd482c4b08355a288213326dc7a8cdcef |https://github.com/apache/cassandra/commit/ecf2c9ccd482c4b08355a288213326dc7a8cdcef]. Thanks [~stefan.miklosovic]! > Add fqltool and auditlogviewer to rpm and deb packages > -- > > Key: CASSANDRA-14712 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14712 > Project: Cassandra > Issue Type: Bug > Components: Packaging >Reporter: Stefan Podkowinski >Assignee: Stefan Miklosovic >Priority: Normal > Labels: Java11, pull-request-available > Fix For: 4.0, 4.0-alpha5 > > Time Spent: 2h 20m > Remaining Estimate: 0h > > Currently it's not possible to build any native packages (.deb/.rpm) for > trunk. > cassandra-builds - docker/*-image.docker > * Add Java11 to debian+centos build image > * (packaged ant scripts won't work with Java 11 on centos, so we may have to > install ant from tarballs) > cassandra-builds - docker/build-*.sh > * set JAVA8_HOME to Java8 > * set JAVA_HOME to Java11 (4.0) or Java8 (<4.0) > cassandra - redhat/cassandra.spec > * Check if patches still apply after CASSANDRA-14707 > * Add fqltool as %files > We may also have to change the version handling in build.xml or build-*.sh, > depending how we plan to release packages during beta, or if we plan to do so > at all before GA. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] branch trunk updated: Add fqltool and auditlogviewer to rpm and deb packages
This is an automated email from the ASF dual-hosted git repository.
mck pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra.git
The following commit(s) were added to refs/heads/trunk by this push:
new ecf2c9c Add fqltool and auditlogviewer to rpm and deb packages
ecf2c9c is described below
commit ecf2c9ccd482c4b08355a288213326dc7a8cdcef
Author: Stefan Miklosovic
AuthorDate: Wed Mar 27 17:32:27 2019 +1100
Add fqltool and auditlogviewer to rpm and deb packages
patch by Stefan Miklosovic; reviewed by Mick Semb Wever for CASSANDRA-14712
---
CHANGES.txt | 1 +
debian/cassandra.install | 2 ++
debian/rules | 4
redhat/cassandra | 2 +-
redhat/cassandra.spec| 5 +
5 files changed, 13 insertions(+), 1 deletion(-)
diff --git a/CHANGES.txt b/CHANGES.txt
index 09635e8..565f362 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
4.0-alpha5
+ * Add fqltool and auditlogviewer to rpm and deb packages (CASSANDRA-14712)
* Include DROPPED_COLUMNS in schema digest computation (CASSANDRA-15843)
* Fix Cassandra restart from rpm install (CASSANDRA-15830)
* Improve handling of 2i initialization failures (CASSANDRA-13606)
diff --git a/debian/cassandra.install b/debian/cassandra.install
index da800a4..be34838 100644
--- a/debian/cassandra.install
+++ b/debian/cassandra.install
@@ -21,6 +21,8 @@ bin/sstablescrub usr/bin
bin/sstableupgrade usr/bin
bin/sstableverify usr/bin
tools/bin/cassandra-stress usr/bin
+tools/bin/fqltool usr/bin
+tools/bin/auditlogviewer usr/bin
lib/*.jar usr/share/cassandra/lib
lib/*.zip usr/share/cassandra/lib
lib/sigar-bin/* usr/share/cassandra/lib/sigar-bin
diff --git a/debian/rules b/debian/rules
index 53874c2..e7fa32f 100755
--- a/debian/rules
+++ b/debian/rules
@@ -51,6 +51,10 @@ install: build
dh_install build/tools/lib/stress.jar \
usr/share/cassandra
+ # Copy fqltool jars
+ dh_install build/tools/lib/fqltool.jar \
+ usr/share/cassandra
+
dh_link usr/share/cassandra/apache-cassandra-$(VERSION).jar \
usr/share/cassandra/apache-cassandra.jar
diff --git a/redhat/cassandra b/redhat/cassandra
index 97a0447..77622b6 100644
--- a/redhat/cassandra
+++ b/redhat/cassandra
@@ -34,7 +34,7 @@ lock_file=/var/lock/subsys/$NAME
CASSANDRA_PROG=/usr/sbin/cassandra
# The first existing directory is used for JAVA_HOME if needed.
-JVM_SEARCH_DIRS="/usr/lib/jvm/jre /usr/lib/jvm/jre-1.7.*
/usr/lib/jvm/java-1.7.*/jre"
+JVM_SEARCH_DIRS="/usr/lib/jvm/jre /usr/lib/jvm/jre-1.8.*
/usr/lib/jvm/java-1.8.*/jre"
# Read configuration variable file if it is present
[ -r /etc/default/$NAME ] && . /etc/default/$NAME
diff --git a/redhat/cassandra.spec b/redhat/cassandra.spec
index 9ca9f4c..f42ab83 100644
--- a/redhat/cassandra.spec
+++ b/redhat/cassandra.spec
@@ -100,6 +100,9 @@ cp -pr lib/* %{buildroot}/usr/share/%{username}/lib/
# copy stress jar
cp -p build/tools/lib/stress.jar %{buildroot}/usr/share/%{username}/
+# copy fqltool jar
+cp -p build/tools/lib/fqltool.jar %{buildroot}/usr/share/%{username}/
+
# copy binaries
mv bin/cassandra %{buildroot}/usr/sbin/
cp -p bin/* %{buildroot}/usr/bin/
@@ -177,6 +180,8 @@ This package contains extra tools for working with
Cassandra clusters.
%attr(755,root,root) %{_bindir}/sstableofflinerelevel
%attr(755,root,root) %{_bindir}/sstablerepairedset
%attr(755,root,root) %{_bindir}/sstablesplit
+%attr(755,root,root) %{_bindir}/auditlogviewer
+%attr(755,root,root) %{_bindir}/fqltool
%changelog
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15845) test_populate_mv_after_insert_wide_rows schema version update race
[
https://issues.apache.org/jira/browse/CASSANDRA-15845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17125051#comment-17125051
]
Berenguer Blasi commented on CASSANDRA-15845:
-
Mmm the fix is no good {{V iewSchemaTest}} blocks now. So there might be legit
cases for node schema agreement while the schema is being worked on. This needs
further digging.
> test_populate_mv_after_insert_wide_rows schema version update race
> --
>
> Key: CASSANDRA-15845
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15845
> Project: Cassandra
> Issue Type: Bug
> Components: Test/dtest
>Reporter: Berenguer Blasi
>Assignee: Berenguer Blasi
>Priority: Normal
> Fix For: 4.0-beta
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> See failure
> [here|https://ci-cassandra.apache.org/job/Cassandra-trunk/155/testReport/dtest-novnode.materialized_views_test/TestMaterializedViews/test_populate_mv_after_insert_wide_rows_2/]
> {quote}Error Message
> assert 160 == 0 + where 160 = Row(count=160).count
> Stacktrace
> self = 0x7fcb232b5760>
> def test_populate_mv_after_insert_wide_rows(self):
> """Test that a view is OK when created with existing data with wide
> rows"""
> session = self.prepare(consistency_level=ConsistencyLevel.QUORUM)
>
> session.execute("CREATE TABLE t (id int, v int, PRIMARY KEY (id, v))")
>
> for i in range(5):
> for j in range(1):
> session.execute("INSERT INTO t (id, v) VALUES ({},
> {})".format(i, j))
>
> session.execute(("CREATE MATERIALIZED VIEW t_by_v AS SELECT * FROM t
> WHERE v IS NOT NULL "
> "AND id IS NOT NULL PRIMARY KEY (v, id)"))
>
> logger.debug("wait for view to build")
> self._wait_for_view("ks", "t_by_v")
>
> logger.debug("wait that all batchlogs are replayed")
> > self._replay_batchlogs()
> materialized_views_test.py:350:
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> _
> self = 0x7fcb232b5760>
> def _replay_batchlogs(self):
> for node in self.cluster.nodelist():
> if node.is_running():
> logger.debug("Replaying batchlog on node
> {}".format(node.name))
> node.nodetool("replaybatchlog")
> # CASSANDRA-13069 - Ensure replayed mutations are removed
> from the batchlog
> node_session = self.patient_exclusive_cql_connection(node)
> result = list(node_session.execute("SELECT count(*) FROM
> system.batches;"))
> > assert result[0].count == 0
> E assert 160 == 0
> E+ where 160 = Row(count=160).count
> materialized_views_test.py:171: AssertionError{quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-15845) test_populate_mv_after_insert_wide_rows schema version update race
[
https://issues.apache.org/jira/browse/CASSANDRA-15845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17125051#comment-17125051
]
Berenguer Blasi edited comment on CASSANDRA-15845 at 6/3/20, 3:24 PM:
--
Mmm the fix is no good {{ViewSchemaTest}} blocks now. So there might be legit
cases for node schema agreement while the schema is being worked on. This needs
further digging.
was (Author: bereng):
Mmm the fix is no good {{V iewSchemaTest}} blocks now. So there might be legit
cases for node schema agreement while the schema is being worked on. This needs
further digging.
> test_populate_mv_after_insert_wide_rows schema version update race
> --
>
> Key: CASSANDRA-15845
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15845
> Project: Cassandra
> Issue Type: Bug
> Components: Test/dtest
>Reporter: Berenguer Blasi
>Assignee: Berenguer Blasi
>Priority: Normal
> Fix For: 4.0-beta
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> See failure
> [here|https://ci-cassandra.apache.org/job/Cassandra-trunk/155/testReport/dtest-novnode.materialized_views_test/TestMaterializedViews/test_populate_mv_after_insert_wide_rows_2/]
> {quote}Error Message
> assert 160 == 0 + where 160 = Row(count=160).count
> Stacktrace
> self = 0x7fcb232b5760>
> def test_populate_mv_after_insert_wide_rows(self):
> """Test that a view is OK when created with existing data with wide
> rows"""
> session = self.prepare(consistency_level=ConsistencyLevel.QUORUM)
>
> session.execute("CREATE TABLE t (id int, v int, PRIMARY KEY (id, v))")
>
> for i in range(5):
> for j in range(1):
> session.execute("INSERT INTO t (id, v) VALUES ({},
> {})".format(i, j))
>
> session.execute(("CREATE MATERIALIZED VIEW t_by_v AS SELECT * FROM t
> WHERE v IS NOT NULL "
> "AND id IS NOT NULL PRIMARY KEY (v, id)"))
>
> logger.debug("wait for view to build")
> self._wait_for_view("ks", "t_by_v")
>
> logger.debug("wait that all batchlogs are replayed")
> > self._replay_batchlogs()
> materialized_views_test.py:350:
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> _
> self = 0x7fcb232b5760>
> def _replay_batchlogs(self):
> for node in self.cluster.nodelist():
> if node.is_running():
> logger.debug("Replaying batchlog on node
> {}".format(node.name))
> node.nodetool("replaybatchlog")
> # CASSANDRA-13069 - Ensure replayed mutations are removed
> from the batchlog
> node_session = self.patient_exclusive_cql_connection(node)
> result = list(node_session.execute("SELECT count(*) FROM
> system.batches;"))
> > assert result[0].count == 0
> E assert 160 == 0
> E+ where 160 = Row(count=160).count
> materialized_views_test.py:171: AssertionError{quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14712) Add fqltool and auditlogviewer to rpm and deb packages
[ https://issues.apache.org/jira/browse/CASSANDRA-14712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Semb Wever updated CASSANDRA-14712: --- Status: Ready to Commit (was: Review In Progress) > Add fqltool and auditlogviewer to rpm and deb packages > -- > > Key: CASSANDRA-14712 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14712 > Project: Cassandra > Issue Type: Bug > Components: Packaging >Reporter: Stefan Podkowinski >Assignee: Stefan Miklosovic >Priority: Normal > Labels: Java11, pull-request-available > Fix For: 4.x > > Time Spent: 2h 20m > Remaining Estimate: 0h > > Currently it's not possible to build any native packages (.deb/.rpm) for > trunk. > cassandra-builds - docker/*-image.docker > * Add Java11 to debian+centos build image > * (packaged ant scripts won't work with Java 11 on centos, so we may have to > install ant from tarballs) > cassandra-builds - docker/build-*.sh > * set JAVA8_HOME to Java8 > * set JAVA_HOME to Java11 (4.0) or Java8 (<4.0) > cassandra - redhat/cassandra.spec > * Check if patches still apply after CASSANDRA-14707 > * Add fqltool as %files > We may also have to change the version handling in build.xml or build-*.sh, > depending how we plan to release packages during beta, or if we plan to do so > at all before GA. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14712) Add fqltool and auditlogviewer to rpm and deb packages
[ https://issues.apache.org/jira/browse/CASSANDRA-14712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Semb Wever updated CASSANDRA-14712: --- Reviewers: Michael Semb Wever, Michael Semb Wever (was: Michael Semb Wever) Michael Semb Wever, Michael Semb Wever (was: Michael Semb Wever) Status: Review In Progress (was: Patch Available) > Add fqltool and auditlogviewer to rpm and deb packages > -- > > Key: CASSANDRA-14712 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14712 > Project: Cassandra > Issue Type: Bug > Components: Packaging >Reporter: Stefan Podkowinski >Assignee: Stefan Miklosovic >Priority: Normal > Labels: Java11, pull-request-available > Fix For: 4.x > > Time Spent: 2h 20m > Remaining Estimate: 0h > > Currently it's not possible to build any native packages (.deb/.rpm) for > trunk. > cassandra-builds - docker/*-image.docker > * Add Java11 to debian+centos build image > * (packaged ant scripts won't work with Java 11 on centos, so we may have to > install ant from tarballs) > cassandra-builds - docker/build-*.sh > * set JAVA8_HOME to Java8 > * set JAVA_HOME to Java11 (4.0) or Java8 (<4.0) > cassandra - redhat/cassandra.spec > * Check if patches still apply after CASSANDRA-14707 > * Add fqltool as %files > We may also have to change the version handling in build.xml or build-*.sh, > depending how we plan to release packages during beta, or if we plan to do so > at all before GA. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14712) Add fqltool and auditlogviewer to rpm and deb packages
[ https://issues.apache.org/jira/browse/CASSANDRA-14712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Semb Wever updated CASSANDRA-14712: --- Test and Documentation Plan: Manually using cassandra-*-packaging.sh scripts. (Will become automatic with CASSANDRA-15838 Status: Patch Available (was: Open) > Add fqltool and auditlogviewer to rpm and deb packages > -- > > Key: CASSANDRA-14712 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14712 > Project: Cassandra > Issue Type: Bug > Components: Packaging >Reporter: Stefan Podkowinski >Assignee: Stefan Miklosovic >Priority: Normal > Labels: Java11, pull-request-available > Fix For: 4.x > > Time Spent: 2h 20m > Remaining Estimate: 0h > > Currently it's not possible to build any native packages (.deb/.rpm) for > trunk. > cassandra-builds - docker/*-image.docker > * Add Java11 to debian+centos build image > * (packaged ant scripts won't work with Java 11 on centos, so we may have to > install ant from tarballs) > cassandra-builds - docker/build-*.sh > * set JAVA8_HOME to Java8 > * set JAVA_HOME to Java11 (4.0) or Java8 (<4.0) > cassandra - redhat/cassandra.spec > * Check if patches still apply after CASSANDRA-14707 > * Add fqltool as %files > We may also have to change the version handling in build.xml or build-*.sh, > depending how we plan to release packages during beta, or if we plan to do so > at all before GA. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14712) Add fqltool and auditlogviewer to rpm and deb packages
[ https://issues.apache.org/jira/browse/CASSANDRA-14712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Semb Wever updated CASSANDRA-14712: --- Test and Documentation Plan: Manually using cassandra-*-packaging.sh scripts. (Will become automatic with CASSANDRA-15838 ) was: Manually using cassandra-*-packaging.sh scripts. (Will become automatic with CASSANDRA-15838 > Add fqltool and auditlogviewer to rpm and deb packages > -- > > Key: CASSANDRA-14712 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14712 > Project: Cassandra > Issue Type: Bug > Components: Packaging >Reporter: Stefan Podkowinski >Assignee: Stefan Miklosovic >Priority: Normal > Labels: Java11, pull-request-available > Fix For: 4.x > > Time Spent: 2h 20m > Remaining Estimate: 0h > > Currently it's not possible to build any native packages (.deb/.rpm) for > trunk. > cassandra-builds - docker/*-image.docker > * Add Java11 to debian+centos build image > * (packaged ant scripts won't work with Java 11 on centos, so we may have to > install ant from tarballs) > cassandra-builds - docker/build-*.sh > * set JAVA8_HOME to Java8 > * set JAVA_HOME to Java11 (4.0) or Java8 (<4.0) > cassandra - redhat/cassandra.spec > * Check if patches still apply after CASSANDRA-14707 > * Add fqltool as %files > We may also have to change the version handling in build.xml or build-*.sh, > depending how we plan to release packages during beta, or if we plan to do so > at all before GA. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14712) Add fqltool and auditlogviewer to rpm and deb packages
[
https://issues.apache.org/jira/browse/CASSANDRA-14712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17125008#comment-17125008
]
Michael Semb Wever commented on CASSANDRA-14712:
A rebase off trunk of #307 is available at
https://github.com/apache/cassandra/compare/trunk...thelastpickle:CASSANDRA-14712
Have tested it. (Had to add back the line {{rm -f bin/*.orig}} to
{{cassandra.spec}}.
+1
> Add fqltool and auditlogviewer to rpm and deb packages
> --
>
> Key: CASSANDRA-14712
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14712
> Project: Cassandra
> Issue Type: Bug
> Components: Packaging
>Reporter: Stefan Podkowinski
>Assignee: Stefan Miklosovic
>Priority: Normal
> Labels: Java11, pull-request-available
> Fix For: 4.x
>
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> Currently it's not possible to build any native packages (.deb/.rpm) for
> trunk.
> cassandra-builds - docker/*-image.docker
> * Add Java11 to debian+centos build image
> * (packaged ant scripts won't work with Java 11 on centos, so we may have to
> install ant from tarballs)
> cassandra-builds - docker/build-*.sh
> * set JAVA8_HOME to Java8
> * set JAVA_HOME to Java11 (4.0) or Java8 (<4.0)
> cassandra - redhat/cassandra.spec
> * Check if patches still apply after CASSANDRA-14707
> * Add fqltool as %files
> We may also have to change the version handling in build.xml or build-*.sh,
> depending how we plan to release packages during beta, or if we plan to do so
> at all before GA.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15845) test_populate_mv_after_insert_wide_rows schema version update race
[
https://issues.apache.org/jira/browse/CASSANDRA-15845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Berenguer Blasi updated CASSANDRA-15845:
Status: In Progress (was: Patch Available)
> test_populate_mv_after_insert_wide_rows schema version update race
> --
>
> Key: CASSANDRA-15845
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15845
> Project: Cassandra
> Issue Type: Bug
> Components: Test/dtest
>Reporter: Berenguer Blasi
>Assignee: Berenguer Blasi
>Priority: Normal
> Fix For: 4.0-beta
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> See failure
> [here|https://ci-cassandra.apache.org/job/Cassandra-trunk/155/testReport/dtest-novnode.materialized_views_test/TestMaterializedViews/test_populate_mv_after_insert_wide_rows_2/]
> {quote}Error Message
> assert 160 == 0 + where 160 = Row(count=160).count
> Stacktrace
> self = 0x7fcb232b5760>
> def test_populate_mv_after_insert_wide_rows(self):
> """Test that a view is OK when created with existing data with wide
> rows"""
> session = self.prepare(consistency_level=ConsistencyLevel.QUORUM)
>
> session.execute("CREATE TABLE t (id int, v int, PRIMARY KEY (id, v))")
>
> for i in range(5):
> for j in range(1):
> session.execute("INSERT INTO t (id, v) VALUES ({},
> {})".format(i, j))
>
> session.execute(("CREATE MATERIALIZED VIEW t_by_v AS SELECT * FROM t
> WHERE v IS NOT NULL "
> "AND id IS NOT NULL PRIMARY KEY (v, id)"))
>
> logger.debug("wait for view to build")
> self._wait_for_view("ks", "t_by_v")
>
> logger.debug("wait that all batchlogs are replayed")
> > self._replay_batchlogs()
> materialized_views_test.py:350:
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> _
> self = 0x7fcb232b5760>
> def _replay_batchlogs(self):
> for node in self.cluster.nodelist():
> if node.is_running():
> logger.debug("Replaying batchlog on node
> {}".format(node.name))
> node.nodetool("replaybatchlog")
> # CASSANDRA-13069 - Ensure replayed mutations are removed
> from the batchlog
> node_session = self.patient_exclusive_cql_connection(node)
> result = list(node_session.execute("SELECT count(*) FROM
> system.batches;"))
> > assert result[0].count == 0
> E assert 160 == 0
> E+ where 160 = Row(count=160).count
> materialized_views_test.py:171: AssertionError{quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15845) test_populate_mv_after_insert_wide_rows schema version update race
[
https://issues.apache.org/jira/browse/CASSANDRA-15845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Berenguer Blasi updated CASSANDRA-15845:
Test and Documentation Plan: See PR
Status: Patch Available (was: In Progress)
> test_populate_mv_after_insert_wide_rows schema version update race
> --
>
> Key: CASSANDRA-15845
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15845
> Project: Cassandra
> Issue Type: Bug
> Components: Test/dtest
>Reporter: Berenguer Blasi
>Assignee: Berenguer Blasi
>Priority: Normal
> Fix For: 4.0-beta
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> See failure
> [here|https://ci-cassandra.apache.org/job/Cassandra-trunk/155/testReport/dtest-novnode.materialized_views_test/TestMaterializedViews/test_populate_mv_after_insert_wide_rows_2/]
> {quote}Error Message
> assert 160 == 0 + where 160 = Row(count=160).count
> Stacktrace
> self = 0x7fcb232b5760>
> def test_populate_mv_after_insert_wide_rows(self):
> """Test that a view is OK when created with existing data with wide
> rows"""
> session = self.prepare(consistency_level=ConsistencyLevel.QUORUM)
>
> session.execute("CREATE TABLE t (id int, v int, PRIMARY KEY (id, v))")
>
> for i in range(5):
> for j in range(1):
> session.execute("INSERT INTO t (id, v) VALUES ({},
> {})".format(i, j))
>
> session.execute(("CREATE MATERIALIZED VIEW t_by_v AS SELECT * FROM t
> WHERE v IS NOT NULL "
> "AND id IS NOT NULL PRIMARY KEY (v, id)"))
>
> logger.debug("wait for view to build")
> self._wait_for_view("ks", "t_by_v")
>
> logger.debug("wait that all batchlogs are replayed")
> > self._replay_batchlogs()
> materialized_views_test.py:350:
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> _
> self = 0x7fcb232b5760>
> def _replay_batchlogs(self):
> for node in self.cluster.nodelist():
> if node.is_running():
> logger.debug("Replaying batchlog on node
> {}".format(node.name))
> node.nodetool("replaybatchlog")
> # CASSANDRA-13069 - Ensure replayed mutations are removed
> from the batchlog
> node_session = self.patient_exclusive_cql_connection(node)
> result = list(node_session.execute("SELECT count(*) FROM
> system.batches;"))
> > assert result[0].count == 0
> E assert 160 == 0
> E+ where 160 = Row(count=160).count
> materialized_views_test.py:171: AssertionError{quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15845) test_populate_mv_after_insert_wide_rows schema version update race
[
https://issues.apache.org/jira/browse/CASSANDRA-15845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17124978#comment-17124978
]
Berenguer Blasi commented on CASSANDRA-15845:
-
Ok I did run this for 1.5h continuously and it's all green except for 2
failures which are 'new' or different from how this was failing before this
fix. I submitted a PR for review. Meanwhile the 'new failure' I would address
in a new ticket.
New failure:
{noformat}
--
Captured stdout setup
--
15:47:37,189 ccm DEBUG Log-watching thread starting.
---
Captured log setup
15:47:37,151 conftest INFO Starting execution of
test_populate_mv_after_insert_wide_rows at 2020-06-03 15:47:37.151758
15:47:37,152 dtest_setup INFO cluster ccm directory: /tmp/dtest-3_i9j_ku
Captured log call
15:47:51,930 cassandra.cluster INFO New Cassandra host
discovered
15:47:51,930 cassandra.cluster INFO New Cassandra host
discovered
15:48:24,880 cassandra.protocol WARNING Server warning: Materialized views are
experimental and are not recommended for production use.
15:48:25,920 cassandra.cluster INFO New Cassandra host
discovered
15:48:25,921 cassandra.cluster INFO New Cassandra host
discovered
15:48:25,947 cassandra.cluster INFO New Cassandra host
discovered
15:48:25,947 cassandra.cluster INFO New Cassandra host
discovered
15:48:25,984 cassandra.cluster INFO New Cassandra host
discovered
15:48:25,984 cassandra.cluster INFO New Cassandra host
discovered
15:48:27,973 cassandra.cluster INFO New Cassandra host
discovered
15:48:27,973 cassandra.cluster INFO New Cassandra host
discovered
15:48:28,108 cassandra.protocol WARNING Server warning: Aggregation query used
without partition key
Captured stdout teardown
-
15:48:28,320 ccm DEBUG Log-watching thread exiting.
--
Captured stdout setup
--
15:48:29,170 ccm DEBUG Log-watching thread starting.
---
Captured log setup
15:48:29,134 conftest INFO Starting execution of
test_populate_mv_after_insert_wide_rows at 2020-06-03 15:48:29.134170
15:48:29,134 dtest_setup INFO cluster ccm directory: /tmp/dtest-a84sugqx
Captured log call
15:48:43,900 cassandra.cluster INFO New Cassandra host
discovered
15:48:43,900 cassandra.cluster INFO New Cassandra host
discovered
15:49:16,846 cassandra.protocol WARNING Server warning: Materialized views are
experimental and are not recommended for production use.
15:49:17,911 cassandra.cluster INFO New Cassandra host
discovered
15:49:17,911 cassandra.cluster INFO New Cassandra host
discovered
15:49:17,946 cassandra.cluster INFO New Cassandra host
discovered
15:49:17,946 cassandra.cluster INFO New Cassandra host
discovered
15:49:17,977 cassandra.cluster INFO New Cassandra host
discovered
15:49:17,977 cassandra.cluster INFO New Cassandra host
discovered
15:49:20,349 cassandra.cluster INFO New Cassandra host
discovered
15:49:20,350 cassandra.cluster INFO New Cassandra host
discovered
15:49:20,614 cassandra.protocol WARNING Server warning: Aggregation query used
without partition key
Captured stdout
[jira] [Updated] (CASSANDRA-15845) test_populate_mv_after_insert_wide_rows schema version update race
[
https://issues.apache.org/jira/browse/CASSANDRA-15845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Berenguer Blasi updated CASSANDRA-15845:
Summary: test_populate_mv_after_insert_wide_rows schema version update race
(was:
dtest-novnode.materialized_views_test.TestMaterializedViews.test_populate_mv_after_insert_wide_rows)
> test_populate_mv_after_insert_wide_rows schema version update race
> --
>
> Key: CASSANDRA-15845
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15845
> Project: Cassandra
> Issue Type: Bug
> Components: Test/dtest
>Reporter: Berenguer Blasi
>Assignee: Berenguer Blasi
>Priority: Normal
> Fix For: 4.0-beta
>
>
> See failure
> [here|https://ci-cassandra.apache.org/job/Cassandra-trunk/155/testReport/dtest-novnode.materialized_views_test/TestMaterializedViews/test_populate_mv_after_insert_wide_rows_2/]
> {quote}Error Message
> assert 160 == 0 + where 160 = Row(count=160).count
> Stacktrace
> self = 0x7fcb232b5760>
> def test_populate_mv_after_insert_wide_rows(self):
> """Test that a view is OK when created with existing data with wide
> rows"""
> session = self.prepare(consistency_level=ConsistencyLevel.QUORUM)
>
> session.execute("CREATE TABLE t (id int, v int, PRIMARY KEY (id, v))")
>
> for i in range(5):
> for j in range(1):
> session.execute("INSERT INTO t (id, v) VALUES ({},
> {})".format(i, j))
>
> session.execute(("CREATE MATERIALIZED VIEW t_by_v AS SELECT * FROM t
> WHERE v IS NOT NULL "
> "AND id IS NOT NULL PRIMARY KEY (v, id)"))
>
> logger.debug("wait for view to build")
> self._wait_for_view("ks", "t_by_v")
>
> logger.debug("wait that all batchlogs are replayed")
> > self._replay_batchlogs()
> materialized_views_test.py:350:
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> _
> self = 0x7fcb232b5760>
> def _replay_batchlogs(self):
> for node in self.cluster.nodelist():
> if node.is_running():
> logger.debug("Replaying batchlog on node
> {}".format(node.name))
> node.nodetool("replaybatchlog")
> # CASSANDRA-13069 - Ensure replayed mutations are removed
> from the batchlog
> node_session = self.patient_exclusive_cql_connection(node)
> result = list(node_session.execute("SELECT count(*) FROM
> system.batches;"))
> > assert result[0].count == 0
> E assert 160 == 0
> E+ where 160 = Row(count=160).count
> materialized_views_test.py:171: AssertionError{quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-15849) Remove generated files from apache-cassandra-*-src.tar.gz artifacts
Michael Semb Wever created CASSANDRA-15849: -- Summary: Remove generated files from apache-cassandra-*-src.tar.gz artifacts Key: CASSANDRA-15849 URL: https://issues.apache.org/jira/browse/CASSANDRA-15849 Project: Cassandra Issue Type: Task Components: Build Reporter: Michael Semb Wever Assignee: Michael Semb Wever The apache-cassandra-*-src.tar.gz source artifacts contain a lot of generated files. These need to be excluded when putting together the source tarball. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15700) Performance regression on internode messaging
[
https://issues.apache.org/jira/browse/CASSANDRA-15700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17124959#comment-17124959
]
Aleksey Yeschenko commented on CASSANDRA-15700:
---
I *think* the way we update times at the end of pruning in this patch is
potentially problematic, but I might be getting this wrong, so pardon me for
any obvious foolishness. Was thinking about this particular scenario:
1. After partial consumption/delivery, thread {{t1}} updates
{{earliestExpiryTime}} with some high value (say, all messages up to now have
been with long timeouts); gets descheduled for a period of time.
2. A different thread, {{t2}}, invokes {{add()}} with an lots of
short-expiration messages, all skipping purging, as the lock is being held by
{{t1}}. Repeatedly updates both {{earliestExpiryTime}} and
{{nextExpirationDeadline}} with a small value
3. Thread {{t1}} is now scheduled again, sets {{nextExpirationDeadline}} to the
high value from step (1)
If there are no {{add()}} incoming and delivery has stalled, we can now hold an
arbitrary # of messages added at step 2, as we use {{nextExpirationDeadline}}
value to determine whether or not pruning is necessary.
> Performance regression on internode messaging
> -
>
> Key: CASSANDRA-15700
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15700
> Project: Cassandra
> Issue Type: Bug
> Components: Messaging/Internode
>Reporter: Sergio Bossa
>Assignee: Sergio Bossa
>Priority: Normal
> Labels: pull-request-available
> Fix For: 4.0-beta
>
> Attachments: Oss40patchedvsOss311.png, Oss40vsOss311.png, oss40.gc,
> oss40_nogc.tar.xz, oss40_system.log
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Me and [~jasonstack] have been investigating a performance regression
> affecting 4.0 during a 3 nodes, RF 3 write throughput test with a timeseries
> like workload, as shown in this plot, where blue is 3.11 and orange is 4.0:
> !Oss40vsOss311.png|width=389,height=214!
> It's been a bit of a long investigation, but two clues ended up standing out:
> 1) An abnormal number of expired messages on 4.0 (as shown in the attached
> system log), while 3.11 has almost none.
> 2) An abnormal GC activity (as shown in the attached gc log).
> Turns out the two are related, as the [on expired
> callback|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/net/OutboundConnection.java#L462]
> creates a huge amount of strings in the {{id()}} call. The next question is
> what causes all those message expirations; we thoroughly reviewed the
> internode messaging code and the only issue we could find so far is related
> to the "batch pruning" calls
> [here|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/net/OutboundMessageQueue.java#L81]
> and
> [here|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/net/OutboundMessageQueue.java#L188]:
> it _seems_ too much time is spent on those, causing the event loop to fall
> behind in processing the rest of the messages, which will end up being
> expired. This is supported by the analysis of the collapsed stacks (after
> fixing the GC issue):
> {noformat}
> (tprint (top-aggregated-calls oss40nogc "EventLoopDelivery:doRun" 5))
> org/apache/cassandra/net/OutboundConnection$EventLoopDelivery:doRun 3456
> org/apache/cassandra/net/OutboundMessageQueue:access$600 1621
> org/apache/cassandra/net/PrunableArrayQueue:prune 1621
> org/apache/cassandra/net/OutboundMessageQueue$WithLock:close 1621
> org/apache/cassandra/net/OutboundMessageQueue:pruneInternalQueueWithLock 1620
> {noformat}
> Those are the top 5 sampled calls from {{EventLoopDelivery#doRun()}} which
> spends half of its time pruning. But only a tiny portion of such pruning time
> is spent actually expiring:
> {noformat}
> (tprint (top-aggregated-calls oss40nogc
> "OutboundMessageQueue:pruneInternalQueueWithLock" 5))
> org/apache/cassandra/net/OutboundMessageQueue:pruneInternalQueueWithLock 1900
> org/apache/cassandra/net/PrunableArrayQueue:prune 1894
> org/apache/cassandra/net/OutboundMessageQueue$1Pruner:onPruned 147
> org/apache/cassandra/net/OutboundConnection$$Lambda$444/740904487:accept 147
> org/apache/cassandra/net/OutboundConnection:onExpired 147
> {noformat}
> And indeed, the {{PrunableArrayQueue:prune()}} self time is dominant:
> {noformat}
> (tprint (top-self-calls oss40nogc "PrunableArrayQueue:prune" 5))
> org/apache/cassandra/net/PrunableArrayQueue:prune 1718
> org/apache/cassandra/net/OutboundConnection:releaseCapacity 27
> java/util/concurrent/ConcurrentHashMap:replaceNode 19
> java/util/concurrent/ConcurrentLinkedQueue:offer 16
> java/util/concurrent/LinkedBlockingQueue:offer 15
> {noformat}
> That said, before
[jira] [Created] (CASSANDRA-15848) Fully purged static row causes NPE in repaired data tracking
Sam Tunnicliffe created CASSANDRA-15848: --- Summary: Fully purged static row causes NPE in repaired data tracking Key: CASSANDRA-15848 URL: https://issues.apache.org/jira/browse/CASSANDRA-15848 Project: Cassandra Issue Type: Bug Components: Legacy/Local Write-Read Paths Reporter: Sam Tunnicliffe Assignee: Sam Tunnicliffe During repaired data tracking, if the result of applying the purge function to a static row is null, an exception is thrown from RepairedDataInfo. This will cause a read exception from the replica and could lead to unavailable results if hit on multiple replicas. A workaround is to disable repaired data tracking. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15848) Fully purged static row causes NPE in repaired data tracking
[ https://issues.apache.org/jira/browse/CASSANDRA-15848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam Tunnicliffe updated CASSANDRA-15848: Bug Category: Parent values: Availability(12983)Level 1 values: Response Crash(12991) Complexity: Low Hanging Fruit Discovered By: User Report Fix Version/s: 4.0-alpha Severity: Normal Status: Open (was: Triage Needed) > Fully purged static row causes NPE in repaired data tracking > > > Key: CASSANDRA-15848 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15848 > Project: Cassandra > Issue Type: Bug > Components: Legacy/Local Write-Read Paths >Reporter: Sam Tunnicliffe >Assignee: Sam Tunnicliffe >Priority: Normal > Fix For: 4.0-alpha > > > During repaired data tracking, if the result of applying the purge function > to a static row is null, an exception is thrown from RepairedDataInfo. This > will cause a read exception from the replica and could lead to unavailable > results if hit on multiple replicas. A workaround is to disable repaired data > tracking. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-15845) dtest-novnode.materialized_views_test.TestMaterializedViews.test_populate_mv_after_insert_wide_rows
[
https://issues.apache.org/jira/browse/CASSANDRA-15845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17124820#comment-17124820
]
Berenguer Blasi edited comment on CASSANDRA-15845 at 6/3/20, 10:11 AM:
---
[~jasonstack] you rock! you saved me a few hours of painful debugging :-) I
have tried to repro locally and I can't anymore. I will leave it running for
lunch just in case as it was a difficult one to repro. I would like to get your
opinion on how to best fix this given I don't know the codebase as well:
A. Make {{SchemaKeyspace#calculateSchemaDigest}} public and check schema
agreement against that as well in {{Gossiper#nodesAgreeOnSchema}} (this is the
current fix that seems to work)
B. Mark the schema somehow 'dirty' and take that into account in all calls to
the version. Not sure how or how much work that would be
C. Block during the 'schema version update' window.
I think B and C sound very nice but I wonder into how much trouble and
deadlocks can we get ourselves into and if it's worth it or reasonably doable.
Maybe fixing {{Gossiper#nodesAgreeOnSchema}} with A is a good compromise and
that method is going to be there for anybody that needs to. Wdyt? :thinking:
was (Author: bereng):
[~jasonstack] you rock! you saved me a few hours of painful debugging :-) I
have tried to repro locally and I can't anymore. I will leave it running for
lunch just in case as it was a difficult one to repro. I would like to get your
opinion on how to best fix this given I don't know the codebase as well:
A. Make {{SchemaKeyspace#calculateSchemaDigest}} public and check schema
agreement against that as well in {{Gossiper#nodesAgreeOnSchema}} (this is the
current fix that seems to work
B. Mark the schema somehow 'dirty' and take that into account in all calls to
the version. Not sure how or how much work that would be
C. Block during the 'schema version update' window.
I think B and C sound very nice but I wonder into how much trouble and
deadlocks can we get ourselves into and if it's worth it or reasonably doable.
Maybe fixing {{Gossiper#nodesAgreeOnSchema}} with A is a good compromise and
that method is going to be there for anybody that needs to. Wdyt? :thinking:
> dtest-novnode.materialized_views_test.TestMaterializedViews.test_populate_mv_after_insert_wide_rows
> ---
>
> Key: CASSANDRA-15845
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15845
> Project: Cassandra
> Issue Type: Bug
> Components: Test/dtest
>Reporter: Berenguer Blasi
>Assignee: Berenguer Blasi
>Priority: Normal
> Fix For: 4.0-beta
>
>
> See failure
> [here|https://ci-cassandra.apache.org/job/Cassandra-trunk/155/testReport/dtest-novnode.materialized_views_test/TestMaterializedViews/test_populate_mv_after_insert_wide_rows_2/]
> {quote}Error Message
> assert 160 == 0 + where 160 = Row(count=160).count
> Stacktrace
> self = 0x7fcb232b5760>
> def test_populate_mv_after_insert_wide_rows(self):
> """Test that a view is OK when created with existing data with wide
> rows"""
> session = self.prepare(consistency_level=ConsistencyLevel.QUORUM)
>
> session.execute("CREATE TABLE t (id int, v int, PRIMARY KEY (id, v))")
>
> for i in range(5):
> for j in range(1):
> session.execute("INSERT INTO t (id, v) VALUES ({},
> {})".format(i, j))
>
> session.execute(("CREATE MATERIALIZED VIEW t_by_v AS SELECT * FROM t
> WHERE v IS NOT NULL "
> "AND id IS NOT NULL PRIMARY KEY (v, id)"))
>
> logger.debug("wait for view to build")
> self._wait_for_view("ks", "t_by_v")
>
> logger.debug("wait that all batchlogs are replayed")
> > self._replay_batchlogs()
> materialized_views_test.py:350:
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> _
> self = 0x7fcb232b5760>
> def _replay_batchlogs(self):
> for node in self.cluster.nodelist():
> if node.is_running():
> logger.debug("Replaying batchlog on node
> {}".format(node.name))
> node.nodetool("replaybatchlog")
> # CASSANDRA-13069 - Ensure replayed mutations are removed
> from the batchlog
> node_session = self.patient_exclusive_cql_connection(node)
> result = list(node_session.execute("SELECT count(*) FROM
> system.batches;"))
> > assert result[0].count == 0
> E assert 160 == 0
> E+ where 160 = Row(count=160).count
> materialized_views_test.py:171: AssertionError{quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CASSANDRA-15845) dtest-novnode.materialized_views_test.TestMaterializedViews.test_populate_mv_after_insert_wide_rows
[
https://issues.apache.org/jira/browse/CASSANDRA-15845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17124820#comment-17124820
]
Berenguer Blasi commented on CASSANDRA-15845:
-
[~jasonstack] you rock! you saved me a few hours of painful debugging :-) I
have tried to repro locally and I can't anymore. I will leave it running for
lunch just in case as it was a difficult one to repro. I would like to get your
opinion on how to best fix this given I don't know the codebase as well:
A. Make {{SchemaKeyspace#calculateSchemaDigest}} public and check schema
agreement against that as well in {{Gossiper#nodesAgreeOnSchema}} (this is the
current fix that seems to work
B. Mark the schema somehow 'dirty' and take that into account in all calls to
the version. Not sure how or how much work that would be
C. Block during the 'schema version update' window.
I think B and C sound very nice but I wonder into how much trouble and
deadlocks can we get ourselves into and if it's worth it or reasonably doable.
Maybe fixing {{Gossiper#nodesAgreeOnSchema}} with A is a good compromise and
that method is going to be there for anybody that needs to. Wdyt? :thinking:
> dtest-novnode.materialized_views_test.TestMaterializedViews.test_populate_mv_after_insert_wide_rows
> ---
>
> Key: CASSANDRA-15845
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15845
> Project: Cassandra
> Issue Type: Bug
> Components: Test/dtest
>Reporter: Berenguer Blasi
>Assignee: Berenguer Blasi
>Priority: Normal
> Fix For: 4.0-beta
>
>
> See failure
> [here|https://ci-cassandra.apache.org/job/Cassandra-trunk/155/testReport/dtest-novnode.materialized_views_test/TestMaterializedViews/test_populate_mv_after_insert_wide_rows_2/]
> {quote}Error Message
> assert 160 == 0 + where 160 = Row(count=160).count
> Stacktrace
> self = 0x7fcb232b5760>
> def test_populate_mv_after_insert_wide_rows(self):
> """Test that a view is OK when created with existing data with wide
> rows"""
> session = self.prepare(consistency_level=ConsistencyLevel.QUORUM)
>
> session.execute("CREATE TABLE t (id int, v int, PRIMARY KEY (id, v))")
>
> for i in range(5):
> for j in range(1):
> session.execute("INSERT INTO t (id, v) VALUES ({},
> {})".format(i, j))
>
> session.execute(("CREATE MATERIALIZED VIEW t_by_v AS SELECT * FROM t
> WHERE v IS NOT NULL "
> "AND id IS NOT NULL PRIMARY KEY (v, id)"))
>
> logger.debug("wait for view to build")
> self._wait_for_view("ks", "t_by_v")
>
> logger.debug("wait that all batchlogs are replayed")
> > self._replay_batchlogs()
> materialized_views_test.py:350:
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> _
> self = 0x7fcb232b5760>
> def _replay_batchlogs(self):
> for node in self.cluster.nodelist():
> if node.is_running():
> logger.debug("Replaying batchlog on node
> {}".format(node.name))
> node.nodetool("replaybatchlog")
> # CASSANDRA-13069 - Ensure replayed mutations are removed
> from the batchlog
> node_session = self.patient_exclusive_cql_connection(node)
> result = list(node_session.execute("SELECT count(*) FROM
> system.batches;"))
> > assert result[0].count == 0
> E assert 160 == 0
> E+ where 160 = Row(count=160).count
> materialized_views_test.py:171: AssertionError{quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15838) Add deb and rpm packaging to artifacts test script
[
https://issues.apache.org/jira/browse/CASSANDRA-15838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17124805#comment-17124805
]
Michael Semb Wever commented on CASSANDRA-15838:
Support for building rpms with jdk11 is also implemented.
And looking at the source artefacts, the biggest current problem is that the
original {{*-src.tar.gz}} artefacts contains thousands of non-source files. I
will address this in a separate ticket.
Patch is complete, at
https://github.com/apache/cassandra-builds/compare/master...thelastpickle:mck/15838
> Add deb and rpm packaging to artifacts test script
> --
>
> Key: CASSANDRA-15838
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15838
> Project: Cassandra
> Issue Type: Task
> Components: Build, CI, Packaging
>Reporter: Michael Semb Wever
>Assignee: Michael Semb Wever
>Priority: Normal
> Fix For: 4.0-beta
>
> Attachments: Screenshot 2020-06-01 at 12.29.21.png, Screenshot
> 2020-06-01 at 23.24.39.png
>
>
> Currently we have no way of testing debian or rpm packaging. This is a
> problem, it hurts when cutting releases as well as afterwards (like
> CASSANDRA-15830).
> This ticket is to add testing of debian or rpm packaging into the artifacts
> test script.
> This will also provide "nightly" build artifacts for the deb and rpm packages.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15838) Add deb and rpm packaging to artifacts test script
[ https://issues.apache.org/jira/browse/CASSANDRA-15838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Semb Wever updated CASSANDRA-15838: --- Test and Documentation Plan: https://issues.apache.org/jira/browse/CASSANDRA-15838?focusedCommentId=17123524=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17123524 Status: Patch Available (was: In Progress) > Add deb and rpm packaging to artifacts test script > -- > > Key: CASSANDRA-15838 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15838 > Project: Cassandra > Issue Type: Task > Components: Build, CI, Packaging >Reporter: Michael Semb Wever >Assignee: Michael Semb Wever >Priority: Normal > Fix For: 4.0-beta > > Attachments: Screenshot 2020-06-01 at 12.29.21.png, Screenshot > 2020-06-01 at 23.24.39.png > > > Currently we have no way of testing debian or rpm packaging. This is a > problem, it hurts when cutting releases as well as afterwards (like > CASSANDRA-15830). > This ticket is to add testing of debian or rpm packaging into the artifacts > test script. > This will also provide "nightly" build artifacts for the deb and rpm packages. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15835) Upgrade-dtests on trunk not working in CircleCI
[
https://issues.apache.org/jira/browse/CASSANDRA-15835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17124754#comment-17124754
]
Eduard Tudenhoefner commented on CASSANDRA-15835:
-
reviewed all of the above PRs and changes LGTM. Waiting on final CI results now.
> Upgrade-dtests on trunk not working in CircleCI
> ---
>
> Key: CASSANDRA-15835
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15835
> Project: Cassandra
> Issue Type: Bug
> Components: CI, Test/dtest
>Reporter: Robert Stupp
>Assignee: Robert Stupp
>Priority: Normal
> Fix For: 4.0-alpha
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> ~3600 Upgrade-dtests are failing in CircleCI for trunk due to the missing
> {{JAVA8_HOME}} environment variable.
> Patching the Docker image is rather simple by creating a new image:
> {code}
> FROM nastra/cassandra-testing-ubuntu1910-java11-w-dependencies:20200406
> ENV JAVA8_HOME=/usr/lib/jvm/java-8-openjdk-amd64
> {code}
> Pushed the above to Docker hub as
> [snazy/cassandra-testing-ubuntu1910-java11-w-dependencies:202005261540|https://hub.docker.com/layers/snazy/cassandra-testing-ubuntu1910-java11-w-dependencies/202005261540/images/sha256-ac8a713be58694f095c491921e006c2d1a7823a3c23299e477198e2c93a6bbd7?context=explore]
> The size of the whole Docker image is a little concerning though (1.85G
> compressed), but that's out of the scope of this ticket.
> I'll prepare a patch soon-ish.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra-builds] branch master updated: Update README
This is an automated email from the ASF dual-hosted git repository. mck pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/cassandra-builds.git The following commit(s) were added to refs/heads/master by this push: new b94a3a1 Update README b94a3a1 is described below commit b94a3a14e53e10dd332ef5a3fe25811cf4e7825e Author: Eduard Tudenhoefner AuthorDate: Wed Jun 3 09:14:49 2020 +0200 Update README --- docker/testing/README.md | 10 +- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docker/testing/README.md b/docker/testing/README.md index 99a88e9..8864391 100644 --- a/docker/testing/README.md +++ b/docker/testing/README.md @@ -2,7 +2,7 @@ Docker files in this directory are used to build images used by CircleCI. These are directly referenced in the `circle.yml` after publishing to dockerhub. There are two types of images: -* Base image for Linux distribution to use for testing (e.g. `ubuntu1810_j11.docker`) +* Base image for Linux distribution to use for testing (e.g. `ubuntu1910_j11.docker`) * Caching image that contains git sources, maven and ccm dependencies ## Building Images @@ -11,11 +11,11 @@ Build images from the parent directory using the following commands. Change tag Base image: -`docker build -t spod/cassandra-testing-ubuntu1810:20180128 -t spod/cassandra-testing-ubuntu1810:latest -f testing/ubuntu1810_j11.docker testing/` +`docker build -t nastra/cassandra-testing-ubuntu1910-java11:20200603 -t nastra/cassandra-testing-ubuntu1910-java11:latest -f ubuntu1910_j11.docker .` Caching image: -`docker build --no-cache -t spod/cassandra-testing-ubuntu1810-java11-w-dependencies:20190306 -t spod/cassandra-testing-ubuntu1810-java11-w-dependencies:latest -f testing/ubuntu1810_j11_w_dependencies.docker testing/` +`docker build --no-cache -t nastra/cassandra-testing-ubuntu1910-java11-w-dependencies:20200603 -t nastra/cassandra-testing-ubuntu1910-java11-w-dependencies:latest -f ubuntu1910_j11_w_dependencies.docker .` Please make sure to always tag also by date, so we can go back to that version in case anything breaks after the next update! @@ -26,8 +26,8 @@ We are using Docker Hub for storing published images. See [Quick Start Guide](ht Push both image references: ``` -docker push spod/cassandra-testing-ubuntu1810-java11-w-dependencies:20190306 -docker push spod/cassandra-testing-ubuntu1810-java11-w-dependencies:latest +docker push nastra/cassandra-testing-ubuntu1910-java11-w-dependencies:20200603 +docker push nastra/cassandra-testing-ubuntu1910-java11-w-dependencies:latest ``` ## Updating circleci.yml - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra-builds] branch master updated: CASSANDRA-15835 Update Docker image w/ JAVA8_HOME via ENV
This is an automated email from the ASF dual-hosted git repository. snazy pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/cassandra-builds.git The following commit(s) were added to refs/heads/master by this push: new 45dc010 CASSANDRA-15835 Update Docker image w/ JAVA8_HOME via ENV 45dc010 is described below commit 45dc010227d75bf05e34617729de5389e62f8f99 Author: Robert Stupp AuthorDate: Tue Jun 2 12:43:30 2020 +0200 CASSANDRA-15835 Update Docker image w/ JAVA8_HOME via ENV --- docker/testing/ubuntu1810_j11.docker | 5 + docker/testing/ubuntu18_j11.docker | 5 + docker/testing/ubuntu1910_j11.docker | 5 + 3 files changed, 15 insertions(+) diff --git a/docker/testing/ubuntu1810_j11.docker b/docker/testing/ubuntu1810_j11.docker index 835d2ad..58fea47 100644 --- a/docker/testing/ubuntu1810_j11.docker +++ b/docker/testing/ubuntu1810_j11.docker @@ -80,6 +80,11 @@ RUN echo 'export ANT_HOME=/usr/share/ant' >> /home/cassandra/.bashrc && \ echo 'export JAVA11_HOME=/usr/lib/jvm/java-11-openjdk-amd64' >> /home/cassandra/.bashrc && \ echo 'export JAVA_HOME=$JAVA8_HOME' >> /home/cassandra/.bashrc +ENV ANT_HOME=/usr/share/ant \ +JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 \ +JAVA8_HOME=/usr/lib/jvm/java-8-openjdk-amd64 \ +JAVA11_HOME=/usr/lib/jvm/java-11-openjdk-amd64 + # run pip commands and setup virtualenv (note we do this after we switch to cassandra user so we # setup the virtualenv for the cassandra user and not the root user by accident) RUN virtualenv --python=python3.6 --no-site-packages env diff --git a/docker/testing/ubuntu18_j11.docker b/docker/testing/ubuntu18_j11.docker index 7e92c0f..3e6fe6b 100644 --- a/docker/testing/ubuntu18_j11.docker +++ b/docker/testing/ubuntu18_j11.docker @@ -80,6 +80,11 @@ RUN echo 'export ANT_HOME=/usr/share/ant' >> /home/cassandra/.bashrc && \ echo 'export JAVA11_HOME=/usr/lib/jvm/java-11-openjdk-amd64' >> /home/cassandra/.bashrc && \ echo 'export JAVA_HOME=$JAVA8_HOME' >> /home/cassandra/.bashrc +ENV ANT_HOME=/usr/share/ant \ +JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 \ +JAVA8_HOME=/usr/lib/jvm/java-8-openjdk-amd64 \ +JAVA11_HOME=/usr/lib/jvm/java-11-openjdk-amd64 + # run pip commands and setup virtualenv (note we do this after we switch to cassandra user so we # setup the virtualenv for the cassandra user and not the root user by accident) RUN virtualenv --python=python3.6 --no-site-packages env diff --git a/docker/testing/ubuntu1910_j11.docker b/docker/testing/ubuntu1910_j11.docker index 38f60bb..70dd44b 100644 --- a/docker/testing/ubuntu1910_j11.docker +++ b/docker/testing/ubuntu1910_j11.docker @@ -92,6 +92,11 @@ RUN echo 'export ANT_HOME=/usr/share/ant' >> /home/cassandra/.bashrc && \ echo 'export JAVA11_HOME=/usr/lib/jvm/java-11-openjdk-amd64' >> /home/cassandra/.bashrc && \ echo 'export JAVA_HOME=$JAVA8_HOME' >> /home/cassandra/.bashrc +ENV ANT_HOME=/usr/share/ant \ +JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 \ +JAVA8_HOME=/usr/lib/jvm/java-8-openjdk-amd64 \ +JAVA11_HOME=/usr/lib/jvm/java-11-openjdk-amd64 + # run pip commands and setup virtualenv (note we do this after we switch to cassandra user so we # setup the virtualenv for the cassandra user and not the root user by accident) for Python 3.6/3.7/3.8 RUN virtualenv --python=python3.6 env3.6 - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra-builds] branch master updated: CASSANDRA-15835 Update Docker image w/ JAVA8_HOME via ENV
This is an automated email from the ASF dual-hosted git repository. snazy pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/cassandra-builds.git The following commit(s) were added to refs/heads/master by this push: new 45dc010 CASSANDRA-15835 Update Docker image w/ JAVA8_HOME via ENV 45dc010 is described below commit 45dc010227d75bf05e34617729de5389e62f8f99 Author: Robert Stupp AuthorDate: Tue Jun 2 12:43:30 2020 +0200 CASSANDRA-15835 Update Docker image w/ JAVA8_HOME via ENV --- docker/testing/ubuntu1810_j11.docker | 5 + docker/testing/ubuntu18_j11.docker | 5 + docker/testing/ubuntu1910_j11.docker | 5 + 3 files changed, 15 insertions(+) diff --git a/docker/testing/ubuntu1810_j11.docker b/docker/testing/ubuntu1810_j11.docker index 835d2ad..58fea47 100644 --- a/docker/testing/ubuntu1810_j11.docker +++ b/docker/testing/ubuntu1810_j11.docker @@ -80,6 +80,11 @@ RUN echo 'export ANT_HOME=/usr/share/ant' >> /home/cassandra/.bashrc && \ echo 'export JAVA11_HOME=/usr/lib/jvm/java-11-openjdk-amd64' >> /home/cassandra/.bashrc && \ echo 'export JAVA_HOME=$JAVA8_HOME' >> /home/cassandra/.bashrc +ENV ANT_HOME=/usr/share/ant \ +JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 \ +JAVA8_HOME=/usr/lib/jvm/java-8-openjdk-amd64 \ +JAVA11_HOME=/usr/lib/jvm/java-11-openjdk-amd64 + # run pip commands and setup virtualenv (note we do this after we switch to cassandra user so we # setup the virtualenv for the cassandra user and not the root user by accident) RUN virtualenv --python=python3.6 --no-site-packages env diff --git a/docker/testing/ubuntu18_j11.docker b/docker/testing/ubuntu18_j11.docker index 7e92c0f..3e6fe6b 100644 --- a/docker/testing/ubuntu18_j11.docker +++ b/docker/testing/ubuntu18_j11.docker @@ -80,6 +80,11 @@ RUN echo 'export ANT_HOME=/usr/share/ant' >> /home/cassandra/.bashrc && \ echo 'export JAVA11_HOME=/usr/lib/jvm/java-11-openjdk-amd64' >> /home/cassandra/.bashrc && \ echo 'export JAVA_HOME=$JAVA8_HOME' >> /home/cassandra/.bashrc +ENV ANT_HOME=/usr/share/ant \ +JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 \ +JAVA8_HOME=/usr/lib/jvm/java-8-openjdk-amd64 \ +JAVA11_HOME=/usr/lib/jvm/java-11-openjdk-amd64 + # run pip commands and setup virtualenv (note we do this after we switch to cassandra user so we # setup the virtualenv for the cassandra user and not the root user by accident) RUN virtualenv --python=python3.6 --no-site-packages env diff --git a/docker/testing/ubuntu1910_j11.docker b/docker/testing/ubuntu1910_j11.docker index 38f60bb..70dd44b 100644 --- a/docker/testing/ubuntu1910_j11.docker +++ b/docker/testing/ubuntu1910_j11.docker @@ -92,6 +92,11 @@ RUN echo 'export ANT_HOME=/usr/share/ant' >> /home/cassandra/.bashrc && \ echo 'export JAVA11_HOME=/usr/lib/jvm/java-11-openjdk-amd64' >> /home/cassandra/.bashrc && \ echo 'export JAVA_HOME=$JAVA8_HOME' >> /home/cassandra/.bashrc +ENV ANT_HOME=/usr/share/ant \ +JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 \ +JAVA8_HOME=/usr/lib/jvm/java-8-openjdk-amd64 \ +JAVA11_HOME=/usr/lib/jvm/java-11-openjdk-amd64 + # run pip commands and setup virtualenv (note we do this after we switch to cassandra user so we # setup the virtualenv for the cassandra user and not the root user by accident) for Python 3.6/3.7/3.8 RUN virtualenv --python=python3.6 env3.6 - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15845) dtest-novnode.materialized_views_test.TestMaterializedViews.test_populate_mv_after_insert_wide_rows
[
https://issues.apache.org/jira/browse/CASSANDRA-15845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17124730#comment-17124730
]
ZhaoYang commented on CASSANDRA-15845:
--
CASSANDRA-14571 should have avoided view builder starting before schema
propagation..
According to the log [~Bereng] uploaded, it looks like node1 is in schema
agreement with other nodes.. My hypothesis is when update view schema at node1,
it starts view builder before updating schema digest..
In {{"Schema#transform"}}"
{code}
merge(diff); // starts view builder asynchronously
updateVersion(); // update schema digest
{code}
bq. Also do you think we should mark this test as a failure in dtests like it's
suggested here until the underlying cause is fixed?
Alternatively we can put a delay in view builder for dtests..
> dtest-novnode.materialized_views_test.TestMaterializedViews.test_populate_mv_after_insert_wide_rows
> ---
>
> Key: CASSANDRA-15845
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15845
> Project: Cassandra
> Issue Type: Bug
> Components: Test/dtest
>Reporter: Berenguer Blasi
>Assignee: Berenguer Blasi
>Priority: Normal
> Fix For: 4.0-beta
>
>
> See failure
> [here|https://ci-cassandra.apache.org/job/Cassandra-trunk/155/testReport/dtest-novnode.materialized_views_test/TestMaterializedViews/test_populate_mv_after_insert_wide_rows_2/]
> {quote}Error Message
> assert 160 == 0 + where 160 = Row(count=160).count
> Stacktrace
> self = 0x7fcb232b5760>
> def test_populate_mv_after_insert_wide_rows(self):
> """Test that a view is OK when created with existing data with wide
> rows"""
> session = self.prepare(consistency_level=ConsistencyLevel.QUORUM)
>
> session.execute("CREATE TABLE t (id int, v int, PRIMARY KEY (id, v))")
>
> for i in range(5):
> for j in range(1):
> session.execute("INSERT INTO t (id, v) VALUES ({},
> {})".format(i, j))
>
> session.execute(("CREATE MATERIALIZED VIEW t_by_v AS SELECT * FROM t
> WHERE v IS NOT NULL "
> "AND id IS NOT NULL PRIMARY KEY (v, id)"))
>
> logger.debug("wait for view to build")
> self._wait_for_view("ks", "t_by_v")
>
> logger.debug("wait that all batchlogs are replayed")
> > self._replay_batchlogs()
> materialized_views_test.py:350:
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> _
> self = 0x7fcb232b5760>
> def _replay_batchlogs(self):
> for node in self.cluster.nodelist():
> if node.is_running():
> logger.debug("Replaying batchlog on node
> {}".format(node.name))
> node.nodetool("replaybatchlog")
> # CASSANDRA-13069 - Ensure replayed mutations are removed
> from the batchlog
> node_session = self.patient_exclusive_cql_connection(node)
> result = list(node_session.execute("SELECT count(*) FROM
> system.batches;"))
> > assert result[0].count == 0
> E assert 160 == 0
> E+ where 160 = Row(count=160).count
> materialized_views_test.py:171: AssertionError{quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-15845) dtest-novnode.materialized_views_test.TestMaterializedViews.test_populate_mv_after_insert_wide_rows
[
https://issues.apache.org/jira/browse/CASSANDRA-15845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17124701#comment-17124701
]
Berenguer Blasi edited comment on CASSANDRA-15845 at 6/3/20, 7:37 AM:
--
Hi [~jasonstack],
thx for the info. What I see is slightly different though. We do indeed
[wait|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/view/ViewBuilderTask.java#L136]
for schema agreement, but adding some logging on node1 we can see it _sees_
schema agreement while the other nodes2&3 don't know about the MV yet. Here is
node1 under the wrong impression on schema agreement:
{noformat}
DEBUG [ViewBuildExecutor:1] 2020-06-02 17:31:46,137 Gossiper.java:2038 - Node:
127.0.0.3:7000 expectedVersion: null remoteVersion:
a17bfd1c-15fd-30ad-9a5e-534b45062856
DEBUG [ViewBuildExecutor:1] 2020-06-02 17:31:46,137 Gossiper.java:2038 - Node:
127.0.0.2:7000 expectedVersion: a17bfd1c-15fd-30ad-9a5e-534b45062856
remoteVersion: a17bfd1c-15fd-30ad-9a5e-534b45062856
DEBUG [ViewBuildExecutor:1] 2020-06-02 17:31:46,137 Gossiper.java:2038 - Node:
127.0.0.1:7000 expectedVersion: a17bfd1c-15fd-30ad-9a5e-534b45062856
remoteVersion: a17bfd1c-15fd-30ad-9a5e-534b45062856
DEBUG [ViewBuildExecutor:1] 2020-06-02 17:31:46,137 Gossiper.java:2047 - Nodes
agree on schema
{noformat}
I don't know if it's worth going down this race rabbit hole or if Is there a
ticket around already I can mark this one a duplicate of? I can't seem to find
one
Also do you think we should mark this test as a failure in dtests like it's
suggested
[here|https://issues.apache.org/jira/browse/CASSANDRA-14571?focusedCommentId=16549702=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16549702]
until the underlying cause is fixed?
was (Author: bereng):
Hi [~jasonstack],
thx for the info. What I see is slightly different though. We do indeed
[wait|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/view/ViewBuilderTask.java#L136]
for schema agreement, but adding some logging on node1 we can see it _sees_
schema agreement while the other nodes2&3 don't know about the MV yet. Here is
node1 under the wrong impression on schema agreement:
{noformat}
DEBUG [ViewBuildExecutor:1] 2020-06-02 17:31:46,137 Gossiper.java:2038 - Node:
127.0.0.3:7000 expectedVersion: null remoteVersion:
a17bfd1c-15fd-30ad-9a5e-534b45062856
DEBUG [ViewBuildExecutor:1] 2020-06-02 17:31:46,137 Gossiper.java:2038 - Node:
127.0.0.2:7000 expectedVersion: a17bfd1c-15fd-30ad-9a5e-534b45062856
remoteVersion: a17bfd1c-15fd-30ad-9a5e-534b45062856
DEBUG [ViewBuildExecutor:1] 2020-06-02 17:31:46,137 Gossiper.java:2038 - Node:
127.0.0.1:7000 expectedVersion: a17bfd1c-15fd-30ad-9a5e-534b45062856
remoteVersion: a17bfd1c-15fd-30ad-9a5e-534b45062856
DEBUG [ViewBuildExecutor:1] 2020-06-02 17:31:46,137 Gossiper.java:2047 - Nodes
agree on schema
{noformat}
I don't know if it's worth going down this race rabbit hole or if Is there a
ticket around already I can mark this one a duplicate of? I can't seem to find
one
> dtest-novnode.materialized_views_test.TestMaterializedViews.test_populate_mv_after_insert_wide_rows
> ---
>
> Key: CASSANDRA-15845
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15845
> Project: Cassandra
> Issue Type: Bug
> Components: Test/dtest
>Reporter: Berenguer Blasi
>Assignee: Berenguer Blasi
>Priority: Normal
> Fix For: 4.0-beta
>
>
> See failure
> [here|https://ci-cassandra.apache.org/job/Cassandra-trunk/155/testReport/dtest-novnode.materialized_views_test/TestMaterializedViews/test_populate_mv_after_insert_wide_rows_2/]
> {quote}Error Message
> assert 160 == 0 + where 160 = Row(count=160).count
> Stacktrace
> self = 0x7fcb232b5760>
> def test_populate_mv_after_insert_wide_rows(self):
> """Test that a view is OK when created with existing data with wide
> rows"""
> session = self.prepare(consistency_level=ConsistencyLevel.QUORUM)
>
> session.execute("CREATE TABLE t (id int, v int, PRIMARY KEY (id, v))")
>
> for i in range(5):
> for j in range(1):
> session.execute("INSERT INTO t (id, v) VALUES ({},
> {})".format(i, j))
>
> session.execute(("CREATE MATERIALIZED VIEW t_by_v AS SELECT * FROM t
> WHERE v IS NOT NULL "
> "AND id IS NOT NULL PRIMARY KEY (v, id)"))
>
> logger.debug("wait for view to build")
> self._wait_for_view("ks", "t_by_v")
>
> logger.debug("wait that all batchlogs are replayed")
> > self._replay_batchlogs()
> materialized_views_test.py:350:
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
[jira] [Commented] (CASSANDRA-15845) dtest-novnode.materialized_views_test.TestMaterializedViews.test_populate_mv_after_insert_wide_rows
[
https://issues.apache.org/jira/browse/CASSANDRA-15845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17124701#comment-17124701
]
Berenguer Blasi commented on CASSANDRA-15845:
-
Hi [~jasonstack],
thx for the info. What I see is slightly different though. We do indeed
[wait|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/view/ViewBuilderTask.java#L136]
for schema agreement, but adding some logging on node1 we can see it _sees_
schema agreement while the other nodes2&3 don't know about the MV yet. Here is
node1 under the wrong impression on schema agreement:
{noformat}
DEBUG [ViewBuildExecutor:1] 2020-06-02 17:31:46,137 Gossiper.java:2038 - Node:
127.0.0.3:7000 expectedVersion: null remoteVersion:
a17bfd1c-15fd-30ad-9a5e-534b45062856
DEBUG [ViewBuildExecutor:1] 2020-06-02 17:31:46,137 Gossiper.java:2038 - Node:
127.0.0.2:7000 expectedVersion: a17bfd1c-15fd-30ad-9a5e-534b45062856
remoteVersion: a17bfd1c-15fd-30ad-9a5e-534b45062856
DEBUG [ViewBuildExecutor:1] 2020-06-02 17:31:46,137 Gossiper.java:2038 - Node:
127.0.0.1:7000 expectedVersion: a17bfd1c-15fd-30ad-9a5e-534b45062856
remoteVersion: a17bfd1c-15fd-30ad-9a5e-534b45062856
DEBUG [ViewBuildExecutor:1] 2020-06-02 17:31:46,137 Gossiper.java:2047 - Nodes
agree on schema
{noformat}
I don't know if it's worth going down this race rabbit hole or if Is there a
ticket around already I can mark this one a duplicate of? I can't seem to find
one
> dtest-novnode.materialized_views_test.TestMaterializedViews.test_populate_mv_after_insert_wide_rows
> ---
>
> Key: CASSANDRA-15845
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15845
> Project: Cassandra
> Issue Type: Bug
> Components: Test/dtest
>Reporter: Berenguer Blasi
>Assignee: Berenguer Blasi
>Priority: Normal
> Fix For: 4.0-beta
>
>
> See failure
> [here|https://ci-cassandra.apache.org/job/Cassandra-trunk/155/testReport/dtest-novnode.materialized_views_test/TestMaterializedViews/test_populate_mv_after_insert_wide_rows_2/]
> {quote}Error Message
> assert 160 == 0 + where 160 = Row(count=160).count
> Stacktrace
> self = 0x7fcb232b5760>
> def test_populate_mv_after_insert_wide_rows(self):
> """Test that a view is OK when created with existing data with wide
> rows"""
> session = self.prepare(consistency_level=ConsistencyLevel.QUORUM)
>
> session.execute("CREATE TABLE t (id int, v int, PRIMARY KEY (id, v))")
>
> for i in range(5):
> for j in range(1):
> session.execute("INSERT INTO t (id, v) VALUES ({},
> {})".format(i, j))
>
> session.execute(("CREATE MATERIALIZED VIEW t_by_v AS SELECT * FROM t
> WHERE v IS NOT NULL "
> "AND id IS NOT NULL PRIMARY KEY (v, id)"))
>
> logger.debug("wait for view to build")
> self._wait_for_view("ks", "t_by_v")
>
> logger.debug("wait that all batchlogs are replayed")
> > self._replay_batchlogs()
> materialized_views_test.py:350:
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> _
> self = 0x7fcb232b5760>
> def _replay_batchlogs(self):
> for node in self.cluster.nodelist():
> if node.is_running():
> logger.debug("Replaying batchlog on node
> {}".format(node.name))
> node.nodetool("replaybatchlog")
> # CASSANDRA-13069 - Ensure replayed mutations are removed
> from the batchlog
> node_session = self.patient_exclusive_cql_connection(node)
> result = list(node_session.execute("SELECT count(*) FROM
> system.batches;"))
> > assert result[0].count == 0
> E assert 160 == 0
> E+ where 160 = Row(count=160).count
> materialized_views_test.py:171: AssertionError{quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-8272) 2ndary indexes can return stale data
[
https://issues.apache.org/jira/browse/CASSANDRA-8272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17124677#comment-17124677
]
Caleb Rackliffe commented on CASSANDRA-8272:
[~adelapena] Given that {{TestRepair.test_dead_sync_initiator}} had already
failed during [CASSANDRA-13606
development|https://ci-cassandra.apache.org/view/all/job/Cassandra-devbranch-dtest/126/parameters/],
it looks like this will be ready to commit once we revert the CircleCI branch
changes?
> 2ndary indexes can return stale data
>
>
> Key: CASSANDRA-8272
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8272
> Project: Cassandra
> Issue Type: Bug
> Components: Feature/2i Index
>Reporter: Sylvain Lebresne
>Assignee: Andres de la Peña
>Priority: Normal
> Labels: pull-request-available
> Fix For: 3.0.x, 3.11.x, 4.0-beta
>
> Time Spent: 5h 20m
> Remaining Estimate: 0h
>
> When replica return 2ndary index results, it's possible for a single replica
> to return a stale result and that result will be sent back to the user,
> potentially failing the CL contract.
> For instance, consider 3 replicas A, B and C, and the following situation:
> {noformat}
> CREATE TABLE test (k int PRIMARY KEY, v text);
> CREATE INDEX ON test(v);
> INSERT INTO test(k, v) VALUES (0, 'foo');
> {noformat}
> with every replica up to date. Now, suppose that the following queries are
> done at {{QUORUM}}:
> {noformat}
> UPDATE test SET v = 'bar' WHERE k = 0;
> SELECT * FROM test WHERE v = 'foo';
> {noformat}
> then, if A and B acknowledge the insert but C respond to the read before
> having applied the insert, then the now stale result will be returned (since
> C will return it and A or B will return nothing).
> A potential solution would be that when we read a tombstone in the index (and
> provided we make the index inherit the gcGrace of it's parent CF), instead of
> skipping that tombstone, we'd insert in the result a corresponding range
> tombstone.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15835) Upgrade-dtests on trunk not working in CircleCI
[
https://issues.apache.org/jira/browse/CASSANDRA-15835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eduard Tudenhoefner updated CASSANDRA-15835:
Reviewers: Eduard Tudenhoefner, Eduard Tudenhoefner (was: Eduard
Tudenhoefner)
Eduard Tudenhoefner, Eduard Tudenhoefner (was: Eduard
Tudenhoefner)
Status: Review In Progress (was: Patch Available)
> Upgrade-dtests on trunk not working in CircleCI
> ---
>
> Key: CASSANDRA-15835
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15835
> Project: Cassandra
> Issue Type: Bug
> Components: CI, Test/dtest
>Reporter: Robert Stupp
>Assignee: Robert Stupp
>Priority: Normal
> Fix For: 4.0-alpha
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> ~3600 Upgrade-dtests are failing in CircleCI for trunk due to the missing
> {{JAVA8_HOME}} environment variable.
> Patching the Docker image is rather simple by creating a new image:
> {code}
> FROM nastra/cassandra-testing-ubuntu1910-java11-w-dependencies:20200406
> ENV JAVA8_HOME=/usr/lib/jvm/java-8-openjdk-amd64
> {code}
> Pushed the above to Docker hub as
> [snazy/cassandra-testing-ubuntu1910-java11-w-dependencies:202005261540|https://hub.docker.com/layers/snazy/cassandra-testing-ubuntu1910-java11-w-dependencies/202005261540/images/sha256-ac8a713be58694f095c491921e006c2d1a7823a3c23299e477198e2c93a6bbd7?context=explore]
> The size of the whole Docker image is a little concerning though (1.85G
> compressed), but that's out of the scope of this ticket.
> I'll prepare a patch soon-ish.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
