[jira] [Commented] (CASSANDRA-15052) Dtests: Add acceptable warnings to offline tool tests in order to pass them
[
https://issues.apache.org/jira/browse/CASSANDRA-15052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16806294#comment-16806294
]
Stefan Miklosovic commented on CASSANDRA-15052:
---
[~djoshi3]
please see the attached log against your trunk and WITHOUT this fix
[^SPICE-15052.txt]
I think the issue is this:
{code:java}
Small commitlog volume detected at /tmp/dtest-78woi6j_/test/node1/commitlogs;
setting commitlog_to...21.000GiB free across all data volumes. Consider adding
more capacity to your cluster or removing obsolete snapshots\n'
...
WARN 10:16:28,580 Small commitlog volume detected at
/tmp/dtest-78woi6j_/test/node1/commitlogs; setting commitlog_total_space_in_mb
to 7428. You can override this in cassandra.yaml
WARN 10:16:28,583 Small cdc volume detected at
/tmp/dtest-78woi6j_/test/node1/cdc_raw; setting cdc_total_space_in_mb to 3714.
You can override this in cassandra.yaml
WARN 10:16:28,584 Only 21.000GiB free across all data volumes. Consider adding
more capacity to your cluster or removing obsolete snapshots
{code}
Which is true, but how did it compute 21 GiB specifically? Seems like it summed
up /run with / and tmpfs, thats rather misleading imho.
{code:java}
(venv) smiklosovic@ip-172-31-8-142:~/temp/cassandra-dtest$ df -h
Filesystem Size Used Avail Use% Mounted on
udev 35G 0 35G 0% /dev
tmpfs 6.9G 876K 6.9G 1% /run
/dev/nvme0n1p1 30G 23G 7.0G 76% /
tmpfs35G 0 35G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs35G 0 35G 0% /sys/fs/cgroup
/dev/loop0 13M 13M 0 100% /snap/amazon-ssm-agent/295
/dev/loop1 18M 18M 0 100% /snap/amazon-ssm-agent/1068
/dev/loop3 91M 91M 0 100% /snap/core/6405
/dev/loop4 92M 92M 0 100% /snap/core/6531
/dev/loop5 90M 90M 0 100% /snap/core/6673
tmpfs 6.9G 0 6.9G 0% /run/user/1000
{code}
What is the minimum here and how is it computed? There is not any mention about
the available space for dtests to allocate to run tests fine. We should either
document what diskspace dtests requires or we should suppress this warning or
we should change the values of these properties. Dtest runs fine if we include
that message among expected warning-ones.
> Dtests: Add acceptable warnings to offline tool tests in order to pass them
> ---
>
> Key: CASSANDRA-15052
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15052
> Project: Cassandra
> Issue Type: Improvement
> Components: Test/dtest
>Reporter: Stefan Miklosovic
>Assignee: Stefan Miklosovic
>Priority: Normal
> Labels: pull-request-available
> Attachments: SPICE-15052.txt
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> I run all dtest suite and test
> offline_tools_test.py::TestOfflineTools::test_sstablelevelreset has failed
> because of additional warning logs which were not added into acceptable ones.
> After adding them, test passed fine. I believe added warning messages have
> nothing to do with test itself, it was reproduced on c5.9xlarge as well as no
> "regular" notebook.
>
> https://github.com/apache/cassandra-dtest/pull/47
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15052) Dtests: Add acceptable warnings to offline tool tests in order to pass them
[ https://issues.apache.org/jira/browse/CASSANDRA-15052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Miklosovic updated CASSANDRA-15052: -- Attachment: SPICE-15052.txt > Dtests: Add acceptable warnings to offline tool tests in order to pass them > --- > > Key: CASSANDRA-15052 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15052 > Project: Cassandra > Issue Type: Improvement > Components: Test/dtest >Reporter: Stefan Miklosovic >Assignee: Stefan Miklosovic >Priority: Normal > Labels: pull-request-available > Attachments: SPICE-15052.txt > > Time Spent: 50m > Remaining Estimate: 0h > > I run all dtest suite and test > offline_tools_test.py::TestOfflineTools::test_sstablelevelreset has failed > because of additional warning logs which were not added into acceptable ones. > After adding them, test passed fine. I believe added warning messages have > nothing to do with test itself, it was reproduced on c5.9xlarge as well as no > "regular" notebook. > > https://github.com/apache/cassandra-dtest/pull/47 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15061) Dtests: tests are failing on too powerful machines, setting more memory per node in dtests
[ https://issues.apache.org/jira/browse/CASSANDRA-15061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16806286#comment-16806286 ] Stefan Miklosovic commented on CASSANDRA-15061: --- FYI there is an option, used in CCM (and in turn in CircleCI too) that overrides default values of MAX_HEAP_SIZE here (1). I was not running dtests with these properties initially so by default it is 512M as I was not overriding anything myself. I have used 1G and tests started to look better but some of them were still failing, after increasing to 4G it seemed to resolve the issue. (1) [https://github.com/apache/cassandra/blob/a85196246c009566aa838156f3f56d00145e76ad/.circleci/config.yml#L84-L85] > Dtests: tests are failing on too powerful machines, setting more memory per > node in dtests > -- > > Key: CASSANDRA-15061 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15061 > Project: Cassandra > Issue Type: Improvement > Components: Test/dtest >Reporter: Stefan Miklosovic >Priority: Normal > Labels: CI, dtest, test > > While running dtests on 32 cores and 64 GB of memory on c5.9xlarge some tests > are failing because they are not able to handle the stress cassandra-stress > is generating for them. > For all examples, there is e.g. this one (1) where we test that a cluster is > able to cope with a boostrapping node. The problem is that node1 is bombed > with cassandra-stress and it is eventually killed and test fails as such > before even proceeding to test itself. > It was said to me that dtests in circleci are running in containers with 8 > cores and 16GB or RAM and I simulated this on my machine > (-Dcassandra.available_processors=8). The core problem is that nodes do not > have enough memory - Xmx and Xms is set to only 512MB and that is very low > figure and they are eventually killed. > Proposed solutions: > 1) Run dtests on less powerful machines so it can not handle stress high > enough so underlying nodes would be killed (rather strange idea) > 2) Increase memory for node - this should be configurable, I saw that 1GB > helps but there are still some timeouts, 2GB is better. 4GB would be the best. > 3) Fix the test in such way it does not fail with 512MB. > > 1) is not viable to me, 3) takes a lot of time to go through and does not > actually solve anything and it would be very cumbersome and clunky to go > through all tests to set them like that. 2) seems to be the best approach but > there is not any way I am aware of how to add more memory to every node all > at once as node and cluster start / creation is scattered all over the > project. > I have raised the issue here (2) too. > Do you guys think that if we manage to somehow fix this in CCM, we could > introduce some switch / flag to dtests as how much memory a node in a cluster > should run with? > (1) > [https://github.com/apache/cassandra-dtest/blob/master/bootstrap_test.py#L419-L470] > (2) [https://github.com/riptano/ccm/issues/696] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-15065) Remove traces of dclocal_read_repair_chance from dtests as it is not used anymore
Stefan Miklosovic created CASSANDRA-15065:
-
Summary: Remove traces of dclocal_read_repair_chance from dtests
as it is not used anymore
Key: CASSANDRA-15065
URL: https://issues.apache.org/jira/browse/CASSANDRA-15065
Project: Cassandra
Issue Type: Task
Components: Test/dtest
Reporter: Stefan Miklosovic
This test as of now fails consistently because property
"dclocal_read_repair_chance" does not exist anymore. It was marked as "not
used" as part of CASSANDRA-13910 and it fails in current trunk in
snitch_test.py TestDynamicEndpointSnitchtest multidatacenter_local_quorum
In test, it was set to 0.0 too so I am not sure what is the benefit of having
that property there (even it is not recognised anymore)
{code:java}
./src/java/org/apache/cassandra/schema/SchemaKeyspace.java:129: +
"dclocal_read_repair_chance double," // no longer used, left for drivers' sake
./src/java/org/apache/cassandra/schema/SchemaKeyspace.java:196: +
"dclocal_read_repair_chance double," // no longer used, left for drivers' sake
./src/java/org/apache/cassandra/schema/SchemaKeyspace.java:560:
.add("dclocal_read_repair_chance", 0.0) // no longer used, left for drivers'
sake{code}
{code:java}
E cassandra.protocol.SyntaxException: {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-15061) Dtests: tests are failing on too powerful machines, setting more memory per node in dtests
Stefan Miklosovic created CASSANDRA-15061: - Summary: Dtests: tests are failing on too powerful machines, setting more memory per node in dtests Key: CASSANDRA-15061 URL: https://issues.apache.org/jira/browse/CASSANDRA-15061 Project: Cassandra Issue Type: Improvement Components: Test/dtest Reporter: Stefan Miklosovic While running dtests on 32 cores and 64 GB of memory on c5.9xlarge some tests are failing because they are not able to handle the stress cassandra-stress is generating for them. For all examples, there is e.g. this one (1) where we test that a cluster is able to cope with a boostrapping node. The problem is that node1 is bombed with cassandra-stress and it is eventually killed and test fails as such before even proceeding to test itself. It was said to me that dtests in circleci are running in containers with 8 cores and 16GB or RAM and I simulated this on my machine (-Dcassandra.available_processors=8). The core problem is that nodes do not have enough memory - Xmx and Xms is set to only 512MB and that is very low figure and they are eventually killed. Proposed solutions: 1) Run dtests on less powerful machines so it can not handle stress high enough so underlying nodes would be killed (rather strange idea) 2) Increase memory for node - this should be configurable, I saw that 1GB helps but there are still some timeouts, 2GB is better. 4GB would be the best. 3) Fix the test in such way it does not fail with 512MB. 1) is not viable to me, 3) takes a lot of time to go through and does not actually solve anything and it would be very cumbersome and clunky to go through all tests to set them like that. 2) seems to be the best approach but there is not any way I am aware of how to add more memory to every node all at once as node and cluster start / creation is scattered all over the project. I have raised the issue here (2) too. Do you guys think that if we manage to somehow fix this in CCM, we could introduce some switch / flag to dtests as how much memory a node in a cluster should run with? (1) [https://github.com/apache/cassandra-dtest/blob/master/bootstrap_test.py#L419-L470] (2) [https://github.com/riptano/ccm/issues/696] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Assigned] (CASSANDRA-14903) Nodetool cfstats prints index name twice
[
https://issues.apache.org/jira/browse/CASSANDRA-14903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Miklosovic reassigned CASSANDRA-14903:
-
Assignee: Stefan Miklosovic (was: Ian Cleasby)
> Nodetool cfstats prints index name twice
>
>
> Key: CASSANDRA-14903
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14903
> Project: Cassandra
> Issue Type: Bug
> Components: Tool/nodetool
>Reporter: Kurt Greaves
>Assignee: Stefan Miklosovic
>Priority: Low
> Fix For: 4.0
>
>
> {code:java}
> CREATE TABLE test.test (
> id int PRIMARY KEY,
> data text
> );
> CREATE INDEX test_data_idx ON test.test (data);
> ccm node1 nodetool cfstats test
> Total number of tables: 40
>
> Keyspace : test
> Read Count: 0
> Read Latency: NaN ms
> Write Count: 0
> Write Latency: NaN ms
> Pending Flushes: 0
> Table (index): test.test_data_idxtest.test_data_idx
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-14903) Nodetool cfstats prints index name twice
[
https://issues.apache.org/jira/browse/CASSANDRA-14903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16810354#comment-16810354
]
Stefan Miklosovic edited comment on CASSANDRA-14903 at 4/4/19 10:38 PM:
Hi [~michaelsembwever] no worries, I ll cover this (I work with [~icleasby]
closely)
was (Author: stefan.miklosovic):
Hi [~michaelsembwever] no worries, I ll cover this.
> Nodetool cfstats prints index name twice
>
>
> Key: CASSANDRA-14903
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14903
> Project: Cassandra
> Issue Type: Bug
> Components: Tool/nodetool
>Reporter: Kurt Greaves
>Assignee: Stefan Miklosovic
>Priority: Low
> Fix For: 4.0
>
>
> {code:java}
> CREATE TABLE test.test (
> id int PRIMARY KEY,
> data text
> );
> CREATE INDEX test_data_idx ON test.test (data);
> ccm node1 nodetool cfstats test
> Total number of tables: 40
>
> Keyspace : test
> Read Count: 0
> Read Latency: NaN ms
> Write Count: 0
> Write Latency: NaN ms
> Pending Flushes: 0
> Table (index): test.test_data_idxtest.test_data_idx
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14903) Nodetool cfstats prints index name twice
[
https://issues.apache.org/jira/browse/CASSANDRA-14903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16810354#comment-16810354
]
Stefan Miklosovic commented on CASSANDRA-14903:
---
Hi [~michaelsembwever] no worries, I ll cover this.
> Nodetool cfstats prints index name twice
>
>
> Key: CASSANDRA-14903
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14903
> Project: Cassandra
> Issue Type: Bug
> Components: Tool/nodetool
>Reporter: Kurt Greaves
>Assignee: Ian Cleasby
>Priority: Low
> Fix For: 4.0
>
>
> {code:java}
> CREATE TABLE test.test (
> id int PRIMARY KEY,
> data text
> );
> CREATE INDEX test_data_idx ON test.test (data);
> ccm node1 nodetool cfstats test
> Total number of tables: 40
>
> Keyspace : test
> Read Count: 0
> Read Latency: NaN ms
> Write Count: 0
> Write Latency: NaN ms
> Pending Flushes: 0
> Table (index): test.test_data_idxtest.test_data_idx
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Assigned] (CASSANDRA-14712) Cassandra 4.0 packaging support
[ https://issues.apache.org/jira/browse/CASSANDRA-14712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Miklosovic reassigned CASSANDRA-14712: - Assignee: Stefan Miklosovic > Cassandra 4.0 packaging support > --- > > Key: CASSANDRA-14712 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14712 > Project: Cassandra > Issue Type: Bug > Components: Packaging >Reporter: Stefan Podkowinski >Assignee: Stefan Miklosovic >Priority: Normal > Labels: Java11, pull-request-available > Fix For: 4.x > > Time Spent: 2h 10m > Remaining Estimate: 0h > > Currently it's not possible to build any native packages (.deb/.rpm) for > trunk. > cassandra-builds - docker/*-image.docker > * Add Java11 to debian+centos build image > * (packaged ant scripts won't work with Java 11 on centos, so we may have to > install ant from tarballs) > cassandra-builds - docker/build-*.sh > * set JAVA8_HOME to Java8 > * set JAVA_HOME to Java11 (4.0) or Java8 (<4.0) > cassandra - redhat/cassandra.spec > * Check if patches still apply after CASSANDRA-14707 > * Add fqltool as %files > We may also have to change the version handling in build.xml or build-*.sh, > depending how we plan to release packages during beta, or if we plan to do so > at all before GA. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14712) Cassandra 4.0 packaging support
[ https://issues.apache.org/jira/browse/CASSANDRA-14712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16804435#comment-16804435 ] Stefan Miklosovic commented on CASSANDRA-14712: --- [~spo...@gmail.com] could you please review both PRs and tell me whatever else you would like to add or modify? I have tested that RPMs and DEBs are installing ok in the latest RHEL7 and Debian Stretch 9 and fqltool as well as auditlogviewer are available to operator. There was not need to package custom Ant (1.10.5) from tarball, Ant packages from repositories which are still of version 1.9 worked just fine. Debian Jesse was in my case so old that backport repository was not existing anymore (it gave me 404 on apt update) so I updated it to Debian Stretch. We could also update version of CentOS to 7.6 instead of the current 7.0 just to be updated. I saw the warning about the deprecation of Python 2.7 but it was still building fine in the end. I am still not completely sure whats the deal with Java 8 / Java 11 combo, as per conversation with [~bdeggleston] here (1) he was able to build trunk in both cases on 8 or 11 only. There is currently still a prerequisite to build trunk on Java 11 to get distribution tarballs and it does not make a lot of sense to me why it is the case if one is reportedly able to build trunk with 8 only. (2) (1) [https://lists.apache.org/thread.html/0501ad77c74f5bc83e84e4d31a063ad09b60a4d9b42112646d97e7e8@%3Cdev.cassandra.apache.org%3E] (2) [https://github.com/apache/cassandra/blob/trunk/build.xml#L1049-L1051] > Cassandra 4.0 packaging support > --- > > Key: CASSANDRA-14712 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14712 > Project: Cassandra > Issue Type: Bug > Components: Packaging >Reporter: Stefan Podkowinski >Assignee: Stefan Miklosovic >Priority: Normal > Labels: Java11, pull-request-available > Fix For: 4.x > > Time Spent: 2h 10m > Remaining Estimate: 0h > > Currently it's not possible to build any native packages (.deb/.rpm) for > trunk. > cassandra-builds - docker/*-image.docker > * Add Java11 to debian+centos build image > * (packaged ant scripts won't work with Java 11 on centos, so we may have to > install ant from tarballs) > cassandra-builds - docker/build-*.sh > * set JAVA8_HOME to Java8 > * set JAVA_HOME to Java11 (4.0) or Java8 (<4.0) > cassandra - redhat/cassandra.spec > * Check if patches still apply after CASSANDRA-14707 > * Add fqltool as %files > We may also have to change the version handling in build.xml or build-*.sh, > depending how we plan to release packages during beta, or if we plan to do so > at all before GA. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15052) Dtests: Add acceptable warnings to offline tool tests in order to pass them
[ https://issues.apache.org/jira/browse/CASSANDRA-15052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Miklosovic updated CASSANDRA-15052: -- Status: Triage Needed (was: Awaiting Feedback) replied > Dtests: Add acceptable warnings to offline tool tests in order to pass them > --- > > Key: CASSANDRA-15052 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15052 > Project: Cassandra > Issue Type: Improvement > Components: Test/dtest >Reporter: Stefan Miklosovic >Assignee: Stefan Miklosovic >Priority: Normal > Labels: pull-request-available > Attachments: SPICE-15052.txt > > Time Spent: 50m > Remaining Estimate: 0h > > I run all dtest suite and test > offline_tools_test.py::TestOfflineTools::test_sstablelevelreset has failed > because of additional warning logs which were not added into acceptable ones. > After adding them, test passed fine. I believe added warning messages have > nothing to do with test itself, it was reproduced on c5.9xlarge as well as no > "regular" notebook. > > https://github.com/apache/cassandra-dtest/pull/47 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-15052) Dtests: Add acceptable warnings to offline tool tests in order to pass them
Stefan Miklosovic created CASSANDRA-15052: - Summary: Dtests: Add acceptable warnings to offline tool tests in order to pass them Key: CASSANDRA-15052 URL: https://issues.apache.org/jira/browse/CASSANDRA-15052 Project: Cassandra Issue Type: Improvement Components: Test/dtest Reporter: Stefan Miklosovic I run all dtest suite and test offline_tools_test.py::TestOfflineTools::test_sstablelevelreset has failed because of additional warning logs which were not added into acceptable ones. After adding them, test passed fine. I believe added warning messages have nothing to do with test itself, it was reproduced on c5.9xlarge as well as no "regular" notebook. https://github.com/apache/cassandra-dtest/pull/47 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13158) Nodetool cleanup throwing exception
[ https://issues.apache.org/jira/browse/CASSANDRA-13158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17038513#comment-17038513 ] Stefan Miklosovic commented on CASSANDRA-13158: --- This is still happening in 4.0-alpha3 > Nodetool cleanup throwing exception > --- > > Key: CASSANDRA-13158 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13158 > Project: Cassandra > Issue Type: Bug > Components: Tool/nodetool > Environment: Fedora 25 x86 >Reporter: Tomas Repik >Assignee: Eduard Tudenhoefner >Priority: Normal > Fix For: 4.0 > > > After running nodetool cleanup I get this exception: > error: Invalid partitioner class org.apache.cassandra.dht.Murmur3Partitioner > -- StackTrace -- > org.apache.cassandra.exceptions.ConfigurationException: Invalid partitioner > class org.apache.cassandra.dht.Murmur3Partitioner > at > org.apache.cassandra.config.DatabaseDescriptor.applyConfig(DatabaseDescriptor.java:383) > at > org.apache.cassandra.config.DatabaseDescriptor.(DatabaseDescriptor.java:125) > at > org.apache.cassandra.cql3.QueryProcessor.(QueryProcessor.java:84) > at org.apache.cassandra.config.CFMetaData.compile(CFMetaData.java:411) > at > org.apache.cassandra.schema.SchemaKeyspace.compile(SchemaKeyspace.java:240) > at > org.apache.cassandra.schema.SchemaKeyspace.(SchemaKeyspace.java:88) > at org.apache.cassandra.config.Schema.(Schema.java:107) > at org.apache.cassandra.config.Schema.(Schema.java:55) > at org.apache.cassandra.tools.nodetool.Cleanup.execute(Cleanup.java:50) > at > org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:251) > at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:165) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15578) Describe keyspace does not work for keyspaces with transient replication
[
https://issues.apache.org/jira/browse/CASSANDRA-15578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Miklosovic updated CASSANDRA-15578:
--
Description:
I can not describe a keyspace which is using transient replication.
{code:java}
admin@cqlsh> CREATE KEYSPACE bar WITH replication = {'class':
'NetworkTopologyStrategy', 'dc1': '3/1'};
admin@cqlsh> DESCRIBE KEYSPACE bar;
'NoneType' object has no attribute 'export_for_schema'
{code}
I am sorry in case this is a duplicate but I have not found similar issue yet.
This was tried on 4.0-alpha3
was:
I can not describe a keyspace which is using transient replication.
{code}
admin@cqlsh> CREATE KEYSPACE bar WITH replication = {'class':
'NetworkTopologyStrategy', 'dc1': '3/1'};
admin@cqlsh> DESCRIBE KEYSPACE bar;
'NoneType' object has no attribute 'export_for_schema'
{code}
I am sorry in case this is a duplicate but I have not found similar issue yet.
> Describe keyspace does not work for keyspaces with transient replication
>
>
> Key: CASSANDRA-15578
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15578
> Project: Cassandra
> Issue Type: Bug
> Components: CQL/Interpreter
>Reporter: Stefan Miklosovic
>Priority: Normal
> Labels: 4.0
>
> I can not describe a keyspace which is using transient replication.
> {code:java}
> admin@cqlsh> CREATE KEYSPACE bar WITH replication = {'class':
> 'NetworkTopologyStrategy', 'dc1': '3/1'};
> admin@cqlsh> DESCRIBE KEYSPACE bar;
> 'NoneType' object has no attribute 'export_for_schema'
> {code}
> I am sorry in case this is a duplicate but I have not found similar issue yet.
>
> This was tried on 4.0-alpha3
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-15578) Describe keyspace does not work for keyspaces with transient replication
Stefan Miklosovic created CASSANDRA-15578:
-
Summary: Describe keyspace does not work for keyspaces with
transient replication
Key: CASSANDRA-15578
URL: https://issues.apache.org/jira/browse/CASSANDRA-15578
Project: Cassandra
Issue Type: Bug
Components: CQL/Interpreter
Reporter: Stefan Miklosovic
I can not describe a keyspace which is using transient replication.
{code}
admin@cqlsh> CREATE KEYSPACE bar WITH replication = {'class':
'NetworkTopologyStrategy', 'dc1': '3/1'};
admin@cqlsh> DESCRIBE KEYSPACE bar;
'NoneType' object has no attribute 'export_for_schema'
{code}
I am sorry in case this is a duplicate but I have not found similar issue yet.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15578) Describe keyspace does not work for keyspaces with transient replication
[
https://issues.apache.org/jira/browse/CASSANDRA-15578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Miklosovic updated CASSANDRA-15578:
--
Labels: 4.0 (was: )
> Describe keyspace does not work for keyspaces with transient replication
>
>
> Key: CASSANDRA-15578
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15578
> Project: Cassandra
> Issue Type: Bug
> Components: CQL/Interpreter
>Reporter: Stefan Miklosovic
>Priority: Normal
> Labels: 4.0
>
> I can not describe a keyspace which is using transient replication.
> {code}
> admin@cqlsh> CREATE KEYSPACE bar WITH replication = {'class':
> 'NetworkTopologyStrategy', 'dc1': '3/1'};
> admin@cqlsh> DESCRIBE KEYSPACE bar;
> 'NoneType' object has no attribute 'export_for_schema'
> {code}
> I am sorry in case this is a duplicate but I have not found similar issue yet.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13158) Nodetool cleanup throwing exception
[ https://issues.apache.org/jira/browse/CASSANDRA-13158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17038895#comment-17038895 ] Stefan Miklosovic commented on CASSANDRA-13158: --- [~dcapwell] I have to admit I am using some "custom" scripts for invoking nodetool but I was trying to be as close as possible to out-of-the-box setting and it is happening. I can try to give you the exact / raw command this issue occurs with once I get to it. Why is this even happening? That error is very strange. What does have murmur in common with some cleanup? Dont you know _why_ is this a thing in the first place? > Nodetool cleanup throwing exception > --- > > Key: CASSANDRA-13158 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13158 > Project: Cassandra > Issue Type: Bug > Components: Tool/nodetool > Environment: Fedora 25 x86 >Reporter: Tomas Repik >Assignee: Eduard Tudenhoefner >Priority: Normal > Fix For: 4.0 > > > After running nodetool cleanup I get this exception: > error: Invalid partitioner class org.apache.cassandra.dht.Murmur3Partitioner > -- StackTrace -- > org.apache.cassandra.exceptions.ConfigurationException: Invalid partitioner > class org.apache.cassandra.dht.Murmur3Partitioner > at > org.apache.cassandra.config.DatabaseDescriptor.applyConfig(DatabaseDescriptor.java:383) > at > org.apache.cassandra.config.DatabaseDescriptor.(DatabaseDescriptor.java:125) > at > org.apache.cassandra.cql3.QueryProcessor.(QueryProcessor.java:84) > at org.apache.cassandra.config.CFMetaData.compile(CFMetaData.java:411) > at > org.apache.cassandra.schema.SchemaKeyspace.compile(SchemaKeyspace.java:240) > at > org.apache.cassandra.schema.SchemaKeyspace.(SchemaKeyspace.java:88) > at org.apache.cassandra.config.Schema.(Schema.java:107) > at org.apache.cassandra.config.Schema.(Schema.java:55) > at org.apache.cassandra.tools.nodetool.Cleanup.execute(Cleanup.java:50) > at > org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:251) > at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:165) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-13158) Nodetool cleanup throwing exception
[ https://issues.apache.org/jira/browse/CASSANDRA-13158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17038895#comment-17038895 ] Stefan Miklosovic edited comment on CASSANDRA-13158 at 2/18/20 8:49 AM: [~dcapwell] I have to admit I am using some "custom" scripts for invoking nodetool but I was trying to be as close as possible to out-of-the-box setting and it is happening. I can try to give you the exact / raw command this issue occurs with once I get to it. Why is this even happening? That error is very strange. What does murmur have in common with some cleanup? Dont you know _why_ is this a thing in the first place? was (Author: stefan.miklosovic): [~dcapwell] I have to admit I am using some "custom" scripts for invoking nodetool but I was trying to be as close as possible to out-of-the-box setting and it is happening. I can try to give you the exact / raw command this issue occurs with once I get to it. Why is this even happening? That error is very strange. What does have murmur in common with some cleanup? Dont you know _why_ is this a thing in the first place? > Nodetool cleanup throwing exception > --- > > Key: CASSANDRA-13158 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13158 > Project: Cassandra > Issue Type: Bug > Components: Tool/nodetool > Environment: Fedora 25 x86 >Reporter: Tomas Repik >Assignee: Eduard Tudenhoefner >Priority: Normal > Fix For: 4.0 > > > After running nodetool cleanup I get this exception: > error: Invalid partitioner class org.apache.cassandra.dht.Murmur3Partitioner > -- StackTrace -- > org.apache.cassandra.exceptions.ConfigurationException: Invalid partitioner > class org.apache.cassandra.dht.Murmur3Partitioner > at > org.apache.cassandra.config.DatabaseDescriptor.applyConfig(DatabaseDescriptor.java:383) > at > org.apache.cassandra.config.DatabaseDescriptor.(DatabaseDescriptor.java:125) > at > org.apache.cassandra.cql3.QueryProcessor.(QueryProcessor.java:84) > at org.apache.cassandra.config.CFMetaData.compile(CFMetaData.java:411) > at > org.apache.cassandra.schema.SchemaKeyspace.compile(SchemaKeyspace.java:240) > at > org.apache.cassandra.schema.SchemaKeyspace.(SchemaKeyspace.java:88) > at org.apache.cassandra.config.Schema.(Schema.java:107) > at org.apache.cassandra.config.Schema.(Schema.java:55) > at org.apache.cassandra.tools.nodetool.Cleanup.execute(Cleanup.java:50) > at > org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:251) > at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:165) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15578) Describe keyspace does not work for keyspaces with transient replication
[
https://issues.apache.org/jira/browse/CASSANDRA-15578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17038924#comment-17038924
]
Stefan Miklosovic commented on CASSANDRA-15578:
---
Hi [~aweisberg], I merely remember you were the one most active behind
transient replication so I guess mentioning you here is a good idea to achieve
some visibility to this issue.
> Describe keyspace does not work for keyspaces with transient replication
>
>
> Key: CASSANDRA-15578
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15578
> Project: Cassandra
> Issue Type: Bug
> Components: CQL/Interpreter
>Reporter: Stefan Miklosovic
>Priority: Normal
> Labels: 4.0
>
> I can not describe a keyspace which is using transient replication.
> {code:java}
> admin@cqlsh> CREATE KEYSPACE bar WITH replication = {'class':
> 'NetworkTopologyStrategy', 'dc1': '3/1'};
> admin@cqlsh> DESCRIBE KEYSPACE bar;
> 'NoneType' object has no attribute 'export_for_schema'
> {code}
> I am sorry in case this is a duplicate but I have not found similar issue yet.
>
> This was tried on 4.0-alpha3
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13158) Nodetool cleanup throwing exception
[ https://issues.apache.org/jira/browse/CASSANDRA-13158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17039394#comment-17039394 ] Stefan Miklosovic commented on CASSANDRA-13158: --- I uploaded error.txt file. this cassandra.config.loader = com.instaclustr.cassandra.k8s.ConcatenatedYamlConfigurationLoader is this: [https://github.com/instaclustr/cassandra-operator/blob/master/java/cassandra-4-k8s-addons/src/main/java/com/instaclustr/cassandra/k8s/ConcatenatedYamlConfigurationLoader.java] It basically just concatenates yaml fragments from multiple files into one file. I do not want to go into this here (if it is not important). > Nodetool cleanup throwing exception > --- > > Key: CASSANDRA-13158 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13158 > Project: Cassandra > Issue Type: Bug > Components: Tool/nodetool > Environment: Fedora 25 x86 >Reporter: Tomas Repik >Assignee: Eduard Tudenhoefner >Priority: Normal > Fix For: 4.0 > > Attachments: error.txt > > > After running nodetool cleanup I get this exception: > error: Invalid partitioner class org.apache.cassandra.dht.Murmur3Partitioner > -- StackTrace -- > org.apache.cassandra.exceptions.ConfigurationException: Invalid partitioner > class org.apache.cassandra.dht.Murmur3Partitioner > at > org.apache.cassandra.config.DatabaseDescriptor.applyConfig(DatabaseDescriptor.java:383) > at > org.apache.cassandra.config.DatabaseDescriptor.(DatabaseDescriptor.java:125) > at > org.apache.cassandra.cql3.QueryProcessor.(QueryProcessor.java:84) > at org.apache.cassandra.config.CFMetaData.compile(CFMetaData.java:411) > at > org.apache.cassandra.schema.SchemaKeyspace.compile(SchemaKeyspace.java:240) > at > org.apache.cassandra.schema.SchemaKeyspace.(SchemaKeyspace.java:88) > at org.apache.cassandra.config.Schema.(Schema.java:107) > at org.apache.cassandra.config.Schema.(Schema.java:55) > at org.apache.cassandra.tools.nodetool.Cleanup.execute(Cleanup.java:50) > at > org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:251) > at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:165) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13158) Nodetool cleanup throwing exception
[
https://issues.apache.org/jira/browse/CASSANDRA-13158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17039343#comment-17039343
]
Stefan Miklosovic commented on CASSANDRA-13158:
---
I followed the code code and quickly ...
{code:java}
// definitely not safe for tools + clients - implicitly instantiates schema
public static void applyPartitioner()
{
/* Hashing strategy */
if (conf.partitioner == null)
{
throw new ConfigurationException("Missing directive: partitioner",
false);
}
try
{
partitioner =
FBUtilities.newPartitioner(System.getProperty(Config.PROPERTY_PREFIX +
"partitioner", conf.partitioner));
}
catch (Exception e)
{
// IT THROWS THIS so try is bad
throw new ConfigurationException("Invalid partitioner class " +
conf.partitioner, false);
}
paritionerName = partitioner.getClass().getCanonicalName();
}
{code}
newPartitioner in FBUtilities does this:
{code:java}
static IPartitioner newPartitioner(String partitionerClassName,
Optional> comparator) throws ConfigurationException
{
if (!partitionerClassName.contains("."))
partitionerClassName = "org.apache.cassandra.dht." +
partitionerClassName;
if
(partitionerClassName.equals("org.apache.cassandra.dht.LocalPartitioner"))
{
assert comparator.isPresent() : "Expected a comparator for local
partitioner";
return new LocalPartitioner(comparator.get());
}
return FBUtilities.instanceOrConstruct(partitionerClassName, "partitioner");
}
{code}
Hence finally, it constructs it like this:
{code:java}
public static T instanceOrConstruct(String classname, String readable)
throws ConfigurationException
{
Class cls = FBUtilities.classForName(classname, readable);
try
{
Field instance = cls.getField("instance");
return cls.cast(instance.get(null));
}
catch (NoSuchFieldException | SecurityException | IllegalArgumentException
| IllegalAccessException e)
{
// Could not get instance field. Try instantiating.
return construct(cls, classname, readable);
}
}
{code}
The only place where it can throw is either in classForName, or in getField or
in cast and finally in construct method. I would bet that it is not in
"cls.getField" because that field exists on murmur instance and it points to
object so it it can not call construct but why? Too bad that we are not
propagaing underlying exception in that first ConfigurationException throw ...
> Nodetool cleanup throwing exception
> ---
>
> Key: CASSANDRA-13158
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13158
> Project: Cassandra
> Issue Type: Bug
> Components: Tool/nodetool
> Environment: Fedora 25 x86
>Reporter: Tomas Repik
>Assignee: Eduard Tudenhoefner
>Priority: Normal
> Fix For: 4.0
>
>
> After running nodetool cleanup I get this exception:
> error: Invalid partitioner class org.apache.cassandra.dht.Murmur3Partitioner
> -- StackTrace --
> org.apache.cassandra.exceptions.ConfigurationException: Invalid partitioner
> class org.apache.cassandra.dht.Murmur3Partitioner
> at
> org.apache.cassandra.config.DatabaseDescriptor.applyConfig(DatabaseDescriptor.java:383)
> at
> org.apache.cassandra.config.DatabaseDescriptor.(DatabaseDescriptor.java:125)
> at
> org.apache.cassandra.cql3.QueryProcessor.(QueryProcessor.java:84)
> at org.apache.cassandra.config.CFMetaData.compile(CFMetaData.java:411)
> at
> org.apache.cassandra.schema.SchemaKeyspace.compile(SchemaKeyspace.java:240)
> at
> org.apache.cassandra.schema.SchemaKeyspace.(SchemaKeyspace.java:88)
> at org.apache.cassandra.config.Schema.(Schema.java:107)
> at org.apache.cassandra.config.Schema.(Schema.java:55)
> at org.apache.cassandra.tools.nodetool.Cleanup.execute(Cleanup.java:50)
> at
> org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:251)
> at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:165)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-13158) Nodetool cleanup throwing exception
[ https://issues.apache.org/jira/browse/CASSANDRA-13158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Miklosovic updated CASSANDRA-13158: -- Attachment: error.txt > Nodetool cleanup throwing exception > --- > > Key: CASSANDRA-13158 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13158 > Project: Cassandra > Issue Type: Bug > Components: Tool/nodetool > Environment: Fedora 25 x86 >Reporter: Tomas Repik >Assignee: Eduard Tudenhoefner >Priority: Normal > Fix For: 4.0 > > Attachments: error.txt > > > After running nodetool cleanup I get this exception: > error: Invalid partitioner class org.apache.cassandra.dht.Murmur3Partitioner > -- StackTrace -- > org.apache.cassandra.exceptions.ConfigurationException: Invalid partitioner > class org.apache.cassandra.dht.Murmur3Partitioner > at > org.apache.cassandra.config.DatabaseDescriptor.applyConfig(DatabaseDescriptor.java:383) > at > org.apache.cassandra.config.DatabaseDescriptor.(DatabaseDescriptor.java:125) > at > org.apache.cassandra.cql3.QueryProcessor.(QueryProcessor.java:84) > at org.apache.cassandra.config.CFMetaData.compile(CFMetaData.java:411) > at > org.apache.cassandra.schema.SchemaKeyspace.compile(SchemaKeyspace.java:240) > at > org.apache.cassandra.schema.SchemaKeyspace.(SchemaKeyspace.java:88) > at org.apache.cassandra.config.Schema.(Schema.java:107) > at org.apache.cassandra.config.Schema.(Schema.java:55) > at org.apache.cassandra.tools.nodetool.Cleanup.execute(Cleanup.java:50) > at > org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:251) > at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:165) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-13158) Nodetool cleanup throwing exception
[ https://issues.apache.org/jira/browse/CASSANDRA-13158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17039394#comment-17039394 ] Stefan Miklosovic edited comment on CASSANDRA-13158 at 2/18/20 7:35 PM: I uploaded error.txt file. this cassandra.config.loader = com.instaclustr.cassandra.k8s.ConcatenatedYamlConfigurationLoader is this: [https://github.com/instaclustr/cassandra-operator/blob/master/java/cassandra-4-k8s-addons/src/main/java/com/instaclustr/cassandra/k8s/ConcatenatedYamlConfigurationLoader.java] It basically just concatenates yaml fragments from multiple files into one file. I do not want to go into this here (if it is not important). >From the console output I see it is using Murmur partitioner. was (Author: stefan.miklosovic): I uploaded error.txt file. this cassandra.config.loader = com.instaclustr.cassandra.k8s.ConcatenatedYamlConfigurationLoader is this: [https://github.com/instaclustr/cassandra-operator/blob/master/java/cassandra-4-k8s-addons/src/main/java/com/instaclustr/cassandra/k8s/ConcatenatedYamlConfigurationLoader.java] It basically just concatenates yaml fragments from multiple files into one file. I do not want to go into this here (if it is not important). > Nodetool cleanup throwing exception > --- > > Key: CASSANDRA-13158 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13158 > Project: Cassandra > Issue Type: Bug > Components: Tool/nodetool > Environment: Fedora 25 x86 >Reporter: Tomas Repik >Assignee: Eduard Tudenhoefner >Priority: Normal > Fix For: 4.0 > > Attachments: error.txt > > > After running nodetool cleanup I get this exception: > error: Invalid partitioner class org.apache.cassandra.dht.Murmur3Partitioner > -- StackTrace -- > org.apache.cassandra.exceptions.ConfigurationException: Invalid partitioner > class org.apache.cassandra.dht.Murmur3Partitioner > at > org.apache.cassandra.config.DatabaseDescriptor.applyConfig(DatabaseDescriptor.java:383) > at > org.apache.cassandra.config.DatabaseDescriptor.(DatabaseDescriptor.java:125) > at > org.apache.cassandra.cql3.QueryProcessor.(QueryProcessor.java:84) > at org.apache.cassandra.config.CFMetaData.compile(CFMetaData.java:411) > at > org.apache.cassandra.schema.SchemaKeyspace.compile(SchemaKeyspace.java:240) > at > org.apache.cassandra.schema.SchemaKeyspace.(SchemaKeyspace.java:88) > at org.apache.cassandra.config.Schema.(Schema.java:107) > at org.apache.cassandra.config.Schema.(Schema.java:55) > at org.apache.cassandra.tools.nodetool.Cleanup.execute(Cleanup.java:50) > at > org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:251) > at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:165) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-13158) Nodetool cleanup throwing exception
[ https://issues.apache.org/jira/browse/CASSANDRA-13158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17040390#comment-17040390 ] Stefan Miklosovic edited comment on CASSANDRA-13158 at 2/19/20 8:08 PM: [~dcapwell] I am eager to try it but I am confused why you have closed that PR? was (Author: stefan.miklosovic): [~dcapwell] I am keen to try it but I am confused why you have closed that PR? > Nodetool cleanup throwing exception > --- > > Key: CASSANDRA-13158 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13158 > Project: Cassandra > Issue Type: Bug > Components: Tool/nodetool > Environment: Fedora 25 x86 >Reporter: Tomas Repik >Assignee: Eduard Tudenhoefner >Priority: Normal > Fix For: 4.0 > > Attachments: error.txt > > > After running nodetool cleanup I get this exception: > error: Invalid partitioner class org.apache.cassandra.dht.Murmur3Partitioner > -- StackTrace -- > org.apache.cassandra.exceptions.ConfigurationException: Invalid partitioner > class org.apache.cassandra.dht.Murmur3Partitioner > at > org.apache.cassandra.config.DatabaseDescriptor.applyConfig(DatabaseDescriptor.java:383) > at > org.apache.cassandra.config.DatabaseDescriptor.(DatabaseDescriptor.java:125) > at > org.apache.cassandra.cql3.QueryProcessor.(QueryProcessor.java:84) > at org.apache.cassandra.config.CFMetaData.compile(CFMetaData.java:411) > at > org.apache.cassandra.schema.SchemaKeyspace.compile(SchemaKeyspace.java:240) > at > org.apache.cassandra.schema.SchemaKeyspace.(SchemaKeyspace.java:88) > at org.apache.cassandra.config.Schema.(Schema.java:107) > at org.apache.cassandra.config.Schema.(Schema.java:55) > at org.apache.cassandra.tools.nodetool.Cleanup.execute(Cleanup.java:50) > at > org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:251) > at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:165) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13158) Nodetool cleanup throwing exception
[ https://issues.apache.org/jira/browse/CASSANDRA-13158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17040390#comment-17040390 ] Stefan Miklosovic commented on CASSANDRA-13158: --- [~dcapwell] I am keen to try it but I am confused why you have closed that PR? > Nodetool cleanup throwing exception > --- > > Key: CASSANDRA-13158 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13158 > Project: Cassandra > Issue Type: Bug > Components: Tool/nodetool > Environment: Fedora 25 x86 >Reporter: Tomas Repik >Assignee: Eduard Tudenhoefner >Priority: Normal > Fix For: 4.0 > > Attachments: error.txt > > > After running nodetool cleanup I get this exception: > error: Invalid partitioner class org.apache.cassandra.dht.Murmur3Partitioner > -- StackTrace -- > org.apache.cassandra.exceptions.ConfigurationException: Invalid partitioner > class org.apache.cassandra.dht.Murmur3Partitioner > at > org.apache.cassandra.config.DatabaseDescriptor.applyConfig(DatabaseDescriptor.java:383) > at > org.apache.cassandra.config.DatabaseDescriptor.(DatabaseDescriptor.java:125) > at > org.apache.cassandra.cql3.QueryProcessor.(QueryProcessor.java:84) > at org.apache.cassandra.config.CFMetaData.compile(CFMetaData.java:411) > at > org.apache.cassandra.schema.SchemaKeyspace.compile(SchemaKeyspace.java:240) > at > org.apache.cassandra.schema.SchemaKeyspace.(SchemaKeyspace.java:88) > at org.apache.cassandra.config.Schema.(Schema.java:107) > at org.apache.cassandra.config.Schema.(Schema.java:55) > at org.apache.cassandra.tools.nodetool.Cleanup.execute(Cleanup.java:50) > at > org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:251) > at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:165) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14939) fix some operational holes in incremental repair
[ https://issues.apache.org/jira/browse/CASSANDRA-14939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17034275#comment-17034275 ] Stefan Miklosovic commented on CASSANDRA-14939: --- Is there any chance this will appear in upcomming 4.x release? Is anybody working on this atm? > fix some operational holes in incremental repair > > > Key: CASSANDRA-14939 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14939 > Project: Cassandra > Issue Type: Bug >Reporter: Blake Eggleston >Assignee: Blake Eggleston >Priority: Normal > Fix For: 4.0 > > > Incremental repair has a few operational rough spots that make it more > difficult to fully automate and operate at scale than it should be. > * Visibility into whether pending repair data exists for a given token range. > * Ability to force promotion/demotion of data for completed sessions instead > of waiting for compaction. > * Get the most recent repairedAt timestamp for a given token range. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13158) Nodetool cleanup throwing exception
[
https://issues.apache.org/jira/browse/CASSANDRA-13158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17039324#comment-17039324
]
Stefan Miklosovic commented on CASSANDRA-13158:
---
[~dcapwell] here you go
{code}
error: Invalid partitioner class org.apache.cassandra.dht.Murmur3Partitioner
-- StackTrace --
org.apache.cassandra.exceptions.ConfigurationException: Invalid partitioner
class org.apache.cassandra.dht.Murmur3Partitioner
at
org.apache.cassandra.config.DatabaseDescriptor.applyPartitioner(DatabaseDescriptor.java:1108)
at
org.apache.cassandra.config.DatabaseDescriptor.toolInitialization(DatabaseDescriptor.java:218)
at
org.apache.cassandra.config.DatabaseDescriptor.toolInitialization(DatabaseDescriptor.java:185)
at org.apache.cassandra.tools.NodeProbe.checkJobs(NodeProbe.java:291)
at
org.apache.cassandra.tools.NodeProbe.forceKeyspaceCleanup(NodeProbe.java:298)
at org.apache.cassandra.tools.nodetool.Cleanup.execute(Cleanup.java:55)
at
org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:341)
at
org.apache.cassandra.tools.NodeTool$NodeToolCmd.accept(NodeTool.java:326)
at
org.apache.cassandra.tools.NodeTool$NodeToolCmd.accept(NodeTool.java:300)
at org.apache.cassandra.tools.NodeTool.execute(NodeTool.java:237)
at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:84)
{code}
> Nodetool cleanup throwing exception
> ---
>
> Key: CASSANDRA-13158
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13158
> Project: Cassandra
> Issue Type: Bug
> Components: Tool/nodetool
> Environment: Fedora 25 x86
>Reporter: Tomas Repik
>Assignee: Eduard Tudenhoefner
>Priority: Normal
> Fix For: 4.0
>
>
> After running nodetool cleanup I get this exception:
> error: Invalid partitioner class org.apache.cassandra.dht.Murmur3Partitioner
> -- StackTrace --
> org.apache.cassandra.exceptions.ConfigurationException: Invalid partitioner
> class org.apache.cassandra.dht.Murmur3Partitioner
> at
> org.apache.cassandra.config.DatabaseDescriptor.applyConfig(DatabaseDescriptor.java:383)
> at
> org.apache.cassandra.config.DatabaseDescriptor.(DatabaseDescriptor.java:125)
> at
> org.apache.cassandra.cql3.QueryProcessor.(QueryProcessor.java:84)
> at org.apache.cassandra.config.CFMetaData.compile(CFMetaData.java:411)
> at
> org.apache.cassandra.schema.SchemaKeyspace.compile(SchemaKeyspace.java:240)
> at
> org.apache.cassandra.schema.SchemaKeyspace.(SchemaKeyspace.java:88)
> at org.apache.cassandra.config.Schema.(Schema.java:107)
> at org.apache.cassandra.config.Schema.(Schema.java:55)
> at org.apache.cassandra.tools.nodetool.Cleanup.execute(Cleanup.java:50)
> at
> org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:251)
> at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:165)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12510) Disallow decommission when number of replicas will drop below configured RF
[ https://issues.apache.org/jira/browse/CASSANDRA-12510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17055361#comment-17055361 ] Stefan Miklosovic commented on CASSANDRA-12510: --- Isnt the same logic applicable to _drain_ ? > Disallow decommission when number of replicas will drop below configured RF > --- > > Key: CASSANDRA-12510 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12510 > Project: Cassandra > Issue Type: Improvement > Components: Legacy/Streaming and Messaging > Environment: C* version 3.3 >Reporter: Atin Sood >Assignee: Kurt Greaves >Priority: Low > Labels: lhf > Fix For: 4.0 > > Attachments: 12510-3.x-v2.patch, 12510-3.x.patch > > > Steps to replicate : > - Create a 3 node cluster in DC1 and create a keyspace test_keyspace with > table test_table with replication strategy NetworkTopologyStrategy , DC1=3 . > Populate some data into this table. > - Add 5 more nodes to this cluster, but in DC2. Also do not alter the > keyspace to add the new DC2 to replication (this is intentional and the > reason why the bug shows up). So the desc keyspace should still list > NetworkTopologyStrategy with DC1=3 as RF > - As expected, this will now be a 8 node cluster with 3 nodes in DC1 and 5 in > DC2 > - Now start decommissioning the nodes in DC1. Note that the decommission runs > fine on all the 3 nodes, but since the new nodes are in DC2 and the RF for > keyspace is restricted to DC1, the new 5 nodes won't get any data. > - You will now end with the 5 node cluster which has no data from the > decommissioned 3 nodes and hence ending up in data loss > I do understand that this problem could have been avoided if we perform an > alter stmt and add DC2 replication before adding the 5 nodes. But the fact > that decommission ran fine on the 3 nodes on DC1 without complaining that > there were no nodes to stream its data seems a little discomforting. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15694) Statistics upon streaming of entire SSTables in Netstats is wrong
[
https://issues.apache.org/jira/browse/CASSANDRA-15694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Miklosovic updated CASSANDRA-15694:
--
Description:
There is a bug in the current code (trunk on 6th April 2020) as if we are
streaming entire SSTables via CassandraEntireSSTableStreamWriter and
CassandraOutgoingFile respectively, there is not any update on particular
components of a SSTable as it counts only "db" file to be there. That
introduces this bug:
{code:java}
Mode: NORMAL
Rebuild 2c0b43f0-735d-11ea-9346-fb0ffe238736
/127.0.0.2 Sending 19 files, 27664559 bytes total. Already sent 133 files,
27664559 bytes total
/tmp/dtests15682026295742741219/node2/data/distributed_test_keyspace/cf-196b3...
{code}
Basically, number of files to be sent is lower than files sent.
The straightforward fix here is to distinguish when we are streaming entire
sstables and in that case include all manifest files into computation.
This issue relates to https://issues.apache.org/jira/browse/CASSANDRA-15657
because the resolution whether we stream entirely or not is got from a method
which is performance sensitive and computed every time. Once CASSANDRA-15657
(hence CASSANDRA-14586) is done, this ticket can be worked on.
branch with fix is here:
[https://github.com/smiklosovic/cassandra/tree/CASSANDRA-15694]
was:
There is a bug in the current code as if we are streaming entire SSTables via
CassandraEntireSSTableStreamWriter and CassandraOutgoingFile respectively,
there is not any update on particular components of a SSTable as it counts only
"db" file to be there. That introduces this bug:
{code:java}
Mode: NORMAL
Rebuild 2c0b43f0-735d-11ea-9346-fb0ffe238736
/127.0.0.2 Sending 19 files, 27664559 bytes total. Already sent 133 files,
27664559 bytes total
/tmp/dtests15682026295742741219/node2/data/distributed_test_keyspace/cf-196b3...
{code}
Basically, number of files to be sent is lower than files sent.
The straightforward fix here is to distinguish when we are streaming entire
sstables and in that case include all manifest files into computation.
This issue relates to https://issues.apache.org/jira/browse/CASSANDRA-15657
because the resolution whether we stream entirely or not is got from a method
which is performance sensitive and computed every time. Once CASSANDRA-15657
(hence CASSANDRA-14586) is done, this ticket can be worked on.
branch with fix is here:
[https://github.com/smiklosovic/cassandra/tree/CASSANDRA-15694]
> Statistics upon streaming of entire SSTables in Netstats is wrong
> -
>
> Key: CASSANDRA-15694
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15694
> Project: Cassandra
> Issue Type: Bug
> Components: Tool/nodetool
>Reporter: Stefan Miklosovic
>Priority: Normal
>
> There is a bug in the current code (trunk on 6th April 2020) as if we are
> streaming entire SSTables via CassandraEntireSSTableStreamWriter and
> CassandraOutgoingFile respectively, there is not any update on particular
> components of a SSTable as it counts only "db" file to be there. That
> introduces this bug:
>
> {code:java}
> Mode: NORMAL
> Rebuild 2c0b43f0-735d-11ea-9346-fb0ffe238736
> /127.0.0.2 Sending 19 files, 27664559 bytes total. Already sent 133
> files, 27664559 bytes total
>
> /tmp/dtests15682026295742741219/node2/data/distributed_test_keyspace/cf-196b3...
>
> {code}
> Basically, number of files to be sent is lower than files sent.
>
> The straightforward fix here is to distinguish when we are streaming entire
> sstables and in that case include all manifest files into computation.
>
> This issue relates to https://issues.apache.org/jira/browse/CASSANDRA-15657
> because the resolution whether we stream entirely or not is got from a method
> which is performance sensitive and computed every time. Once CASSANDRA-15657
> (hence CASSANDRA-14586) is done, this ticket can be worked on.
>
> branch with fix is here:
> [https://github.com/smiklosovic/cassandra/tree/CASSANDRA-15694]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15682) Missing commas between endpoints in nodetool describering
[
https://issues.apache.org/jira/browse/CASSANDRA-15682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17076401#comment-17076401
]
Stefan Miklosovic commented on CASSANDRA-15682:
---
PR with the fix here [https://github.com/apache/cassandra/pull/517]
> Missing commas between endpoints in nodetool describering
> -
>
> Key: CASSANDRA-15682
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15682
> Project: Cassandra
> Issue Type: Bug
> Components: Tool/nodetool
>Reporter: Aleksandr Sorokoumov
>Assignee: Stefan Miklosovic
>Priority: Normal
> Fix For: 4.0
>
>
> *Setup*
> 3-node cluster created with ccm
> {noformat}
> cqlsh> create keyspace ks with replication = {'class': 'SimpleStrategy',
> 'replication_factor': 2};
> {noformat}
> *trunk*:
> {noformat}
> $ bin/nodetool describering ks --port 7100
> Schema Version:295e8142-fc9f-3f76-b6db-24430ff572e5
> TokenRange:
> TokenRange(start_token:-9223372036854775808,
> end_token:-3074457345618258603endpoints:[127.0.0.2,
> 127.0.0.3]rpc_endpoints:[127.0.0.2,
> 127.0.0.3]endpoint_details:[EndpointDetails(host:127.0.0.2, datacenter:d
> atacenter1, rack:rack1), EndpointDetails(host:127.0.0.3,
> datacenter:datacenter1, rack:rack1)])
> TokenRange(start_token:-3074457345618258603,
> end_token:3074457345618258602endpoints:[127.0.0.3,
> 127.0.0.1]rpc_endpoints:[127.0.0.3,
> /127.0.0.1]endpoint_details:[EndpointDetails(host:127.0.0.3, datacenter:d
> atacenter1, rack:rack1), EndpointDetails(host:127.0.0.1,
> datacenter:datacenter1, rack:rack1)])
> TokenRange(start_token:3074457345618258602,
> end_token:-9223372036854775808endpoints:[127.0.0.1,
> 127.0.0.2]rpc_endpoints:[/127.0.0.1,
> 127.0.0.2]endpoint_details:[EndpointDetails(host:127.0.0.1, datacenter:d
> atacenter1, rack:rack1), EndpointDetails(host:127.0.0.2,
> datacenter:datacenter1, rack:rack1)])
> {noformat}
> *3.11* (correct output)
> {noformat}
> bin/nodetool describering ks --port 7100
> Schema Version:c8fd35ea-6f49-3e77-85e7-a92e79df8696
> TokenRange:
> TokenRange(start_token:-9223372036854775808,
> end_token:-3074457345618258603, endpoints:[127.0.0.2, 127.0.0.3],
> rpc_endpoints:[127.0.0.2, 127.0.0.3],
> endpoint_details:[EndpointDetails(host:127.0.0.2, datace
> nter:datacenter1, rack:rack1), EndpointDetails(host:127.0.0.3,
> datacenter:datacenter1, rack:rack1)])
> TokenRange(start_token:-3074457345618258603,
> end_token:3074457345618258602, endpoints:[127.0.0.3, 127.0.0.1],
> rpc_endpoints:[127.0.0.3, 127.0.0.1],
> endpoint_details:[EndpointDetails(host:127.0.0.3, datacen
> ter:datacenter1, rack:rack1), EndpointDetails(host:127.0.0.1,
> datacenter:datacenter1, rack:rack1)])
> TokenRange(start_token:3074457345618258602,
> end_token:-9223372036854775808, endpoints:[127.0.0.1, 127.0.0.2],
> rpc_endpoints:[127.0.0.1, 127.0.0.2],
> endpoint_details:[EndpointDetails(host:127.0.0.1, datacen
> ter:datacenter1, rack:rack1), EndpointDetails(host:127.0.0.2,
> datacenter:datacenter1, rack:rack1)])
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15697) cqlsh -e parsing bug
[
https://issues.apache.org/jira/browse/CASSANDRA-15697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17076707#comment-17076707
]
Stefan Miklosovic commented on CASSANDRA-15697:
---
[~jrwest] I have already hit this and it is solved here
https://issues.apache.org/jira/browse/CASSANDRA-15660, this should be closed /
resolved as a duplicate.
> cqlsh -e parsing bug
>
>
> Key: CASSANDRA-15697
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15697
> Project: Cassandra
> Issue Type: Bug
> Components: Tool/cqlsh
>Reporter: Jordan West
>Priority: Normal
> Fix For: 4.0-alpha
>
>
> {{cqlsh -e}} no longer works on trunk after the introduction of python 3
> support (CASSANDRA-10190). Examples below.
> {code}
> $ ./bin/cqlsh -e 'select * from foo;'
> Usage: cqlsh.py [options] [host [port]]
> cqlsh.py: error: ‘CHANGES.txt' is not a valid port number.
> $ ./bin/cqlsh -e 'select id from foo;'
> Usage: cqlsh.py [options] [host [port]]
> cqlsh.py: error: 'from' is not a valid port number.
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15694) Statistics upon streaming of entire SSTables in Netstats is wrong
[
https://issues.apache.org/jira/browse/CASSANDRA-15694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17087019#comment-17087019
]
Stefan Miklosovic commented on CASSANDRA-15694:
---
[~jasonstack] yes I noticed that, thanks. [~djoshi] could you please review
this and merge? I think this is your area of expertise as you have written that
whole SSTable streaming ...
> Statistics upon streaming of entire SSTables in Netstats is wrong
> -
>
> Key: CASSANDRA-15694
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15694
> Project: Cassandra
> Issue Type: Bug
> Components: Tool/nodetool
>Reporter: Stefan Miklosovic
>Priority: Normal
>
> There is a bug in the current code (trunk on 6th April 2020) as if we are
> streaming entire SSTables via CassandraEntireSSTableStreamWriter and
> CassandraOutgoingFile respectively, there is not any update on particular
> components of a SSTable as it counts only "db" file to be there. That
> introduces this bug:
>
> {code:java}
> Mode: NORMAL
> Rebuild 2c0b43f0-735d-11ea-9346-fb0ffe238736
> /127.0.0.2 Sending 19 files, 27664559 bytes total. Already sent 133
> files, 27664559 bytes total
>
> /tmp/dtests15682026295742741219/node2/data/distributed_test_keyspace/cf-196b3...
>
> {code}
> Basically, number of files to be sent is lower than files sent.
>
> The straightforward fix here is to distinguish when we are streaming entire
> sstables and in that case include all manifest files into computation.
>
> This issue depends on CASSANDRA-15657 because the resolution whether we
> stream entirely or not is got from a method which is performance sensitive
> and computed every time. Once CASSANDRA-15657 (hence CASSANDRA-14586) is
> done, this ticket can be worked on.
>
> branch with fix is here:
> [https://github.com/smiklosovic/cassandra/tree/CASSANDRA-15694]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Assigned] (CASSANDRA-15694) Statistics upon streaming of entire SSTables in Netstats is wrong
[
https://issues.apache.org/jira/browse/CASSANDRA-15694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Miklosovic reassigned CASSANDRA-15694:
-
Assignee: Stefan Miklosovic
> Statistics upon streaming of entire SSTables in Netstats is wrong
> -
>
> Key: CASSANDRA-15694
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15694
> Project: Cassandra
> Issue Type: Bug
> Components: Tool/nodetool
>Reporter: Stefan Miklosovic
>Assignee: Stefan Miklosovic
>Priority: Normal
>
> There is a bug in the current code (trunk on 6th April 2020) as if we are
> streaming entire SSTables via CassandraEntireSSTableStreamWriter and
> CassandraOutgoingFile respectively, there is not any update on particular
> components of a SSTable as it counts only "db" file to be there. That
> introduces this bug:
>
> {code:java}
> Mode: NORMAL
> Rebuild 2c0b43f0-735d-11ea-9346-fb0ffe238736
> /127.0.0.2 Sending 19 files, 27664559 bytes total. Already sent 133
> files, 27664559 bytes total
>
> /tmp/dtests15682026295742741219/node2/data/distributed_test_keyspace/cf-196b3...
>
> {code}
> Basically, number of files to be sent is lower than files sent.
>
> The straightforward fix here is to distinguish when we are streaming entire
> sstables and in that case include all manifest files into computation.
>
> This issue depends on CASSANDRA-15657 because the resolution whether we
> stream entirely or not is got from a method which is performance sensitive
> and computed every time. Once CASSANDRA-15657 (hence CASSANDRA-14586) is
> done, this ticket can be worked on.
>
> branch with fix is here:
> [https://github.com/smiklosovic/cassandra/tree/CASSANDRA-15694]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15694) Statistics upon streaming of entire SSTables in Netstats is wrong
[
https://issues.apache.org/jira/browse/CASSANDRA-15694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17087482#comment-17087482
]
Stefan Miklosovic commented on CASSANDRA-15694:
---
[~djoshi] PR prepared here, please let me know if there is anything I should do
to make this happen.
[https://github.com/apache/cassandra/pull/546]
> Statistics upon streaming of entire SSTables in Netstats is wrong
> -
>
> Key: CASSANDRA-15694
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15694
> Project: Cassandra
> Issue Type: Bug
> Components: Tool/nodetool
>Reporter: Stefan Miklosovic
>Assignee: Stefan Miklosovic
>Priority: Normal
> Time Spent: 10m
> Remaining Estimate: 0h
>
> There is a bug in the current code (trunk on 6th April 2020) as if we are
> streaming entire SSTables via CassandraEntireSSTableStreamWriter and
> CassandraOutgoingFile respectively, there is not any update on particular
> components of a SSTable as it counts only "db" file to be there. That
> introduces this bug:
>
> {code:java}
> Mode: NORMAL
> Rebuild 2c0b43f0-735d-11ea-9346-fb0ffe238736
> /127.0.0.2 Sending 19 files, 27664559 bytes total. Already sent 133
> files, 27664559 bytes total
>
> /tmp/dtests15682026295742741219/node2/data/distributed_test_keyspace/cf-196b3...
>
> {code}
> Basically, number of files to be sent is lower than files sent.
>
> The straightforward fix here is to distinguish when we are streaming entire
> sstables and in that case include all manifest files into computation.
>
> This issue depends on CASSANDRA-15657 because the resolution whether we
> stream entirely or not is got from a method which is performance sensitive
> and computed every time. Once CASSANDRA-15657 (hence CASSANDRA-14586) is
> done, this ticket can be worked on.
>
> branch with fix is here:
> [https://github.com/smiklosovic/cassandra/tree/CASSANDRA-15694]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15406) Add command to show the progress of data streaming and index build
[ https://issues.apache.org/jira/browse/CASSANDRA-15406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17076115#comment-17076115 ] Stefan Miklosovic commented on CASSANDRA-15406: --- This issue should be blocked to work on until the underlying bug when streaming of entire sstables is resolved as the figures for the computation of progress would not make sense anyway. > Add command to show the progress of data streaming and index build > --- > > Key: CASSANDRA-15406 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15406 > Project: Cassandra > Issue Type: Improvement > Components: Consistency/Streaming, Legacy/Streaming and Messaging, > Tool/nodetool >Reporter: maxwellguo >Assignee: Stefan Miklosovic >Priority: Normal > Fix For: 4.0, 4.x > > > I found that we should supply a command to show the progress of streaming > when we do the operation of bootstrap/move/decommission/removenode. For when > do data streaming , noboday knows which steps there program are in , so I > think a command to show the joing/leaving node's is needed . -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Assigned] (CASSANDRA-15694) Statistics upon streaming of entire SSTables in Netstats is wrong
[
https://issues.apache.org/jira/browse/CASSANDRA-15694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Miklosovic reassigned CASSANDRA-15694:
-
Assignee: (was: Stefan Miklosovic)
> Statistics upon streaming of entire SSTables in Netstats is wrong
> -
>
> Key: CASSANDRA-15694
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15694
> Project: Cassandra
> Issue Type: Bug
> Components: Tool/nodetool
>Reporter: Stefan Miklosovic
>Priority: Normal
>
> There is a bug in the current code as if we are streaming entire SSTables via
> CassandraEntireSSTableStreamWriter and CassandraOutgoingFile respectively,
> there is not any update on particular components of a SSTable as it counts
> only "db" file to be there. That introduces this bug:
>
> {code:java}
> Mode: NORMAL
> Rebuild 2c0b43f0-735d-11ea-9346-fb0ffe238736
> /127.0.0.2 Sending 19 files, 27664559 bytes total. Already sent 133
> files, 27664559 bytes total
>
> /tmp/dtests15682026295742741219/node2/data/distributed_test_keyspace/cf-196b3...
>
> {code}
> Basically, number of files to be sent is lower than files sent.
>
> The straightforward fix here is to distinguish when we are streaming entire
> sstables and in that case include all manifest files into computation.
>
> This issue relates to https://issues.apache.org/jira/browse/CASSANDRA-15657
> because the resolution whether we stream entirely or not is got from a method
> which is performance sensitive and computed every time. Once CASSANDRA-15657
> (hence CASSANDRA-14586) is done, this ticket can be worked on.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-15694) Statistics upon streaming of entire SSTables in Netstats is wrong
Stefan Miklosovic created CASSANDRA-15694:
-
Summary: Statistics upon streaming of entire SSTables in Netstats
is wrong
Key: CASSANDRA-15694
URL: https://issues.apache.org/jira/browse/CASSANDRA-15694
Project: Cassandra
Issue Type: Bug
Components: Tool/nodetool
Reporter: Stefan Miklosovic
Assignee: Stefan Miklosovic
There is a bug in the current code as if we are streaming entire SSTables via
CassandraEntireSSTableStreamWriter and CassandraOutgoingFile respectively,
there is not any update on particular components of a SSTable as it counts only
"db" file to be there. That introduces this bug:
{code:java}
Mode: NORMAL
Rebuild 2c0b43f0-735d-11ea-9346-fb0ffe238736
/127.0.0.2 Sending 19 files, 27664559 bytes total. Already sent 133 files,
27664559 bytes total
/tmp/dtests15682026295742741219/node2/data/distributed_test_keyspace/cf-196b3...
{code}
Basically, number of files to be sent is lower than files sent.
The straightforward fix here is to distinguish when we are streaming entire
sstables and in that case include all manifest files into computation.
This issue relates to https://issues.apache.org/jira/browse/CASSANDRA-15657
because the resolution whether we stream entirely or not is got from a method
which is performance sensitive and computed every time. Once CASSANDRA-15657
(hence CASSANDRA-14586) is done, this ticket can be worked on.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15694) Statistics upon streaming of entire SSTables in Netstats is wrong
[
https://issues.apache.org/jira/browse/CASSANDRA-15694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Miklosovic updated CASSANDRA-15694:
--
Description:
There is a bug in the current code as if we are streaming entire SSTables via
CassandraEntireSSTableStreamWriter and CassandraOutgoingFile respectively,
there is not any update on particular components of a SSTable as it counts only
"db" file to be there. That introduces this bug:
{code:java}
Mode: NORMAL
Rebuild 2c0b43f0-735d-11ea-9346-fb0ffe238736
/127.0.0.2 Sending 19 files, 27664559 bytes total. Already sent 133 files,
27664559 bytes total
/tmp/dtests15682026295742741219/node2/data/distributed_test_keyspace/cf-196b3...
{code}
Basically, number of files to be sent is lower than files sent.
The straightforward fix here is to distinguish when we are streaming entire
sstables and in that case include all manifest files into computation.
This issue relates to https://issues.apache.org/jira/browse/CASSANDRA-15657
because the resolution whether we stream entirely or not is got from a method
which is performance sensitive and computed every time. Once CASSANDRA-15657
(hence CASSANDRA-14586) is done, this ticket can be worked on.
branch with fix is here:
[https://github.com/smiklosovic/cassandra/tree/CASSANDRA-15694]
was:
There is a bug in the current code as if we are streaming entire SSTables via
CassandraEntireSSTableStreamWriter and CassandraOutgoingFile respectively,
there is not any update on particular components of a SSTable as it counts only
"db" file to be there. That introduces this bug:
{code:java}
Mode: NORMAL
Rebuild 2c0b43f0-735d-11ea-9346-fb0ffe238736
/127.0.0.2 Sending 19 files, 27664559 bytes total. Already sent 133 files,
27664559 bytes total
/tmp/dtests15682026295742741219/node2/data/distributed_test_keyspace/cf-196b3...
{code}
Basically, number of files to be sent is lower than files sent.
The straightforward fix here is to distinguish when we are streaming entire
sstables and in that case include all manifest files into computation.
This issue relates to https://issues.apache.org/jira/browse/CASSANDRA-15657
because the resolution whether we stream entirely or not is got from a method
which is performance sensitive and computed every time. Once CASSANDRA-15657
(hence CASSANDRA-14586) is done, this ticket can be worked on.
> Statistics upon streaming of entire SSTables in Netstats is wrong
> -
>
> Key: CASSANDRA-15694
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15694
> Project: Cassandra
> Issue Type: Bug
> Components: Tool/nodetool
>Reporter: Stefan Miklosovic
>Priority: Normal
>
> There is a bug in the current code as if we are streaming entire SSTables via
> CassandraEntireSSTableStreamWriter and CassandraOutgoingFile respectively,
> there is not any update on particular components of a SSTable as it counts
> only "db" file to be there. That introduces this bug:
>
> {code:java}
> Mode: NORMAL
> Rebuild 2c0b43f0-735d-11ea-9346-fb0ffe238736
> /127.0.0.2 Sending 19 files, 27664559 bytes total. Already sent 133
> files, 27664559 bytes total
>
> /tmp/dtests15682026295742741219/node2/data/distributed_test_keyspace/cf-196b3...
>
> {code}
> Basically, number of files to be sent is lower than files sent.
>
> The straightforward fix here is to distinguish when we are streaming entire
> sstables and in that case include all manifest files into computation.
>
> This issue relates to https://issues.apache.org/jira/browse/CASSANDRA-15657
> because the resolution whether we stream entirely or not is got from a method
> which is performance sensitive and computed every time. Once CASSANDRA-15657
> (hence CASSANDRA-14586) is done, this ticket can be worked on.
>
> branch with fix is here:
> [https://github.com/smiklosovic/cassandra/tree/CASSANDRA-15694]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Assigned] (CASSANDRA-15406) Add command to show the progress of data streaming and index build
[ https://issues.apache.org/jira/browse/CASSANDRA-15406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Miklosovic reassigned CASSANDRA-15406: - Assignee: (was: Stefan Miklosovic) > Add command to show the progress of data streaming and index build > --- > > Key: CASSANDRA-15406 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15406 > Project: Cassandra > Issue Type: Improvement > Components: Consistency/Streaming, Legacy/Streaming and Messaging, > Tool/nodetool >Reporter: maxwellguo >Priority: Normal > Fix For: 4.0, 4.x > > > I found that we should supply a command to show the progress of streaming > when we do the operation of bootstrap/move/decommission/removenode. For when > do data streaming , noboday knows which steps there program are in , so I > think a command to show the joing/leaving node's is needed . -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-13158) Nodetool cleanup throwing exception
[ https://issues.apache.org/jira/browse/CASSANDRA-13158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17067091#comment-17067091 ] Stefan Miklosovic edited comment on CASSANDRA-13158 at 3/25/20, 9:11 PM: - [~dcapwell] I tested this again and I was not able to reproduce it. I think I got my scripts wrong so I did it right way and it works. I still think that issue is still there burried in some complex "this is not set and this is" but user will not experience it if he does not do something really strange. Good there is cause of that exception given so we know what is going on if somebody ever hit it again. was (Author: stefan.miklosovic): [~dcapwell] I tested this again and I was not able to replicate it. I think I got my scripts wrong so I did it right way and it works. I still think that issue is still there burried in some complex "this is not set and this is" but user will not experience it if he does not do something really strange. Good there is cause of that exception given so we know what is going on if somebody ever hit it anymore. > Nodetool cleanup throwing exception > --- > > Key: CASSANDRA-13158 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13158 > Project: Cassandra > Issue Type: Bug > Components: Tool/nodetool > Environment: Fedora 25 x86 >Reporter: Tomas Repik >Assignee: Eduard Tudenhoefner >Priority: Normal > Fix For: 4.0 > > Attachments: error.txt > > > After running nodetool cleanup I get this exception: > error: Invalid partitioner class org.apache.cassandra.dht.Murmur3Partitioner > -- StackTrace -- > org.apache.cassandra.exceptions.ConfigurationException: Invalid partitioner > class org.apache.cassandra.dht.Murmur3Partitioner > at > org.apache.cassandra.config.DatabaseDescriptor.applyConfig(DatabaseDescriptor.java:383) > at > org.apache.cassandra.config.DatabaseDescriptor.(DatabaseDescriptor.java:125) > at > org.apache.cassandra.cql3.QueryProcessor.(QueryProcessor.java:84) > at org.apache.cassandra.config.CFMetaData.compile(CFMetaData.java:411) > at > org.apache.cassandra.schema.SchemaKeyspace.compile(SchemaKeyspace.java:240) > at > org.apache.cassandra.schema.SchemaKeyspace.(SchemaKeyspace.java:88) > at org.apache.cassandra.config.Schema.(Schema.java:107) > at org.apache.cassandra.config.Schema.(Schema.java:55) > at org.apache.cassandra.tools.nodetool.Cleanup.execute(Cleanup.java:50) > at > org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:251) > at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:165) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13158) Nodetool cleanup throwing exception
[ https://issues.apache.org/jira/browse/CASSANDRA-13158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17067091#comment-17067091 ] Stefan Miklosovic commented on CASSANDRA-13158: --- [~dcapwell] I tested this again and I was not able to replicate it. I think I got my scripts wrong so I did it right way and it works. I still think that issue is still there burried in some complex "this is not set and this is" but user will not experience it if he does not do something really strange. Good there is cause of that exception given so we know what is going on if somebody ever hit it anymore. > Nodetool cleanup throwing exception > --- > > Key: CASSANDRA-13158 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13158 > Project: Cassandra > Issue Type: Bug > Components: Tool/nodetool > Environment: Fedora 25 x86 >Reporter: Tomas Repik >Assignee: Eduard Tudenhoefner >Priority: Normal > Fix For: 4.0 > > Attachments: error.txt > > > After running nodetool cleanup I get this exception: > error: Invalid partitioner class org.apache.cassandra.dht.Murmur3Partitioner > -- StackTrace -- > org.apache.cassandra.exceptions.ConfigurationException: Invalid partitioner > class org.apache.cassandra.dht.Murmur3Partitioner > at > org.apache.cassandra.config.DatabaseDescriptor.applyConfig(DatabaseDescriptor.java:383) > at > org.apache.cassandra.config.DatabaseDescriptor.(DatabaseDescriptor.java:125) > at > org.apache.cassandra.cql3.QueryProcessor.(QueryProcessor.java:84) > at org.apache.cassandra.config.CFMetaData.compile(CFMetaData.java:411) > at > org.apache.cassandra.schema.SchemaKeyspace.compile(SchemaKeyspace.java:240) > at > org.apache.cassandra.schema.SchemaKeyspace.(SchemaKeyspace.java:88) > at org.apache.cassandra.config.Schema.(Schema.java:107) > at org.apache.cassandra.config.Schema.(Schema.java:55) > at org.apache.cassandra.tools.nodetool.Cleanup.execute(Cleanup.java:50) > at > org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:251) > at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:165) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15659) Better support of Python 3 for cqlsh
[ https://issues.apache.org/jira/browse/CASSANDRA-15659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17068506#comment-17068506 ] Stefan Miklosovic commented on CASSANDRA-15659: --- [~yukim] That sounds good! Thanks a lot. > Better support of Python 3 for cqlsh > > > Key: CASSANDRA-15659 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15659 > Project: Cassandra > Issue Type: Task > Components: Tool/cqlsh >Reporter: Stefan Miklosovic >Priority: Normal > Fix For: 4.0-alpha > > > From mailing list: > [https://lists.apache.org/thread.html/r377099b632c62b641e4feef5b738084fc5369b0c7157fae867853597%40%3Cdev.cassandra.apache.org%3E] > > As of today (24/3/2020) and current trunk, there is Python 3.6 supported (1) > but there is not any 3.6 version ootb in Debian for example. E.g. Buster has > Python 3.7 and other (recent) releases have version 2.7. This means that if > one wants to use Python 3 in Debian, he has to use 3.6 but it is not in the > repository so he has to download / compile / install it on his own. > There should be some sane Python 3 version supported which is as well present > in Debian repository (or requirement to run with 3.6 should be relaxed) . > (1) > [https://github.com/apache/cassandra/blob/bf9a1d487b9ba469e8d740cf7d1cd419535a7e79/bin/cqlsh#L57-L65] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-15406) Add command to show the progress of data streaming and index build
[ https://issues.apache.org/jira/browse/CASSANDRA-15406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17072227#comment-17072227 ] Stefan Miklosovic edited comment on CASSANDRA-15406 at 4/1/20, 9:07 AM: Hi [~dcapwell], I think that this ticket will be more or less transformed just into a bug fix. I do not think that there is a need for a "command" for nodetool and having a lot of work before release anyway, it is not a good idea to introduce something completely new. As I see it, this ticket will be just about the introduction of global percentage into netstats output + I think I hit a bug when we stream entire SSTables and netstats is displaying it weirdly so that should be fixed too. was (Author: stefan.miklosovic): Hi [~dcapwell], I think that this ticket will be more or less transformed just into a bug fix. I do not thing that there is a need for a "command" for nodetool and having a lot of work before release anyway, it is not a good idea to introduce something completely new. As I see it, this ticket will be just about the introduction of global percentage into netstats output + I think I hit a bug when we stream entire SSTables and netstats is displaying it weirdly so that should be fixed too. > Add command to show the progress of data streaming and index build > --- > > Key: CASSANDRA-15406 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15406 > Project: Cassandra > Issue Type: Improvement > Components: Consistency/Streaming, Legacy/Streaming and Messaging, > Tool/nodetool >Reporter: maxwellguo >Assignee: Stefan Miklosovic >Priority: Normal > Fix For: 4.0, 4.x > > > I found that we should supply a command to show the progress of streaming > when we do the operation of bootstrap/move/decommission/removenode. For when > do data streaming , noboday knows which steps there program are in , so I > think a command to show the joing/leaving node's is needed . -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Assigned] (CASSANDRA-15682) Missing commas between endpoints in nodetool describering
[
https://issues.apache.org/jira/browse/CASSANDRA-15682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Miklosovic reassigned CASSANDRA-15682:
-
Assignee: Stefan Miklosovic
> Missing commas between endpoints in nodetool describering
> -
>
> Key: CASSANDRA-15682
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15682
> Project: Cassandra
> Issue Type: Bug
> Components: Tool/nodetool
>Reporter: Aleksandr Sorokoumov
>Assignee: Stefan Miklosovic
>Priority: Normal
> Fix For: 4.0
>
>
> *Setup*
> 3-node cluster created with ccm
> {noformat}
> cqlsh> create keyspace ks with replication = {'class': 'SimpleStrategy',
> 'replication_factor': 2};
> {noformat}
> *trunk*:
> {noformat}
> $ bin/nodetool describering ks --port 7100
> Schema Version:295e8142-fc9f-3f76-b6db-24430ff572e5
> TokenRange:
> TokenRange(start_token:-9223372036854775808,
> end_token:-3074457345618258603endpoints:[127.0.0.2,
> 127.0.0.3]rpc_endpoints:[127.0.0.2,
> 127.0.0.3]endpoint_details:[EndpointDetails(host:127.0.0.2, datacenter:d
> atacenter1, rack:rack1), EndpointDetails(host:127.0.0.3,
> datacenter:datacenter1, rack:rack1)])
> TokenRange(start_token:-3074457345618258603,
> end_token:3074457345618258602endpoints:[127.0.0.3,
> 127.0.0.1]rpc_endpoints:[127.0.0.3,
> /127.0.0.1]endpoint_details:[EndpointDetails(host:127.0.0.3, datacenter:d
> atacenter1, rack:rack1), EndpointDetails(host:127.0.0.1,
> datacenter:datacenter1, rack:rack1)])
> TokenRange(start_token:3074457345618258602,
> end_token:-9223372036854775808endpoints:[127.0.0.1,
> 127.0.0.2]rpc_endpoints:[/127.0.0.1,
> 127.0.0.2]endpoint_details:[EndpointDetails(host:127.0.0.1, datacenter:d
> atacenter1, rack:rack1), EndpointDetails(host:127.0.0.2,
> datacenter:datacenter1, rack:rack1)])
> {noformat}
> *3.11* (correct output)
> {noformat}
> bin/nodetool describering ks --port 7100
> Schema Version:c8fd35ea-6f49-3e77-85e7-a92e79df8696
> TokenRange:
> TokenRange(start_token:-9223372036854775808,
> end_token:-3074457345618258603, endpoints:[127.0.0.2, 127.0.0.3],
> rpc_endpoints:[127.0.0.2, 127.0.0.3],
> endpoint_details:[EndpointDetails(host:127.0.0.2, datace
> nter:datacenter1, rack:rack1), EndpointDetails(host:127.0.0.3,
> datacenter:datacenter1, rack:rack1)])
> TokenRange(start_token:-3074457345618258603,
> end_token:3074457345618258602, endpoints:[127.0.0.3, 127.0.0.1],
> rpc_endpoints:[127.0.0.3, 127.0.0.1],
> endpoint_details:[EndpointDetails(host:127.0.0.3, datacen
> ter:datacenter1, rack:rack1), EndpointDetails(host:127.0.0.1,
> datacenter:datacenter1, rack:rack1)])
> TokenRange(start_token:3074457345618258602,
> end_token:-9223372036854775808, endpoints:[127.0.0.1, 127.0.0.2],
> rpc_endpoints:[127.0.0.1, 127.0.0.2],
> endpoint_details:[EndpointDetails(host:127.0.0.1, datacen
> ter:datacenter1, rack:rack1), EndpointDetails(host:127.0.0.2,
> datacenter:datacenter1, rack:rack1)])
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13254) move compaction strategies to dedicated pages and expand on details
[ https://issues.apache.org/jira/browse/CASSANDRA-13254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17074089#comment-17074089 ] Stefan Miklosovic commented on CASSANDRA-13254: --- this is in trunk as I lastly checked and might be closed > move compaction strategies to dedicated pages and expand on details > --- > > Key: CASSANDRA-13254 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13254 > Project: Cassandra > Issue Type: Improvement > Components: Legacy/Documentation and Website >Reporter: Jon Haddad >Priority: Normal > Fix For: 4.0 > > > current sections on compactions are lacking details. we should split them > out into their own sections and improve each of them. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15406) Add command to show the progress of data streaming and index build
[ https://issues.apache.org/jira/browse/CASSANDRA-15406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17074400#comment-17074400 ] Stefan Miklosovic commented on CASSANDRA-15406: --- [~djoshi] yes I can do that but without fixing the underlying issue when the streaming of entire sstable is happening, it does not make sense to continue with the computation and rendering of the progress in percentage as the computed figure would not make any sense and we would have to yet do some workarounds which is imho not desirable. Hence, the course of action will be: 1) create new ticket as you suggested describing the problem of not updated figures properly when sstable is streamed in its entirety 2) momentarilly abandoning this ticket, marking it as dependant to the newly created one and returning to it once the former is resolved. > Add command to show the progress of data streaming and index build > --- > > Key: CASSANDRA-15406 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15406 > Project: Cassandra > Issue Type: Improvement > Components: Consistency/Streaming, Legacy/Streaming and Messaging, > Tool/nodetool >Reporter: maxwellguo >Assignee: Stefan Miklosovic >Priority: Normal > Fix For: 4.0, 4.x > > > I found that we should supply a command to show the progress of streaming > when we do the operation of bootstrap/move/decommission/removenode. For when > do data streaming , noboday knows which steps there program are in , so I > think a command to show the joing/leaving node's is needed . -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15406) Add command to show the progress of data streaming and index build
[ https://issues.apache.org/jira/browse/CASSANDRA-15406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17074404#comment-17074404 ] Stefan Miklosovic commented on CASSANDRA-15406: --- For the reference this is a branch it is fixed, with test checking that figures do match [https://github.com/smiklosovic/cassandra/commit/e98338082a73e5f360c6d9eb234710810eba5cea] > Add command to show the progress of data streaming and index build > --- > > Key: CASSANDRA-15406 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15406 > Project: Cassandra > Issue Type: Improvement > Components: Consistency/Streaming, Legacy/Streaming and Messaging, > Tool/nodetool >Reporter: maxwellguo >Assignee: Stefan Miklosovic >Priority: Normal > Fix For: 4.0, 4.x > > > I found that we should supply a command to show the progress of streaming > when we do the operation of bootstrap/move/decommission/removenode. For when > do data streaming , noboday knows which steps there program are in , so I > think a command to show the joing/leaving node's is needed . -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15586) 4.0 quality testing: Cluster Setup and Maintenance
[ https://issues.apache.org/jira/browse/CASSANDRA-15586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17071776#comment-17071776 ] Stefan Miklosovic commented on CASSANDRA-15586: --- I contacted FreeBSD guy and he said that he is already tracking 4.0 branch and he follows this quite closely and he is planning to port it as soon as it is out. I asked him to approach us if he ever encounters bugs and issues which make Cassandra 4 on that system buggy. > 4.0 quality testing: Cluster Setup and Maintenance > -- > > Key: CASSANDRA-15586 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15586 > Project: Cassandra > Issue Type: Task > Components: Test/dtest >Reporter: Josh McKenzie >Assignee: Ekaterina Dimitrova >Priority: Normal > Labels: 4.0-QA > Fix For: 4.0-rc > > > We want 4.0 to be easy for users to setup out of the box and just work. This > means having low friction when users download the Cassandra package and start > running it. For example, users should be able to easily configure and start > new 4.0 clusters and have tokens distributed evenly. Another example is > packaging, it should be easy to install Cassandra on all supported platforms > (e.g. packaging) and have Cassandra use standard platform integrations. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-15586) 4.0 quality testing: Cluster Setup and Maintenance
[ https://issues.apache.org/jira/browse/CASSANDRA-15586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17071776#comment-17071776 ] Stefan Miklosovic edited comment on CASSANDRA-15586 at 3/31/20, 1:20 PM: - I contacted FreeBSD guy and he said that he is already tracking 4.0 branch and he follows this quite closely and he is planning to port it as soon as it is out. I asked him to approach us if he ever encountered bugs and issues which would make Cassandra 4 on that system buggy. was (Author: stefan.miklosovic): I contacted FreeBSD guy and he said that he is already tracking 4.0 branch and he follows this quite closely and he is planning to port it as soon as it is out. I asked him to approach us if he ever encounters bugs and issues which make Cassandra 4 on that system buggy. > 4.0 quality testing: Cluster Setup and Maintenance > -- > > Key: CASSANDRA-15586 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15586 > Project: Cassandra > Issue Type: Task > Components: Test/dtest >Reporter: Josh McKenzie >Assignee: Ekaterina Dimitrova >Priority: Normal > Labels: 4.0-QA > Fix For: 4.0-rc > > > We want 4.0 to be easy for users to setup out of the box and just work. This > means having low friction when users download the Cassandra package and start > running it. For example, users should be able to easily configure and start > new 4.0 clusters and have tokens distributed evenly. Another example is > packaging, it should be easy to install Cassandra on all supported platforms > (e.g. packaging) and have Cassandra use standard platform integrations. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15660) Unable to specify -e/--execute flag in cqlsh
[
https://issues.apache.org/jira/browse/CASSANDRA-15660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17071940#comment-17071940
]
Stefan Miklosovic commented on CASSANDRA-15660:
---
tested and works as expected
> Unable to specify -e/--execute flag in cqlsh
>
>
> Key: CASSANDRA-15660
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15660
> Project: Cassandra
> Issue Type: Bug
> Components: Tool/cqlsh
>Reporter: Stefan Miklosovic
>Assignee: ZhaoYang
>Priority: Normal
> Labels: pull-request-available
> Fix For: 4.0-alpha
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> From mailing list:
> [https://lists.apache.org/thread.html/r377099b632c62b641e4feef5b738084fc5369b0c7157fae867853597%40%3Cdev.cassandra.apache.org%3E]
> The bug looks like this:
> {code:java}
> $ /usr/bin/cqlsh -e 'describe keyspaces' -u cassandra -p cassandra 127.0.0.1
> Usage: cqlsh.py [options] [host [port]]cqlsh.py: error: '127.0.0.1' is not a
> valid port number.
> {code}
> This is working in 3.x releases just fine but fails on 4.
> The workaround for 4.x code as of today is to put these statements into file
> and use "-f" flag.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15406) Add command to show the progress of data streaming and index build
[
https://issues.apache.org/jira/browse/CASSANDRA-15406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17071884#comment-17071884
]
Stefan Miklosovic commented on CASSANDRA-15406:
---
I have added "global percentage" functionality into netstats both for sending
and receiving. It looks like this:
{code:java}
Mode: NORMAL
Rebuild 2c0b43f0-735d-11ea-9346-fb0ffe238736
/127.0.0.1
Receiving 133 files, 27664559 bytes total. Already received 21 files
(15.79%), 918834 bytes total (3.32%)
/tmp/dtests15682026295742741219/node2/data/distributed_test_keyspace/cf-196b3...
{code}
On sending:
{code:java}
Mode: NORMAL
Rebuild 2c0b43f0-735d-11ea-9346-fb0ffe238736
/127.0.0.2
Sending 133 files, 27664559 bytes total. Already sent 42 files
(31.58%), 1841015 bytes total (6.65%)
/tmp/dtests15682026295742741219/node2/data/distributed_test_keyspace/cf-196b3...
{code}
There is a bug in the current code as if we are streaming entire SSTables via
CassandraEntireSSTableStreamWriter and CassandraOutgoingFile respectively,
there is not any update on particular components of a SSTable as it counts only
"db" file to be there. That introduces this bug:
{code:java}
Mode: NORMAL
Rebuild 2c0b43f0-735d-11ea-9346-fb0ffe238736
/127.0.0.2 Sending 19 files, 27664559 bytes total. Already sent 133 files,
27664559 bytes total
/tmp/dtests15682026295742741219/node2/data/distributed_test_keyspace/cf-196b3...
{code}
Basically, number of files to be sent is lower than files sent.
Maybe something for [~djoshi] as food for thought.
I have fixed it here
[https://github.com/apache/cassandra/compare/trunk...smiklosovic:CASSANDRA-15406]
(forget the test, will be finished).
[~jasonstack], this fix would be way better if we manage to merge yours
CASSANDRA-15657 because mine is checking if SSTable is meant to be streamed
entirely and this is not performance-friendly. With your patch it would be just
one call to already cached value because you have introduced a field for that.
Could somebody please give me some feedback here if this is what we want?
Thanks
@
> Add command to show the progress of data streaming and index build
> ---
>
> Key: CASSANDRA-15406
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15406
> Project: Cassandra
> Issue Type: Improvement
> Components: Consistency/Streaming, Legacy/Streaming and Messaging,
> Tool/nodetool
>Reporter: maxwellguo
>Assignee: Stefan Miklosovic
>Priority: Normal
> Fix For: 4.0, 4.x
>
>
> I found that we should supply a command to show the progress of streaming
> when we do the operation of bootstrap/move/decommission/removenode. For when
> do data streaming , noboday knows which steps there program are in , so I
> think a command to show the joing/leaving node's is needed .
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-15657) Improve zero-copy-streaming containment check by using file sections
[ https://issues.apache.org/jira/browse/CASSANDRA-15657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17066128#comment-17066128 ] Stefan Miklosovic edited comment on CASSANDRA-15657 at 3/31/20, 3:29 PM: - [~djoshi] I think this is the duplicate of Jira of yours https://issues.apache.org/jira/browse/CASSANDRA-14586 was (Author: stefan.miklosovic): [~bhagat_dineshbe2006] I think this is the duplicate of Jira of yours https://issues.apache.org/jira/browse/CASSANDRA-14586 > Improve zero-copy-streaming containment check by using file sections > > > Key: CASSANDRA-15657 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15657 > Project: Cassandra > Issue Type: Improvement > Components: Legacy/Streaming and Messaging >Reporter: ZhaoYang >Assignee: ZhaoYang >Priority: Normal > Fix For: 4.0 > > > Currently zero copy streaming is only enabled for leveled-compaction strategy > and it checks if all keys in the sstables are included in the transferred > ranges. > This is very inefficient. The containment check can be improved by checking > if transferred sections (the transferred file positions) cover entire sstable. > I also enabled ZCS for all compaction strategies since the new containment > check is very fast.. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15406) Add command to show the progress of data streaming and index build
[ https://issues.apache.org/jira/browse/CASSANDRA-15406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17072227#comment-17072227 ] Stefan Miklosovic commented on CASSANDRA-15406: --- Hi [~dcapwell], I think that this ticket will be more or less transformed just into a bug fix. I do not thing that there is a need for a "command" for nodetool and having a lot of work before release anyway, it is not a good idea to introduce something completely new. As I see it, this ticket will be just about the introduction of global percentage into netstats output + I think I hit a bug when we stream entire SSTables and netstats is displaying it weirdly so that should be fixed too. > Add command to show the progress of data streaming and index build > --- > > Key: CASSANDRA-15406 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15406 > Project: Cassandra > Issue Type: Improvement > Components: Consistency/Streaming, Legacy/Streaming and Messaging, > Tool/nodetool >Reporter: maxwellguo >Assignee: Stefan Miklosovic >Priority: Normal > Fix For: 4.0, 4.x > > > I found that we should supply a command to show the progress of streaming > when we do the operation of bootstrap/move/decommission/removenode. For when > do data streaming , noboday knows which steps there program are in , so I > think a command to show the joing/leaving node's is needed . -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-15586) 4.0 quality testing: Cluster Setup and Maintenance
[ https://issues.apache.org/jira/browse/CASSANDRA-15586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17070863#comment-17070863 ] Stefan Miklosovic edited comment on CASSANDRA-15586 at 3/30/20, 10:52 AM: -- I was testing alpha3 on FreeBSD 12.1 and openjdk 11 and it worked with some caveats. As you probably know, there is a system of "ports" in FreeBSD which are basically vanilla binary packages / sources which are taken and repackaged to be compatible what FreeBSD runs with. There is currently only Cassandra 3 port of version 3.11.6 which is taken care of by one of a FreeBSD port commiter. I think that the porting work will be done similarly for 4.0 once it is out but I can contact that person on my behalf to know more details and maybe I will even contribute that myself. When it comes to tarball of alpha3, it worked but there was one stacktrace about not having epoll on FreeBSD and that it is skipped but otherwise I was able to connect to it and so on and it seemed to behave just fine. I was trying to install Linux compatibility layer (official thingy from FreeBSD) but the result was just same. My line of thought here is to try to provide Cassandra 4.0 port as there is for Cassandra 3 already. If nobody objects and seems this effort worthy, I might start to hack around that ... FreeBSD port: [https://svnweb.freebsd.org/ports/head/databases/cassandra3/] [~e.dimitrova] feel free to reach me to put head together so we can test stuff according to common plans so we dont reinvent the wheel here. I am more or less fully dedicated to this on everyday basis. was (Author: stefan.miklosovic): I was testing alpha3 on FreeBSD 12.1 and openjdk 11 and it worked with some caveats. As you probably know, there is a system of "ports" in FreeBSD which are basically vanilla binary packages / sources which are taken and repackaged to be compatible what FreeBSD runs with. There is currently only Cassandra 3 port of version 3.11.6 which is taken care of by one of a FreeBSD port commiter. I think that the porting work will be done similarly for 4.0 once it is out but I can contact that person on my behalf to know more details and maybe I will even contribute that myself. When it comes to tarball of alpha3, it worked but there was one stacktrace about not having epoll on FreeBSD and that it is skipped but otherwise I was able to connect to it and so on and it seemed to behave just fine. I was trying to install Linux compatibility layer (official thingy from FreeBSD) but the result was just same. My line of thought here is to try to provide Cassandra 4.0 port as there is for Cassandra 3 already. If nobody objects and seems this effort worthy, I might start to hack around that ... FreeBSD port: [https://svnweb.freebsd.org/ports/head/databases/cassandra3/] > 4.0 quality testing: Cluster Setup and Maintenance > -- > > Key: CASSANDRA-15586 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15586 > Project: Cassandra > Issue Type: Task > Components: Test/dtest >Reporter: Josh McKenzie >Assignee: Ekaterina Dimitrova >Priority: Normal > Labels: 4.0-QA > Fix For: 4.0-rc > > > We want 4.0 to be easy for users to setup out of the box and just work. This > means having low friction when users download the Cassandra package and start > running it. For example, users should be able to easily configure and start > new 4.0 clusters and have tokens distributed evenly. Another example is > packaging, it should be easy to install Cassandra on all supported platforms > (e.g. packaging) and have Cassandra use standard platform integrations. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15406) Add command to show the progress of data streaming and index build
[ https://issues.apache.org/jira/browse/CASSANDRA-15406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17070860#comment-17070860 ] Stefan Miklosovic commented on CASSANDRA-15406: --- [~dcapwell] where do we stand with this? I was reading your pull request for 15399 and do I understand it right that one is postponed indefinitely? (at least until 4.0 is out) I would like to work on this but I am not sure if there is some reasonable probability it will eventually and actually end up being merged before 4.0 is out. Could you elaborate more about what parts of this code would be similar with your issue which ended up postponed so we would not duplicate things? Next, [~maxwellguo], I am not completely sure how you want to actually measure the progress of a bootstrap. As I understand it, if you bootstrap, Cassandra is not completely up so one might only query it via JMX or is it true that virtual tables would work too? [~dcapwell] do you know some entry points in the code base which I could navigate into so I would somehow start to dig deeper? > Add command to show the progress of data streaming and index build > --- > > Key: CASSANDRA-15406 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15406 > Project: Cassandra > Issue Type: Improvement > Components: Consistency/Streaming, Legacy/Streaming and Messaging, > Tool/nodetool >Reporter: maxwellguo >Priority: Normal > Fix For: 4.0, 4.x > > > I found that we should supply a command to show the progress of streaming > when we do the operation of bootstrap/move/decommission/removenode. For when > do data streaming , noboday knows which steps there program are in , so I > think a command to show the joing/leaving node's is needed . -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-15586) 4.0 quality testing: Cluster Setup and Maintenance
[ https://issues.apache.org/jira/browse/CASSANDRA-15586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17070863#comment-17070863 ] Stefan Miklosovic edited comment on CASSANDRA-15586 at 3/30/20, 10:36 AM: -- I was testing alpha3 on FreeBSD 12.1 and it worked with some caveats. As you probably know, there is a system of "ports" in FreeBSD which are basically vanilla binary packages / sources which are taken and repackaged to be comaptible what FreeBSD runs with. There is currently only Cassandra 3 port of version 3.11.6 which is taken care of by one of a FreeBSD port commiter. I think that the porting work will be done similarly for 4.0 once it is out but I can contact that person on my behalf to know more details and maybe I will even contribute that myself. When it comes to tarball of alpha3, it worked but there was one stacktrace about not having epoll on FreeBSD and that it is skipped but otherwise I was able to connect to it and so on and it seemed to behave just fine. I was trying to install Linux compatibility layer (official thingy from FreeBSD) but the result was just same. My line of thought here is to try to provide Cassandra 4.0 port as there is for Cassandra 3 already. If nobody objects and seems this effort worthy, I might start to hack around that ... FreeBSD port: [https://svnweb.freebsd.org/ports/head/databases/cassandra3/] was (Author: stefan.miklosovic): I was testing alpha3 on FreeBSD 12.1 and it worked with some caveats. As you probably know, there is a system of "ports" in FreeBSD which are basically vanilla binary packages / sources which are taken and repackaged to be comaptible what FreeBSD runs with. There is currently only Cassandra 3 port of version 3.11.6 which is taken care of by one of a FreeBSD port commiter. I think that the porting work will be done similarly for 4.0 once it is out but I can contact that person on my behalf to know more details and maybe I will even contribute that myself. When it comes to tarball of alpha3, it worked but there was one stacktrace about not having epoll on FreeBSD and that it is skipped but otherwise I was able to connect to it and so on and it seemed to behave just fine. My line of thought here is to try to provide Cassandra 4.0 port as there is for Cassandra 3 already. If nobody objects and seems this effort worthy, I might start to hack around that ... FreeBSD port: [https://svnweb.freebsd.org/ports/head/databases/cassandra3/] > 4.0 quality testing: Cluster Setup and Maintenance > -- > > Key: CASSANDRA-15586 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15586 > Project: Cassandra > Issue Type: Task > Components: Test/dtest >Reporter: Josh McKenzie >Assignee: Ekaterina Dimitrova >Priority: Normal > Labels: 4.0-QA > Fix For: 4.0-rc > > > We want 4.0 to be easy for users to setup out of the box and just work. This > means having low friction when users download the Cassandra package and start > running it. For example, users should be able to easily configure and start > new 4.0 clusters and have tokens distributed evenly. Another example is > packaging, it should be easy to install Cassandra on all supported platforms > (e.g. packaging) and have Cassandra use standard platform integrations. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-15586) 4.0 quality testing: Cluster Setup and Maintenance
[ https://issues.apache.org/jira/browse/CASSANDRA-15586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17070863#comment-17070863 ] Stefan Miklosovic edited comment on CASSANDRA-15586 at 3/30/20, 10:37 AM: -- I was testing alpha3 on FreeBSD 12.1 and openjdk 11 and it worked with some caveats. As you probably know, there is a system of "ports" in FreeBSD which are basically vanilla binary packages / sources which are taken and repackaged to be comaptible what FreeBSD runs with. There is currently only Cassandra 3 port of version 3.11.6 which is taken care of by one of a FreeBSD port commiter. I think that the porting work will be done similarly for 4.0 once it is out but I can contact that person on my behalf to know more details and maybe I will even contribute that myself. When it comes to tarball of alpha3, it worked but there was one stacktrace about not having epoll on FreeBSD and that it is skipped but otherwise I was able to connect to it and so on and it seemed to behave just fine. I was trying to install Linux compatibility layer (official thingy from FreeBSD) but the result was just same. My line of thought here is to try to provide Cassandra 4.0 port as there is for Cassandra 3 already. If nobody objects and seems this effort worthy, I might start to hack around that ... FreeBSD port: [https://svnweb.freebsd.org/ports/head/databases/cassandra3/] was (Author: stefan.miklosovic): I was testing alpha3 on FreeBSD 12.1 and it worked with some caveats. As you probably know, there is a system of "ports" in FreeBSD which are basically vanilla binary packages / sources which are taken and repackaged to be comaptible what FreeBSD runs with. There is currently only Cassandra 3 port of version 3.11.6 which is taken care of by one of a FreeBSD port commiter. I think that the porting work will be done similarly for 4.0 once it is out but I can contact that person on my behalf to know more details and maybe I will even contribute that myself. When it comes to tarball of alpha3, it worked but there was one stacktrace about not having epoll on FreeBSD and that it is skipped but otherwise I was able to connect to it and so on and it seemed to behave just fine. I was trying to install Linux compatibility layer (official thingy from FreeBSD) but the result was just same. My line of thought here is to try to provide Cassandra 4.0 port as there is for Cassandra 3 already. If nobody objects and seems this effort worthy, I might start to hack around that ... FreeBSD port: [https://svnweb.freebsd.org/ports/head/databases/cassandra3/] > 4.0 quality testing: Cluster Setup and Maintenance > -- > > Key: CASSANDRA-15586 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15586 > Project: Cassandra > Issue Type: Task > Components: Test/dtest >Reporter: Josh McKenzie >Assignee: Ekaterina Dimitrova >Priority: Normal > Labels: 4.0-QA > Fix For: 4.0-rc > > > We want 4.0 to be easy for users to setup out of the box and just work. This > means having low friction when users download the Cassandra package and start > running it. For example, users should be able to easily configure and start > new 4.0 clusters and have tokens distributed evenly. Another example is > packaging, it should be easy to install Cassandra on all supported platforms > (e.g. packaging) and have Cassandra use standard platform integrations. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15399) Add ability to track state in repair
[
https://issues.apache.org/jira/browse/CASSANDRA-15399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17070861#comment-17070861
]
Stefan Miklosovic commented on CASSANDRA-15399:
---
[~dcapwell] is this your definitive answer in regard of postponing this feature
until 4.0 is out? Since CASSANDRA-15406 is there too which you say it is
related to, does it mean that feature is not going to happen either or I am
free do dig into it? I do not want to reinvent the wheel so you telling me what
parts of the code are common (as you say in 15406) would be very beneficial and
I would be grateful for that a lot.
> Add ability to track state in repair
>
>
> Key: CASSANDRA-15399
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15399
> Project: Cassandra
> Issue Type: Improvement
> Components: Consistency/Repair
>Reporter: David Capwell
>Assignee: David Capwell
>Priority: Normal
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> To enhance the visibility in repair, we should expose internal state via
> virtual tables; the state should include coordinator as well as participant
> state (validation, sync, etc.)
> I propose the following tables:
> repairs - high level summary of the global state of repair; this should be
> called on the coordinator.
> {code:sql}
> CREATE TABLE repairs (
> id uuid,
> keyspace_name text,
> table_names frozen>,
> ranges frozen>,
> coordinator text,
> participants frozen>,
> state text,
> progress_percentage float,
> last_updated_at_millis bigint,
> duration_micro bigint,
> failure_cause text,
> PRIMARY KEY ( (id) )
> )
> {code}
> repair_tasks - represents RepairJob and participants state. This will show
> if validations are running on participants and the progress they are making;
> this should be called on the coordinator.
> {code:sql}
> CREATE TABLE repair_tasks (
> id uuid,
> session_id uuid,
> keyspace_name text,
> table_name text,
> ranges frozen>,
> coordinator text,
> participant text,
> state text,
> state_description text,
> progress_percentage float, -- between 0.0 and 100.0
> last_updated_at_millis bigint,
> duration_micro bigint,
> failure_cause text,
> PRIMARY KEY ( (id), session_id, table_name, participant )
> )
> {code}
> repair_validations - shows the state of the validation task and updated
> periodically while validation is running; this should be called on the
> participants.
> {code:sql}
> CREATE TABLE repair_validations (
> id uuid,
> session_id uuid,
> ranges frozen>,
> keyspace_name text,
> table_name text,
> initiator text,
> state text,
> progress_percentage float,
> queue_duration_ms bigint,
> runtime_duration_ms bigint,
> total_duration_ms bigint,
> estimated_partitions bigint,
> partitions_processed bigint,
> estimated_total_bytes bigint,
> failure_cause text,
> PRIMARY KEY ( (id), session_id, table_name )
> )
> {code}
> The main reason for exposing virtual tables rather than exposing through
> durable tables is to make sure what is exposed is accurate. In cases of
> write failures or node failures, the durable tables could become in-accurate
> and could add edge cases where the repair is not running but the tables say
> it is; by relying on repair's internal in-memory bookkeeping, these problems
> go away.
> This jira does not try to solve the following:
> 1) repair resiliency - there are edge cases where repair hits an error and
> runs forever (at least from nodetool's perspective).
> 2) repair stream tracking - I have not learned the streaming side yet and
> what I see is multiple implementations exist, so seems like high scope. My
> hope is to punt from this jira and tackle separately.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15586) 4.0 quality testing: Cluster Setup and Maintenance
[ https://issues.apache.org/jira/browse/CASSANDRA-15586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17070863#comment-17070863 ] Stefan Miklosovic commented on CASSANDRA-15586: --- I was testing alpha3 on FreeBSD 12.1 and it worked with some caveats. As you probably know, there is a system of "ports" in FreeBSD which are basically vanilla binary packages / sources which are taken and repackaged to be comaptible what FreeBSD runs with. There is currently only Cassandra 3 port of version 3.11.6 which is taken care of by one of a FreeBSD port commiter. I think that the porting work will be done similarly for 4.0 once it is out but I can contact that person on my behalf to know more details and maybe I will even contribute that myself. When it comes to tarball of alpha3, it worked but there was one stacktrace about not having epoll on FreeBSD and that it is skipped but otherwise I was able to connect to it and so on and it seemed to behave just fine. My line of thought here is to try to provide Cassandra 4.0 port as there is for Cassandra 3 already. If nobody objects and seems this effort worthy, I might start to hack around that ... FreeBSD port: [https://svnweb.freebsd.org/ports/head/databases/cassandra3/] > 4.0 quality testing: Cluster Setup and Maintenance > -- > > Key: CASSANDRA-15586 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15586 > Project: Cassandra > Issue Type: Task > Components: Test/dtest >Reporter: Josh McKenzie >Assignee: Ekaterina Dimitrova >Priority: Normal > Labels: 4.0-QA > Fix For: 4.0-rc > > > We want 4.0 to be easy for users to setup out of the box and just work. This > means having low friction when users download the Cassandra package and start > running it. For example, users should be able to easily configure and start > new 4.0 clusters and have tokens distributed evenly. Another example is > packaging, it should be easy to install Cassandra on all supported platforms > (e.g. packaging) and have Cassandra use standard platform integrations. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-15586) 4.0 quality testing: Cluster Setup and Maintenance
[ https://issues.apache.org/jira/browse/CASSANDRA-15586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17070863#comment-17070863 ] Stefan Miklosovic edited comment on CASSANDRA-15586 at 3/30/20, 10:42 AM: -- I was testing alpha3 on FreeBSD 12.1 and openjdk 11 and it worked with some caveats. As you probably know, there is a system of "ports" in FreeBSD which are basically vanilla binary packages / sources which are taken and repackaged to be compatible what FreeBSD runs with. There is currently only Cassandra 3 port of version 3.11.6 which is taken care of by one of a FreeBSD port commiter. I think that the porting work will be done similarly for 4.0 once it is out but I can contact that person on my behalf to know more details and maybe I will even contribute that myself. When it comes to tarball of alpha3, it worked but there was one stacktrace about not having epoll on FreeBSD and that it is skipped but otherwise I was able to connect to it and so on and it seemed to behave just fine. I was trying to install Linux compatibility layer (official thingy from FreeBSD) but the result was just same. My line of thought here is to try to provide Cassandra 4.0 port as there is for Cassandra 3 already. If nobody objects and seems this effort worthy, I might start to hack around that ... FreeBSD port: [https://svnweb.freebsd.org/ports/head/databases/cassandra3/] was (Author: stefan.miklosovic): I was testing alpha3 on FreeBSD 12.1 and openjdk 11 and it worked with some caveats. As you probably know, there is a system of "ports" in FreeBSD which are basically vanilla binary packages / sources which are taken and repackaged to be comaptible what FreeBSD runs with. There is currently only Cassandra 3 port of version 3.11.6 which is taken care of by one of a FreeBSD port commiter. I think that the porting work will be done similarly for 4.0 once it is out but I can contact that person on my behalf to know more details and maybe I will even contribute that myself. When it comes to tarball of alpha3, it worked but there was one stacktrace about not having epoll on FreeBSD and that it is skipped but otherwise I was able to connect to it and so on and it seemed to behave just fine. I was trying to install Linux compatibility layer (official thingy from FreeBSD) but the result was just same. My line of thought here is to try to provide Cassandra 4.0 port as there is for Cassandra 3 already. If nobody objects and seems this effort worthy, I might start to hack around that ... FreeBSD port: [https://svnweb.freebsd.org/ports/head/databases/cassandra3/] > 4.0 quality testing: Cluster Setup and Maintenance > -- > > Key: CASSANDRA-15586 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15586 > Project: Cassandra > Issue Type: Task > Components: Test/dtest >Reporter: Josh McKenzie >Assignee: Ekaterina Dimitrova >Priority: Normal > Labels: 4.0-QA > Fix For: 4.0-rc > > > We want 4.0 to be easy for users to setup out of the box and just work. This > means having low friction when users download the Cassandra package and start > running it. For example, users should be able to easily configure and start > new 4.0 clusters and have tokens distributed evenly. Another example is > packaging, it should be easy to install Cassandra on all supported platforms > (e.g. packaging) and have Cassandra use standard platform integrations. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-15406) Add command to show the progress of data streaming and index build
[ https://issues.apache.org/jira/browse/CASSANDRA-15406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17070979#comment-17070979 ] Stefan Miklosovic edited comment on CASSANDRA-15406 at 3/30/20, 1:50 PM: - I will assign this to myself temporarily (as it was unassigned), is somebody objects please feel free to raise your opinion here. Thank you. was (Author: stefan.miklosovic): I will assign this to myself temporarily, is somebody objects please feel free to raise your opinion here. Thank you. > Add command to show the progress of data streaming and index build > --- > > Key: CASSANDRA-15406 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15406 > Project: Cassandra > Issue Type: Improvement > Components: Consistency/Streaming, Legacy/Streaming and Messaging, > Tool/nodetool >Reporter: maxwellguo >Assignee: Stefan Miklosovic >Priority: Normal > Fix For: 4.0, 4.x > > > I found that we should supply a command to show the progress of streaming > when we do the operation of bootstrap/move/decommission/removenode. For when > do data streaming , noboday knows which steps there program are in , so I > think a command to show the joing/leaving node's is needed . -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Assigned] (CASSANDRA-15406) Add command to show the progress of data streaming and index build
[ https://issues.apache.org/jira/browse/CASSANDRA-15406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Miklosovic reassigned CASSANDRA-15406: - Assignee: Stefan Miklosovic > Add command to show the progress of data streaming and index build > --- > > Key: CASSANDRA-15406 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15406 > Project: Cassandra > Issue Type: Improvement > Components: Consistency/Streaming, Legacy/Streaming and Messaging, > Tool/nodetool >Reporter: maxwellguo >Assignee: Stefan Miklosovic >Priority: Normal > Fix For: 4.0, 4.x > > > I found that we should supply a command to show the progress of streaming > when we do the operation of bootstrap/move/decommission/removenode. For when > do data streaming , noboday knows which steps there program are in , so I > think a command to show the joing/leaving node's is needed . -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15406) Add command to show the progress of data streaming and index build
[ https://issues.apache.org/jira/browse/CASSANDRA-15406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17070979#comment-17070979 ] Stefan Miklosovic commented on CASSANDRA-15406: --- I will assign this to myself temporarily, is somebody objects please feel free to raise your opinion here. Thank you. > Add command to show the progress of data streaming and index build > --- > > Key: CASSANDRA-15406 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15406 > Project: Cassandra > Issue Type: Improvement > Components: Consistency/Streaming, Legacy/Streaming and Messaging, > Tool/nodetool >Reporter: maxwellguo >Priority: Normal > Fix For: 4.0, 4.x > > > I found that we should supply a command to show the progress of streaming > when we do the operation of bootstrap/move/decommission/removenode. For when > do data streaming , noboday knows which steps there program are in , so I > think a command to show the joing/leaving node's is needed . -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-15586) 4.0 quality testing: Cluster Setup and Maintenance
[ https://issues.apache.org/jira/browse/CASSANDRA-15586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17070863#comment-17070863 ] Stefan Miklosovic edited comment on CASSANDRA-15586 at 3/30/20, 11:05 AM: -- I was testing alpha3 on FreeBSD 12.1 and openjdk 11 and Python 2 and it worked with some caveats. As you probably know, there is a system of "ports" in FreeBSD which are basically vanilla binary packages / sources which are taken and repackaged to be compatible what FreeBSD runs with. There is currently only Cassandra 3 port of version 3.11.6 which is taken care of by one of a FreeBSD port commiter. I think that the porting work will be done similarly for 4.0 once it is out but I can contact that person on my behalf to know more details and maybe I will even contribute that myself. When it comes to tarball of alpha3, it worked but there was one stacktrace about not having epoll on FreeBSD and that it is skipped but otherwise I was able to connect to it and so on and it seemed to behave just fine. I was trying to install Linux compatibility layer (official thingy from FreeBSD) but the result was just same. My line of thought here is to try to provide Cassandra 4.0 port as there is for Cassandra 3 already. If nobody objects and seems this effort worthy, I might start to hack around that ... FreeBSD port: [https://svnweb.freebsd.org/ports/head/databases/cassandra3/] [~e.dimitrova] feel free to reach me to put head together so we can test stuff according to common plans so we dont reinvent the wheel here. I am more or less fully dedicated to this on everyday basis. was (Author: stefan.miklosovic): I was testing alpha3 on FreeBSD 12.1 and openjdk 11 and it worked with some caveats. As you probably know, there is a system of "ports" in FreeBSD which are basically vanilla binary packages / sources which are taken and repackaged to be compatible what FreeBSD runs with. There is currently only Cassandra 3 port of version 3.11.6 which is taken care of by one of a FreeBSD port commiter. I think that the porting work will be done similarly for 4.0 once it is out but I can contact that person on my behalf to know more details and maybe I will even contribute that myself. When it comes to tarball of alpha3, it worked but there was one stacktrace about not having epoll on FreeBSD and that it is skipped but otherwise I was able to connect to it and so on and it seemed to behave just fine. I was trying to install Linux compatibility layer (official thingy from FreeBSD) but the result was just same. My line of thought here is to try to provide Cassandra 4.0 port as there is for Cassandra 3 already. If nobody objects and seems this effort worthy, I might start to hack around that ... FreeBSD port: [https://svnweb.freebsd.org/ports/head/databases/cassandra3/] [~e.dimitrova] feel free to reach me to put head together so we can test stuff according to common plans so we dont reinvent the wheel here. I am more or less fully dedicated to this on everyday basis. > 4.0 quality testing: Cluster Setup and Maintenance > -- > > Key: CASSANDRA-15586 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15586 > Project: Cassandra > Issue Type: Task > Components: Test/dtest >Reporter: Josh McKenzie >Assignee: Ekaterina Dimitrova >Priority: Normal > Labels: 4.0-QA > Fix For: 4.0-rc > > > We want 4.0 to be easy for users to setup out of the box and just work. This > means having low friction when users download the Cassandra package and start > running it. For example, users should be able to easily configure and start > new 4.0 clusters and have tokens distributed evenly. Another example is > packaging, it should be easy to install Cassandra on all supported platforms > (e.g. packaging) and have Cassandra use standard platform integrations. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15657) Improve zero-copy-streaming containment check by using file sections
[ https://issues.apache.org/jira/browse/CASSANDRA-15657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17073070#comment-17073070 ] Stefan Miklosovic commented on CASSANDRA-15657: --- [~tjake] would you mind to look at this? > Improve zero-copy-streaming containment check by using file sections > > > Key: CASSANDRA-15657 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15657 > Project: Cassandra > Issue Type: Improvement > Components: Legacy/Streaming and Messaging >Reporter: ZhaoYang >Assignee: ZhaoYang >Priority: Normal > Fix For: 4.0 > > > Currently zero copy streaming is only enabled for leveled-compaction strategy > and it checks if all keys in the sstables are included in the transferred > ranges. > This is very inefficient. The containment check can be improved by checking > if transferred sections (the transferred file positions) cover entire sstable. > I also enabled ZCS for all compaction strategies since the new containment > check is very fast.. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-15657) Improve zero-copy-streaming containment check by using file sections
[ https://issues.apache.org/jira/browse/CASSANDRA-15657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17073070#comment-17073070 ] Stefan Miklosovic edited comment on CASSANDRA-15657 at 4/1/20, 6:31 PM: [~tjake] would you mind to look at this please? was (Author: stefan.miklosovic): [~tjake] would you mind to look at this? > Improve zero-copy-streaming containment check by using file sections > > > Key: CASSANDRA-15657 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15657 > Project: Cassandra > Issue Type: Improvement > Components: Legacy/Streaming and Messaging >Reporter: ZhaoYang >Assignee: ZhaoYang >Priority: Normal > Fix For: 4.0 > > > Currently zero copy streaming is only enabled for leveled-compaction strategy > and it checks if all keys in the sstables are included in the transferred > ranges. > This is very inefficient. The containment check can be improved by checking > if transferred sections (the transferred file positions) cover entire sstable. > I also enabled ZCS for all compaction strategies since the new containment > check is very fast.. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14606) Add documentation for java 11 support
[ https://issues.apache.org/jira/browse/CASSANDRA-14606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17067775#comment-17067775 ] Stefan Miklosovic commented on CASSANDRA-14606: --- This was likely already covered? I see Java 11 related documentation and build steps here [https://cassandra.apache.org/doc/latest/new/java11.html] > Add documentation for java 11 support > - > > Key: CASSANDRA-14606 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14606 > Project: Cassandra > Issue Type: Improvement > Components: Legacy/Documentation and Website >Reporter: Jason Brown >Priority: Low > Labels: Java11 > Fix For: 4.0 > > > Let's add some documentation for operators around the java 11 support that > was introduced in CASSANDRA-9608. Also, we should point out changes in the > scripts that might affect automation that operators have in place. > Parking on [~snazy] just 'cuz ;) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-15158) Wait for schema agreement rather then in flight schema requests when bootstrapping
[ https://issues.apache.org/jira/browse/CASSANDRA-15158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17093752#comment-17093752 ] Stefan Miklosovic edited comment on CASSANDRA-15158 at 4/28/20, 7:35 PM: - Hi [~bdeggleston] I took the patch and rewored it little bit [https://github.com/smiklosovic/cassandra/tree/CASSANDRA-15158-2] Looking forward to have some feedback! was (Author: stefan.miklosovic): Hi [~bdeggleston] I took the patch and rewored it little bit [https://github.com/smiklosovic/cassandra/commits/CASSANDRA-15158] You have said that there is not any way how to confirm that any in flight migration tasks have been completed and applied. I do not know how to verify this was indeed done but what I did is that I have exposed the information if a particular migration task has failed or not to process (based on onFailure callback) so we can work on this further if necessary. Logically, it is same stuff as it was before in the original patch but the code is reorganised a bit. The "escape hatch" is one global bootstrap timeout, if it is passed and schemas are still not in agreement, it is still uknown to me what we want to do - either fail completely and halt that node or we allow to proceed with big fat warning. Looking forward to have some feedback! > Wait for schema agreement rather then in flight schema requests when > bootstrapping > -- > > Key: CASSANDRA-15158 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15158 > Project: Cassandra > Issue Type: Bug > Components: Cluster/Gossip, Cluster/Schema >Reporter: Vincent White >Assignee: Ben Bromhead >Priority: Normal > > Currently when a node is bootstrapping we use a set of latches > (org.apache.cassandra.service.MigrationTask#inflightTasks) to keep track of > in-flight schema pull requests, and we don't proceed with > bootstrapping/stream until all the latches are released (or we timeout > waiting for each one). One issue with this is that if we have a large schema, > or the retrieval of the schema from the other nodes was unexpectedly slow > then we have no explicit check in place to ensure we have actually received a > schema before we proceed. > While it's possible to increase "migration_task_wait_in_seconds" to force the > node to wait on each latche longer, there are cases where this doesn't help > because the callbacks for the schema pull requests have expired off the > messaging service's callback map > (org.apache.cassandra.net.MessagingService#callbacks) after > request_timeout_in_ms (default 10 seconds) before the other nodes were able > to respond to the new node. > This patch checks for schema agreement between the bootstrapping node and the > rest of the live nodes before proceeding with bootstrapping. It also adds a > check to prevent the new node from flooding existing nodes with simultaneous > schema pull requests as can happen in large clusters. > Removing the latch system should also prevent new nodes in large clusters > getting stuck for extended amounts of time as they wait > `migration_task_wait_in_seconds` on each of the latches left orphaned by the > timed out callbacks. > > ||3.11|| > |[PoC|https://github.com/apache/cassandra/compare/cassandra-3.11...vincewhite:check_for_schema]| > |[dtest|https://github.com/apache/cassandra-dtest/compare/master...vincewhite:wait_for_schema_agreement]| > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15158) Wait for schema agreement rather then in flight schema requests when bootstrapping
[ https://issues.apache.org/jira/browse/CASSANDRA-15158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17093752#comment-17093752 ] Stefan Miklosovic commented on CASSANDRA-15158: --- Hi [~bdeggleston] I took the patch and rewored it little bit [https://github.com/smiklosovic/cassandra/commits/CASSANDRA-15158] You have said that there is not any way how to confirm that any in flight migration tasks have been completed and applied. I do not know how to verify this was indeed done but what I did is that I have exposed the information if a particular migration task has failed or nor to process (based on onFailure callback) so we can work on this further if necessary. Logically, it is same stuff as it was before in the original patch but the code is reorganised a bit. The "escape hatch" is one global bootstrap timeout, if it is passed and schemas are still not in agreement, it is still uknown to me what we want to do - either fail completely and halt that service or we allow to proceed with big fat warning. Looking forward to have some feedback! > Wait for schema agreement rather then in flight schema requests when > bootstrapping > -- > > Key: CASSANDRA-15158 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15158 > Project: Cassandra > Issue Type: Bug > Components: Cluster/Gossip, Cluster/Schema >Reporter: Vincent White >Assignee: Ben Bromhead >Priority: Normal > > Currently when a node is bootstrapping we use a set of latches > (org.apache.cassandra.service.MigrationTask#inflightTasks) to keep track of > in-flight schema pull requests, and we don't proceed with > bootstrapping/stream until all the latches are released (or we timeout > waiting for each one). One issue with this is that if we have a large schema, > or the retrieval of the schema from the other nodes was unexpectedly slow > then we have no explicit check in place to ensure we have actually received a > schema before we proceed. > While it's possible to increase "migration_task_wait_in_seconds" to force the > node to wait on each latche longer, there are cases where this doesn't help > because the callbacks for the schema pull requests have expired off the > messaging service's callback map > (org.apache.cassandra.net.MessagingService#callbacks) after > request_timeout_in_ms (default 10 seconds) before the other nodes were able > to respond to the new node. > This patch checks for schema agreement between the bootstrapping node and the > rest of the live nodes before proceeding with bootstrapping. It also adds a > check to prevent the new node from flooding existing nodes with simultaneous > schema pull requests as can happen in large clusters. > Removing the latch system should also prevent new nodes in large clusters > getting stuck for extended amounts of time as they wait > `migration_task_wait_in_seconds` on each of the latches left orphaned by the > timed out callbacks. > > ||3.11|| > |[PoC|https://github.com/apache/cassandra/compare/cassandra-3.11...vincewhite:check_for_schema]| > |[dtest|https://github.com/apache/cassandra-dtest/compare/master...vincewhite:wait_for_schema_agreement]| > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-15158) Wait for schema agreement rather then in flight schema requests when bootstrapping
[ https://issues.apache.org/jira/browse/CASSANDRA-15158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17093752#comment-17093752 ] Stefan Miklosovic edited comment on CASSANDRA-15158 at 4/27/20, 7:19 PM: - Hi [~bdeggleston] I took the patch and rewored it little bit [https://github.com/smiklosovic/cassandra/commits/CASSANDRA-15158] You have said that there is not any way how to confirm that any in flight migration tasks have been completed and applied. I do not know how to verify this was indeed done but what I did is that I have exposed the information if a particular migration task has failed or not to process (based on onFailure callback) so we can work on this further if necessary. Logically, it is same stuff as it was before in the original patch but the code is reorganised a bit. The "escape hatch" is one global bootstrap timeout, if it is passed and schemas are still not in agreement, it is still uknown to me what we want to do - either fail completely and halt that node or we allow to proceed with big fat warning. Looking forward to have some feedback! was (Author: stefan.miklosovic): Hi [~bdeggleston] I took the patch and rewored it little bit [https://github.com/smiklosovic/cassandra/commits/CASSANDRA-15158] You have said that there is not any way how to confirm that any in flight migration tasks have been completed and applied. I do not know how to verify this was indeed done but what I did is that I have exposed the information if a particular migration task has failed or nor to process (based on onFailure callback) so we can work on this further if necessary. Logically, it is same stuff as it was before in the original patch but the code is reorganised a bit. The "escape hatch" is one global bootstrap timeout, if it is passed and schemas are still not in agreement, it is still uknown to me what we want to do - either fail completely and halt that service or we allow to proceed with big fat warning. Looking forward to have some feedback! > Wait for schema agreement rather then in flight schema requests when > bootstrapping > -- > > Key: CASSANDRA-15158 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15158 > Project: Cassandra > Issue Type: Bug > Components: Cluster/Gossip, Cluster/Schema >Reporter: Vincent White >Assignee: Ben Bromhead >Priority: Normal > > Currently when a node is bootstrapping we use a set of latches > (org.apache.cassandra.service.MigrationTask#inflightTasks) to keep track of > in-flight schema pull requests, and we don't proceed with > bootstrapping/stream until all the latches are released (or we timeout > waiting for each one). One issue with this is that if we have a large schema, > or the retrieval of the schema from the other nodes was unexpectedly slow > then we have no explicit check in place to ensure we have actually received a > schema before we proceed. > While it's possible to increase "migration_task_wait_in_seconds" to force the > node to wait on each latche longer, there are cases where this doesn't help > because the callbacks for the schema pull requests have expired off the > messaging service's callback map > (org.apache.cassandra.net.MessagingService#callbacks) after > request_timeout_in_ms (default 10 seconds) before the other nodes were able > to respond to the new node. > This patch checks for schema agreement between the bootstrapping node and the > rest of the live nodes before proceeding with bootstrapping. It also adds a > check to prevent the new node from flooding existing nodes with simultaneous > schema pull requests as can happen in large clusters. > Removing the latch system should also prevent new nodes in large clusters > getting stuck for extended amounts of time as they wait > `migration_task_wait_in_seconds` on each of the latches left orphaned by the > timed out callbacks. > > ||3.11|| > |[PoC|https://github.com/apache/cassandra/compare/cassandra-3.11...vincewhite:check_for_schema]| > |[dtest|https://github.com/apache/cassandra-dtest/compare/master...vincewhite:wait_for_schema_agreement]| > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Issue Comment Deleted] (CASSANDRA-15158) Wait for schema agreement rather then in flight schema requests when bootstrapping
[ https://issues.apache.org/jira/browse/CASSANDRA-15158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Miklosovic updated CASSANDRA-15158: -- Comment: was deleted (was: Hi [~bdeggleston] I took the patch and rewored it little bit [https://github.com/smiklosovic/cassandra/tree/CASSANDRA-15158-2] Looking forward to have some feedback!) > Wait for schema agreement rather then in flight schema requests when > bootstrapping > -- > > Key: CASSANDRA-15158 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15158 > Project: Cassandra > Issue Type: Bug > Components: Cluster/Gossip, Cluster/Schema >Reporter: Vincent White >Assignee: Ben Bromhead >Priority: Normal > > Currently when a node is bootstrapping we use a set of latches > (org.apache.cassandra.service.MigrationTask#inflightTasks) to keep track of > in-flight schema pull requests, and we don't proceed with > bootstrapping/stream until all the latches are released (or we timeout > waiting for each one). One issue with this is that if we have a large schema, > or the retrieval of the schema from the other nodes was unexpectedly slow > then we have no explicit check in place to ensure we have actually received a > schema before we proceed. > While it's possible to increase "migration_task_wait_in_seconds" to force the > node to wait on each latche longer, there are cases where this doesn't help > because the callbacks for the schema pull requests have expired off the > messaging service's callback map > (org.apache.cassandra.net.MessagingService#callbacks) after > request_timeout_in_ms (default 10 seconds) before the other nodes were able > to respond to the new node. > This patch checks for schema agreement between the bootstrapping node and the > rest of the live nodes before proceeding with bootstrapping. It also adds a > check to prevent the new node from flooding existing nodes with simultaneous > schema pull requests as can happen in large clusters. > Removing the latch system should also prevent new nodes in large clusters > getting stuck for extended amounts of time as they wait > `migration_task_wait_in_seconds` on each of the latches left orphaned by the > timed out callbacks. > > ||3.11|| > |[PoC|https://github.com/apache/cassandra/compare/cassandra-3.11...vincewhite:check_for_schema]| > |[dtest|https://github.com/apache/cassandra-dtest/compare/master...vincewhite:wait_for_schema_agreement]| > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15759) Add progress column in percents for system_views.sstable_tasks
[
https://issues.apache.org/jira/browse/CASSANDRA-15759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17092322#comment-17092322
]
Stefan Miklosovic commented on CASSANDRA-15759:
---
Hi [~cnlwsu], [~aleksey], I am tagging you Chris as you have implemented
SSTableTasksTable in CASSANDRA-14457 and Aleksey has reviewed that. This one
would be very handy to have in 4.0 indeed! Is there any chance this would go
in?
> Add progress column in percents for system_views.sstable_tasks
> --
>
> Key: CASSANDRA-15759
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15759
> Project: Cassandra
> Issue Type: Improvement
> Components: Feature/Virtual Tables
>Reporter: Stefan Miklosovic
>Priority: Normal
> Time Spent: 10m
> Remaining Estimate: 0h
>
> It would be very handy to have a percentage column in
> system_views.sstable_tasks which would say how far a respective task is.
> Indeed, there are currently "progress" and "total" columns but honestly, for
> every day usage, it is rather strange to expect that humans will divide these
> numbers in head if they want to roughly know what the overall progress is.
> One just does not have a rough estimation of the task progress when he is
> presented with two quite big numbers and to estimate the progress from the.
> In the following output, the field "progress_in_percents" is introduced.
> {code:java}
> admin@cqlsh> select * from system_views.sstable_tasks ;
> @ Row 1
> ---+--
> keyspace_name | mykeyspace
> table_name| mytable
> task_id | 0db5d9b1-8726-11ea-8a6c-b92f3be367bb
> kind | secondary index build
> progress | 19456965
> progress_in_percents | 8.17
> total | 238208674
> unit | bytes
> @ Row 2
> --+--
> keyspace_name| mykeyspace
> table_name | mytable.mytable_surname_idx
> task_id | 1817ee71-8726-11ea-8a6c-b92f3be367bb
> kind | compaction
> progress | 284396233
> progress_in_percents | 75.92
> total| 374598446
> unit | bytes
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-15759) Add progress column in percents for system_views.sstable_tasks
Stefan Miklosovic created CASSANDRA-15759:
-
Summary: Add progress column in percents for
system_views.sstable_tasks
Key: CASSANDRA-15759
URL: https://issues.apache.org/jira/browse/CASSANDRA-15759
Project: Cassandra
Issue Type: Improvement
Components: Feature/Virtual Tables
Reporter: Stefan Miklosovic
It would be very handy to have a percentage column in
system_views.sstable_tasks which would say how far a respective task is.
Indeed, there are currently "progress" and "total" columns but honestly, for
every day usage, it is rather strange to expect that humans will divide these
numbers in head if they want to roughly know what the overall progress is. One
just does not have a rough estimation of the task progress when he is presented
with two quite big numbers and to estimate the progress from the.
In the following output, the field "progress_in_percents" is introduced.
{code:java}
admin@cqlsh> select * from system_views.sstable_tasks ;
@ Row 1
---+--
keyspace_name | mykeyspace
table_name| mytable
task_id | 0db5d9b1-8726-11ea-8a6c-b92f3be367bb
kind | secondary index build
progress | 19456965
progress_in_percents | 8.17
total | 238208674
unit | bytes
@ Row 2
--+--
keyspace_name| mykeyspace
table_name | mytable.mytable_surname_idx
task_id | 1817ee71-8726-11ea-8a6c-b92f3be367bb
kind | compaction
progress | 284396233
progress_in_percents | 75.92
total| 374598446
unit | bytes
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15759) Add progress column in percents for system_views.sstable_tasks
[
https://issues.apache.org/jira/browse/CASSANDRA-15759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Miklosovic updated CASSANDRA-15759:
--
Description:
It would be very handy to have a percentage column in
system_views.sstable_tasks which would say how far a respective task is.
Indeed, there are currently "progress" and "total" columns but honestly, for
every day usage, it is rather strange to expect that humans will divide these
numbers in head if they want to roughly know what the overall progress is. One
just does not have a rough estimation of the task progress when he is presented
with two quite big numbers and to estimate the progress from the.
In the following output, the field "progress_in_percents" is introduced.
PR is here [https://github.com/apache/cassandra/pull/566]
{code:java}
admin@cqlsh> select * from system_views.sstable_tasks ;
@ Row 1
---+--
keyspace_name | mykeyspace
table_name| mytable
task_id | 0db5d9b1-8726-11ea-8a6c-b92f3be367bb
kind | secondary index build
progress | 19456965
progress_in_percents | 8.17
total | 238208674
unit | bytes
@ Row 2
--+--
keyspace_name| mykeyspace
table_name | mytable.mytable_surname_idx
task_id | 1817ee71-8726-11ea-8a6c-b92f3be367bb
kind | compaction
progress | 284396233
progress_in_percents | 75.92
total| 374598446
unit | bytes
{code}
was:
It would be very handy to have a percentage column in
system_views.sstable_tasks which would say how far a respective task is.
Indeed, there are currently "progress" and "total" columns but honestly, for
every day usage, it is rather strange to expect that humans will divide these
numbers in head if they want to roughly know what the overall progress is. One
just does not have a rough estimation of the task progress when he is presented
with two quite big numbers and to estimate the progress from the.
In the following output, the field "progress_in_percents" is introduced.
{code:java}
admin@cqlsh> select * from system_views.sstable_tasks ;
@ Row 1
---+--
keyspace_name | mykeyspace
table_name| mytable
task_id | 0db5d9b1-8726-11ea-8a6c-b92f3be367bb
kind | secondary index build
progress | 19456965
progress_in_percents | 8.17
total | 238208674
unit | bytes
@ Row 2
--+--
keyspace_name| mykeyspace
table_name | mytable.mytable_surname_idx
task_id | 1817ee71-8726-11ea-8a6c-b92f3be367bb
kind | compaction
progress | 284396233
progress_in_percents | 75.92
total| 374598446
unit | bytes
{code}
> Add progress column in percents for system_views.sstable_tasks
> --
>
> Key: CASSANDRA-15759
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15759
> Project: Cassandra
> Issue Type: Improvement
> Components: Feature/Virtual Tables
>Reporter: Stefan Miklosovic
>Priority: Normal
> Time Spent: 10m
> Remaining Estimate: 0h
>
> It would be very handy to have a percentage column in
> system_views.sstable_tasks which would say how far a respective task is.
> Indeed, there are currently "progress" and "total" columns but honestly, for
> every day usage, it is rather strange to expect that humans will divide these
> numbers in head if they want to roughly know what the overall progress is.
> One just does not have a rough estimation of the task progress when he is
> presented with two quite big numbers and to estimate the progress from the.
> In the following output, the field "progress_in_percents" is introduced.
>
> PR is here [https://github.com/apache/cassandra/pull/566]
>
> {code:java}
> admin@cqlsh> select * from system_views.sstable_tasks ;
> @ Row 1
> ---+--
> keyspace_name | mykeyspace
> table_name| mytable
> task_id | 0db5d9b1-8726-11ea-8a6c-b92f3be367bb
> kind | secondary index build
> progress | 19456965
> progress_in_percents | 8.17
> total | 238208674
> unit | bytes
> @ Row 2
> --+--
> keyspace_name| mykeyspace
> table_name | mytable.mytable_surname_idx
> task_id |
[jira] [Commented] (CASSANDRA-15406) Add command to show the progress of data streaming and index build
[ https://issues.apache.org/jira/browse/CASSANDRA-15406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17092324#comment-17092324 ] Stefan Miklosovic commented on CASSANDRA-15406: --- For progress of index building, I have created this issue and respective patch https://issues.apache.org/jira/browse/CASSANDRA-15759 I think we should reuse system views as much as possible once we have it. > Add command to show the progress of data streaming and index build > --- > > Key: CASSANDRA-15406 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15406 > Project: Cassandra > Issue Type: Improvement > Components: Consistency/Streaming, Legacy/Streaming and Messaging, > Tool/nodetool >Reporter: maxwellguo >Assignee: Stefan Miklosovic >Priority: Normal > Fix For: 4.0, 4.x > > Time Spent: 10m > Remaining Estimate: 0h > > I found that we should supply a command to show the progress of streaming > when we do the operation of bootstrap/move/decommission/removenode. For when > do data streaming , noboday knows which steps there program are in , so I > think a command to show the joing/leaving node's is needed . > > PR [https://github.com/apache/cassandra/pull/558] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15759) Add progress column in percents for system_views.sstable_tasks
[
https://issues.apache.org/jira/browse/CASSANDRA-15759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Miklosovic updated CASSANDRA-15759:
--
Description:
It would be very handy to have a percentage column in
system_views.sstable_tasks which would say how far a respective task is.
Indeed, there are currently "progress" and "total" columns but honestly, for
every day usage, it is rather strange to expect that humans will divide these
numbers in head if they want to roughly know what the overall progress is. One
just does not have a rough estimation of the task progress when he is presented
with two quite big numbers and to estimate the progress from the.
In the following output, the field "progress_in_percents" is introduced.
{code:java}
admin@cqlsh> select * from system_views.sstable_tasks ;
@ Row 1
---+--
keyspace_name | mykeyspace
table_name| mytable
task_id | 0db5d9b1-8726-11ea-8a6c-b92f3be367bb
kind | secondary index build
progress | 19456965
progress_in_percents | 8.17
total | 238208674
unit | bytes
@ Row 2
--+--
keyspace_name| mykeyspace
table_name | mytable.mytable_surname_idx
task_id | 1817ee71-8726-11ea-8a6c-b92f3be367bb
kind | compaction
progress | 284396233
progress_in_percents | 75.92
total| 374598446
unit | bytes
{code}
was:
It would be very handy to have a percentage column in
system_views.sstable_tasks which would say how far a respective task is.
Indeed, there are currently "progress" and "total" columns but honestly, for
every day usage, it is rather strange to expect that humans will divide these
numbers in head if they want to roughly know what the overall progress is. One
just does not have a rough estimation of the task progress when he is presented
with two quite big numbers and to estimate the progress from the.
In the following output, the field "progress_in_percents" is introduced.
{code:java}
admin@cqlsh> select * from system_views.sstable_tasks ;
@ Row 1
---+--
keyspace_name | mykeyspace
table_name| mytable
task_id | 0db5d9b1-8726-11ea-8a6c-b92f3be367bb
kind | secondary index build
progress | 19456965
progress_in_percents | 8.17
total | 238208674
unit | bytes
@ Row 2
--+--
keyspace_name| mykeyspace
table_name | mytable.mytable_surname_idx
task_id | 1817ee71-8726-11ea-8a6c-b92f3be367bb
kind | compaction
progress | 284396233
progress_in_percents | 75.92
total| 374598446
unit | bytes
{code}
> Add progress column in percents for system_views.sstable_tasks
> --
>
> Key: CASSANDRA-15759
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15759
> Project: Cassandra
> Issue Type: Improvement
> Components: Feature/Virtual Tables
>Reporter: Stefan Miklosovic
>Priority: Normal
>
> It would be very handy to have a percentage column in
> system_views.sstable_tasks which would say how far a respective task is.
> Indeed, there are currently "progress" and "total" columns but honestly, for
> every day usage, it is rather strange to expect that humans will divide these
> numbers in head if they want to roughly know what the overall progress is.
> One just does not have a rough estimation of the task progress when he is
> presented with two quite big numbers and to estimate the progress from the.
> In the following output, the field "progress_in_percents" is introduced.
> {code:java}
> admin@cqlsh> select * from system_views.sstable_tasks ;
> @ Row 1
> ---+--
> keyspace_name | mykeyspace
> table_name| mytable
> task_id | 0db5d9b1-8726-11ea-8a6c-b92f3be367bb
> kind | secondary index build
> progress | 19456965
> progress_in_percents | 8.17
> total | 238208674
> unit | bytes
> @ Row 2
> --+--
> keyspace_name| mykeyspace
> table_name | mytable.mytable_surname_idx
> task_id | 1817ee71-8726-11ea-8a6c-b92f3be367bb
> kind | compaction
> progress | 284396233
> progress_in_percents | 75.92
> total| 374598446
> unit
[jira] [Created] (CASSANDRA-15748) Provide ability to configure IAuditLogger
Stefan Miklosovic created CASSANDRA-15748: - Summary: Provide ability to configure IAuditLogger Key: CASSANDRA-15748 URL: https://issues.apache.org/jira/browse/CASSANDRA-15748 Project: Cassandra Issue Type: Improvement Reporter: Stefan Miklosovic Assignee: Stefan Miklosovic Currently there is a parameterless constructor expected for IAuditLogger instances. There is not any way how to "inject" configuration properties from cassandra.yaml into these concrete classes. This would be very handy for custom IAuditLogger implementations. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15748) Provide ability to configure IAuditLogger
[ https://issues.apache.org/jira/browse/CASSANDRA-15748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Miklosovic updated CASSANDRA-15748: -- Description: Currently there is a parameterless constructor expected for IAuditLogger instances. There is not any way how to "inject" configuration properties from cassandra.yaml into these concrete classes. This would be very handy for custom IAuditLogger implementations. implementation: [https://github.com/smiklosovic/cassandra/tree/CASSANDRA-15748] was:Currently there is a parameterless constructor expected for IAuditLogger instances. There is not any way how to "inject" configuration properties from cassandra.yaml into these concrete classes. This would be very handy for custom IAuditLogger implementations. > Provide ability to configure IAuditLogger > - > > Key: CASSANDRA-15748 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15748 > Project: Cassandra > Issue Type: Improvement >Reporter: Stefan Miklosovic >Assignee: Stefan Miklosovic >Priority: Normal > > Currently there is a parameterless constructor expected for IAuditLogger > instances. There is not any way how to "inject" configuration properties from > cassandra.yaml into these concrete classes. This would be very handy for > custom IAuditLogger implementations. > > implementation: > [https://github.com/smiklosovic/cassandra/tree/CASSANDRA-15748] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15406) Add command to show the progress of data streaming and index build
[ https://issues.apache.org/jira/browse/CASSANDRA-15406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17090530#comment-17090530 ] Stefan Miklosovic commented on CASSANDRA-15406: --- [~djoshi] as we resolved CASSANDRA-15694 there is nothing which prevents us from merging this as well. Would you mind to go over this please? (PR in description of this issue). Do we want to back-port this progress into Cassandra 3.x as well? I am fine having this just in 4 only. > Add command to show the progress of data streaming and index build > --- > > Key: CASSANDRA-15406 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15406 > Project: Cassandra > Issue Type: Improvement > Components: Consistency/Streaming, Legacy/Streaming and Messaging, > Tool/nodetool >Reporter: maxwellguo >Assignee: Stefan Miklosovic >Priority: Normal > Fix For: 4.0, 4.x > > Time Spent: 10m > Remaining Estimate: 0h > > I found that we should supply a command to show the progress of streaming > when we do the operation of bootstrap/move/decommission/removenode. For when > do data streaming , noboday knows which steps there program are in , so I > think a command to show the joing/leaving node's is needed . > > PR [https://github.com/apache/cassandra/pull/558] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15406) Add command to show the progress of data streaming and index build
[ https://issues.apache.org/jira/browse/CASSANDRA-15406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Miklosovic updated CASSANDRA-15406: -- Description: I found that we should supply a command to show the progress of streaming when we do the operation of bootstrap/move/decommission/removenode. For when do data streaming , noboday knows which steps there program are in , so I think a command to show the joing/leaving node's is needed . PR [https://github.com/apache/cassandra/pull/558] was:I found that we should supply a command to show the progress of streaming when we do the operation of bootstrap/move/decommission/removenode. For when do data streaming , noboday knows which steps there program are in , so I think a command to show the joing/leaving node's is needed . > Add command to show the progress of data streaming and index build > --- > > Key: CASSANDRA-15406 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15406 > Project: Cassandra > Issue Type: Improvement > Components: Consistency/Streaming, Legacy/Streaming and Messaging, > Tool/nodetool >Reporter: maxwellguo >Assignee: Stefan Miklosovic >Priority: Normal > Fix For: 4.0, 4.x > > Time Spent: 10m > Remaining Estimate: 0h > > I found that we should supply a command to show the progress of streaming > when we do the operation of bootstrap/move/decommission/removenode. For when > do data streaming , noboday knows which steps there program are in , so I > think a command to show the joing/leaving node's is needed . > > PR [https://github.com/apache/cassandra/pull/558] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Assigned] (CASSANDRA-15406) Add command to show the progress of data streaming and index build
[ https://issues.apache.org/jira/browse/CASSANDRA-15406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Miklosovic reassigned CASSANDRA-15406: - Assignee: Stefan Miklosovic > Add command to show the progress of data streaming and index build > --- > > Key: CASSANDRA-15406 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15406 > Project: Cassandra > Issue Type: Improvement > Components: Consistency/Streaming, Legacy/Streaming and Messaging, > Tool/nodetool >Reporter: maxwellguo >Assignee: Stefan Miklosovic >Priority: Normal > Fix For: 4.0, 4.x > > > I found that we should supply a command to show the progress of streaming > when we do the operation of bootstrap/move/decommission/removenode. For when > do data streaming , noboday knows which steps there program are in , so I > think a command to show the joing/leaving node's is needed . -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15694) Statistics upon streaming of entire SSTables in Netstats is wrong
[
https://issues.apache.org/jira/browse/CASSANDRA-15694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17089306#comment-17089306
]
Stefan Miklosovic commented on CASSANDRA-15694:
---
[~djoshi] good stuff! Yes, I went through your changes and I do not have any
objections, looks pretty solid. Please go ahead with the committing.
> Statistics upon streaming of entire SSTables in Netstats is wrong
> -
>
> Key: CASSANDRA-15694
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15694
> Project: Cassandra
> Issue Type: Bug
> Components: Tool/nodetool
>Reporter: Stefan Miklosovic
>Assignee: Stefan Miklosovic
>Priority: Normal
> Time Spent: 10m
> Remaining Estimate: 0h
>
> There is a bug in the current code (trunk on 6th April 2020) as if we are
> streaming entire SSTables via CassandraEntireSSTableStreamWriter and
> CassandraOutgoingFile respectively, there is not any update on particular
> components of a SSTable as it counts only "db" file to be there. That
> introduces this bug:
>
> {code:java}
> Mode: NORMAL
> Rebuild 2c0b43f0-735d-11ea-9346-fb0ffe238736
> /127.0.0.2 Sending 19 files, 27664559 bytes total. Already sent 133
> files, 27664559 bytes total
>
> /tmp/dtests15682026295742741219/node2/data/distributed_test_keyspace/cf-196b3...
>
> {code}
> Basically, number of files to be sent is lower than files sent.
>
> The straightforward fix here is to distinguish when we are streaming entire
> sstables and in that case include all manifest files into computation.
>
> This issue depends on CASSANDRA-15657 because the resolution whether we
> stream entirely or not is got from a method which is performance sensitive
> and computed every time. Once CASSANDRA-15657 (hence CASSANDRA-14586) is
> done, this ticket can be worked on.
>
> branch with fix is here:
> [https://github.com/smiklosovic/cassandra/tree/CASSANDRA-15694]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-15052) Dtests: Add acceptable warnings to offline tool tests in order to pass them
[ https://issues.apache.org/jira/browse/CASSANDRA-15052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17089291#comment-17089291 ] Stefan Miklosovic edited comment on CASSANDRA-15052 at 4/22/20, 5:19 AM: - [~djoshi] I dont think this is issue anymore. I think I have tested this on unrealistically low disk space and that is what emitted these warnings. was (Author: stefan.miklosovic): [~djoshi] I dont think this is issue anymore. I think I have tested this on unrealistically low disk usage and that is what emitted these warnings. > Dtests: Add acceptable warnings to offline tool tests in order to pass them > --- > > Key: CASSANDRA-15052 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15052 > Project: Cassandra > Issue Type: Improvement > Components: Test/dtest >Reporter: Stefan Miklosovic >Assignee: Stefan Miklosovic >Priority: Normal > Labels: pull-request-available > Attachments: SPICE-15052.txt > > Time Spent: 50m > Remaining Estimate: 0h > > I run all dtest suite and test > offline_tools_test.py::TestOfflineTools::test_sstablelevelreset has failed > because of additional warning logs which were not added into acceptable ones. > After adding them, test passed fine. I believe added warning messages have > nothing to do with test itself, it was reproduced on c5.9xlarge as well as no > "regular" notebook. > > https://github.com/apache/cassandra-dtest/pull/47 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15052) Dtests: Add acceptable warnings to offline tool tests in order to pass them
[ https://issues.apache.org/jira/browse/CASSANDRA-15052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17089291#comment-17089291 ] Stefan Miklosovic commented on CASSANDRA-15052: --- I dont think this is issue anymore. I think I have tested this on unrealistically low disk usage and that is what emitted these warnings. > Dtests: Add acceptable warnings to offline tool tests in order to pass them > --- > > Key: CASSANDRA-15052 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15052 > Project: Cassandra > Issue Type: Improvement > Components: Test/dtest >Reporter: Stefan Miklosovic >Assignee: Stefan Miklosovic >Priority: Normal > Labels: pull-request-available > Attachments: SPICE-15052.txt > > Time Spent: 50m > Remaining Estimate: 0h > > I run all dtest suite and test > offline_tools_test.py::TestOfflineTools::test_sstablelevelreset has failed > because of additional warning logs which were not added into acceptable ones. > After adding them, test passed fine. I believe added warning messages have > nothing to do with test itself, it was reproduced on c5.9xlarge as well as no > "regular" notebook. > > https://github.com/apache/cassandra-dtest/pull/47 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-15052) Dtests: Add acceptable warnings to offline tool tests in order to pass them
[ https://issues.apache.org/jira/browse/CASSANDRA-15052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17089291#comment-17089291 ] Stefan Miklosovic edited comment on CASSANDRA-15052 at 4/22/20, 5:19 AM: - [~djoshi] I dont think this is issue anymore. I think I have tested this on unrealistically low disk usage and that is what emitted these warnings. was (Author: stefan.miklosovic): I dont think this is issue anymore. I think I have tested this on unrealistically low disk usage and that is what emitted these warnings. > Dtests: Add acceptable warnings to offline tool tests in order to pass them > --- > > Key: CASSANDRA-15052 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15052 > Project: Cassandra > Issue Type: Improvement > Components: Test/dtest >Reporter: Stefan Miklosovic >Assignee: Stefan Miklosovic >Priority: Normal > Labels: pull-request-available > Attachments: SPICE-15052.txt > > Time Spent: 50m > Remaining Estimate: 0h > > I run all dtest suite and test > offline_tools_test.py::TestOfflineTools::test_sstablelevelreset has failed > because of additional warning logs which were not added into acceptable ones. > After adding them, test passed fine. I believe added warning messages have > nothing to do with test itself, it was reproduced on c5.9xlarge as well as no > "regular" notebook. > > https://github.com/apache/cassandra-dtest/pull/47 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15748) Provide ability to configure IAuditLogger
[ https://issues.apache.org/jira/browse/CASSANDRA-15748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17089571#comment-17089571 ] Stefan Miklosovic commented on CASSANDRA-15748: --- Hi [~marcuse], would you mind to go over my PR? Thanks a lot. > Provide ability to configure IAuditLogger > - > > Key: CASSANDRA-15748 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15748 > Project: Cassandra > Issue Type: Improvement >Reporter: Stefan Miklosovic >Assignee: Stefan Miklosovic >Priority: Normal > Time Spent: 10m > Remaining Estimate: 0h > > Currently there is a parameterless constructor expected for IAuditLogger > instances. There is not any way how to "inject" configuration properties from > cassandra.yaml into these concrete classes. This would be very handy for > custom IAuditLogger implementations. > > PR: [https://github.com/apache/cassandra/pull/555] > [|https://github.com/smiklosovic/cassandra/tree/CASSANDRA-15748] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15748) Provide ability to configure IAuditLogger
[ https://issues.apache.org/jira/browse/CASSANDRA-15748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Miklosovic updated CASSANDRA-15748: -- Description: Currently there is a parameterless constructor expected for IAuditLogger instances. There is not any way how to "inject" configuration properties from cassandra.yaml into these concrete classes. This would be very handy for custom IAuditLogger implementations. PR: [https://github.com/apache/cassandra/pull/555] [|https://github.com/smiklosovic/cassandra/tree/CASSANDRA-15748] was: Currently there is a parameterless constructor expected for IAuditLogger instances. There is not any way how to "inject" configuration properties from cassandra.yaml into these concrete classes. This would be very handy for custom IAuditLogger implementations. implementation: [https://github.com/smiklosovic/cassandra/tree/CASSANDRA-15748] > Provide ability to configure IAuditLogger > - > > Key: CASSANDRA-15748 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15748 > Project: Cassandra > Issue Type: Improvement >Reporter: Stefan Miklosovic >Assignee: Stefan Miklosovic >Priority: Normal > Time Spent: 10m > Remaining Estimate: 0h > > Currently there is a parameterless constructor expected for IAuditLogger > instances. There is not any way how to "inject" configuration properties from > cassandra.yaml into these concrete classes. This would be very handy for > custom IAuditLogger implementations. > > PR: [https://github.com/apache/cassandra/pull/555] > [|https://github.com/smiklosovic/cassandra/tree/CASSANDRA-15748] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15158) Wait for schema agreement rather then in flight schema requests when bootstrapping
[ https://issues.apache.org/jira/browse/CASSANDRA-15158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17095596#comment-17095596 ] Stefan Miklosovic commented on CASSANDRA-15158: --- Hi [~bdeggleston], please review this one [https://github.com/smiklosovic/cassandra/tree/CASSANDRA-15158] Sorry for the fuss but this seems to be more problematic than I was thinking initially. I do not think that we have to _check_ that respective migration request is successful or not, we should just check it all schemas match ... The logic is based on repeated checking if schemas agree or not and only in case all schemas are equal we proceed, otherwise an exception is thrown (this might be reworked in such sense that we just log this). There is a global timeout for this check, it will be obvious from reading the code. There is a test added too, because of the nature of the test, I had to "inject" these callbacks into respective methods so I could modify their default behaviour and test their state. This is against 3.11, we need to backport this to 3.0 for our customer. > Wait for schema agreement rather then in flight schema requests when > bootstrapping > -- > > Key: CASSANDRA-15158 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15158 > Project: Cassandra > Issue Type: Bug > Components: Cluster/Gossip, Cluster/Schema >Reporter: Vincent White >Assignee: Ben Bromhead >Priority: Normal > > Currently when a node is bootstrapping we use a set of latches > (org.apache.cassandra.service.MigrationTask#inflightTasks) to keep track of > in-flight schema pull requests, and we don't proceed with > bootstrapping/stream until all the latches are released (or we timeout > waiting for each one). One issue with this is that if we have a large schema, > or the retrieval of the schema from the other nodes was unexpectedly slow > then we have no explicit check in place to ensure we have actually received a > schema before we proceed. > While it's possible to increase "migration_task_wait_in_seconds" to force the > node to wait on each latche longer, there are cases where this doesn't help > because the callbacks for the schema pull requests have expired off the > messaging service's callback map > (org.apache.cassandra.net.MessagingService#callbacks) after > request_timeout_in_ms (default 10 seconds) before the other nodes were able > to respond to the new node. > This patch checks for schema agreement between the bootstrapping node and the > rest of the live nodes before proceeding with bootstrapping. It also adds a > check to prevent the new node from flooding existing nodes with simultaneous > schema pull requests as can happen in large clusters. > Removing the latch system should also prevent new nodes in large clusters > getting stuck for extended amounts of time as they wait > `migration_task_wait_in_seconds` on each of the latches left orphaned by the > timed out callbacks. > > ||3.11|| > |[PoC|https://github.com/apache/cassandra/compare/cassandra-3.11...vincewhite:check_for_schema]| > |[dtest|https://github.com/apache/cassandra-dtest/compare/master...vincewhite:wait_for_schema_agreement]| > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15158) Wait for schema agreement rather then in flight schema requests when bootstrapping
[ https://issues.apache.org/jira/browse/CASSANDRA-15158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17098952#comment-17098952 ] Stefan Miklosovic commented on CASSANDRA-15158: --- code for 3.0 [https://github.com/smiklosovic/cassandra/tree/CASSANDRA-15158-cassandra-3] > Wait for schema agreement rather then in flight schema requests when > bootstrapping > -- > > Key: CASSANDRA-15158 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15158 > Project: Cassandra > Issue Type: Bug > Components: Cluster/Gossip, Cluster/Schema >Reporter: Vincent White >Assignee: Ben Bromhead >Priority: Normal > > Currently when a node is bootstrapping we use a set of latches > (org.apache.cassandra.service.MigrationTask#inflightTasks) to keep track of > in-flight schema pull requests, and we don't proceed with > bootstrapping/stream until all the latches are released (or we timeout > waiting for each one). One issue with this is that if we have a large schema, > or the retrieval of the schema from the other nodes was unexpectedly slow > then we have no explicit check in place to ensure we have actually received a > schema before we proceed. > While it's possible to increase "migration_task_wait_in_seconds" to force the > node to wait on each latche longer, there are cases where this doesn't help > because the callbacks for the schema pull requests have expired off the > messaging service's callback map > (org.apache.cassandra.net.MessagingService#callbacks) after > request_timeout_in_ms (default 10 seconds) before the other nodes were able > to respond to the new node. > This patch checks for schema agreement between the bootstrapping node and the > rest of the live nodes before proceeding with bootstrapping. It also adds a > check to prevent the new node from flooding existing nodes with simultaneous > schema pull requests as can happen in large clusters. > Removing the latch system should also prevent new nodes in large clusters > getting stuck for extended amounts of time as they wait > `migration_task_wait_in_seconds` on each of the latches left orphaned by the > timed out callbacks. > > ||3.11|| > |[PoC|https://github.com/apache/cassandra/compare/cassandra-3.11...vincewhite:check_for_schema]| > |[dtest|https://github.com/apache/cassandra-dtest/compare/master...vincewhite:wait_for_schema_agreement]| > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-15748) Provide ability to configure IAuditLogger
[ https://issues.apache.org/jira/browse/CASSANDRA-15748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17106097#comment-17106097 ] Stefan Miklosovic edited comment on CASSANDRA-15748 at 5/13/20, 8:32 AM: - [~vinaykumarcse] yes, this was done on purpose, I think I had this discussion with Marcus privately, the thing is that I dont know how to propagate map from nodetool's enableauditlogging - there is not any way to pass map from the command line (at least the straightforward one). If you look closely into StorageServiceMBean, that patch contains another enableAuditLog method which can accept a map. It is there for cases somebody really wants to pass some parameters via map there via JMX. There is as well the original method without that map, because in tooling like JConsole, that method is not invokable anymore if it contains a map, so there is not any way to invoke it via JMX in JConsole, hence the original method is left there for this purpose. [~marcuse] also raised an interesting issue that it feels overall "slightly wrong" to have this enable / disable method there at all. One could just disable logging, do something bad and enable it again. Hence the question is if that nodetool command is actually relevant at all. was (Author: stefan.miklosovic): [~vinaykumarcse] yes, this was done on purpose, I think I had this discussion with Marcus privately, the thing is that I dont know how to propagate map from nodetool's enableauditlogging - there is not any way to pass map from the command line (at least the straightforward one). If you look closely into StorageServiceMBean, that patch contains another enableAuditLog method which can accept a map. It is there for cases somebody really wants to pass some parameters via map there via JMX. There is as well the original method without that map, because in tooling like JConsole, that method is not invokable anymore if it contains a map, so there is not any way to invoke it via JMX, hence the original method is left there for this purpose. [~marcuse] also raised an interesting issue that it feels overall "slightly wrong" to have this enable / disable method there at all. One could just disable logging, do something bad and enable it again. Hence the question is if that nodetool command is actually relevant at all. > Provide ability to configure IAuditLogger > - > > Key: CASSANDRA-15748 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15748 > Project: Cassandra > Issue Type: Improvement > Components: Local/Other >Reporter: Stefan Miklosovic >Assignee: Stefan Miklosovic >Priority: Normal > Fix For: 4.0-beta > > Time Spent: 50m > Remaining Estimate: 0h > > Currently there is a parameterless constructor expected for IAuditLogger > instances. There is not any way how to "inject" configuration properties from > cassandra.yaml into these concrete classes. This would be very handy for > custom IAuditLogger implementations. > > PR: [https://github.com/apache/cassandra/pull/555] > [|https://github.com/smiklosovic/cassandra/tree/CASSANDRA-15748] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-15748) Provide ability to configure IAuditLogger
[ https://issues.apache.org/jira/browse/CASSANDRA-15748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17106097#comment-17106097 ] Stefan Miklosovic edited comment on CASSANDRA-15748 at 5/13/20, 8:31 AM: - [~vinaykumarcse] yes, this was done on purpose, I think I had this discussion with Marcus privately, the thing is that I dont know how to propagate map from nodetool's enableauditlogging - there is not any way to pass map from the command line (at least the straightforward one). If you look closely into StorageServiceMBean, that patch contains another enableAuditLog method which can accept a map. It is there for cases somebody really wants to pass some parameters via map there via JMX. There is as well the original method without that map, because in tooling like JConsole, that method is not invokable anymore if it contains a map, so there is not any way to invoke it via JMX, hence the original method is left there for this purpose. [~marcuse] also raised an interesting issue that it feels overall "slightly wrong" to have this enable / disable method there at all. One could just disable logging, do something bad and enable it again. Hence the question is if that nodetool command is actually relevant at all. was (Author: stefan.miklosovic): [~vinaykumarcse] yes, this was done on purpose, I think I had this discussion with Marcus privately, the thing is that I dont know how to propagate map from nodetool's enableauditlogging - there is not any way to pass map from the command line (at least the straightforward one). If you look closely into NodeProbe, that patch contains another enableAuditLog method which can accept a map. It is there for cases somebody really wants to pass some parameters via map there via JMX. There is as well the original method without that map, because in tooling like JConsole, that method is not invokable anymore if it contains a map, so there is not any way to invoke it via JMX, hence the original method is left there for this purpose. [~marcuse] also raised an interesting issue that it feels overall "slightly wrong" to have this enable / disable method there at all. One could just disable logging, do something bad and enable it again. Hence the question is if that nodetool command is actually relevant at all. > Provide ability to configure IAuditLogger > - > > Key: CASSANDRA-15748 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15748 > Project: Cassandra > Issue Type: Improvement > Components: Local/Other >Reporter: Stefan Miklosovic >Assignee: Stefan Miklosovic >Priority: Normal > Fix For: 4.0-beta > > Time Spent: 50m > Remaining Estimate: 0h > > Currently there is a parameterless constructor expected for IAuditLogger > instances. There is not any way how to "inject" configuration properties from > cassandra.yaml into these concrete classes. This would be very handy for > custom IAuditLogger implementations. > > PR: [https://github.com/apache/cassandra/pull/555] > [|https://github.com/smiklosovic/cassandra/tree/CASSANDRA-15748] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15748) Provide ability to configure IAuditLogger
[ https://issues.apache.org/jira/browse/CASSANDRA-15748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17106097#comment-17106097 ] Stefan Miklosovic commented on CASSANDRA-15748: --- [~vinaykumarcse] yes, this was done on purpose, I think I had this discussion with Marcus privately, the thing is that I dont know how to propagate map from nodetool's enableauditlogging - there is not any way to pass map from the command line (at least the straightforward one). If you look closely into NodeProbe, that patch contains another enableAuditLog method which can accept a map. It is there for cases somebody really wants to pass some parameters via map there. There is as well the original method with that map, because in tooling like JConsole, that method is not invokable anymore if it contains a map, so there is not any way to invoke it via JMX, hence the original method is left there for this purpose. [~marcuse] also raised an interesting issue that it feels overall "slightly wrong" to have this enable / disable method there at all. One could just disable logging, do something bad and enable it again. Hence the question is if that nodetool command is actually relevant at all. > Provide ability to configure IAuditLogger > - > > Key: CASSANDRA-15748 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15748 > Project: Cassandra > Issue Type: Improvement > Components: Local/Other >Reporter: Stefan Miklosovic >Assignee: Stefan Miklosovic >Priority: Normal > Fix For: 4.0-beta > > Time Spent: 50m > Remaining Estimate: 0h > > Currently there is a parameterless constructor expected for IAuditLogger > instances. There is not any way how to "inject" configuration properties from > cassandra.yaml into these concrete classes. This would be very handy for > custom IAuditLogger implementations. > > PR: [https://github.com/apache/cassandra/pull/555] > [|https://github.com/smiklosovic/cassandra/tree/CASSANDRA-15748] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-15748) Provide ability to configure IAuditLogger
[ https://issues.apache.org/jira/browse/CASSANDRA-15748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17106097#comment-17106097 ] Stefan Miklosovic edited comment on CASSANDRA-15748 at 5/13/20, 8:29 AM: - [~vinaykumarcse] yes, this was done on purpose, I think I had this discussion with Marcus privately, the thing is that I dont know how to propagate map from nodetool's enableauditlogging - there is not any way to pass map from the command line (at least the straightforward one). If you look closely into NodeProbe, that patch contains another enableAuditLog method which can accept a map. It is there for cases somebody really wants to pass some parameters via map there via JMX. There is as well the original method without that map, because in tooling like JConsole, that method is not invokable anymore if it contains a map, so there is not any way to invoke it via JMX, hence the original method is left there for this purpose. [~marcuse] also raised an interesting issue that it feels overall "slightly wrong" to have this enable / disable method there at all. One could just disable logging, do something bad and enable it again. Hence the question is if that nodetool command is actually relevant at all. was (Author: stefan.miklosovic): [~vinaykumarcse] yes, this was done on purpose, I think I had this discussion with Marcus privately, the thing is that I dont know how to propagate map from nodetool's enableauditlogging - there is not any way to pass map from the command line (at least the straightforward one). If you look closely into NodeProbe, that patch contains another enableAuditLog method which can accept a map. It is there for cases somebody really wants to pass some parameters via map there. There is as well the original method with that map, because in tooling like JConsole, that method is not invokable anymore if it contains a map, so there is not any way to invoke it via JMX, hence the original method is left there for this purpose. [~marcuse] also raised an interesting issue that it feels overall "slightly wrong" to have this enable / disable method there at all. One could just disable logging, do something bad and enable it again. Hence the question is if that nodetool command is actually relevant at all. > Provide ability to configure IAuditLogger > - > > Key: CASSANDRA-15748 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15748 > Project: Cassandra > Issue Type: Improvement > Components: Local/Other >Reporter: Stefan Miklosovic >Assignee: Stefan Miklosovic >Priority: Normal > Fix For: 4.0-beta > > Time Spent: 50m > Remaining Estimate: 0h > > Currently there is a parameterless constructor expected for IAuditLogger > instances. There is not any way how to "inject" configuration properties from > cassandra.yaml into these concrete classes. This would be very handy for > custom IAuditLogger implementations. > > PR: [https://github.com/apache/cassandra/pull/555] > [|https://github.com/smiklosovic/cassandra/tree/CASSANDRA-15748] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15748) Provide ability to configure IAuditLogger
[ https://issues.apache.org/jira/browse/CASSANDRA-15748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Miklosovic updated CASSANDRA-15748: -- Test and Documentation Plan: Status: Patch Available (was: Open) https://github.com/apache/cassandra/pull/555 > Provide ability to configure IAuditLogger > - > > Key: CASSANDRA-15748 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15748 > Project: Cassandra > Issue Type: Improvement > Components: Local/Other >Reporter: Stefan Miklosovic >Assignee: Stefan Miklosovic >Priority: Normal > Fix For: 4.0-beta > > Time Spent: 10m > Remaining Estimate: 0h > > Currently there is a parameterless constructor expected for IAuditLogger > instances. There is not any way how to "inject" configuration properties from > cassandra.yaml into these concrete classes. This would be very handy for > custom IAuditLogger implementations. > > PR: [https://github.com/apache/cassandra/pull/555] > [|https://github.com/smiklosovic/cassandra/tree/CASSANDRA-15748] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15406) Add command to show the progress of data streaming and index build
[ https://issues.apache.org/jira/browse/CASSANDRA-15406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Miklosovic updated CASSANDRA-15406: -- Test and Documentation Plan: https://github.com/apache/cassandra/pull/558 Status: Patch Available (was: Open) https://github.com/apache/cassandra/pull/558 > Add command to show the progress of data streaming and index build > --- > > Key: CASSANDRA-15406 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15406 > Project: Cassandra > Issue Type: Improvement > Components: Consistency/Streaming, Legacy/Streaming and Messaging, > Tool/nodetool >Reporter: maxwellguo >Assignee: Stefan Miklosovic >Priority: Normal > Fix For: 4.0, 4.x > > Time Spent: 10m > Remaining Estimate: 0h > > I found that we should supply a command to show the progress of streaming > when we do the operation of bootstrap/move/decommission/removenode. For when > do data streaming , noboday knows which steps there program are in , so I > think a command to show the joing/leaving node's is needed . > > PR [https://github.com/apache/cassandra/pull/558] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15158) Wait for schema agreement rather then in flight schema requests when bootstrapping
[ https://issues.apache.org/jira/browse/CASSANDRA-15158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Miklosovic updated CASSANDRA-15158: -- Authors: Stefan Miklosovic (was: Ben Bromhead) Test and Documentation Plan: There are unit tests as part of PR testing this feature. Status: Patch Available (was: Open) for 3.11 [https://github.com/smiklosovic/cassandra/tree/CASSANDRA-15158] for 3.0 https://github.com/smiklosovic/cassandra/tree/CASSANDRA-15158-cassandra-3 > Wait for schema agreement rather then in flight schema requests when > bootstrapping > -- > > Key: CASSANDRA-15158 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15158 > Project: Cassandra > Issue Type: Bug > Components: Cluster/Gossip, Cluster/Schema >Reporter: Vincent White >Assignee: Ben Bromhead >Priority: Normal > > Currently when a node is bootstrapping we use a set of latches > (org.apache.cassandra.service.MigrationTask#inflightTasks) to keep track of > in-flight schema pull requests, and we don't proceed with > bootstrapping/stream until all the latches are released (or we timeout > waiting for each one). One issue with this is that if we have a large schema, > or the retrieval of the schema from the other nodes was unexpectedly slow > then we have no explicit check in place to ensure we have actually received a > schema before we proceed. > While it's possible to increase "migration_task_wait_in_seconds" to force the > node to wait on each latche longer, there are cases where this doesn't help > because the callbacks for the schema pull requests have expired off the > messaging service's callback map > (org.apache.cassandra.net.MessagingService#callbacks) after > request_timeout_in_ms (default 10 seconds) before the other nodes were able > to respond to the new node. > This patch checks for schema agreement between the bootstrapping node and the > rest of the live nodes before proceeding with bootstrapping. It also adds a > check to prevent the new node from flooding existing nodes with simultaneous > schema pull requests as can happen in large clusters. > Removing the latch system should also prevent new nodes in large clusters > getting stuck for extended amounts of time as they wait > `migration_task_wait_in_seconds` on each of the latches left orphaned by the > timed out callbacks. > > ||3.11|| > |[PoC|https://github.com/apache/cassandra/compare/cassandra-3.11...vincewhite:check_for_schema]| > |[dtest|https://github.com/apache/cassandra-dtest/compare/master...vincewhite:wait_for_schema_agreement]| > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15748) Provide ability to configure IAuditLogger
[ https://issues.apache.org/jira/browse/CASSANDRA-15748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17100246#comment-17100246 ] Stefan Miklosovic commented on CASSANDRA-15748: --- [~jmeredithco] no worries, as long as we do not forget, all good :) > Provide ability to configure IAuditLogger > - > > Key: CASSANDRA-15748 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15748 > Project: Cassandra > Issue Type: Improvement > Components: Local/Other >Reporter: Stefan Miklosovic >Assignee: Stefan Miklosovic >Priority: Normal > Fix For: 4.0-alpha > > Time Spent: 10m > Remaining Estimate: 0h > > Currently there is a parameterless constructor expected for IAuditLogger > instances. There is not any way how to "inject" configuration properties from > cassandra.yaml into these concrete classes. This would be very handy for > custom IAuditLogger implementations. > > PR: [https://github.com/apache/cassandra/pull/555] > [|https://github.com/smiklosovic/cassandra/tree/CASSANDRA-15748] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-15158) Wait for schema agreement rather then in flight schema requests when bootstrapping
[ https://issues.apache.org/jira/browse/CASSANDRA-15158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102543#comment-17102543 ] Stefan Miklosovic edited comment on CASSANDRA-15158 at 5/8/20, 12:49 PM: - It seems to me that one aspect of the PR was overlooked so I just iterate on that one. The mechanim how to not flood nodes with schema pull messages is incorporated in the loop over callbacks. If you notice it, there are sleeps of various lenghts based on a request being already sent or not. This sleep will actually "delay" the next schema pull from the other node because during this time of a sleep, some schema could come from the node we just sent a message to so on the next iteration when another node is compared on schema equality, it may happen that there is not any need to pull it anymore because they are on par. Hence we are not blindly sending messages to all nodes. If there are some discrepancies, there is the global timeout set after which whole bootstrapping process will be evaluated as errorneous and (in the current code) we throw a ConfigurationException. This behaviour might be relaxed but I consider it more appropriate to just throw it there. was (Author: stefan.miklosovic): It seems to me that one aspect of the PR was overlooked so I just iterate on that one. The mechanims how to not flood nodes with schema pull messages is incorporated in a loop over callbacks. If you notice it, there are sleeps of various lenghts based on a request being already sent or not. This sleep will actually "delay" the next schema pull from the other node because during this time of a sleep, a schema could come in so on the next iteration when another node is compared on schema equality, it may happen that there is not any need to pull it anymore because they are on par. Hence we are not blindly sending messages to all nodes. If there are some discrepancies, there is the global timeout set after which whole bootstrapping process will be evaluated as errorneous and (in the current code) we throw a ConfigurationException. This behaviour might be relaxed but I consider it more appropriate to just throw it there. > Wait for schema agreement rather then in flight schema requests when > bootstrapping > -- > > Key: CASSANDRA-15158 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15158 > Project: Cassandra > Issue Type: Bug > Components: Cluster/Gossip, Cluster/Schema >Reporter: Vincent White >Assignee: Ben Bromhead >Priority: Normal > > Currently when a node is bootstrapping we use a set of latches > (org.apache.cassandra.service.MigrationTask#inflightTasks) to keep track of > in-flight schema pull requests, and we don't proceed with > bootstrapping/stream until all the latches are released (or we timeout > waiting for each one). One issue with this is that if we have a large schema, > or the retrieval of the schema from the other nodes was unexpectedly slow > then we have no explicit check in place to ensure we have actually received a > schema before we proceed. > While it's possible to increase "migration_task_wait_in_seconds" to force the > node to wait on each latche longer, there are cases where this doesn't help > because the callbacks for the schema pull requests have expired off the > messaging service's callback map > (org.apache.cassandra.net.MessagingService#callbacks) after > request_timeout_in_ms (default 10 seconds) before the other nodes were able > to respond to the new node. > This patch checks for schema agreement between the bootstrapping node and the > rest of the live nodes before proceeding with bootstrapping. It also adds a > check to prevent the new node from flooding existing nodes with simultaneous > schema pull requests as can happen in large clusters. > Removing the latch system should also prevent new nodes in large clusters > getting stuck for extended amounts of time as they wait > `migration_task_wait_in_seconds` on each of the latches left orphaned by the > timed out callbacks. > > ||3.11|| > |[PoC|https://github.com/apache/cassandra/compare/cassandra-3.11...vincewhite:check_for_schema]| > |[dtest|https://github.com/apache/cassandra-dtest/compare/master...vincewhite:wait_for_schema_agreement]| > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15158) Wait for schema agreement rather then in flight schema requests when bootstrapping
[ https://issues.apache.org/jira/browse/CASSANDRA-15158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102543#comment-17102543 ] Stefan Miklosovic commented on CASSANDRA-15158: --- It seems to me that one aspect of the PR was overlooked so I just iterate on that one. The mechanims how to not flood nodes with schema pull messages is incorporated in a loop over callbacks. If you notice it, there are sleeps of various lenghts based on a request being already sent or not. This sleep will actually "delay" the next schema pull from the other node because during this time of a sleep, a schema could come in so on the next iteration when another node is compared on schema equality, it may happen that there is not any need to pull it anymore because they are on par. Hence we are not blindly sending messages to all nodes. If there are some discrepancies, there is the global timeout set after which whole bootstrapping process will be evaluated as errorneous and (in the current code) we throw a ConfigurationException. This behaviour might be relaxed but I consider it more appropriate to just throw it there. > Wait for schema agreement rather then in flight schema requests when > bootstrapping > -- > > Key: CASSANDRA-15158 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15158 > Project: Cassandra > Issue Type: Bug > Components: Cluster/Gossip, Cluster/Schema >Reporter: Vincent White >Assignee: Ben Bromhead >Priority: Normal > > Currently when a node is bootstrapping we use a set of latches > (org.apache.cassandra.service.MigrationTask#inflightTasks) to keep track of > in-flight schema pull requests, and we don't proceed with > bootstrapping/stream until all the latches are released (or we timeout > waiting for each one). One issue with this is that if we have a large schema, > or the retrieval of the schema from the other nodes was unexpectedly slow > then we have no explicit check in place to ensure we have actually received a > schema before we proceed. > While it's possible to increase "migration_task_wait_in_seconds" to force the > node to wait on each latche longer, there are cases where this doesn't help > because the callbacks for the schema pull requests have expired off the > messaging service's callback map > (org.apache.cassandra.net.MessagingService#callbacks) after > request_timeout_in_ms (default 10 seconds) before the other nodes were able > to respond to the new node. > This patch checks for schema agreement between the bootstrapping node and the > rest of the live nodes before proceeding with bootstrapping. It also adds a > check to prevent the new node from flooding existing nodes with simultaneous > schema pull requests as can happen in large clusters. > Removing the latch system should also prevent new nodes in large clusters > getting stuck for extended amounts of time as they wait > `migration_task_wait_in_seconds` on each of the latches left orphaned by the > timed out callbacks. > > ||3.11|| > |[PoC|https://github.com/apache/cassandra/compare/cassandra-3.11...vincewhite:check_for_schema]| > |[dtest|https://github.com/apache/cassandra-dtest/compare/master...vincewhite:wait_for_schema_agreement]| > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15158) Wait for schema agreement rather then in flight schema requests when bootstrapping
[
https://issues.apache.org/jira/browse/CASSANDRA-15158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102374#comment-17102374
]
Stefan Miklosovic commented on CASSANDRA-15158:
---
Hi [~bdeggleston],
commenting on design issues, I am not completely sure if these issues you are
talking about are related to this patch or they are already existing? We could
indeed focus on the points you raised but it seems to me that the current
(comitted) code is worse without this patch than with as I guess these problems
are already there?
Isn't the goal here to have all nodes on same versions? Isn't the very fact
that there are multiple versions pretty strange to begin with so we should not
even try to join a node if they mismatch hence there is nothing to deal with in
the first place?
{quote}It will only wait until it has _some_ schema to begin bootstrapping, not
all
{quote}
This is the most likely not true unless I am not getting something. The node to
be bootstrapped will never advance in doing so unless all nodes have same
versions.
{quote} For instance, if a single node is reporting a schema version that no
one else has, but the node is unreachable, what do we do?
{quote}
We should fail whole bootstrapping and one should go and fix it.
{quote}Next, I like how this limits the number of messages sent to a given
endpoint, but we should also limit the number of messages we send out for a
given schema version. If we have a large cluster, and all nodes are reporting
the same version, we don't need to ask every node for it's schema.{quote}
I am sorry, I am not following what you say here, in particular the very last
sentence. I think the schema is ever pull (message is sent) _only_ in case
that reported schema version from Gossipper is different, only after that we
are ever sending a message.
When it comes to testing, I admit that adding isRunningForcibly method feels
like a hack but I had very hard time to test this stuff out. It was basically
the only reasonable way possible at the time I was coding it, if you know of
more better version, please tell me otherwise I am not sure what might be
better here and we could stick with this for a time being? The whole testing
methodology was based on these callbacks and checking their inner state which
results into having a methods which are accepting them so we can elaborate on
their state. Without "injecting" them from outside, I would not be able to do
that.
> Wait for schema agreement rather then in flight schema requests when
> bootstrapping
> --
>
> Key: CASSANDRA-15158
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15158
> Project: Cassandra
> Issue Type: Bug
> Components: Cluster/Gossip, Cluster/Schema
>Reporter: Vincent White
>Assignee: Ben Bromhead
>Priority: Normal
>
> Currently when a node is bootstrapping we use a set of latches
> (org.apache.cassandra.service.MigrationTask#inflightTasks) to keep track of
> in-flight schema pull requests, and we don't proceed with
> bootstrapping/stream until all the latches are released (or we timeout
> waiting for each one). One issue with this is that if we have a large schema,
> or the retrieval of the schema from the other nodes was unexpectedly slow
> then we have no explicit check in place to ensure we have actually received a
> schema before we proceed.
> While it's possible to increase "migration_task_wait_in_seconds" to force the
> node to wait on each latche longer, there are cases where this doesn't help
> because the callbacks for the schema pull requests have expired off the
> messaging service's callback map
> (org.apache.cassandra.net.MessagingService#callbacks) after
> request_timeout_in_ms (default 10 seconds) before the other nodes were able
> to respond to the new node.
> This patch checks for schema agreement between the bootstrapping node and the
> rest of the live nodes before proceeding with bootstrapping. It also adds a
> check to prevent the new node from flooding existing nodes with simultaneous
> schema pull requests as can happen in large clusters.
> Removing the latch system should also prevent new nodes in large clusters
> getting stuck for extended amounts of time as they wait
> `migration_task_wait_in_seconds` on each of the latches left orphaned by the
> timed out callbacks.
>
> ||3.11||
> |[PoC|https://github.com/apache/cassandra/compare/cassandra-3.11...vincewhite:check_for_schema]|
> |[dtest|https://github.com/apache/cassandra-dtest/compare/master...vincewhite:wait_for_schema_agreement]|
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail:
