[jira] [Comment Edited] (CASSANDRA-15213) DecayingEstimatedHistogramReservoir Inefficiencies
[
https://issues.apache.org/jira/browse/CASSANDRA-15213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17011493#comment-17011493
]
Jordan West edited comment on CASSANDRA-15213 at 1/9/20 7:09 AM:
-
Thanks for the clarification. I will take a stab at something to replace the
usage of {{LongAdder[]}} along the lines of what you suggested and share when
ready. I agree re: benchmarking and will try and come up with a more complete
set -- clearly we used the existing benchmarks before without finding this
issue.
Regarding the binary search, I am still a bit confused. If you run:
https://github.com/jrwest/cassandra/blob/jwest/15213/test/unit/org/apache/cassandra/metrics/DecayingEstimatedHistogramReservoirTest.java#L58
you will see that the proposed estimation (using the code provided) would
result in several cases where a linear search would lead to over 10-50 accesses
(as value grows the approximation gets more inaccurate) and
{{Arrays.binarySearch(offsets, value) <= fastLog12(value)}} always.
Fwiw, the proposed fast log code does improve performance of my local
modifications when benchmarked but basically brings it back inline w/ the
existing implementation (caveat: benchmarking issues already mentioned).
was (Author: jrwest):
Thanks for the clarification. I will take a stab at something to replace the
usage of {{LongAdder[]}} along the lines of what you suggested and share when
ready. I agree re: benchmarking and will try and come up with a more complete
set -- clearly we used the existing benchmarks before without finding this
issue.
Regarding the binary search, I am still a bit confused. If you run:
https://github.com/jrwest/cassandra/blob/jwest/15213/test/unit/org/apache/cassandra/metrics/DecayingEstimatedHistogramReservoirTest.java#L58
you will see that the proposed estimation (using the code provided) would
result in several cases where a linear search would lead to 10-50 accesses (as
value grows the approximation gets more inaccurate) and
{{Arrays.binarySearch(offsets, value) <= fastLog12(value)}} always.
Fwiw, the proposed fast log code does improve performance of my local
modifications when benchmarked but basically brings it back inline w/ the
existing implementation (caveat: benchmarking issues already mentioned).
> DecayingEstimatedHistogramReservoir Inefficiencies
> --
>
> Key: CASSANDRA-15213
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15213
> Project: Cassandra
> Issue Type: Bug
> Components: Observability/Metrics
>Reporter: Benedict Elliott Smith
>Assignee: Jordan West
>Priority: Normal
> Fix For: 4.0-beta
>
>
> * {{LongAdder}} introduced to trunk consumes 9MiB of heap without user
> schemas, and this will grow significantly under contention and user schemas
> with many tables. This is because {{LongAdder}} is a very heavy class
> designed for single contended values.
> ** This can likely be improved significantly, without significant loss of
> performance in the contended case, by simply increasing the size of our
> primitive backing array and providing multiple buckets, with each thread
> picking a bucket to increment, or simply multiple backing arrays. Probably a
> better way still to do this would be to introduce some competition detection
> to the update, much like {{LongAdder}} utilises, that increases the number of
> backing arrays under competition.
> ** To save memory this approach could partition the space into chunks that
> are likely to be updated together, so that we do not need to duplicate the
> entire array under competition.
> * Similarly, binary search is costly and a measurable cost as a share of the
> new networking work (without filtering it was > 10% of the CPU used overall).
> We can compute an approximation floor(log2 n / log2 1.2) extremely cheaply,
> to save the random memory access costs.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-15213) DecayingEstimatedHistogramReservoir Inefficiencies
[
https://issues.apache.org/jira/browse/CASSANDRA-15213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17011493#comment-17011493
]
Jordan West edited comment on CASSANDRA-15213 at 1/9/20 7:13 AM:
-
Thanks for the clarification. I will take a stab at something to replace the
usage of {{LongAdder[]}} along the lines of what you suggested and share when
ready. I agree re: benchmarking and will try and come up with a more complete
set -- clearly we used the existing benchmarks before without finding this
issue.
Regarding the binary search, I am still a bit confused. If you run:
https://github.com/jrwest/cassandra/blob/jwest/15213/test/unit/org/apache/cassandra/metrics/DecayingEstimatedHistogramReservoirTest.java#L58
you will see that the proposed estimation (using the code provided) would
result in several cases where a linear search would lead to over 10-50 accesses
(as value grows the approximation gets more inaccurate) and
{{Arrays.binarySearch(offsets, value) <= fastLog12(value)}} always (even when
correctly adjusting the return value of the binary search for negative values).
Fwiw, the proposed fast log code does improve performance of my local
modifications when benchmarked but basically brings it back inline w/ the
existing implementation (caveat: benchmarking issues already mentioned).
was (Author: jrwest):
Thanks for the clarification. I will take a stab at something to replace the
usage of {{LongAdder[]}} along the lines of what you suggested and share when
ready. I agree re: benchmarking and will try and come up with a more complete
set -- clearly we used the existing benchmarks before without finding this
issue.
Regarding the binary search, I am still a bit confused. If you run:
https://github.com/jrwest/cassandra/blob/jwest/15213/test/unit/org/apache/cassandra/metrics/DecayingEstimatedHistogramReservoirTest.java#L58
you will see that the proposed estimation (using the code provided) would
result in several cases where a linear search would lead to over 10-50 accesses
(as value grows the approximation gets more inaccurate) and
{{Arrays.binarySearch(offsets, value) <= fastLog12(value)}} always.
Fwiw, the proposed fast log code does improve performance of my local
modifications when benchmarked but basically brings it back inline w/ the
existing implementation (caveat: benchmarking issues already mentioned).
> DecayingEstimatedHistogramReservoir Inefficiencies
> --
>
> Key: CASSANDRA-15213
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15213
> Project: Cassandra
> Issue Type: Bug
> Components: Observability/Metrics
>Reporter: Benedict Elliott Smith
>Assignee: Jordan West
>Priority: Normal
> Fix For: 4.0-beta
>
>
> * {{LongAdder}} introduced to trunk consumes 9MiB of heap without user
> schemas, and this will grow significantly under contention and user schemas
> with many tables. This is because {{LongAdder}} is a very heavy class
> designed for single contended values.
> ** This can likely be improved significantly, without significant loss of
> performance in the contended case, by simply increasing the size of our
> primitive backing array and providing multiple buckets, with each thread
> picking a bucket to increment, or simply multiple backing arrays. Probably a
> better way still to do this would be to introduce some competition detection
> to the update, much like {{LongAdder}} utilises, that increases the number of
> backing arrays under competition.
> ** To save memory this approach could partition the space into chunks that
> are likely to be updated together, so that we do not need to duplicate the
> entire array under competition.
> * Similarly, binary search is costly and a measurable cost as a share of the
> new networking work (without filtering it was > 10% of the CPU used overall).
> We can compute an approximation floor(log2 n / log2 1.2) extremely cheaply,
> to save the random memory access costs.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15213) DecayingEstimatedHistogramReservoir Inefficiencies
[
https://issues.apache.org/jira/browse/CASSANDRA-15213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17011493#comment-17011493
]
Jordan West commented on CASSANDRA-15213:
-
Thanks for the clarification. I will take a stab at something to replace the
usage of {{LongAdder[]}} along the lines of what you suggested and share when
ready. I agree re: benchmarking and will try and come up with a more complete
set -- clearly we used the existing benchmarks before without finding this
issue.
Regarding the binary search, I am still a bit confused. If you run:
https://github.com/jrwest/cassandra/blob/jwest/15213/test/unit/org/apache/cassandra/metrics/DecayingEstimatedHistogramReservoirTest.java#L58
you will see that the proposed estimation (using the code provided) would
result in several cases where a linear search would lead to 10-50 accesses (as
value grows the approximation gets more inaccurate).
Fwiw, the proposed fast log code does improve performance of my local
modifications when benchmarked but basically brings it back inline w/ the
existing implementation (caveat: benchmarking issues already mentioned).
> DecayingEstimatedHistogramReservoir Inefficiencies
> --
>
> Key: CASSANDRA-15213
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15213
> Project: Cassandra
> Issue Type: Bug
> Components: Observability/Metrics
>Reporter: Benedict Elliott Smith
>Assignee: Jordan West
>Priority: Normal
> Fix For: 4.0-beta
>
>
> * {{LongAdder}} introduced to trunk consumes 9MiB of heap without user
> schemas, and this will grow significantly under contention and user schemas
> with many tables. This is because {{LongAdder}} is a very heavy class
> designed for single contended values.
> ** This can likely be improved significantly, without significant loss of
> performance in the contended case, by simply increasing the size of our
> primitive backing array and providing multiple buckets, with each thread
> picking a bucket to increment, or simply multiple backing arrays. Probably a
> better way still to do this would be to introduce some competition detection
> to the update, much like {{LongAdder}} utilises, that increases the number of
> backing arrays under competition.
> ** To save memory this approach could partition the space into chunks that
> are likely to be updated together, so that we do not need to duplicate the
> entire array under competition.
> * Similarly, binary search is costly and a measurable cost as a share of the
> new networking work (without filtering it was > 10% of the CPU used overall).
> We can compute an approximation floor(log2 n / log2 1.2) extremely cheaply,
> to save the random memory access costs.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-15213) DecayingEstimatedHistogramReservoir Inefficiencies
[
https://issues.apache.org/jira/browse/CASSANDRA-15213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17011493#comment-17011493
]
Jordan West edited comment on CASSANDRA-15213 at 1/9/20 7:07 AM:
-
Thanks for the clarification. I will take a stab at something to replace the
usage of {{LongAdder[]}} along the lines of what you suggested and share when
ready. I agree re: benchmarking and will try and come up with a more complete
set -- clearly we used the existing benchmarks before without finding this
issue.
Regarding the binary search, I am still a bit confused. If you run:
https://github.com/jrwest/cassandra/blob/jwest/15213/test/unit/org/apache/cassandra/metrics/DecayingEstimatedHistogramReservoirTest.java#L58
you will see that the proposed estimation (using the code provided) would
result in several cases where a linear search would lead to 10-50 accesses (as
value grows the approximation gets more inaccurate) and
{{Arrays.binarySearch(offsets, value) <= fastLog12(value)}} always.
Fwiw, the proposed fast log code does improve performance of my local
modifications when benchmarked but basically brings it back inline w/ the
existing implementation (caveat: benchmarking issues already mentioned).
was (Author: jrwest):
Thanks for the clarification. I will take a stab at something to replace the
usage of {{LongAdder[]}} along the lines of what you suggested and share when
ready. I agree re: benchmarking and will try and come up with a more complete
set -- clearly we used the existing benchmarks before without finding this
issue.
Regarding the binary search, I am still a bit confused. If you run:
https://github.com/jrwest/cassandra/blob/jwest/15213/test/unit/org/apache/cassandra/metrics/DecayingEstimatedHistogramReservoirTest.java#L58
you will see that the proposed estimation (using the code provided) would
result in several cases where a linear search would lead to 10-50 accesses (as
value grows the approximation gets more inaccurate).
Fwiw, the proposed fast log code does improve performance of my local
modifications when benchmarked but basically brings it back inline w/ the
existing implementation (caveat: benchmarking issues already mentioned).
> DecayingEstimatedHistogramReservoir Inefficiencies
> --
>
> Key: CASSANDRA-15213
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15213
> Project: Cassandra
> Issue Type: Bug
> Components: Observability/Metrics
>Reporter: Benedict Elliott Smith
>Assignee: Jordan West
>Priority: Normal
> Fix For: 4.0-beta
>
>
> * {{LongAdder}} introduced to trunk consumes 9MiB of heap without user
> schemas, and this will grow significantly under contention and user schemas
> with many tables. This is because {{LongAdder}} is a very heavy class
> designed for single contended values.
> ** This can likely be improved significantly, without significant loss of
> performance in the contended case, by simply increasing the size of our
> primitive backing array and providing multiple buckets, with each thread
> picking a bucket to increment, or simply multiple backing arrays. Probably a
> better way still to do this would be to introduce some competition detection
> to the update, much like {{LongAdder}} utilises, that increases the number of
> backing arrays under competition.
> ** To save memory this approach could partition the space into chunks that
> are likely to be updated together, so that we do not need to duplicate the
> entire array under competition.
> * Similarly, binary search is costly and a measurable cost as a share of the
> new networking work (without filtering it was > 10% of the CPU used overall).
> We can compute an approximation floor(log2 n / log2 1.2) extremely cheaply,
> to save the random memory access costs.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Assigned] (CASSANDRA-15428) First of the new features - Support for Java 11
[ https://issues.apache.org/jira/browse/CASSANDRA-15428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DeepakVohra reassigned CASSANDRA-15428: --- Assignee: DeepakVohra (was: Dinesh Joshi) > First of the new features - Support for Java 11 > --- > > Key: CASSANDRA-15428 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15428 > Project: Cassandra > Issue Type: Sub-task > Components: Legacy/Documentation and Website >Reporter: DeepakVohra >Assignee: DeepakVohra >Priority: Normal > Fix For: 4.0 > > > A pull request for first of the new features in 4.0. > - As 4.0 is a major release created a new folder called "new" > - Added a _index.rst_ to the _new_ folder > - Added _new/index_ to the top-level _index.rst_ > - Added _java11.rst_ to the _new_ folder for the new feature _Support for > Java 11_ > https://github.com/apache/cassandra/pull/383 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Assigned] (CASSANDRA-15473) New feature - Virtual Tables
[ https://issues.apache.org/jira/browse/CASSANDRA-15473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DeepakVohra reassigned CASSANDRA-15473: --- Authors: Deepak Vohra (was: Chris Lohfink) Reviewers: Chris Lohfink Assignee: DeepakVohra (was: Chris Lohfink) > New feature - Virtual Tables > > > Key: CASSANDRA-15473 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15473 > Project: Cassandra > Issue Type: Sub-task > Components: Documentation/Website >Reporter: DeepakVohra >Assignee: DeepakVohra >Priority: Normal > > Added a page on virtual tables, a new feature. > https://github.com/apache/cassandra/pull/402 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Assigned] (CASSANDRA-15474) Audit Logging - New Feature
[ https://issues.apache.org/jira/browse/CASSANDRA-15474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DeepakVohra reassigned CASSANDRA-15474: --- Authors: Deepak Vohra (was: Vinay Chella) Reviewers: Vinay Chella Assignee: DeepakVohra (was: Vinay Chella) > Audit Logging - New Feature > --- > > Key: CASSANDRA-15474 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15474 > Project: Cassandra > Issue Type: Sub-task > Components: Documentation/Website >Reporter: DeepakVohra >Assignee: DeepakVohra >Priority: Normal > > Added a page on audit logging, a new feature. > https://github.com/apache/cassandra/pull/403 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Assigned] (CASSANDRA-15476) Transient Replication - New Feature
[ https://issues.apache.org/jira/browse/CASSANDRA-15476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DeepakVohra reassigned CASSANDRA-15476: --- Authors: Deepak Vohra (was: Blake Eggleston) Reviewers: Blake Eggleston Assignee: DeepakVohra (was: Blake Eggleston) > Transient Replication - New Feature > --- > > Key: CASSANDRA-15476 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15476 > Project: Cassandra > Issue Type: Sub-task > Components: Documentation/Website >Reporter: DeepakVohra >Assignee: DeepakVohra >Priority: Normal > > Added a page on transient replication, a new feature. > https://github.com/apache/cassandra/pull/405 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Assigned] (CASSANDRA-15483) Overview
[ https://issues.apache.org/jira/browse/CASSANDRA-15483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DeepakVohra reassigned CASSANDRA-15483: --- Authors: Deepak Vohra (was: Nate McCall) Reviewers: Nate McCall Assignee: DeepakVohra (was: Nate McCall) > Overview > > > Key: CASSANDRA-15483 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15483 > Project: Cassandra > Issue Type: Sub-task > Components: Documentation/Website >Reporter: DeepakVohra >Assignee: DeepakVohra >Priority: Normal > > Added page on Overview. > https://github.com/apache/cassandra/pull/412 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Assigned] (CASSANDRA-15479) Backups - updated
[ https://issues.apache.org/jira/browse/CASSANDRA-15479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DeepakVohra reassigned CASSANDRA-15479: --- Authors: Deepak Vohra (was: Chris Lohfink) Reviewers: Chris Lohfink Assignee: DeepakVohra (was: Chris Lohfink) > Backups - updated > - > > Key: CASSANDRA-15479 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15479 > Project: Cassandra > Issue Type: Sub-task > Components: Documentation/Website >Reporter: DeepakVohra >Assignee: DeepakVohra >Priority: Normal > > Added Backups page. > https://github.com/apache/cassandra/pull/408 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Assigned] (CASSANDRA-15480) Bulk Loading
[ https://issues.apache.org/jira/browse/CASSANDRA-15480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DeepakVohra reassigned CASSANDRA-15480: --- Authors: Deepak Vohra (was: Nate McCall) Reviewers: Nate McCall Assignee: DeepakVohra (was: Nate McCall) > Bulk Loading > > > Key: CASSANDRA-15480 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15480 > Project: Cassandra > Issue Type: Sub-task > Components: Documentation/Website >Reporter: DeepakVohra >Assignee: DeepakVohra >Priority: Normal > > Added bulk loading page. > https://github.com/apache/cassandra/pull/409 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Assigned] (CASSANDRA-15492) DDL update
[ https://issues.apache.org/jira/browse/CASSANDRA-15492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DeepakVohra reassigned CASSANDRA-15492: --- Authors: Deepak Vohra (was: Dinesh Joshi) Reviewers: Dinesh Joshi Assignee: DeepakVohra (was: Dinesh Joshi) > DDL update > -- > > Key: CASSANDRA-15492 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15492 > Project: Cassandra > Issue Type: Sub-task > Components: Documentation/Website >Reporter: DeepakVohra >Assignee: DeepakVohra >Priority: Normal > > - Added more detail and examples to some sections such as Speculative retry > - Added table option 'cdc' > https://github.com/apache/cassandra/pull/420 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Assigned] (CASSANDRA-15485) Added Full Repair Example
[ https://issues.apache.org/jira/browse/CASSANDRA-15485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DeepakVohra reassigned CASSANDRA-15485: --- Authors: Deepak Vohra (was: Blake Eggleston) Reviewers: Blake Eggleston Assignee: DeepakVohra (was: Blake Eggleston) > Added Full Repair Example > - > > Key: CASSANDRA-15485 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15485 > Project: Cassandra > Issue Type: Sub-task > Components: Documentation/Website >Reporter: DeepakVohra >Assignee: DeepakVohra >Priority: Normal > > Added full repair example. > https://github.com/apache/cassandra/pull/414 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Assigned] (CASSANDRA-15491) Hints
[ https://issues.apache.org/jira/browse/CASSANDRA-15491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DeepakVohra reassigned CASSANDRA-15491: --- Authors: Deepak Vohra (was: Joey Lynch) Reviewers: Joey Lynch Assignee: DeepakVohra (was: Joey Lynch) > Hints > - > > Key: CASSANDRA-15491 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15491 > Project: Cassandra > Issue Type: Sub-task > Components: Documentation/Website >Reporter: DeepakVohra >Assignee: DeepakVohra >Priority: Normal > > Added page for hints. > https://github.com/apache/cassandra/pull/419 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15468) Unable to start embedded cassandra on JDK 11
[
https://issues.apache.org/jira/browse/CASSANDRA-15468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17011085#comment-17011085
]
Gus Heck commented on CASSANDRA-15468:
--
Worth noting that I've realized that "Add-Opens" does not exist in the world of
jar Manifests AFAICT... I assumed it would if "Add-Export" did, but it didn't
work and I find no references to it on the web. It might be possible for me to
use that info in Uno-Jar anyway since I hope to add support for module layers
such that dependencies on different versions can co-exist, and java.lang.Module
has an addOpens() method, so while I'm there playing with modules anyway... :).
> Unable to start embedded cassandra on JDK 11
>
>
> Key: CASSANDRA-15468
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15468
> Project: Cassandra
> Issue Type: Bug
>Reporter: Gus Heck
>Assignee: Gus Heck
>Priority: Normal
> Attachments: CASSANDRA-15468.patch
>
>
> Cassandra fails to start without the following jvm options from
> jvm-11-server.options:
> {code:java}
> -Djdk.attach.allowAttachSelf=true
> --add-exports java.base/jdk.internal.misc=ALL-UNNAMED
> --add-exports java.base/jdk.internal.ref=ALL-UNNAMED
> --add-exports java.base/sun.nio.ch=ALL-UNNAMED
> --add-exports java.management.rmi/com.sun.jmx.remote.internal.rmi=ALL-UNNAMED
> --add-exports java.rmi/sun.rmi.registry=ALL-UNNAMED
> --add-exports java.rmi/sun.rmi.server=ALL-UNNAMED
> --add-exports java.sql/java.sql=ALL-UNNAMED
> --add-opens java.base/java.lang.module=ALL-UNNAMED
> --add-opens java.base/jdk.internal.loader=ALL-UNNAMED
> --add-opens java.base/jdk.internal.ref=ALL-UNNAMED
> --add-opens java.base/jdk.internal.reflect=ALL-UNNAMED
> --add-opens java.base/jdk.internal.math=ALL-UNNAMED
> --add-opens java.base/jdk.internal.module=ALL-UNNAMED
> --add-opens java.base/jdk.internal.util.jar=ALL-UNNAMED
> --add-opens jdk.management/com.sun.management.internal=ALL-UNNAMED
> {code}
> Without these options the following stack trace appears and prevents startup
> {code:java}
> 2020-01-05T17:42:15,009 ERROR main
> org.apache.cassandra.service.CassandraDaemon Exception encountered during
> startup java.lang.ExceptionInInitializerError
> at
> org.apache.cassandra.config.DatabaseDescriptor.guessFileStore(DatabaseDescriptor.java:1127)
> at
> org.apache.cassandra.config.DatabaseDescriptor.applySimpleConfig(DatabaseDescriptor.java:541)
> at
> org.apache.cassandra.config.DatabaseDescriptor.applyAll(DatabaseDescriptor.java:340)
> at
> org.apache.cassandra.config.DatabaseDescriptor.daemonInitialization(DatabaseDescriptor.java:168)
> at
> org.apache.cassandra.config.DatabaseDescriptor.daemonInitialization(DatabaseDescriptor.java:152)
> at
> org.apache.cassandra.service.CassandraDaemon.applyConfig(CassandraDaemon.java:665)
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:609)
> at org.jesterj.ingest.persistence.Cassandra.start(Cassandra.java:135)
> at
> org.jesterj.ingest.scanners.SimpleFileScannerImplFTITest.testScanWithMemory(SimpleFileScannerImplFTITest.java:63)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
> at org.junit.runners.ParentRunner.run(Parent
[jira] [Updated] (CASSANDRA-15494) Support additional Jar url scheme
[
https://issues.apache.org/jira/browse/CASSANDRA-15494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gus Heck updated CASSANDRA-15494:
-
Attachment: CASSANDRA-15494.patch
> Support additional Jar url scheme
> -
>
> Key: CASSANDRA-15494
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15494
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Gus Heck
>Priority: Normal
> Attachments: CASSANDRA-15494.patch
>
>
> This code in ThreadAwareSecurityManager causes issues for embedded cassandra
> launched from a one-jar or uno-jar.
> {code:java}
> switch (codesource.getLocation().getProtocol())
> {
> case "file":
> // All JARs and class files reside on the file system
> - we can safely
> // assume that these classes are "good".
> return true;
> }
> {code}
> Attaching patch to support protocol of 'jar' as well. AFAICT this won't
> introduce any added risk for UDF's discussed in CASSANDRA-9402 since a jar
> (or one/uno jar) must also be on the filesystem.
> New code would be (in 2 places):
> {code:java}
> switch (codesource.getLocation().getProtocol())
> {
> case "jar":
> case "file":
> // All JARs and class files reside on the file system
> - we can safely
> // assume that these classes are "good".
> return true;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-15494) Support additional Jar url scheme
Gus Heck created CASSANDRA-15494:
Summary: Support additional Jar url scheme
Key: CASSANDRA-15494
URL: https://issues.apache.org/jira/browse/CASSANDRA-15494
Project: Cassandra
Issue Type: Improvement
Reporter: Gus Heck
This code in ThreadAwareSecurityManager causes issues for embedded cassandra
launched from a one-jar or uno-jar.
{code:java}
switch (codesource.getLocation().getProtocol())
{
case "file":
// All JARs and class files reside on the file system -
we can safely
// assume that these classes are "good".
return true;
}
{code}
Attaching patch to support protocol of 'jar' as well. AFAICT this won't
introduce any added risk for UDF's discussed in CASSANDRA-9402 since a jar (or
one/uno jar) must also be on the filesystem.
New code would be (in 2 places):
{code:java}
switch (codesource.getLocation().getProtocol())
{
case "jar":
case "file":
// All JARs and class files reside on the file system -
we can safely
// assume that these classes are "good".
return true;
}
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15213) DecayingEstimatedHistogramReservoir Inefficiencies
[
https://issues.apache.org/jira/browse/CASSANDRA-15213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17011029#comment-17011029
]
Benedict Elliott Smith commented on CASSANDRA-15213:
FWIW, I think when benchmarking something like this you need to create a few
hundred MiB worth of backing arrays, and cycle through them for each test. Or,
at least, there are different ways to achieve this but you ideally want tests
that include memory latency and this is a simple mechanism to achieve that.
> DecayingEstimatedHistogramReservoir Inefficiencies
> --
>
> Key: CASSANDRA-15213
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15213
> Project: Cassandra
> Issue Type: Bug
> Components: Observability/Metrics
>Reporter: Benedict Elliott Smith
>Assignee: Jordan West
>Priority: Normal
> Fix For: 4.0-beta
>
>
> * {{LongAdder}} introduced to trunk consumes 9MiB of heap without user
> schemas, and this will grow significantly under contention and user schemas
> with many tables. This is because {{LongAdder}} is a very heavy class
> designed for single contended values.
> ** This can likely be improved significantly, without significant loss of
> performance in the contended case, by simply increasing the size of our
> primitive backing array and providing multiple buckets, with each thread
> picking a bucket to increment, or simply multiple backing arrays. Probably a
> better way still to do this would be to introduce some competition detection
> to the update, much like {{LongAdder}} utilises, that increases the number of
> backing arrays under competition.
> ** To save memory this approach could partition the space into chunks that
> are likely to be updated together, so that we do not need to duplicate the
> entire array under competition.
> * Similarly, binary search is costly and a measurable cost as a share of the
> new networking work (without filtering it was > 10% of the CPU used overall).
> We can compute an approximation floor(log2 n / log2 1.2) extremely cheaply,
> to save the random memory access costs.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-12993) License headers missing in some source files
[ https://issues.apache.org/jira/browse/CASSANDRA-12993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Venkata Harikrishna Nukala updated CASSANDRA-12993: --- Test and Documentation Plan: Changes are related to adding license text only. Ran unit test cases. Here is the CI: [https://app.circleci.com/github/nvharikrishna/cassandra/pipelines/257856a5-2c72-4efc-9162-6acba2c61c1f/workflows/e87fcb23-ff95-4ecc-a3f0-47bc5b44d64c] Status: Patch Available (was: Open) > License headers missing in some source files > > > Key: CASSANDRA-12993 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12993 > Project: Cassandra > Issue Type: Improvement > Components: Packaging >Reporter: Tomas Repik >Assignee: Venkata Harikrishna Nukala >Priority: Normal > Fix For: 4.0 > > > The following source files are without license headers: > doc/source/_static/extra.css > src/java/org/apache/cassandra/db/commitlog/IntervalSet.java > src/java/org/apache/cassandra/utils/IntegerInterval.java > test/unit/org/apache/cassandra/db/commitlog/CommitLogCQLTest.java > test/unit/org/apache/cassandra/utils/IntegerIntervalsTest.java > tools/stress/src/org/apache/cassandra/stress/WorkManager.java > Could you please confirm the licensing of code and/or content/s, and add > license headers? -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-15213) DecayingEstimatedHistogramReservoir Inefficiencies
[
https://issues.apache.org/jira/browse/CASSANDRA-15213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17010990#comment-17010990
]
Benedict Elliott Smith edited comment on CASSANDRA-15213 at 1/8/20 8:24 PM:
bq. Or are you suggesting increasing the bucket resolution
bq. Do you mean having multiple buckets per offset?
Right. I meant one option was to essentially inline the striping by
introducing, e.g., 4x as many buckets as requested. Each logical bucket can be
represented by the sum of that many buckets. The buckets would be offset from
each other by at least two cache lines, and all buckets would ideally be more
uniformly distributed in memory (by e.g. {{Integer.reverse(idx)}}), so that
adjacent buckets that are most likely to be used together don't also compete
for cache locks.
bq. Additionally, re: your LongAdder reference, are you proposing creating new
buckets for the same offset under contention (like LongAdder does w/ Cell)?
I meant an alternative approach might be to batch a range of buckets together -
say 16 or more, to amortise the object overhead - and to perform
{{LongAdder}}’s inflation of the number of buckets on the detection of
competition, so that we can gradually increase the number of cells needed.
This might have the added benefit of not wasting space for striping of buckets
that are rarely used, but at the cost of greater complexity, object overheads
and indirection.
bq. Regarding the binary search suggestion, are you suggesting using the
approximation to replace the maximum index
I meant to use it as the floor for a linear search for the correct position, so
that it would ordinarily be 1 or 2 comparisons, but typically only one cache
miss.
Specifically, I meant to use an integer approximation, along the lines of...
{code}
private static final int TABLE_BITS = 4;
private static final int TABLE_MASK = -1 >>> (32 - TABLE_BITS);
private static final float[] LOG2_TABLE = computeTable(TABLE_BITS);
private static final float log2_12 = (float) slowLog2(1.2d);
private static final float log2_12_recp = (float) (1d / slowLog2(1.2d));
private static float[] computeTable(int bits)
{
float[] table = new float[1 << bits];
for (int i = 1 ; i < 1<>> (32 - TABLE_BITS);
private static final float[] LOG2_TABLE = computeTable(TABLE_BITS);
private static final float log2_12 = (float) slowLog2(1.2d);
private static final float log2_12_recp = (float) (1d / slowLog2(1.2d));
private static float[] computeTable(int bits)
{
float[] table = new float[1 << bits];
for (int i = 1 ; i < 1< DecayingEstimatedHistogramReservoir Inefficiencies
> --
>
> Key: CASSANDRA-15213
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15213
> Project: Cassandra
> Issue Type: Bug
> Components: Observability/Metrics
>Reporter: Benedict Elliott Smith
>Assignee: Jordan West
>Priority: Normal
> Fix For: 4.0-beta
>
>
> * {{LongAdder}} introduced to trunk consumes 9MiB of heap without user
> schemas, and this will grow significantly under contention and user schemas
> with many tables. This is because {{LongAdder}} is a very heavy class
> designed for single contended values.
> ** This can likely be improved significantly, without significant loss of
> performance in the contended case, by simply increasing the size of our
> primitive backing array and providing multiple buckets, with each thread
> picking a bucket to increment, or simply multiple backing arrays. Probably a
> better way still to do this would be to introduce some competition detection
> to the update, much like {{LongAdder}} utilises, that increases the number of
> backing arrays under competition.
> ** To save memory this approach could partition the space into chunks that
> are likely to be updated together, so that we do not need to duplicate the
> entire array under competition.
> * Similarly, binary search is costly and a measurable cost as a share of the
> new networking work (without filtering it was > 10% of the CPU used overall).
> We can compute an approximation floor(log2 n / log2 1.2) extremely cheaply,
> to save the random memory access costs.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15213) DecayingEstimatedHistogramReservoir Inefficiencies
[
https://issues.apache.org/jira/browse/CASSANDRA-15213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17010990#comment-17010990
]
Benedict Elliott Smith commented on CASSANDRA-15213:
bq. Or are you suggesting increasing the bucket resolution
bq. Do you mean having multiple buckets per offset?
Right. I meant one option was to essentially inline the striping by
introducing, e.g., 4x as many buckets as requested. Each logical bucket can be
represented by the sum of that many buckets. The buckets would be offset from
each other by at least two cache lines, and all buckets would ideally be more
uniformly distributed in memory (by e.g. {{Integer.reverse(idx)}}), so that
adjacent buckets that are most likely to be used together don't also compete
for cache locks.
bq. Additionally, re: your LongAdder reference, are you proposing creating new
buckets for the same offset under contention (like LongAdder does w/ Cell)?
I meant an alternative approach might be to batch a range of buckets together -
say 16 or more, to amortise the object overhead - and to perform
{{LongAdder}}’s inflation of the number of buckets on the detection of
competition, so that we can gradually increase the number of cells needed.
This might have the added benefit of not wasting space for striping of buckets
that are rarely used, but at the cost of greater complexity, object overheads
and indirection.
bq. Regarding the binary search suggestion, are you suggesting using the
approximation to replace the maximum index
I meant to use it as the floor for a linear search for the correct position, so
that it would ordinarily be 1 or 2 comparisons, but typically only one cache
miss.
Specifically, I meant to use an integer approximation, along the lines of...
{code}
private static final int TABLE_BITS = 4;
private static final int TABLE_MASK = -1 >>> (32 - TABLE_BITS);
private static final float[] LOG2_TABLE = computeTable(TABLE_BITS);
private static final float log2_12 = (float) slowLog2(1.2d);
private static final float log2_12_recp = (float) (1d / slowLog2(1.2d));
private static float[] computeTable(int bits)
{
float[] table = new float[1 << bits];
for (int i = 1 ; i < 1< DecayingEstimatedHistogramReservoir Inefficiencies
> --
>
> Key: CASSANDRA-15213
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15213
> Project: Cassandra
> Issue Type: Bug
> Components: Observability/Metrics
>Reporter: Benedict Elliott Smith
>Assignee: Jordan West
>Priority: Normal
> Fix For: 4.0-beta
>
>
> * {{LongAdder}} introduced to trunk consumes 9MiB of heap without user
> schemas, and this will grow significantly under contention and user schemas
> with many tables. This is because {{LongAdder}} is a very heavy class
> designed for single contended values.
> ** This can likely be improved significantly, without significant loss of
> performance in the contended case, by simply increasing the size of our
> primitive backing array and providing multiple buckets, with each thread
> picking a bucket to increment, or simply multiple backing arrays. Probably a
> better way still to do this would be to introduce some competition detection
> to the update, much like {{LongAdder}} utilises, that increases the number of
> backing arrays under competition.
> ** To save memory this approach could partition the space into chunks that
> are likely to be updated together, so that we do not need to duplicate the
> entire array under competition.
> * Similarly, binary search is costly and a measurable cost as a share of the
> new networking work (without filtering it was > 10% of the CPU used overall).
> We can compute an approximation floor(log2 n / log2 1.2) extremely cheaply,
> to save the random memory access costs.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15470) Potential Overflow in DatabaseDescriptor Functions That Convert Between KB/MB & Bytes
[
https://issues.apache.org/jira/browse/CASSANDRA-15470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17010989#comment-17010989
]
Jordan West commented on CASSANDRA-15470:
-
[~mallika] that seems like a reasonable range. When {{input}} is KiB thats 2Tib
which is extremely large for any of those values. I agree the validation would
be beneficial in the setters since some of them can be changed by operators at
runtime (or may be made to do so in the future) or tests.
I would recommend *not* performing additional validation in the getters. The
reasons for doing the getter validation you noted are correct but the getters
may be used in a code hot path where minimal work is desired. Looking at the
code {{getRawConfig()}} is only used for testing (and perhaps should be marked
as {{@VisibleForTesting}}
> Potential Overflow in DatabaseDescriptor Functions That Convert Between KB/MB
> & Bytes
> -
>
> Key: CASSANDRA-15470
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15470
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Config
>Reporter: Jordan West
>Assignee: Mallika Kulkarni
>Priority: Normal
> Fix For: 4.0-rc
>
>
> {{DatabaseDescriptor}} has several functions that convert between user
> supplied sizes in KB/MB and bytes. These are implemented without much
> consistency and, while unlikely, several have the potential to overflow since
> validation on the input is missing. Meanwhile, some widen the number to a
> long correctly. Options include: widening in all places or simply doing
> better validation on start up — currently only the lower bound of the valid
> range is checked for many of these fields.
> List of Affected {{DatabaseDescriptor}} Methods:
> * {{getColumnIndexSize}}
> * {{getColumnIndexCacheSize}}
> * {{getBatchSizeWarnThreshold}}
> * {{getNativeTransportFrameBlockSize}}
> * {{getRepairSessionSpaceInMegabytes}}
> * {{getNativeTransportMaxFrameSize}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-15213) DecayingEstimatedHistogramReservoir Inefficiencies
[
https://issues.apache.org/jira/browse/CASSANDRA-15213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17010956#comment-17010956
]
Jordan West edited comment on CASSANDRA-15213 at 1/8/20 7:41 PM:
-
I’ve verified the reported {{LongAdder}} memory consumption. I’m seeing 13.5MiB
right after startup.
A few questions about your proposed solutions?
* By the backing array do you mean the original AtomicLongArray? Two issues I
see with increasing the number of buckets: the maximum we can create are 237
(then the offset overflows) and increasing the number of buckets doesn’t
necessarily reduce contention. This is because the resolution of the buckets
doesn’t change except for the last bucket (which for microseconds is over 200
days and for nanoseconds is over 5 hours). Or are you suggesting increasing the
bucket resolution by decreasing the factor in {{newOffsets}} when adding more
buckets?
* I’m not sure what you mean by “providing multiple buckets” but I suspect it
might be what I am missing to better understand the proposal. Do you mean
having multiple buckets per offset? Additionally, re: your {{LongAdder}}
reference, are you proposing creating new buckets for the same offset under
contention (like {{LongAdder}} does w/ {{Cell}})?
* Regarding the binary search suggestion, are you suggesting using the
approximation to replace the maximum index (e.g. binarySearch(offsets, 0,
Math.floor(….) + 1, value)? Or to outright use that approximation? I wasn’t
sure of your exact proposal but it does seem like using the equation you gave
as an upper bound estimate and {{Math.floor(Math.log(Math.floor(n / 1.2)) /
Math.log(1.2))}} as a lower bound estimate can reduce the space of the array we
have to search by 25-60% (I saw up to 75% in one case with DEFAULT_BUCKET_COUNT
but haven’t reproduced that high of a reduction since). I’ve added a
QuickTheories test to show this here:
[https://github.com/jrwest/cassandra/commit/a4795bc651cd0e0bd582a8df2414e312782b5020].
I should note however that this doesn't translate into a performance
improvement in the microbenchmark -- perhaps because it really only reduces the
number of accesses slightly at the cost of more calculations.
* We could also circumvent the search entirely in the lowest buckets by using
{{if value < bucketCount && bucketOffsets[value - 1] == value}} although given
our use case its unlikely we will see sub 165 ns operations often.
was (Author: jrwest):
I’ve verified the reported {{LongAdder}} memory consumption. I’m seeing 13.5MiB
right after startup.
A few questions about your proposed solutions?
* By the backing array do you mean the original AtomicLongArray? Two issues I
see with increasing the number of buckets: the maximum we can create are 237
(then the offset overflows) and increasing the number of buckets doesn’t
necessarily reduce contention. This is because the resolution of the buckets
doesn’t change except for the last bucket (which for microseconds is over 200
days and for nanoseconds is over 5 hours). Or are you suggesting increasing the
bucket resolution by decreasing the factor in {{newOffsets}} when adding more
buckets?
* I’m not sure what you mean by “providing multiple buckets” but I suspect it
might be what I am missing to better understand the proposal. Do you mean
having multiple buckets per offset? Additionally, re: your {{LongAdder}}
reference, are you proposing creating new buckets for the same offset under
contention (like {{LongAdder}} does w/ {{Cell}})?
* Regarding the binary search suggestion, are you suggesting using the
approximation to replace the maximum index (e.g. binarySearch(offsets, 0,
Math.floor(….) + 1, value)? Or to outright use that approximation? I wasn’t
sure of your exact proposal but it does seem like using the equation you gave
as an upper bound estimate and {{Math.floor(Math.log(Math.floor(n / 1.2)) /
Math.log(1.2))}} as a lower bound estimate can reduce the space of the array we
have to search by 25-60% (I saw up to 75% in one case with DEFAULT_BUCKET_COUNT
but haven’t reproduced that high of a reduction since). I’ve added a
QuickTheories test to show this here:
[https://github.com/jrwest/cassandra/commit/a4795bc651cd0e0bd582a8df2414e312782b5020].
I should note however that this doesn't translate into a performance
improvement in the microbenchmark -- perhaps because it really only reduces the
number of accesses slightly at the cost of more calculations.
* We could also circumvent the search entirely in the lowest buckets by using
{{if value < bucketCount && bucketOffsets[value - 1] == value}} although given
our use case I’m not sure how often we will see sub 165 ns operations.
> DecayingEstimatedHistogramReservoir Inefficiencies
> --
>
> Key: CASSANDRA-15213
> URL:
[jira] [Comment Edited] (CASSANDRA-15213) DecayingEstimatedHistogramReservoir Inefficiencies
[
https://issues.apache.org/jira/browse/CASSANDRA-15213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17010956#comment-17010956
]
Jordan West edited comment on CASSANDRA-15213 at 1/8/20 7:37 PM:
-
I’ve verified the reported {{LongAdder}} memory consumption. I’m seeing 13.5MiB
right after startup.
A few questions about your proposed solutions?
* By the backing array do you mean the original AtomicLongArray? Two issues I
see with increasing the number of buckets: the maximum we can create are 237
(then the offset overflows) and increasing the number of buckets doesn’t
necessarily reduce contention. This is because the resolution of the buckets
doesn’t change except for the last bucket (which for microseconds is over 200
days and for nanoseconds is over 5 hours). Or are you suggesting increasing the
bucket resolution by decreasing the factor in {{newOffsets}} when adding more
buckets?
* I’m not sure what you mean by “providing multiple buckets” but I suspect it
might be what I am missing to better understand the proposal. Do you mean
having multiple buckets per offset? Additionally, re: your {{LongAdder}}
reference, are you proposing creating new buckets for the same offset under
contention (like {{LongAdder}} does w/ {{Cell}})?
* Regarding the binary search suggestion, are you suggesting using the
approximation to replace the maximum index (e.g. binarySearch(offsets, 0,
Math.floor(….) + 1, value)? Or to outright use that approximation? I wasn’t
sure of your exact proposal but it does seem like using the equation you gave
as an upper bound estimate and {{Math.floor(Math.log(Math.floor(n / 1.2)) /
Math.log(1.2))}} as a lower bound estimate can reduce the space of the array we
have to search by 25-60% (I saw up to 75% in one case with DEFAULT_BUCKET_COUNT
but haven’t reproduced that high of a reduction since). I’ve added a
QuickTheories test to show this here:
[https://github.com/jrwest/cassandra/commit/a4795bc651cd0e0bd582a8df2414e312782b5020].
I should note however that this doesn't translate into a performance
improvement in the microbenchmark -- perhaps because it really only reduces the
number of accesses slightly at the cost of more calculations.
* We could also circumvent the search entirely in the lowest buckets by using
{{if value < bucketCount && bucketOffsets[value - 1] == value}} although given
our use case I’m not sure how often we will see sub 165 ns operations.
was (Author: jrwest):
I’ve verified the reported {{LongAdder}} memory consumption. I’m seeing 13.5MiB
right after startup.
A few questions about your proposed solutions?
* By the backing array do you mean the original AtomicLongArray? Two issues I
see with increasing the number of buckets: the maximum we can create are 237
(then the offset overflows) and increasing the number of buckets doesn’t
necessarily reduce contention. This is because the resolution of the buckets
doesn’t change except for the last bucket (which for microseconds is over 200
days and for nanoseconds is over 5 hours). Or are you suggesting increasing the
bucket resolution by decreasing the factor in {{newOffsets}} when adding more
buckets?
* I’m not sure what you mean by “providing multiple buckets” but I suspect it
might be what I am missing to better understand the proposal. Do you mean
having multiple buckets per offset? Additionally, re: your {{LongAdder}}
reference, are you proposing creating new buckets for the same offset under
contention (like {{LongAdder}} does w/ {{Cell}})?
* Regarding the binary search suggestion, are you suggesting using the
approximation to replace the maximum index (e.g. binarySearch(offsets, 0,
Math.floor(….) + 1, value)? Or to outright use that approximation? I wasn’t
sure of your exact proposal but it does seem like using the equation you gave
as an upper bound estimate and {{Math.floor(Math.log(Math.floor(n / 1.2)) /
Math.log(1.2))}} as a lower bound estimate can reduce the space of the array we
have to search by 25-60% (I saw up to 75% in one case with DEFAULT_BUCKET_COUNT
but haven’t reproduced that high of a reduction since). I’ve added a
QuickTheories test to show this here:
[https://github.com/jrwest/cassandra/commit/a4795bc651cd0e0bd582a8df2414e312782b5020].
* We could also circumvent the search entirely in the lowest buckets by using
{{if value < bucketCount && bucketOffsets[value - 1] == value}} although given
our use case I’m not sure how often we will see sub 165 ns operations.
> DecayingEstimatedHistogramReservoir Inefficiencies
> --
>
> Key: CASSANDRA-15213
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15213
> Project: Cassandra
> Issue Type: Bug
> Components: Observability/Metrics
>Reporter: Benedict Elliott Smith
>
[jira] [Comment Edited] (CASSANDRA-12993) License headers missing in some source files
[ https://issues.apache.org/jira/browse/CASSANDRA-12993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17010962#comment-17010962 ] Venkata Harikrishna Nukala edited comment on CASSANDRA-12993 at 1/8/20 7:36 PM: Added license text for the files mentioned in ticket description. Here is the patch on trunk: [https://github.com/nvharikrishna/cassandra/commit/96326245de7a5157452fb753075dff1870a2def6] was (Author: n.v.harikrishna): Added license text for the files mentioned above. Here is the patch on trunk: [https://github.com/nvharikrishna/cassandra/commit/96326245de7a5157452fb753075dff1870a2def6] > License headers missing in some source files > > > Key: CASSANDRA-12993 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12993 > Project: Cassandra > Issue Type: Improvement > Components: Packaging >Reporter: Tomas Repik >Assignee: Venkata Harikrishna Nukala >Priority: Normal > Fix For: 4.0 > > > The following source files are without license headers: > doc/source/_static/extra.css > src/java/org/apache/cassandra/db/commitlog/IntervalSet.java > src/java/org/apache/cassandra/utils/IntegerInterval.java > test/unit/org/apache/cassandra/db/commitlog/CommitLogCQLTest.java > test/unit/org/apache/cassandra/utils/IntegerIntervalsTest.java > tools/stress/src/org/apache/cassandra/stress/WorkManager.java > Could you please confirm the licensing of code and/or content/s, and add > license headers? -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12993) License headers missing in some source files
[ https://issues.apache.org/jira/browse/CASSANDRA-12993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17010962#comment-17010962 ] Venkata Harikrishna Nukala commented on CASSANDRA-12993: Added license text for the files mentioned above. Here is the patch on trunk: [https://github.com/nvharikrishna/cassandra/commit/96326245de7a5157452fb753075dff1870a2def6] > License headers missing in some source files > > > Key: CASSANDRA-12993 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12993 > Project: Cassandra > Issue Type: Improvement > Components: Packaging >Reporter: Tomas Repik >Assignee: Venkata Harikrishna Nukala >Priority: Normal > Fix For: 4.0 > > > The following source files are without license headers: > doc/source/_static/extra.css > src/java/org/apache/cassandra/db/commitlog/IntervalSet.java > src/java/org/apache/cassandra/utils/IntegerInterval.java > test/unit/org/apache/cassandra/db/commitlog/CommitLogCQLTest.java > test/unit/org/apache/cassandra/utils/IntegerIntervalsTest.java > tools/stress/src/org/apache/cassandra/stress/WorkManager.java > Could you please confirm the licensing of code and/or content/s, and add > license headers? -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15213) DecayingEstimatedHistogramReservoir Inefficiencies
[
https://issues.apache.org/jira/browse/CASSANDRA-15213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17010956#comment-17010956
]
Jordan West commented on CASSANDRA-15213:
-
I’ve verified the reported {{LongAdder}} memory consumption. I’m seeing 13.5MiB
right after startup.
A few questions about your proposed solutions?
* By the backing array do you mean the original AtomicLongArray? Two issues I
see with increasing the number of buckets: the maximum we can create are 237
(then the offset overflows) and increasing the number of buckets doesn’t
necessarily reduce contention. This is because the resolution of the buckets
doesn’t change except for the last bucket (which for microseconds is over 200
days and for nanoseconds is over 5 hours). Or are you suggesting increasing the
bucket resolution by decreasing the factor in {{newOffsets}} when adding more
buckets?
* I’m not sure what you mean by “providing multiple buckets” but I suspect it
might be what I am missing to better understand the proposal. Do you mean
having multiple buckets per offset? Additionally, re: your {{LongAdder}}
reference, are you proposing creating new buckets for the same offset under
contention (like {{LongAdder}} does w/ {{Cell}})?
* Regarding the binary search suggestion, are you suggesting using the
approximation to replace the maximum index (e.g. binarySearch(offsets, 0,
Math.floor(….) + 1, value)? Or to outright use that approximation? I wasn’t
sure of your exact proposal but it does seem like using the equation you gave
as an upper bound estimate and {{Math.floor(Math.log(Math.floor(n / 1.2)) /
Math.log(1.2))}} as a lower bound estimate can reduce the space of the array we
have to search by 25-60% (I saw up to 75% in one case with DEFAULT_BUCKET_COUNT
but haven’t reproduced that high of a reduction since). I’ve added a
QuickTheories test to show this here:
[https://github.com/jrwest/cassandra/commit/a4795bc651cd0e0bd582a8df2414e312782b5020].
* We could also circumvent the search entirely in the lowest buckets by using
{{if value < bucketCount && bucketOffsets[value - 1] == value}} although given
our use case I’m not sure how often we will see sub 165 ns operations.
> DecayingEstimatedHistogramReservoir Inefficiencies
> --
>
> Key: CASSANDRA-15213
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15213
> Project: Cassandra
> Issue Type: Bug
> Components: Observability/Metrics
>Reporter: Benedict Elliott Smith
>Assignee: Jordan West
>Priority: Normal
> Fix For: 4.0-beta
>
>
> * {{LongAdder}} introduced to trunk consumes 9MiB of heap without user
> schemas, and this will grow significantly under contention and user schemas
> with many tables. This is because {{LongAdder}} is a very heavy class
> designed for single contended values.
> ** This can likely be improved significantly, without significant loss of
> performance in the contended case, by simply increasing the size of our
> primitive backing array and providing multiple buckets, with each thread
> picking a bucket to increment, or simply multiple backing arrays. Probably a
> better way still to do this would be to introduce some competition detection
> to the update, much like {{LongAdder}} utilises, that increases the number of
> backing arrays under competition.
> ** To save memory this approach could partition the space into chunks that
> are likely to be updated together, so that we do not need to duplicate the
> entire array under competition.
> * Similarly, binary search is costly and a measurable cost as a share of the
> new networking work (without filtering it was > 10% of the CPU used overall).
> We can compute an approximation floor(log2 n / log2 1.2) extremely cheaply,
> to save the random memory access costs.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14781) Log message when mutation passed to CommitLog#add(Mutation) is too large is not descriptive enough
[ https://issues.apache.org/jira/browse/CASSANDRA-14781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17010954#comment-17010954 ] Venkata Harikrishna Nukala commented on CASSANDRA-14781: [~jrwest] Made the changes. Here is the [updated patch|https://github.com/nvharikrishna/cassandra/commit/1eb9a9846187f669516c88c85fa3550e4efb08f7] and [CI|https://app.circleci.com/jobs/github/nvharikrishna/cassandra/120]. Summary of changes: * Removed SizeHolder * MutationExceededMaxSizeException ** Avoided calculating size again. ** Changed constant for limiting no.of keys to size of the message. We are mostly concerned about dumping huge message to log. No.of keys to log has to vary based on its size and there won't be an ideal config by no.of keys. So changed it to message size (1kb for now, we can increase it further). * IMutation ** removed getMaxMutationSize and replaced it with constant from CommitLog. * Replaced Mutation.serializer.serializedSize with mutation.serializedSize. > Log message when mutation passed to CommitLog#add(Mutation) is too large is > not descriptive enough > -- > > Key: CASSANDRA-14781 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14781 > Project: Cassandra > Issue Type: Bug > Components: Consistency/Hints, Local/Commit Log, Messaging/Client >Reporter: Jordan West >Assignee: Tom Petracca >Priority: Normal > Labels: protocolv5 > Fix For: 4.0-beta > > Attachments: CASSANDRA-14781.patch, CASSANDRA-14781_3.0.patch, > CASSANDRA-14781_3.11.patch > > > When hitting > [https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/db/commitlog/CommitLog.java#L256-L257], > the log message produced does not help the operator track down what data is > being written. At a minimum the keyspace and cfIds involved would be useful > (and are available) – more detail might not be reasonable to include. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15389) Minimize BTree iterator allocations
[ https://issues.apache.org/jira/browse/CASSANDRA-15389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benedict Elliott Smith updated CASSANDRA-15389: --- Status: Changes Suggested (was: Review In Progress) > Minimize BTree iterator allocations > --- > > Key: CASSANDRA-15389 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15389 > Project: Cassandra > Issue Type: Sub-task > Components: Local/Compaction >Reporter: Blake Eggleston >Assignee: Blake Eggleston >Priority: Normal > Fix For: 4.0 > > > Allocations of BTree iterators contribute a lot amount of garbage to the > compaction and read paths. > This patch removes most btree iterator allocations on hot paths by: > • using Row#apply where appropriate on frequently called methods > (Row#digest, Row#validateData > • adding BTree accumulate method. Like the apply method, this method walks > the btree with a function that takes and returns a long argument, this > eliminates iterator allocations without adding helper object allocations > (BTreeRow#hasComplex, BTreeRow#hasInvalidDeletions, BTreeRow#dataSize, > BTreeRow#unsharedHeapSizeExcludingData, Rows#collectStats, > UnfilteredSerializer#serializedRowBodySize) as well as eliminating the > allocation of helper objects in places where apply was used previously^[1]^. > • Create map of columns in SerializationHeader, this lets us avoid > allocating a btree search iterator for each row we serialize. > These optimizations reduce garbage created during compaction by up to 13.5% > > [1] the memory test does measure memory allocated by lambdas capturing objects -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15389) Minimize BTree iterator allocations
[
https://issues.apache.org/jira/browse/CASSANDRA-15389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17010900#comment-17010900
]
Benedict Elliott Smith commented on CASSANDRA-15389:
Thanks, the patch is looking really good. Just some minor suggestions to
consider.
AbstractRow
* If we introduce a {{BiFunction}} variant of {{apply}}, in which we can
supply the second argument, then we can go garbage-free for e.g. {{digest}} by
calling {{apply(ColumnData::digest, hasher)}}
BTreeRow
* {{hasComplexDeletion}} does not need the {{if (isSimple)}} branch anymore, I
think?
* {{hasComplexDeletion}} seems to be testing the wrong sentinel return value
from {{accumulate}}?
* Does {{HAS_INVALID_DELETIONS}} need to test {{cd != null}}?
Row
* {{collectStats}} could delegate to a new {{ComplexColumnData.accumulate}},
and pass itself as the parameter (though we would need to test {{!(cd
instanceof ComplexColumnData)}} instead of {{cd.column.isSimple()}}
* This might mean we can remove the {{hasCells()}} test, as we will be
garbage-free and can let {{accumulate}} make the right decision for us
BTree
* Having been playing with a lot of BTree methods recently, I personally find
it aids clarity to have totally separate bits of code implementing behavior for
leaves and branches, with a guard at-or-near the start of the method switching
behaviour. It could invoke a {{methodLeaf}} variant, or do it inline, but I
think it could make it somewhat easier to follow the logic in {{accumulate}}.
It’s only a small amount of code to duplicate, but the reader doesn’t have to
try and play out the switches while following the flow.
* I also find that when iterating branches, it is most clear to simply do one
key/child pair per loop, and a final (or initial) unrolled copy of a child
visit. Again, a small amount of duplication but I think easier for a reader to
digest (at least, I found this helped when reading my own methods). This
_should_ also lead to fewer executed instructions, and perhaps better branch
predictor behaviour (hard to say, many of them might be well predicted anyway).
nits:
* {{AbstractRow.validateData}} should probably use method reference for
clarity, i.e. {{apply(ColumnData::validate)}}
* {{BTreeRow.HAS_INVALID_DELETIONS}} can be declared using lambda syntax
* {{BTreeRow}} unused imports
* {{BTreeRow:345}} unused {{ALWAYS_TRUE}}
* {{Row}} unused imports
* {{Rows}} unused {{addColumn}}
* {{UnfilteredSerializer}} unused import
* {{SerializationHeader}} unused imports
* {{BTree}} unused import
> Minimize BTree iterator allocations
> ---
>
> Key: CASSANDRA-15389
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15389
> Project: Cassandra
> Issue Type: Sub-task
> Components: Local/Compaction
>Reporter: Blake Eggleston
>Assignee: Blake Eggleston
>Priority: Normal
> Fix For: 4.0
>
>
> Allocations of BTree iterators contribute a lot amount of garbage to the
> compaction and read paths.
> This patch removes most btree iterator allocations on hot paths by:
> • using Row#apply where appropriate on frequently called methods
> (Row#digest, Row#validateData
> • adding BTree accumulate method. Like the apply method, this method walks
> the btree with a function that takes and returns a long argument, this
> eliminates iterator allocations without adding helper object allocations
> (BTreeRow#hasComplex, BTreeRow#hasInvalidDeletions, BTreeRow#dataSize,
> BTreeRow#unsharedHeapSizeExcludingData, Rows#collectStats,
> UnfilteredSerializer#serializedRowBodySize) as well as eliminating the
> allocation of helper objects in places where apply was used previously^[1]^.
> • Create map of columns in SerializationHeader, this lets us avoid
> allocating a btree search iterator for each row we serialize.
> These optimizations reduce garbage created during compaction by up to 13.5%
>
> [1] the memory test does measure memory allocated by lambdas capturing objects
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-15392) Pool Merge Iterators
[
https://issues.apache.org/jira/browse/CASSANDRA-15392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17010853#comment-17010853
]
Benedict Elliott Smith edited comment on CASSANDRA-15392 at 1/8/20 5:34 PM:
MergeIteratorPool:
* As Jordan suggests, storing the pool inside the \{{MergeIterator}} makes
sense - more so because it can avoid incurring the cost of
\{{ThreadLocal.get()}} on \{{returnToPool}}, by stashing a link to an inner
object (that guards put on the identity of the \{{Thread}} owning the queue)
* Perhaps an \{{ArrayList}} is a better idea for collection type, as it will
be generally garbage-free, and occupy less space than a \{{LinkedList}}? An
\{{ArrayDeque}} could maintain the current behaviour, or you could pop from the
back
* Perhaps the queue would be better with fixed size, permitting at most two or
three items to be stored?
* As it happens, I’ve been working on very similar parts of the codebase, and
I have implemented my own \{{TinyThreadLocalPool}} in the patches I hope to
land soon, that does the above but has lower fixed-overhead and less
indirection by permitting at most 3 items, stored in in-line properties (i.e.
\{{val0}}, \{{val1}} and \{{val2}}). It might be worthwhile introducing that
class here for consideration/comparison, and perhaps some amalgamation of the
two used in whichever patch lands first (which I fully expect to be this one)?
{\{MergeIterator.OneToOne}}:
* While we’re here optimising, it probably makes sense to merge \{{OneToOne}}
and \{{TrivialOneToOne}}, simply permitting the reducer in \{{OneToOne}} to be
\{{null}}, and to switch behaviour based on a null check. This should permit
call-sites to become bimorphic that are presently megamorphic, and the branch
should be near perfectly predicted.
{\{MergeIterator.ManyToOne}}:
* Instead of using a separate \{{candidates}} array for reusing a
\{{Candidate}}, could we avoid setting the last heap element to null by instead
placing the now-defunct item there?
* Do we need \{{active}} when it is implied by \{{MergeIterator.iterators !=
null}} and \{{Candidate.iter != null}}?
* Do we need to set primitives to \{{-1}} or \{{false}} on \{{close}}, given
we will throw \{{NullPointerException}} if we haven’t invoked \{{reset}}?
Benchmarking:
* It’s annoying work, but alongside the garbage benchmarks, it would be nice
to see some simple JMH throughput benchmarks introduced, to see what the
overall impact of the change is. I don’t mind helping out here.
nits:
* \{{ManyToOne.DEFAULT_SIZE}} -> \{{ManyToOne.POOLED_MERGE_LIMIT}}?
* +1 Jordan’s nit about naming of {{getSimple}}
* \{{MergeIteratorPool.queues.initialValue}} should not declare {{throw
Exception}}
* {{MergeIteratorPool.initialValue}} looks to be a duplicate
* {{TokenTree:380}} errant new line
was (Author: benedict):
MergeIteratorPool:
* As Jordan suggests, storing the pool inside the \{{MergeIterator}} makes
sense - more so because it can avoid incurring the cost of
\{{ThreadLocal.get()}} on \{{returnToPool}}, by stashing a link to an inner
object (that guards put on the identity of the \{{Thread}} owning the queue)
* Perhaps an \{{ArrayList}} is a better idea for collection type, as it will
be generally garbage-free, and occupy less space than a \{{LinkedList}}? An
\{{ArrayDeque}} could maintain the current behaviour, or you could pop from the
back
* Perhaps the queue would be better with fixed size, permitting at most two or
three items to be stored?
* As it happens, I’ve been working on very similar parts of the codebase, and
I have implemented my own \{{TinyThreadLocalPool}} in the patches I hope to
land soon, that does the above but has lower fixed-overhead and less
indirection by permitting at most 3 items, stored in in-line properties (i.e.
\{{val0}}, \{{val1}} and \{{val2}}). It might be worthwhile introducing that
class here for consideration/comparison, and perhaps some amalgamation of the
two used in whichever patch lands first (which I fully expect to be this one)?
{\{MergeIterator.OneToOne}}:
* While we’re here optimising, it probably makes sense to merge \{{OneToOne}}
and \{{TrivialOneToOne}}, simply permitting the reducer in \{{OneToOne}} to be
\{{null}}, and to switch behaviour based on a null check. This should permit
call-sites to become bimorphic that are presently megamorphic, and the branch
should be near perfectly predicted.
{\{MergeIterator.ManyToOne}}:
* Instead of using a separate \{{candidates}} array for reusing a
\{{Candidate}}, could we avoid setting the last heap element to null by instead
placing the now-defunct item there?
* Do we need \{{active}} when it is implied by \{{MergeIterator.iterators !=
null}} and \{{Candidate.iter != null}}?
* Do we need to set primitives to \{{-1}} or \{{false}} on \{{close}}, giv
[jira] [Commented] (CASSANDRA-15392) Pool Merge Iterators
[
https://issues.apache.org/jira/browse/CASSANDRA-15392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17010853#comment-17010853
]
Benedict Elliott Smith commented on CASSANDRA-15392:
MergeIteratorPool:
* As Jordan suggests, storing the pool inside the \{{MergeIterator}} makes
sense - more so because it can avoid incurring the cost of
\{{ThreadLocal.get()}} on \{{returnToPool}}, by stashing a link to an inner
object (that guards put on the identity of the \{{Thread}} owning the queue)
* Perhaps an \{{ArrayList}} is a better idea for collection type, as it will
be generally garbage-free, and occupy less space than a \{{LinkedList}}? An
\{{ArrayDeque}} could maintain the current behaviour, or you could pop from the
back
* Perhaps the queue would be better with fixed size, permitting at most two or
three items to be stored?
* As it happens, I’ve been working on very similar parts of the codebase, and
I have implemented my own \{{TinyThreadLocalPool}} in the patches I hope to
land soon, that does the above but has lower fixed-overhead and less
indirection by permitting at most 3 items, stored in in-line properties (i.e.
\{{val0}}, \{{val1}} and \{{val2}}). It might be worthwhile introducing that
class here for consideration/comparison, and perhaps some amalgamation of the
two used in whichever patch lands first (which I fully expect to be this one)?
{\{MergeIterator.OneToOne}}:
* While we’re here optimising, it probably makes sense to merge \{{OneToOne}}
and \{{TrivialOneToOne}}, simply permitting the reducer in \{{OneToOne}} to be
\{{null}}, and to switch behaviour based on a null check. This should permit
call-sites to become bimorphic that are presently megamorphic, and the branch
should be near perfectly predicted.
{\{MergeIterator.ManyToOne}}:
* Instead of using a separate \{{candidates}} array for reusing a
\{{Candidate}}, could we avoid setting the last heap element to null by instead
placing the now-defunct item there?
* Do we need \{{active}} when it is implied by \{{MergeIterator.iterators !=
null}} and \{{Candidate.iter != null}}?
* Do we need to set primitives to \{{-1}} or \{{false}} on \{{close}}, given
we will throw \{{NullPointerException}} if we haven’t invoked \{{reset}}?
Benchmarking:
* It’s annoying work, but alongside the garbage benchmarks, it would be nice
to see some simple JMH throughput benchmarks introduced, to see what the
overall impact of the change is. I don’t mind helping out here.
nits:
* {\{ManyToOne.DEFAULT_SIZE}} -> \{{ManyToOne.POOLED_MERGE_LIMIT}}?
* +1 Jordan’s nit about naming of \{{getSimple}}
* {\{MergeIteratorPool.queues.initialValue}} should not declare \{{throw
Exception}}
* {\{MergeIteratorPool.initialValue}} looks to be a duplicate
* {\{TokenTree:380}} errant new line
> Pool Merge Iterators
>
>
> Key: CASSANDRA-15392
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15392
> Project: Cassandra
> Issue Type: Sub-task
> Components: Local/Compaction
>Reporter: Blake Eggleston
>Assignee: Blake Eggleston
>Priority: Normal
> Fix For: 4.0
>
>
> By pooling merge iterators, instead of creating new ones each time we need
> them, we can reduce garbage on the compaction and read paths under relevant
> workloads by ~4% in many cases.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14740) BlockingReadRepair does not maintain monotonicity during range movements
[ https://issues.apache.org/jira/browse/CASSANDRA-14740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benedict Elliott Smith updated CASSANDRA-14740: --- Fix Version/s: 4.0 > BlockingReadRepair does not maintain monotonicity during range movements > > > Key: CASSANDRA-14740 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14740 > Project: Cassandra > Issue Type: Bug > Components: Legacy/Coordination >Reporter: Benedict Elliott Smith >Assignee: Benedict Elliott Smith >Priority: Urgent > Labels: correctness > Fix For: 4.0, 4.0-beta > > > The BlockingReadRepair code introduced by CASSANDRA-10726 requires that each > of the queried nodes are written to, but pending nodes are not considered. > If there is a pending range movement, one of these writes may be ‘lost’ when > the range movement completes. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14740) BlockingReadRepair does not maintain monotonicity during range movements
[ https://issues.apache.org/jira/browse/CASSANDRA-14740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benedict Elliott Smith updated CASSANDRA-14740: --- Fix Version/s: (was: 4.0) 4.0-beta > BlockingReadRepair does not maintain monotonicity during range movements > > > Key: CASSANDRA-14740 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14740 > Project: Cassandra > Issue Type: Bug > Components: Legacy/Coordination >Reporter: Benedict Elliott Smith >Assignee: Benedict Elliott Smith >Priority: Urgent > Labels: correctness > Fix For: 4.0-beta > > > The BlockingReadRepair code introduced by CASSANDRA-10726 requires that each > of the queried nodes are written to, but pending nodes are not considered. > If there is a pending range movement, one of these writes may be ‘lost’ when > the range movement completes. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-13917) COMPACT STORAGE queries on dense static tables accept hidden column1 and value columns
[
https://issues.apache.org/jira/browse/CASSANDRA-13917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alex Petrov updated CASSANDRA-13917:
Since Version: 3.0.19
Source Control Link:
https://github.com/apache/cassandra/commit/0c54cc98595b4879c9a634737674fd36fd1c46d0
Resolution: Fixed
Status: Resolved (was: Ready to Commit)
> COMPACT STORAGE queries on dense static tables accept hidden column1 and
> value columns
> --
>
> Key: CASSANDRA-13917
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13917
> Project: Cassandra
> Issue Type: Bug
> Components: Legacy/Core
>Reporter: Alex Petrov
>Assignee: Aleksandr Sorokoumov
>Priority: Low
> Labels: lhf
> Fix For: 3.0.x, 3.11.x
>
> Attachments: 13917-3.0-testall-13.12.2019, 13917-3.0-testall-2.png,
> 13917-3.0-testall-20.11.2019.png, 13917-3.0.png,
> 13917-3.11-testall-13.12.2019, 13917-3.11-testall-2.png,
> 13917-3.11-testall-20.11.2019.png, 13917-3.11.png
>
>
> Test for the issue:
> {code}
> @Test
> public void testCompactStorage() throws Throwable
> {
> createTable("CREATE TABLE %s (a int PRIMARY KEY, b int, c int) WITH
> COMPACT STORAGE");
> assertInvalid("INSERT INTO %s (a, b, c, column1) VALUES (?, ?, ?,
> ?)", 1, 1, 1, ByteBufferUtil.bytes('a'));
> // This one fails with Some clustering keys are missing: column1,
> which is still wrong
> assertInvalid("INSERT INTO %s (a, b, c, value) VALUES (?, ?, ?, ?)",
> 1, 1, 1, ByteBufferUtil.bytes('a'));
> assertInvalid("INSERT INTO %s (a, b, c, column1, value) VALUES (?, ?,
> ?, ?, ?)", 1, 1, 1, ByteBufferUtil.bytes('a'), ByteBufferUtil.bytes('b'));
> assertEmpty(execute("SELECT * FROM %s"));
> }
> {code}
> Gladly, these writes are no-op, even though they succeed.
> {{value}} and {{column1}} should be completely hidden. Fixing this one should
> be as easy as just adding validations.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-13917) COMPACT STORAGE queries on dense static tables accept hidden column1 and value columns
[
https://issues.apache.org/jira/browse/CASSANDRA-13917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alex Petrov updated CASSANDRA-13917:
Status: Patch Available (was: Review In Progress)
> COMPACT STORAGE queries on dense static tables accept hidden column1 and
> value columns
> --
>
> Key: CASSANDRA-13917
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13917
> Project: Cassandra
> Issue Type: Bug
> Components: Legacy/Core
>Reporter: Alex Petrov
>Assignee: Aleksandr Sorokoumov
>Priority: Low
> Labels: lhf
> Fix For: 3.0.x, 3.11.x
>
> Attachments: 13917-3.0-testall-13.12.2019, 13917-3.0-testall-2.png,
> 13917-3.0-testall-20.11.2019.png, 13917-3.0.png,
> 13917-3.11-testall-13.12.2019, 13917-3.11-testall-2.png,
> 13917-3.11-testall-20.11.2019.png, 13917-3.11.png
>
>
> Test for the issue:
> {code}
> @Test
> public void testCompactStorage() throws Throwable
> {
> createTable("CREATE TABLE %s (a int PRIMARY KEY, b int, c int) WITH
> COMPACT STORAGE");
> assertInvalid("INSERT INTO %s (a, b, c, column1) VALUES (?, ?, ?,
> ?)", 1, 1, 1, ByteBufferUtil.bytes('a'));
> // This one fails with Some clustering keys are missing: column1,
> which is still wrong
> assertInvalid("INSERT INTO %s (a, b, c, value) VALUES (?, ?, ?, ?)",
> 1, 1, 1, ByteBufferUtil.bytes('a'));
> assertInvalid("INSERT INTO %s (a, b, c, column1, value) VALUES (?, ?,
> ?, ?, ?)", 1, 1, 1, ByteBufferUtil.bytes('a'), ByteBufferUtil.bytes('b'));
> assertEmpty(execute("SELECT * FROM %s"));
> }
> {code}
> Gladly, these writes are no-op, even though they succeed.
> {{value}} and {{column1}} should be completely hidden. Fixing this one should
> be as easy as just adding validations.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-13917) COMPACT STORAGE queries on dense static tables accept hidden column1 and value columns
[
https://issues.apache.org/jira/browse/CASSANDRA-13917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alex Petrov updated CASSANDRA-13917:
Reviewers: Alex Petrov, Alex Petrov (was: Alex Petrov)
Alex Petrov, Alex Petrov (was: Alex Petrov)
Status: Review In Progress (was: Patch Available)
> COMPACT STORAGE queries on dense static tables accept hidden column1 and
> value columns
> --
>
> Key: CASSANDRA-13917
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13917
> Project: Cassandra
> Issue Type: Bug
> Components: Legacy/Core
>Reporter: Alex Petrov
>Assignee: Aleksandr Sorokoumov
>Priority: Low
> Labels: lhf
> Fix For: 3.0.x, 3.11.x
>
> Attachments: 13917-3.0-testall-13.12.2019, 13917-3.0-testall-2.png,
> 13917-3.0-testall-20.11.2019.png, 13917-3.0.png,
> 13917-3.11-testall-13.12.2019, 13917-3.11-testall-2.png,
> 13917-3.11-testall-20.11.2019.png, 13917-3.11.png
>
>
> Test for the issue:
> {code}
> @Test
> public void testCompactStorage() throws Throwable
> {
> createTable("CREATE TABLE %s (a int PRIMARY KEY, b int, c int) WITH
> COMPACT STORAGE");
> assertInvalid("INSERT INTO %s (a, b, c, column1) VALUES (?, ?, ?,
> ?)", 1, 1, 1, ByteBufferUtil.bytes('a'));
> // This one fails with Some clustering keys are missing: column1,
> which is still wrong
> assertInvalid("INSERT INTO %s (a, b, c, value) VALUES (?, ?, ?, ?)",
> 1, 1, 1, ByteBufferUtil.bytes('a'));
> assertInvalid("INSERT INTO %s (a, b, c, column1, value) VALUES (?, ?,
> ?, ?, ?)", 1, 1, 1, ByteBufferUtil.bytes('a'), ByteBufferUtil.bytes('b'));
> assertEmpty(execute("SELECT * FROM %s"));
> }
> {code}
> Gladly, these writes are no-op, even though they succeed.
> {{value}} and {{column1}} should be completely hidden. Fixing this one should
> be as easy as just adding validations.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-13917) COMPACT STORAGE queries on dense static tables accept hidden column1 and value columns
[
https://issues.apache.org/jira/browse/CASSANDRA-13917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alex Petrov updated CASSANDRA-13917:
Status: Ready to Commit (was: Review In Progress)
> COMPACT STORAGE queries on dense static tables accept hidden column1 and
> value columns
> --
>
> Key: CASSANDRA-13917
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13917
> Project: Cassandra
> Issue Type: Bug
> Components: Legacy/Core
>Reporter: Alex Petrov
>Assignee: Aleksandr Sorokoumov
>Priority: Low
> Labels: lhf
> Fix For: 3.0.x, 3.11.x
>
> Attachments: 13917-3.0-testall-13.12.2019, 13917-3.0-testall-2.png,
> 13917-3.0-testall-20.11.2019.png, 13917-3.0.png,
> 13917-3.11-testall-13.12.2019, 13917-3.11-testall-2.png,
> 13917-3.11-testall-20.11.2019.png, 13917-3.11.png
>
>
> Test for the issue:
> {code}
> @Test
> public void testCompactStorage() throws Throwable
> {
> createTable("CREATE TABLE %s (a int PRIMARY KEY, b int, c int) WITH
> COMPACT STORAGE");
> assertInvalid("INSERT INTO %s (a, b, c, column1) VALUES (?, ?, ?,
> ?)", 1, 1, 1, ByteBufferUtil.bytes('a'));
> // This one fails with Some clustering keys are missing: column1,
> which is still wrong
> assertInvalid("INSERT INTO %s (a, b, c, value) VALUES (?, ?, ?, ?)",
> 1, 1, 1, ByteBufferUtil.bytes('a'));
> assertInvalid("INSERT INTO %s (a, b, c, column1, value) VALUES (?, ?,
> ?, ?, ?)", 1, 1, 1, ByteBufferUtil.bytes('a'), ByteBufferUtil.bytes('b'));
> assertEmpty(execute("SELECT * FROM %s"));
> }
> {code}
> Gladly, these writes are no-op, even though they succeed.
> {{value}} and {{column1}} should be completely hidden. Fixing this one should
> be as easy as just adding validations.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-13917) COMPACT STORAGE queries on dense static tables accept hidden column1 and value columns
[
https://issues.apache.org/jira/browse/CASSANDRA-13917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alex Petrov updated CASSANDRA-13917:
Status: Review In Progress (was: Changes Suggested)
> COMPACT STORAGE queries on dense static tables accept hidden column1 and
> value columns
> --
>
> Key: CASSANDRA-13917
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13917
> Project: Cassandra
> Issue Type: Bug
> Components: Legacy/Core
>Reporter: Alex Petrov
>Assignee: Aleksandr Sorokoumov
>Priority: Low
> Labels: lhf
> Fix For: 3.0.x, 3.11.x
>
> Attachments: 13917-3.0-testall-13.12.2019, 13917-3.0-testall-2.png,
> 13917-3.0-testall-20.11.2019.png, 13917-3.0.png,
> 13917-3.11-testall-13.12.2019, 13917-3.11-testall-2.png,
> 13917-3.11-testall-20.11.2019.png, 13917-3.11.png
>
>
> Test for the issue:
> {code}
> @Test
> public void testCompactStorage() throws Throwable
> {
> createTable("CREATE TABLE %s (a int PRIMARY KEY, b int, c int) WITH
> COMPACT STORAGE");
> assertInvalid("INSERT INTO %s (a, b, c, column1) VALUES (?, ?, ?,
> ?)", 1, 1, 1, ByteBufferUtil.bytes('a'));
> // This one fails with Some clustering keys are missing: column1,
> which is still wrong
> assertInvalid("INSERT INTO %s (a, b, c, value) VALUES (?, ?, ?, ?)",
> 1, 1, 1, ByteBufferUtil.bytes('a'));
> assertInvalid("INSERT INTO %s (a, b, c, column1, value) VALUES (?, ?,
> ?, ?, ?)", 1, 1, 1, ByteBufferUtil.bytes('a'), ByteBufferUtil.bytes('b'));
> assertEmpty(execute("SELECT * FROM %s"));
> }
> {code}
> Gladly, these writes are no-op, even though they succeed.
> {{value}} and {{column1}} should be completely hidden. Fixing this one should
> be as easy as just adding validations.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15468) Unable to start embedded cassandra on JDK 11
[
https://issues.apache.org/jira/browse/CASSANDRA-15468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sam Tunnicliffe updated CASSANDRA-15468:
Test and Documentation Plan: No additional testing required
Status: Patch Available (was: Open)
> Unable to start embedded cassandra on JDK 11
>
>
> Key: CASSANDRA-15468
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15468
> Project: Cassandra
> Issue Type: Bug
>Reporter: Gus Heck
>Assignee: Gus Heck
>Priority: Normal
> Attachments: CASSANDRA-15468.patch
>
>
> Cassandra fails to start without the following jvm options from
> jvm-11-server.options:
> {code:java}
> -Djdk.attach.allowAttachSelf=true
> --add-exports java.base/jdk.internal.misc=ALL-UNNAMED
> --add-exports java.base/jdk.internal.ref=ALL-UNNAMED
> --add-exports java.base/sun.nio.ch=ALL-UNNAMED
> --add-exports java.management.rmi/com.sun.jmx.remote.internal.rmi=ALL-UNNAMED
> --add-exports java.rmi/sun.rmi.registry=ALL-UNNAMED
> --add-exports java.rmi/sun.rmi.server=ALL-UNNAMED
> --add-exports java.sql/java.sql=ALL-UNNAMED
> --add-opens java.base/java.lang.module=ALL-UNNAMED
> --add-opens java.base/jdk.internal.loader=ALL-UNNAMED
> --add-opens java.base/jdk.internal.ref=ALL-UNNAMED
> --add-opens java.base/jdk.internal.reflect=ALL-UNNAMED
> --add-opens java.base/jdk.internal.math=ALL-UNNAMED
> --add-opens java.base/jdk.internal.module=ALL-UNNAMED
> --add-opens java.base/jdk.internal.util.jar=ALL-UNNAMED
> --add-opens jdk.management/com.sun.management.internal=ALL-UNNAMED
> {code}
> Without these options the following stack trace appears and prevents startup
> {code:java}
> 2020-01-05T17:42:15,009 ERROR main
> org.apache.cassandra.service.CassandraDaemon Exception encountered during
> startup java.lang.ExceptionInInitializerError
> at
> org.apache.cassandra.config.DatabaseDescriptor.guessFileStore(DatabaseDescriptor.java:1127)
> at
> org.apache.cassandra.config.DatabaseDescriptor.applySimpleConfig(DatabaseDescriptor.java:541)
> at
> org.apache.cassandra.config.DatabaseDescriptor.applyAll(DatabaseDescriptor.java:340)
> at
> org.apache.cassandra.config.DatabaseDescriptor.daemonInitialization(DatabaseDescriptor.java:168)
> at
> org.apache.cassandra.config.DatabaseDescriptor.daemonInitialization(DatabaseDescriptor.java:152)
> at
> org.apache.cassandra.service.CassandraDaemon.applyConfig(CassandraDaemon.java:665)
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:609)
> at org.jesterj.ingest.persistence.Cassandra.start(Cassandra.java:135)
> at
> org.jesterj.ingest.scanners.SimpleFileScannerImplFTITest.testScanWithMemory(SimpleFileScannerImplFTITest.java:63)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:157)
> at
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
> at
> com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
> at
> com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:
[jira] [Updated] (CASSANDRA-15468) Unable to start embedded cassandra on JDK 11
[
https://issues.apache.org/jira/browse/CASSANDRA-15468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sam Tunnicliffe updated CASSANDRA-15468:
Reviewers: Sam Tunnicliffe (was: beobal, Sam Tunnicliffe)
> Unable to start embedded cassandra on JDK 11
>
>
> Key: CASSANDRA-15468
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15468
> Project: Cassandra
> Issue Type: Bug
>Reporter: Gus Heck
>Assignee: Gus Heck
>Priority: Normal
> Attachments: CASSANDRA-15468.patch
>
>
> Cassandra fails to start without the following jvm options from
> jvm-11-server.options:
> {code:java}
> -Djdk.attach.allowAttachSelf=true
> --add-exports java.base/jdk.internal.misc=ALL-UNNAMED
> --add-exports java.base/jdk.internal.ref=ALL-UNNAMED
> --add-exports java.base/sun.nio.ch=ALL-UNNAMED
> --add-exports java.management.rmi/com.sun.jmx.remote.internal.rmi=ALL-UNNAMED
> --add-exports java.rmi/sun.rmi.registry=ALL-UNNAMED
> --add-exports java.rmi/sun.rmi.server=ALL-UNNAMED
> --add-exports java.sql/java.sql=ALL-UNNAMED
> --add-opens java.base/java.lang.module=ALL-UNNAMED
> --add-opens java.base/jdk.internal.loader=ALL-UNNAMED
> --add-opens java.base/jdk.internal.ref=ALL-UNNAMED
> --add-opens java.base/jdk.internal.reflect=ALL-UNNAMED
> --add-opens java.base/jdk.internal.math=ALL-UNNAMED
> --add-opens java.base/jdk.internal.module=ALL-UNNAMED
> --add-opens java.base/jdk.internal.util.jar=ALL-UNNAMED
> --add-opens jdk.management/com.sun.management.internal=ALL-UNNAMED
> {code}
> Without these options the following stack trace appears and prevents startup
> {code:java}
> 2020-01-05T17:42:15,009 ERROR main
> org.apache.cassandra.service.CassandraDaemon Exception encountered during
> startup java.lang.ExceptionInInitializerError
> at
> org.apache.cassandra.config.DatabaseDescriptor.guessFileStore(DatabaseDescriptor.java:1127)
> at
> org.apache.cassandra.config.DatabaseDescriptor.applySimpleConfig(DatabaseDescriptor.java:541)
> at
> org.apache.cassandra.config.DatabaseDescriptor.applyAll(DatabaseDescriptor.java:340)
> at
> org.apache.cassandra.config.DatabaseDescriptor.daemonInitialization(DatabaseDescriptor.java:168)
> at
> org.apache.cassandra.config.DatabaseDescriptor.daemonInitialization(DatabaseDescriptor.java:152)
> at
> org.apache.cassandra.service.CassandraDaemon.applyConfig(CassandraDaemon.java:665)
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:609)
> at org.jesterj.ingest.persistence.Cassandra.start(Cassandra.java:135)
> at
> org.jesterj.ingest.scanners.SimpleFileScannerImplFTITest.testScanWithMemory(SimpleFileScannerImplFTITest.java:63)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:157)
> at
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
> at
> com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
> at
> com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
> at
> com.intellij.rt.execution.junit.JUnitStarter
[jira] [Updated] (CASSANDRA-15468) Unable to start embedded cassandra on JDK 11
[
https://issues.apache.org/jira/browse/CASSANDRA-15468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sam Tunnicliffe updated CASSANDRA-15468:
Reviewers: beobal, Sam Tunnicliffe
Status: Review In Progress (was: Patch Available)
> Unable to start embedded cassandra on JDK 11
>
>
> Key: CASSANDRA-15468
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15468
> Project: Cassandra
> Issue Type: Bug
>Reporter: Gus Heck
>Assignee: Gus Heck
>Priority: Normal
> Attachments: CASSANDRA-15468.patch
>
>
> Cassandra fails to start without the following jvm options from
> jvm-11-server.options:
> {code:java}
> -Djdk.attach.allowAttachSelf=true
> --add-exports java.base/jdk.internal.misc=ALL-UNNAMED
> --add-exports java.base/jdk.internal.ref=ALL-UNNAMED
> --add-exports java.base/sun.nio.ch=ALL-UNNAMED
> --add-exports java.management.rmi/com.sun.jmx.remote.internal.rmi=ALL-UNNAMED
> --add-exports java.rmi/sun.rmi.registry=ALL-UNNAMED
> --add-exports java.rmi/sun.rmi.server=ALL-UNNAMED
> --add-exports java.sql/java.sql=ALL-UNNAMED
> --add-opens java.base/java.lang.module=ALL-UNNAMED
> --add-opens java.base/jdk.internal.loader=ALL-UNNAMED
> --add-opens java.base/jdk.internal.ref=ALL-UNNAMED
> --add-opens java.base/jdk.internal.reflect=ALL-UNNAMED
> --add-opens java.base/jdk.internal.math=ALL-UNNAMED
> --add-opens java.base/jdk.internal.module=ALL-UNNAMED
> --add-opens java.base/jdk.internal.util.jar=ALL-UNNAMED
> --add-opens jdk.management/com.sun.management.internal=ALL-UNNAMED
> {code}
> Without these options the following stack trace appears and prevents startup
> {code:java}
> 2020-01-05T17:42:15,009 ERROR main
> org.apache.cassandra.service.CassandraDaemon Exception encountered during
> startup java.lang.ExceptionInInitializerError
> at
> org.apache.cassandra.config.DatabaseDescriptor.guessFileStore(DatabaseDescriptor.java:1127)
> at
> org.apache.cassandra.config.DatabaseDescriptor.applySimpleConfig(DatabaseDescriptor.java:541)
> at
> org.apache.cassandra.config.DatabaseDescriptor.applyAll(DatabaseDescriptor.java:340)
> at
> org.apache.cassandra.config.DatabaseDescriptor.daemonInitialization(DatabaseDescriptor.java:168)
> at
> org.apache.cassandra.config.DatabaseDescriptor.daemonInitialization(DatabaseDescriptor.java:152)
> at
> org.apache.cassandra.service.CassandraDaemon.applyConfig(CassandraDaemon.java:665)
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:609)
> at org.jesterj.ingest.persistence.Cassandra.start(Cassandra.java:135)
> at
> org.jesterj.ingest.scanners.SimpleFileScannerImplFTITest.testScanWithMemory(SimpleFileScannerImplFTITest.java:63)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:157)
> at
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
> at
> com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
> at
> com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
> at
> com.intell
[jira] [Updated] (CASSANDRA-15433) Pending ranges are not recalculated on keyspace creation
[
https://issues.apache.org/jira/browse/CASSANDRA-15433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alex Petrov updated CASSANDRA-15433:
Reviewers: Alex Petrov
> Pending ranges are not recalculated on keyspace creation
>
>
> Key: CASSANDRA-15433
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15433
> Project: Cassandra
> Issue Type: Bug
> Components: Cluster/Membership
>Reporter: Josh Snyder
>Assignee: Sumanth Pasupuleti
>Priority: Normal
>
> When a node begins bootstrapping, Cassandra recalculates pending tokens for
> each keyspace that exists when the state change is observed (in
> StorageService:handleState*). When new keyspaces are created, we do not
> recalculate pending ranges (around Schema:merge). As a result, writes for new
> keyspaces are not received by nodes in BOOT or BOOT_REPLACE modes. When
> bootstrapping finishes, the node which just bootstrapped will not have data
> for the newly created keyspace.
> Consider a ring with bootstrapped nodes A, B, and C. Node D is pending, and
> when it finishes bootstrapping, C will cede ownership of some ranges to D. A
> quorum write is acknowledged by C and A. B missed the write, and the
> coordinator didn't send it to D at all. When D finishes bootstrapping, the
> quorum B+D will not contain the mutation.
> Steps to reproduce:
> # Join a node in BOOT mode
> # Create a keyspace
> # Send writes to that keyspace
> # On the joining node, observe that {{nodetool cfstats}} records zero writes
> to the new keyspace
> I have observed this directly in Cassandra 3.0, and based on my reading the
> code, I believe it affects up through trunk.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Assigned] (CASSANDRA-15468) Unable to start embedded cassandra on JDK 11
[
https://issues.apache.org/jira/browse/CASSANDRA-15468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sam Tunnicliffe reassigned CASSANDRA-15468:
---
Assignee: Gus Heck
> Unable to start embedded cassandra on JDK 11
>
>
> Key: CASSANDRA-15468
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15468
> Project: Cassandra
> Issue Type: Bug
>Reporter: Gus Heck
>Assignee: Gus Heck
>Priority: Normal
> Attachments: CASSANDRA-15468.patch
>
>
> Cassandra fails to start without the following jvm options from
> jvm-11-server.options:
> {code:java}
> -Djdk.attach.allowAttachSelf=true
> --add-exports java.base/jdk.internal.misc=ALL-UNNAMED
> --add-exports java.base/jdk.internal.ref=ALL-UNNAMED
> --add-exports java.base/sun.nio.ch=ALL-UNNAMED
> --add-exports java.management.rmi/com.sun.jmx.remote.internal.rmi=ALL-UNNAMED
> --add-exports java.rmi/sun.rmi.registry=ALL-UNNAMED
> --add-exports java.rmi/sun.rmi.server=ALL-UNNAMED
> --add-exports java.sql/java.sql=ALL-UNNAMED
> --add-opens java.base/java.lang.module=ALL-UNNAMED
> --add-opens java.base/jdk.internal.loader=ALL-UNNAMED
> --add-opens java.base/jdk.internal.ref=ALL-UNNAMED
> --add-opens java.base/jdk.internal.reflect=ALL-UNNAMED
> --add-opens java.base/jdk.internal.math=ALL-UNNAMED
> --add-opens java.base/jdk.internal.module=ALL-UNNAMED
> --add-opens java.base/jdk.internal.util.jar=ALL-UNNAMED
> --add-opens jdk.management/com.sun.management.internal=ALL-UNNAMED
> {code}
> Without these options the following stack trace appears and prevents startup
> {code:java}
> 2020-01-05T17:42:15,009 ERROR main
> org.apache.cassandra.service.CassandraDaemon Exception encountered during
> startup java.lang.ExceptionInInitializerError
> at
> org.apache.cassandra.config.DatabaseDescriptor.guessFileStore(DatabaseDescriptor.java:1127)
> at
> org.apache.cassandra.config.DatabaseDescriptor.applySimpleConfig(DatabaseDescriptor.java:541)
> at
> org.apache.cassandra.config.DatabaseDescriptor.applyAll(DatabaseDescriptor.java:340)
> at
> org.apache.cassandra.config.DatabaseDescriptor.daemonInitialization(DatabaseDescriptor.java:168)
> at
> org.apache.cassandra.config.DatabaseDescriptor.daemonInitialization(DatabaseDescriptor.java:152)
> at
> org.apache.cassandra.service.CassandraDaemon.applyConfig(CassandraDaemon.java:665)
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:609)
> at org.jesterj.ingest.persistence.Cassandra.start(Cassandra.java:135)
> at
> org.jesterj.ingest.scanners.SimpleFileScannerImplFTITest.testScanWithMemory(SimpleFileScannerImplFTITest.java:63)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:157)
> at
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
> at
> com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
> at
> com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
> at
> com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
> Cau
[jira] [Updated] (CASSANDRA-15468) Unable to start embedded cassandra on JDK 11
[
https://issues.apache.org/jira/browse/CASSANDRA-15468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sam Tunnicliffe updated CASSANDRA-15468:
Status: Open (was: Resolved)
> Unable to start embedded cassandra on JDK 11
>
>
> Key: CASSANDRA-15468
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15468
> Project: Cassandra
> Issue Type: Bug
>Reporter: Gus Heck
>Priority: Normal
> Attachments: CASSANDRA-15468.patch
>
>
> Cassandra fails to start without the following jvm options from
> jvm-11-server.options:
> {code:java}
> -Djdk.attach.allowAttachSelf=true
> --add-exports java.base/jdk.internal.misc=ALL-UNNAMED
> --add-exports java.base/jdk.internal.ref=ALL-UNNAMED
> --add-exports java.base/sun.nio.ch=ALL-UNNAMED
> --add-exports java.management.rmi/com.sun.jmx.remote.internal.rmi=ALL-UNNAMED
> --add-exports java.rmi/sun.rmi.registry=ALL-UNNAMED
> --add-exports java.rmi/sun.rmi.server=ALL-UNNAMED
> --add-exports java.sql/java.sql=ALL-UNNAMED
> --add-opens java.base/java.lang.module=ALL-UNNAMED
> --add-opens java.base/jdk.internal.loader=ALL-UNNAMED
> --add-opens java.base/jdk.internal.ref=ALL-UNNAMED
> --add-opens java.base/jdk.internal.reflect=ALL-UNNAMED
> --add-opens java.base/jdk.internal.math=ALL-UNNAMED
> --add-opens java.base/jdk.internal.module=ALL-UNNAMED
> --add-opens java.base/jdk.internal.util.jar=ALL-UNNAMED
> --add-opens jdk.management/com.sun.management.internal=ALL-UNNAMED
> {code}
> Without these options the following stack trace appears and prevents startup
> {code:java}
> 2020-01-05T17:42:15,009 ERROR main
> org.apache.cassandra.service.CassandraDaemon Exception encountered during
> startup java.lang.ExceptionInInitializerError
> at
> org.apache.cassandra.config.DatabaseDescriptor.guessFileStore(DatabaseDescriptor.java:1127)
> at
> org.apache.cassandra.config.DatabaseDescriptor.applySimpleConfig(DatabaseDescriptor.java:541)
> at
> org.apache.cassandra.config.DatabaseDescriptor.applyAll(DatabaseDescriptor.java:340)
> at
> org.apache.cassandra.config.DatabaseDescriptor.daemonInitialization(DatabaseDescriptor.java:168)
> at
> org.apache.cassandra.config.DatabaseDescriptor.daemonInitialization(DatabaseDescriptor.java:152)
> at
> org.apache.cassandra.service.CassandraDaemon.applyConfig(CassandraDaemon.java:665)
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:609)
> at org.jesterj.ingest.persistence.Cassandra.start(Cassandra.java:135)
> at
> org.jesterj.ingest.scanners.SimpleFileScannerImplFTITest.testScanWithMemory(SimpleFileScannerImplFTITest.java:63)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:157)
> at
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
> at
> com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
> at
> com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
> at
> com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
> Caused by: java.lang.RuntimeExc
[jira] [Commented] (CASSANDRA-13917) COMPACT STORAGE queries on dense static tables accept hidden column1 and value columns
[
https://issues.apache.org/jira/browse/CASSANDRA-13917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17010798#comment-17010798
]
Alex Petrov commented on CASSANDRA-13917:
-
Thank you for the patch! Committed to 3.0 with
[0c54cc98595b4879c9a634737674fd36fd1c46d0|https://github.com/apache/cassandra/commit/0c54cc98595b4879c9a634737674fd36fd1c46d0],
and merged up to
[3.11|https://github.com/apache/cassandra/commit/9635e55f73598c0f7aece56d5e603198ca464fcc]
and trunk (with {{-s ours}})
> COMPACT STORAGE queries on dense static tables accept hidden column1 and
> value columns
> --
>
> Key: CASSANDRA-13917
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13917
> Project: Cassandra
> Issue Type: Bug
> Components: Legacy/Core
>Reporter: Alex Petrov
>Assignee: Aleksandr Sorokoumov
>Priority: Low
> Labels: lhf
> Fix For: 3.0.x, 3.11.x
>
> Attachments: 13917-3.0-testall-13.12.2019, 13917-3.0-testall-2.png,
> 13917-3.0-testall-20.11.2019.png, 13917-3.0.png,
> 13917-3.11-testall-13.12.2019, 13917-3.11-testall-2.png,
> 13917-3.11-testall-20.11.2019.png, 13917-3.11.png
>
>
> Test for the issue:
> {code}
> @Test
> public void testCompactStorage() throws Throwable
> {
> createTable("CREATE TABLE %s (a int PRIMARY KEY, b int, c int) WITH
> COMPACT STORAGE");
> assertInvalid("INSERT INTO %s (a, b, c, column1) VALUES (?, ?, ?,
> ?)", 1, 1, 1, ByteBufferUtil.bytes('a'));
> // This one fails with Some clustering keys are missing: column1,
> which is still wrong
> assertInvalid("INSERT INTO %s (a, b, c, value) VALUES (?, ?, ?, ?)",
> 1, 1, 1, ByteBufferUtil.bytes('a'));
> assertInvalid("INSERT INTO %s (a, b, c, column1, value) VALUES (?, ?,
> ?, ?, ?)", 1, 1, 1, ByteBufferUtil.bytes('a'), ByteBufferUtil.bytes('b'));
> assertEmpty(execute("SELECT * FROM %s"));
> }
> {code}
> Gladly, these writes are no-op, even though they succeed.
> {{value}} and {{column1}} should be completely hidden. Fixing this one should
> be as easy as just adding validations.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] branch cassandra-3.11 updated (00316e9 -> 9635e55)
This is an automated email from the ASF dual-hosted git repository. ifesdjeen pushed a change to branch cassandra-3.11 in repository https://gitbox.apache.org/repos/asf/cassandra.git. from 00316e9 Merge branch 'cassandra-3.0' into cassandra-3.11 new 0c54cc9 Queries on hidden columns act as if these columns do not exist new 9635e55 Merge branch 'cassandra-3.0' into cassandra-3.11 The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../org/apache/cassandra/config/CFMetaData.java| 30 - .../cql3/validation/operations/AlterTest.java | 27 .../cql3/validation/operations/CreateTest.java | 13 .../cql3/validation/operations/DeleteTest.java | 64 ++ .../cql3/validation/operations/InsertTest.java | 68 +++ .../cql3/validation/operations/SelectTest.java | 76 ++ .../cql3/validation/operations/UpdateTest.java | 57 .../cassandra/db/PartitionRangeReadTest.java | 6 -- 8 files changed, 334 insertions(+), 7 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] branch trunk updated (cfe2081 -> 63b3de9)
This is an automated email from the ASF dual-hosted git repository. ifesdjeen pushed a change to branch trunk in repository https://gitbox.apache.org/repos/asf/cassandra.git. from cfe2081 Merge branch 'cassandra-3.11' into trunk new 0c54cc9 Queries on hidden columns act as if these columns do not exist new 9635e55 Merge branch 'cassandra-3.0' into cassandra-3.11 new 63b3de9 Merge branch 'cassandra-3.11' into trunk The 3 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] 01/01: Merge branch 'cassandra-3.0' into cassandra-3.11
This is an automated email from the ASF dual-hosted git repository.
ifesdjeen pushed a commit to branch cassandra-3.11
in repository https://gitbox.apache.org/repos/asf/cassandra.git
commit 9635e55f73598c0f7aece56d5e603198ca464fcc
Merge: 00316e9 0c54cc9
Author: Alex Petrov
AuthorDate: Wed Jan 8 17:02:15 2020 +0100
Merge branch 'cassandra-3.0' into cassandra-3.11
.../org/apache/cassandra/config/CFMetaData.java| 30 -
.../cql3/validation/operations/AlterTest.java | 27
.../cql3/validation/operations/CreateTest.java | 13
.../cql3/validation/operations/DeleteTest.java | 64 ++
.../cql3/validation/operations/InsertTest.java | 68 +++
.../cql3/validation/operations/SelectTest.java | 76 ++
.../cql3/validation/operations/UpdateTest.java | 57
.../cassandra/db/PartitionRangeReadTest.java | 6 --
8 files changed, 334 insertions(+), 7 deletions(-)
diff --cc src/java/org/apache/cassandra/config/CFMetaData.java
index 632c530,5888f42..992357d
--- a/src/java/org/apache/cassandra/config/CFMetaData.java
+++ b/src/java/org/apache/cassandra/config/CFMetaData.java
@@@ -125,11 -120,8 +125,13 @@@ public final class CFMetaDat
// for those tables in practice).
private volatile ColumnDefinition compactValueColumn;
+ private volatile Set hiddenColumns;
+
+public final DataResource resource;
+
+//For hot path serialization it's often easier to store this info here
+private volatile ColumnFilter allColumnFilter;
+
/**
* These two columns are "virtual" (e.g. not persisted together with
schema).
*
@@@ -398,7 -386,23 +400,25 @@@
else
this.comparator = new
ClusteringComparator(extractTypes(clusteringColumns));
+ Set hiddenColumns;
+ if (isCompactTable() && isDense &&
CompactTables.hasEmptyCompactValue(this))
+ {
+ hiddenColumns = Collections.singleton(compactValueColumn);
+ }
+ else if (isCompactTable() && !isDense && !isSuper)
+ {
+ hiddenColumns =
Sets.newHashSetWithExpectedSize(clusteringColumns.size() + 1);
+ hiddenColumns.add(compactValueColumn);
+ hiddenColumns.addAll(clusteringColumns);
+
+ }
+ else
+ {
+ hiddenColumns = Collections.emptySet();
+ }
+ this.hiddenColumns = hiddenColumns;
++
+this.allColumnFilter = ColumnFilter.all(this);
}
public Indexes getIndexes()
diff --cc
test/unit/org/apache/cassandra/cql3/validation/operations/CreateTest.java
index 66078c5,1f436b9..edb6668
--- a/test/unit/org/apache/cassandra/cql3/validation/operations/CreateTest.java
+++ b/test/unit/org/apache/cassandra/cql3/validation/operations/CreateTest.java
@@@ -669,6 -494,19 +669,19 @@@ public class CreateTest extends CQLTest
assertRows(execute("SELECT * FROM %s WHERE b = ?", 4), row(2, 4));
}
+ /**
+ * Test for CASSANDRA-13917
+ */
+ @Test
+ public void testCreateIndextWithCompactStaticFormat() throws Throwable
+ {
+ createTable("CREATE TABLE %s (a int PRIMARY KEY, b int, c int) WITH
COMPACT STORAGE");
-assertInvalidMessage("No column definition found for column column1",
++assertInvalidMessage("Undefined column name column1",
+ "CREATE INDEX column1_index on %s (column1)");
-assertInvalidMessage("No column definition found for column value",
++assertInvalidMessage("Undefined column name value",
+ "CREATE INDEX value_index on %s (value)");
+ }
+
@Test
// tests CASSANDRA-9565
public void testDoubleWith() throws Throwable
diff --cc
test/unit/org/apache/cassandra/cql3/validation/operations/DeleteTest.java
index ef1b5ac,9d495b3..66554da
--- a/test/unit/org/apache/cassandra/cql3/validation/operations/DeleteTest.java
+++ b/test/unit/org/apache/cassandra/cql3/validation/operations/DeleteTest.java
@@@ -1506,6 -1452,70 +1506,70 @@@ public class DeleteTest extends CQLTest
execute("DELETE FROM %s WHERE k = ? AND a >= ? AND a < ?", "a", 0, 2);
}
+ /**
+ * Test for CASSANDRA-13917
+ */
+ @Test
+ public void testWithCompactStaticFormat() throws Throwable
+ {
+ createTable("CREATE TABLE %s (a int PRIMARY KEY, b int, c int) WITH
COMPACT STORAGE");
+ testWithCompactFormat();
+
+ // if column1 is present, hidden column is called column2
+ createTable("CREATE TABLE %s (a int PRIMARY KEY, b int, c int,
column1 int) WITH COMPACT STORAGE");
-assertInvalidMessage("Undefined name column2 in where clause
('column2 = 1')",
++assertInvalidMessage("Undefined column name column2",
+ "DELETE FROM %s WHERE a = 1 AND column2= 1");
-assertInvalidMessage("Undefined name column2 in where clau
[cassandra] 01/01: Merge branch 'cassandra-3.11' into trunk
This is an automated email from the ASF dual-hosted git repository. ifesdjeen pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/cassandra.git commit 63b3de97142d3e32dc03e80a12f1edcb38abd5e0 Merge: cfe2081 9635e55 Author: Alex Petrov AuthorDate: Wed Jan 8 17:08:26 2020 +0100 Merge branch 'cassandra-3.11' into trunk - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] branch cassandra-3.0 updated: Queries on hidden columns act as if these columns do not exist
This is an automated email from the ASF dual-hosted git repository.
ifesdjeen pushed a commit to branch cassandra-3.0
in repository https://gitbox.apache.org/repos/asf/cassandra.git
The following commit(s) were added to refs/heads/cassandra-3.0 by this push:
new 0c54cc9 Queries on hidden columns act as if these columns do not exist
0c54cc9 is described below
commit 0c54cc98595b4879c9a634737674fd36fd1c46d0
Author: Aleksandr Sorokoumov
AuthorDate: Fri Oct 13 17:32:50 2017 +0200
Queries on hidden columns act as if these columns do not exist
Patch by Aleksandr Sorokoumov, reviewed by Alex Petrov for CASSANDRA-13917.
---
.../org/apache/cassandra/config/CFMetaData.java| 30 -
.../cql3/validation/operations/AlterTest.java | 27
.../cql3/validation/operations/CreateTest.java | 13
.../cql3/validation/operations/DeleteTest.java | 64 ++
.../cql3/validation/operations/InsertTest.java | 68 +++
.../cql3/validation/operations/SelectTest.java | 76 ++
.../cql3/validation/operations/UpdateTest.java | 57
.../cassandra/db/PartitionRangeReadTest.java | 6 --
8 files changed, 334 insertions(+), 7 deletions(-)
diff --git a/src/java/org/apache/cassandra/config/CFMetaData.java
b/src/java/org/apache/cassandra/config/CFMetaData.java
index ecb0fbf..5888f42 100644
--- a/src/java/org/apache/cassandra/config/CFMetaData.java
+++ b/src/java/org/apache/cassandra/config/CFMetaData.java
@@ -120,6 +120,8 @@ public final class CFMetaData
// for those tables in practice).
private volatile ColumnDefinition compactValueColumn;
+private volatile Set hiddenColumns;
+
/**
* These two columns are "virtual" (e.g. not persisted together with
schema).
*
@@ -383,6 +385,24 @@ public final class CFMetaData
this.comparator = new
ClusteringComparator(clusteringColumns.get(0).type);
else
this.comparator = new
ClusteringComparator(extractTypes(clusteringColumns));
+
+Set hiddenColumns;
+if (isCompactTable() && isDense &&
CompactTables.hasEmptyCompactValue(this))
+{
+hiddenColumns = Collections.singleton(compactValueColumn);
+}
+else if (isCompactTable() && !isDense && !isSuper)
+{
+hiddenColumns =
Sets.newHashSetWithExpectedSize(clusteringColumns.size() + 1);
+hiddenColumns.add(compactValueColumn);
+hiddenColumns.addAll(clusteringColumns);
+
+}
+else
+{
+hiddenColumns = Collections.emptySet();
+}
+this.hiddenColumns = hiddenColumns;
}
public Indexes getIndexes()
@@ -762,6 +782,11 @@ public final class CFMetaData
return compactValueColumn;
}
+private boolean isHiddenColumn(ColumnDefinition def)
+{
+return hiddenColumns.contains(def);
+}
+
public ClusteringComparator getKeyValidatorAsClusteringComparator()
{
boolean isCompound = keyValidator instanceof CompositeType;
@@ -963,7 +988,10 @@ public final class CFMetaData
// for instance) so...
public ColumnDefinition getColumnDefinition(ByteBuffer name)
{
-return columnMetadata.get(name);
+ColumnDefinition cd = columnMetadata.get(name);
+if (cd == null || isHiddenColumn(cd))
+return null;
+return cd;
}
public static boolean isNameValid(String name)
diff --git
a/test/unit/org/apache/cassandra/cql3/validation/operations/AlterTest.java
b/test/unit/org/apache/cassandra/cql3/validation/operations/AlterTest.java
index 1262258..da27dac 100644
--- a/test/unit/org/apache/cassandra/cql3/validation/operations/AlterTest.java
+++ b/test/unit/org/apache/cassandra/cql3/validation/operations/AlterTest.java
@@ -476,4 +476,31 @@ public class AlterTest extends CQLTester
assertEmpty(execute("SELECT * FROM %s"));
}
+
+/**
+ * Test for CASSANDRA-13917
+ */
+@Test
+public void testAlterWithCompactStaticFormat() throws Throwable
+{
+createTable("CREATE TABLE %s (a int PRIMARY KEY, b int, c int) WITH
COMPACT STORAGE");
+
+assertInvalidMessage("Cannot rename unknown column column1 in
keyspace",
+ "ALTER TABLE %s RENAME column1 TO column2");
+}
+
+/**
+ * Test for CASSANDRA-13917
+ */
+@Test
+public void testAlterWithCompactNonStaticFormat() throws Throwable
+{
+createTable("CREATE TABLE %s (a int, b int, PRIMARY KEY (a, b)) WITH
COMPACT STORAGE");
+assertInvalidMessage("Cannot rename unknown column column1 in
keyspace",
+ "ALTER TABLE %s RENAME column1 TO column2");
+
+createTable("CREATE TABLE %s (a int, b int, v int, PRIMARY KEY (a, b))
WITH COMPACT STORAGE");
+assertInvalidMessage("Cannot rename unknown column column1 in
keyspace",
+
[jira] [Updated] (CASSANDRA-15464) Inserts to set slow due to AtomicBTreePartition for ComplexColumnData.dataSize
[
https://issues.apache.org/jira/browse/CASSANDRA-15464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Benedict Elliott Smith updated CASSANDRA-15464:
---
Bug Category: Parent values: Degradation(12984)Level 1 values: Performance
Bug/Regression(12997)
Complexity: Normal
Discovered By: User Report
Severity: Normal
Status: Open (was: Triage Needed)
> Inserts to set slow due to AtomicBTreePartition for
> ComplexColumnData.dataSize
>
>
> Key: CASSANDRA-15464
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15464
> Project: Cassandra
> Issue Type: Bug
> Components: Legacy/Core
>Reporter: Eric Jacobsen
>Priority: Normal
>
> Concurrent inserts to set can cause client timeouts and excessive CPU
> due to compare and swap in AtomicBTreePartition for
> ComplexColumnData.dataSize. As the length of the set gets longer, the
> probability of doing the compare decreases.
> The problem we saw in production was with insertions into a set with
> len(set) hundreds to thousands. Because of the semantics of what we
> store in the set, we had not anticipated the length being more than about 10.
> (Almost all rows have length <= 6, the largest observed was 7032. Total
> number of rows < 4000. 3 machines were used.)
> The bad behavior we saw was all machines went to 100% cpu on all cores, and
> clients were timing out. Our immediate solution in production was adding more
> machines (went from 3 machines to 6 machines). The stack included
> partitions.AtomicBTreePartition.addAllWithSizeDelta …
> ComplexColumnData.dataSize.
> The AtomicBTreePartition code uses a Compare And Swap approach, yet the time
> between compares is dependent on the length of the set. When the length of
> the set is long, with concurrent updates, each loop is unlikely to make
> forward progress and can be delayed looping.
> Here is one example call stack:
> {noformat}
> "SharedPool-Worker-40" #167 daemon prio=10 os_prio=0 tid=0x7f9bb4032800
> nid=0x2ee5 runnable [0x7f9b067f4000]
> java.lang.Thread.State: RUNNABLE
> at
> org.apache.cassandra.db.rows.ComplexColumnData.dataSize(ComplexColumnData.java:114)
> at org.apache.cassandra.db.rows.BTreeRow.dataSize(BTreeRow.java:373)
> at
> org.apache.cassandra.db.partitions.AtomicBTreePartition$RowUpdater.apply(AtomicBTreePartition.java:292)
> at
> org.apache.cassandra.db.partitions.AtomicBTreePartition$RowUpdater.apply(AtomicBTreePartition.java:235)
> at org.apache.cassandra.utils.btree.NodeBuilder.update(NodeBuilder.java:159)
> at org.apache.cassandra.utils.btree.TreeBuilder.update(TreeBuilder.java:73)
> at org.apache.cassandra.utils.btree.BTree.update(BTree.java:181)
> at
> org.apache.cassandra.db.partitions.AtomicBTreePartition.addAllWithSizeDelta(AtomicBTreePartition.java:155)
> at org.apache.cassandra.db.Memtable.put(Memtable.java:254)
> at
> org.apache.cassandra.db.ColumnFamilyStore.apply(ColumnFamilyStore.java:1204)
> at org.apache.cassandra.db.Keyspace.applyInternal(Keyspace.java:573)
> at org.apache.cassandra.db.Keyspace.applyFuture(Keyspace.java:384)
> at org.apache.cassandra.db.Mutation.applyFuture(Mutation.java:205)
> at org.apache.cassandra.hints.Hint.applyFuture(Hint.java:99)
> at org.apache.cassandra.hints.HintVerbHandler.doVerb(HintVerbHandler.java:95)
> at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:67)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
> at
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136)
> at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
> In a test program to repro the problem, we raise the number of concurrent
> users and lower the think time between queries. Updating elements of
> low-length sets can occur without errors, and with long-length sets, clients
> time out with errors and there are periods with all cores 99.x% CPU and with
> jstack shows time going to ComplexColumnData.dataSize.
> Here is the schema. Our long term application solution was to just have the
> set elements be part of the primary key and avoid using set, thus
> guaranteeing the code does not go through ComplexColumnData.dataSize
> {noformat}
> CREATE TABLE x.x (
> x int PRIMARY KEY,
> y set ) ...
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-ma
[jira] [Updated] (CASSANDRA-15493) Allow cassandra-diff to be run via in-jvm java code rather than only via spark-submit
[ https://issues.apache.org/jira/browse/CASSANDRA-15493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated CASSANDRA-15493: --- Labels: pull-request-available (was: ) > Allow cassandra-diff to be run via in-jvm java code rather than only via > spark-submit > - > > Key: CASSANDRA-15493 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15493 > Project: Cassandra > Issue Type: Improvement > Components: Tool/diff >Reporter: Doug Rohrer >Assignee: Doug Rohrer >Priority: Normal > Labels: pull-request-available > > It should be possible to run the cassandra-diff tool via java code, and run > it multiple times within the same JVM, rather than only being able to execute > it via spark-submit. Make the necessary changes to allow for this use-case. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-15493) Allow cassandra-diff to be run via in-jvm java code rather than only via spark-submit
Doug Rohrer created CASSANDRA-15493: --- Summary: Allow cassandra-diff to be run via in-jvm java code rather than only via spark-submit Key: CASSANDRA-15493 URL: https://issues.apache.org/jira/browse/CASSANDRA-15493 Project: Cassandra Issue Type: Improvement Components: Tool/diff Reporter: Doug Rohrer Assignee: Doug Rohrer It should be possible to run the cassandra-diff tool via java code, and run it multiple times within the same JVM, rather than only being able to execute it via spark-submit. Make the necessary changes to allow for this use-case. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15450) in-jvm dtest cluster uncaughtExceptions propagation of exception goes to the wrong instance, it uses cluster generation when it should be using the instance id
[ https://issues.apache.org/jira/browse/CASSANDRA-15450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17010772#comment-17010772 ] Alex Petrov commented on CASSANDRA-15450: - +1 for not adding test. Patch LGTM as-is. Thank you for filing a jira for back port, too! > in-jvm dtest cluster uncaughtExceptions propagation of exception goes to the > wrong instance, it uses cluster generation when it should be using the > instance id > --- > > Key: CASSANDRA-15450 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15450 > Project: Cassandra > Issue Type: Bug > Components: Test/dtest >Reporter: David Capwell >Assignee: David Capwell >Priority: Normal > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > In AbstractCluster.uncaughtExceptions, we attempt to get the instance from > the class loader and used the “generation”. This value is actually the > cluster id, so causes tests to fail when multiple tests share the same JVM; > it should be using the “id” field which represents the instance id relative > to the cluster. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15490) in-jvm dtest upgrade tests don't support 4.0 versioning
[
https://issues.apache.org/jira/browse/CASSANDRA-15490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alex Petrov updated CASSANDRA-15490:
Description:
The current 4.0 versioning has a tag to denote that 4.0 isn’t in public release
yet (currently beta3). dtest does version parsing which is much more restive
and doesn’t support preRelease tags (like the client version parsing does), so
skips the 4.0 jar. This then leads to a exception while running the upgrade
tests
{code}
java.lang.RuntimeException: No v4 versions found
at
org.apache.cassandra.distributed.impl.Versions.lambda$getLatest$2(Versions.java:166)
at java.util.Optional.orElseThrow(Optional.java:290)
at
org.apache.cassandra.distributed.impl.Versions.getLatest(Versions.java:166)
at
java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at
java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
at
java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:546)
at
java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260)
at
java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:505)
at
org.apache.cassandra.distributed.upgrade.UpgradeTestBase$TestCase.upgrade(UpgradeTestBase.java:92)
at
org.apache.cassandra.distributed.upgrade.UpgradeTest.upgradeTest(UpgradeTest.java:35)
{code}
was:
The current 4.0 versioning has a tag to denote that 4.0 isn’t in public release
yet (currently beta3). dtest does version parsing which is much more restive
and doesn’t support preRelease tags (like the client version parsing does), so
skips the 4.0 jar. This then leads to a exception while running the upgrade
tests
java.lang.RuntimeException: No v4 versions found
at
org.apache.cassandra.distributed.impl.Versions.lambda$getLatest$2(Versions.java:166)
at java.util.Optional.orElseThrow(Optional.java:290)
at
org.apache.cassandra.distributed.impl.Versions.getLatest(Versions.java:166)
at
java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at
java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
at
java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:546)
at
java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260)
at
java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:505)
at
org.apache.cassandra.distributed.upgrade.UpgradeTestBase$TestCase.upgrade(UpgradeTestBase.java:92)
at
org.apache.cassandra.distributed.upgrade.UpgradeTest.upgradeTest(UpgradeTest.java:35)
> in-jvm dtest upgrade tests don't support 4.0 versioning
> ---
>
> Key: CASSANDRA-15490
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15490
> Project: Cassandra
> Issue Type: Bug
> Components: Test/dtest
>Reporter: David Capwell
>Priority: Normal
>
> The current 4.0 versioning has a tag to denote that 4.0 isn’t in public
> release yet (currently beta3). dtest does version parsing which is much more
> restive and doesn’t support preRelease tags (like the client version parsing
> does), so skips the 4.0 jar. This then leads to a exception while running
> the upgrade tests
>
> {code}
> java.lang.RuntimeException: No v4 versions found
> at
> org.apache.cassandra.distributed.impl.Versions.lambda$getLatest$2(Versions.java:166)
> at java.util.Optional.orElseThrow(Optional.java:290)
> at
> org.apache.cassandra.distributed.impl.Versions.getLatest(Versions.java:166)
> at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> at
> java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)
> at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
> at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
> at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:546)
> at
> java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260)
> at
> java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:505)
> at
> org.apache.cassandra.distributed.upgrade.UpgradeTestBase$TestCase.upgrade(UpgradeTestBase.java:92)
> at
> org.apache.cassandra.distributed.upgrade.UpgradeTest.upgradeTest(UpgradeTest.java:35)
> {cod
[jira] [Assigned] (CASSANDRA-15213) DecayingEstimatedHistogramReservoir Inefficiencies
[
https://issues.apache.org/jira/browse/CASSANDRA-15213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jordan West reassigned CASSANDRA-15213:
---
Assignee: Jordan West
> DecayingEstimatedHistogramReservoir Inefficiencies
> --
>
> Key: CASSANDRA-15213
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15213
> Project: Cassandra
> Issue Type: Bug
> Components: Observability/Metrics
>Reporter: Benedict Elliott Smith
>Assignee: Jordan West
>Priority: Normal
> Fix For: 4.0-beta
>
>
> * {{LongAdder}} introduced to trunk consumes 9MiB of heap without user
> schemas, and this will grow significantly under contention and user schemas
> with many tables. This is because {{LongAdder}} is a very heavy class
> designed for single contended values.
> ** This can likely be improved significantly, without significant loss of
> performance in the contended case, by simply increasing the size of our
> primitive backing array and providing multiple buckets, with each thread
> picking a bucket to increment, or simply multiple backing arrays. Probably a
> better way still to do this would be to introduce some competition detection
> to the update, much like {{LongAdder}} utilises, that increases the number of
> backing arrays under competition.
> ** To save memory this approach could partition the space into chunks that
> are likely to be updated together, so that we do not need to duplicate the
> entire array under competition.
> * Similarly, binary search is costly and a measurable cost as a share of the
> new networking work (without filtering it was > 10% of the CPU used overall).
> We can compute an approximation floor(log2 n / log2 1.2) extremely cheaply,
> to save the random memory access costs.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
