[jira] [Commented] (CASSANDRA-14325) Java executable check succeeds despite no java on PATH
[
https://issues.apache.org/jira/browse/CASSANDRA-14325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17376785#comment-17376785
]
Alexey Zotov commented on CASSANDRA-14325:
--
The patch looks good to me, however, I have two small comments:
1. Even though the patch's logic seems to be working perfectly well, I feel
the same can be achieved in a bit more compact way:
{code:java}
diff --git a/bin/cassandra.in.sh b/bin/cassandra.in.sh
index 58b4dd2896..5d13cdae70 100644
--- a/bin/cassandra.in.sh
+++ b/bin/cassandra.in.sh
@@ -95,7 +95,7 @@ if [ -n "$JAVA_HOME" ]; then
fi
done
else
-JAVA=java
+JAVA=`command -v java 2> /dev/null`
fi
if [ -z $JAVA ] ; then
{code}
If there is no _java_ executable available then _JAVA_ variable will be empty
and _if [ -z $JAVA ]_ condition will match to trigger the error.
2. I can see two more similar scripts and I believe they need to be updated as
well:
{code:java}
redhat/cassandra.in.sh
tools/bin/cassandra.in.sh
{code}
PS:
I'm not a project committer. I've just chimed in after seeing [~blerer]'s email
seeking for reviewers.
> Java executable check succeeds despite no java on PATH
> --
>
> Key: CASSANDRA-14325
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14325
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Startup and Shutdown
>Reporter: Angelo Polo
>Assignee: Angelo Polo
>Priority: Low
> Fix For: 3.0.x, 3.11.x, 4.0.x
>
> Attachments: bin_cassandra.patch
>
>
> The check -z $JAVA on line 102 of bin/cassandra currently always succeeds if
> JAVA_HOME is not set since in this case JAVA gets set directly to 'java'. The
> error message "_Unable to find java executable. Check JAVA_HOME and PATH
> environment variables._" will never be echoed on a PATH misconfiguration. If
> java isn't on the PATH the failure will instead occur on line 95 of
> cassandra-env.sh at the java version check.
> It would be better to check consistently for the java executable in one place
> in bin/cassandra. Also we don't want users to mistakenly think they have a
> java version problem when they in fact have a PATH problem.
> See proposed patch.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15669) LeveledCompactionStrategy compact last level throw an ArrayIndexOutOfBoundsException
[ https://issues.apache.org/jira/browse/CASSANDRA-15669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17354616#comment-17354616 ] Alexey Zotov commented on CASSANDRA-15669: -- Thanks [~yifanc]! > LeveledCompactionStrategy compact last level throw an > ArrayIndexOutOfBoundsException > > > Key: CASSANDRA-15669 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15669 > Project: Cassandra > Issue Type: Bug > Components: Local/Compaction/LCS >Reporter: sunhaihong >Assignee: Alexey Zotov >Priority: Normal > Fix For: 3.11.x, 4.0.x > > Attachments: 15669-3.11.txt, 15669-4.0.txt, 15669-trunk.txt, > cfs_compaction_info.png, error_info.png > > Time Spent: 1h 50m > Remaining Estimate: 0h > > Cassandra will throw an ArrayIndexOutOfBoundsException when compact last > level. > My test is as follows： > # Create a table with LeveledCompactionStrategy and its params are > 'enabled': 'true', 'fanout_size': '2', 'max_threshold': '32', > 'min_threshold': '4', 'sstable_size_in_mb': '2'（fanout_size and > sstable_size_in_mb are too small just to make it easier to reproduce the > problem）; > # Insert data into the table by stress; > # Cassandra throw an ArrayIndexOutOfBoundsException when compact level9 > sstables(this level score bigger than 1.001) > ERROR [CompactionExecutor:4] 2020-03-28 08:59:00,990 CassandraDaemon.java:442 > - Exception in thread Thread[CompactionExecutor:4,1,main] > java.lang.ArrayIndexOutOfBoundsException: 9 > at > org.apache.cassandra.db.compaction.LeveledManifest.getLevel(LeveledManifest.java:814) > at > org.apache.cassandra.db.compaction.LeveledManifest.getCandidatesFor(LeveledManifest.java:746) > at > org.apache.cassandra.db.compaction.LeveledManifest.getCompactionCandidates(LeveledManifest.java:398) > at > org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getNextBackgroundTask(LeveledCompactionStrategy.java:131) > at > org.apache.cassandra.db.compaction.CompactionStrategyHolder.lambda$getBackgroundTaskSuppliers$0(CompactionStrategyHolder.java:109) > at > org.apache.cassandra.db.compaction.AbstractStrategyHolder$TaskSupplier.getTask(AbstractStrategyHolder.java:66) > at > org.apache.cassandra.db.compaction.CompactionStrategyManager.getNextBackgroundTask(CompactionStrategyManager.java:214) > at > org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:289) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266) > at java.util.concurrent.FutureTask.run(FutureTask.java) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at > io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) > at java.lang.Thread.run(Thread.java:748) > I tested it on cassandra version 3.11.3 & 4.0-alpha3. The exception all > happened. > once it triggers, level1- leveln compaction no longer works, level0 is still > valid > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-15669) LeveledCompactionStrategy compact last level throw an ArrayIndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/CASSANDRA-15669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17354107#comment-17354107
]
Alexey Zotov edited comment on CASSANDRA-15669 at 5/30/21, 7:57 PM:
[~yifanc] [~marcuse]
I'm attaching patches for 3.11, 4.0 and trunk branches. In order to make the
unit test working in 3.11 branch, I've had to onboard these changes:
*
[https://github.com/apache/cassandra/commit/546331037ee8218208b61a26ecbc12ab8dd7eace#diff-d2993d01b021fb5e2ed23171c0bf5f1abb2f13d1d9ef67d089bb2cf8108ff64c]
*
[https://github.com/apache/cassandra/commit/54d297a192ca452dab5640f33fd6c22fd31e2f9c#diff-d2993d01b021fb5e2ed23171c0bf5f1abb2f13d1d9ef67d089bb2cf8108ff64c]
I run the {{LeveledCompactionStrategyTest}} on every branch and it worked fine
for me locally. Please, let me know if you encounter any issues with that test
on CI.
PS:
Initially I misread the comment and tried to prepare a patch for 3.0 version
:D It turned out that it is pretty to hard to migrate the unit test to that
branch because _fanout_ parameter is not yet configurable. Consequently it
requires to generate a bunch of sstables (to reach _max level size_ for L8)
which make the test long and clumsy. It took some time to realize that it is
better to go without having that test in 3.0. And finally I found out that the
patch for 3.0 is not required.
was (Author: azotcsit):
[~yifanc] [~marcuse]
I'm attaching patches for 3.11, 4.0 and trunk branches. In order to make the
unit test working in 3.11 branch, I've had to onboard these changes:
*
[https://github.com/apache/cassandra/commit/546331037ee8218208b61a26ecbc12ab8dd7eace#diff-d2993d01b021fb5e2ed23171c0bf5f1abb2f13d1d9ef67d089bb2cf8108ff64c]
*
[https://github.com/apache/cassandra/commit/54d297a192ca452dab5640f33fd6c22fd31e2f9c#diff-d2993d01b021fb5e2ed23171c0bf5f1abb2f13d1d9ef67d089bb2cf8108ff64c]
I run the {{LeveledCompactionStrategyTest}} on every branch and it worked fine
for me locally. Please, let me know if encounter any issues with that test on
CI.
PS:
Initially I misread the comment and tried to prepare a patch for 3.0 version
:D It turned out that it is pretty to hard to migrate the unit test to that
branch because _fanout_ parameter is not yet configurable. Consequently it
requires to generate a bunch of sstables (to reach _max level size_ for L8)
which make the test long and clumsy. It took some time to realize that it is
better to go without having that test in 3.0. And finally I found out that the
patch for 3.0 is not required.
> LeveledCompactionStrategy compact last level throw an
> ArrayIndexOutOfBoundsException
>
>
> Key: CASSANDRA-15669
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15669
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Compaction/LCS
>Reporter: sunhaihong
>Assignee: Alexey Zotov
>Priority: Normal
> Fix For: 3.11.x, 4.0.x
>
> Attachments: 15669-3.11.txt, 15669-4.0.txt, 15669-trunk.txt,
> cfs_compaction_info.png, error_info.png
>
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> Cassandra will throw an ArrayIndexOutOfBoundsException when compact last
> level.
> My test is as follows：
> # Create a table with LeveledCompactionStrategy and its params are
> 'enabled': 'true', 'fanout_size': '2', 'max_threshold': '32',
> 'min_threshold': '4', 'sstable_size_in_mb': '2'（fanout_size and
> sstable_size_in_mb are too small just to make it easier to reproduce the
> problem）;
> # Insert data into the table by stress;
> # Cassandra throw an ArrayIndexOutOfBoundsException when compact level9
> sstables(this level score bigger than 1.001)
> ERROR [CompactionExecutor:4] 2020-03-28 08:59:00,990 CassandraDaemon.java:442
> - Exception in thread Thread[CompactionExecutor:4,1,main]
> java.lang.ArrayIndexOutOfBoundsException: 9
> at
> org.apache.cassandra.db.compaction.LeveledManifest.getLevel(LeveledManifest.java:814)
> at
> org.apache.cassandra.db.compaction.LeveledManifest.getCandidatesFor(LeveledManifest.java:746)
> at
> org.apache.cassandra.db.compaction.LeveledManifest.getCompactionCandidates(LeveledManifest.java:398)
> at
> org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getNextBackgroundTask(LeveledCompactionStrategy.java:131)
> at
> org.apache.cassandra.db.compaction.CompactionStrategyHolder.lambda$getBackgroundTaskSuppliers$0(CompactionStrategyHolder.java:109)
> at
> org.apache.cassandra.db.compaction.AbstractStrategyHolder$TaskSupplier.getTask(AbstractStrategyHolder.java:66)
> at
> org.apache.cassandra.db.compaction.CompactionStrategyManager.getNextBackgroundTask(CompactionStrategyManager.java:214)
> at
>
[jira] [Updated] (CASSANDRA-15669) LeveledCompactionStrategy compact last level throw an ArrayIndexOutOfBoundsException
[ https://issues.apache.org/jira/browse/CASSANDRA-15669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Zotov updated CASSANDRA-15669: - Attachment: 15669-trunk.txt 15669-4.0.txt 15669-3.11.txt > LeveledCompactionStrategy compact last level throw an > ArrayIndexOutOfBoundsException > > > Key: CASSANDRA-15669 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15669 > Project: Cassandra > Issue Type: Bug > Components: Local/Compaction/LCS >Reporter: sunhaihong >Assignee: Alexey Zotov >Priority: Normal > Fix For: 3.11.x, 4.0.x > > Attachments: 15669-3.11.txt, 15669-4.0.txt, 15669-trunk.txt, > cfs_compaction_info.png, error_info.png > > Time Spent: 1h 50m > Remaining Estimate: 0h > > Cassandra will throw an ArrayIndexOutOfBoundsException when compact last > level. > My test is as follows： > # Create a table with LeveledCompactionStrategy and its params are > 'enabled': 'true', 'fanout_size': '2', 'max_threshold': '32', > 'min_threshold': '4', 'sstable_size_in_mb': '2'（fanout_size and > sstable_size_in_mb are too small just to make it easier to reproduce the > problem）; > # Insert data into the table by stress; > # Cassandra throw an ArrayIndexOutOfBoundsException when compact level9 > sstables(this level score bigger than 1.001) > ERROR [CompactionExecutor:4] 2020-03-28 08:59:00,990 CassandraDaemon.java:442 > - Exception in thread Thread[CompactionExecutor:4,1,main] > java.lang.ArrayIndexOutOfBoundsException: 9 > at > org.apache.cassandra.db.compaction.LeveledManifest.getLevel(LeveledManifest.java:814) > at > org.apache.cassandra.db.compaction.LeveledManifest.getCandidatesFor(LeveledManifest.java:746) > at > org.apache.cassandra.db.compaction.LeveledManifest.getCompactionCandidates(LeveledManifest.java:398) > at > org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getNextBackgroundTask(LeveledCompactionStrategy.java:131) > at > org.apache.cassandra.db.compaction.CompactionStrategyHolder.lambda$getBackgroundTaskSuppliers$0(CompactionStrategyHolder.java:109) > at > org.apache.cassandra.db.compaction.AbstractStrategyHolder$TaskSupplier.getTask(AbstractStrategyHolder.java:66) > at > org.apache.cassandra.db.compaction.CompactionStrategyManager.getNextBackgroundTask(CompactionStrategyManager.java:214) > at > org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:289) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266) > at java.util.concurrent.FutureTask.run(FutureTask.java) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at > io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) > at java.lang.Thread.run(Thread.java:748) > I tested it on cassandra version 3.11.3 & 4.0-alpha3. The exception all > happened. > once it triggers, level1- leveln compaction no longer works, level0 is still > valid > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15669) LeveledCompactionStrategy compact last level throw an ArrayIndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/CASSANDRA-15669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17354107#comment-17354107
]
Alexey Zotov commented on CASSANDRA-15669:
--
[~yifanc] [~marcuse]
I'm attaching patches for 3.11, 4.0 and trunk branches. In order to make the
unit test working in 3.11 branch, I've had to onboard these changes:
*
[https://github.com/apache/cassandra/commit/546331037ee8218208b61a26ecbc12ab8dd7eace#diff-d2993d01b021fb5e2ed23171c0bf5f1abb2f13d1d9ef67d089bb2cf8108ff64c]
*
[https://github.com/apache/cassandra/commit/54d297a192ca452dab5640f33fd6c22fd31e2f9c#diff-d2993d01b021fb5e2ed23171c0bf5f1abb2f13d1d9ef67d089bb2cf8108ff64c]
I run the {{LeveledCompactionStrategyTest}} on every branch and it worked fine
for me locally. Please, let me know if encounter any issues with that test on
CI.
PS:
Initially I misread the comment and tried to prepare a patch for 3.0 version
:D It turned out that it is pretty to hard to migrate the unit test to that
branch because _fanout_ parameter is not yet configurable. Consequently it
requires to generate a bunch of sstables (to reach _max level size_ for L8)
which make the test long and clumsy. It took some time to realize that it is
better to go without having that test in 3.0. And finally I found out that the
patch for 3.0 is not required.
> LeveledCompactionStrategy compact last level throw an
> ArrayIndexOutOfBoundsException
>
>
> Key: CASSANDRA-15669
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15669
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Compaction/LCS
>Reporter: sunhaihong
>Assignee: Alexey Zotov
>Priority: Normal
> Fix For: 3.11.x, 4.0.x
>
> Attachments: 15669-3.11.txt, 15669-4.0.txt, 15669-trunk.txt,
> cfs_compaction_info.png, error_info.png
>
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> Cassandra will throw an ArrayIndexOutOfBoundsException when compact last
> level.
> My test is as follows：
> # Create a table with LeveledCompactionStrategy and its params are
> 'enabled': 'true', 'fanout_size': '2', 'max_threshold': '32',
> 'min_threshold': '4', 'sstable_size_in_mb': '2'（fanout_size and
> sstable_size_in_mb are too small just to make it easier to reproduce the
> problem）;
> # Insert data into the table by stress;
> # Cassandra throw an ArrayIndexOutOfBoundsException when compact level9
> sstables(this level score bigger than 1.001)
> ERROR [CompactionExecutor:4] 2020-03-28 08:59:00,990 CassandraDaemon.java:442
> - Exception in thread Thread[CompactionExecutor:4,1,main]
> java.lang.ArrayIndexOutOfBoundsException: 9
> at
> org.apache.cassandra.db.compaction.LeveledManifest.getLevel(LeveledManifest.java:814)
> at
> org.apache.cassandra.db.compaction.LeveledManifest.getCandidatesFor(LeveledManifest.java:746)
> at
> org.apache.cassandra.db.compaction.LeveledManifest.getCompactionCandidates(LeveledManifest.java:398)
> at
> org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getNextBackgroundTask(LeveledCompactionStrategy.java:131)
> at
> org.apache.cassandra.db.compaction.CompactionStrategyHolder.lambda$getBackgroundTaskSuppliers$0(CompactionStrategyHolder.java:109)
> at
> org.apache.cassandra.db.compaction.AbstractStrategyHolder$TaskSupplier.getTask(AbstractStrategyHolder.java:66)
> at
> org.apache.cassandra.db.compaction.CompactionStrategyManager.getNextBackgroundTask(CompactionStrategyManager.java:214)
> at
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:289)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
> at java.util.concurrent.FutureTask.run(FutureTask.java)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> at java.lang.Thread.run(Thread.java:748)
> I tested it on cassandra version 3.11.3 & 4.0-alpha3. The exception all
> happened.
> once it triggers, level1- leveln compaction no longer works, level0 is still
> valid
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16404) Provide a nodetool way of invalidating auth caches
[ https://issues.apache.org/jira/browse/CASSANDRA-16404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17352652#comment-17352652 ] Alexey Zotov commented on CASSANDRA-16404: -- [~blerer] [~sumanth.pasupuleti] Have you had a chance to check the changes? > Provide a nodetool way of invalidating auth caches > -- > > Key: CASSANDRA-16404 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16404 > Project: Cassandra > Issue Type: Improvement > Components: Feature/Authorization >Reporter: Sumanth Pasupuleti >Assignee: Alexey Zotov >Priority: Normal > Fix For: 4.x > > Time Spent: 0.5h > Remaining Estimate: 0h > > We currently have nodetool commands to invalidate certain caches like > KeyCache, RowCache and CounterCache. > Being able to invalidate auth caches as well can come in handy in situations > where, critical backend auth changes may need to be in effect right away for > all the connections, especially in configurations where cache validity is > chosen to be for a longer duration. An example can be that an authenticated > user "User1" is no longer authorized to access a table resource "table1" and > it is vital that this change is reflected right away, without having to wait > for cache expiry/refresh to trigger. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15669) LeveledCompactionStrategy compact last level throw an ArrayIndexOutOfBoundsException
[ https://issues.apache.org/jira/browse/CASSANDRA-15669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17352648#comment-17352648 ] Alexey Zotov commented on CASSANDRA-15669: -- [~blerer] [~marcuse] could you please take a look to the PR and let me know whether it is good to merge or any other changes are required. > LeveledCompactionStrategy compact last level throw an > ArrayIndexOutOfBoundsException > > > Key: CASSANDRA-15669 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15669 > Project: Cassandra > Issue Type: Bug > Components: Local/Compaction/LCS >Reporter: sunhaihong >Assignee: Alexey Zotov >Priority: Normal > Fix For: 3.11.x, 4.0.x > > Attachments: cfs_compaction_info.png, error_info.png > > Time Spent: 1h 50m > Remaining Estimate: 0h > > Cassandra will throw an ArrayIndexOutOfBoundsException when compact last > level. > My test is as follows： > # Create a table with LeveledCompactionStrategy and its params are > 'enabled': 'true', 'fanout_size': '2', 'max_threshold': '32', > 'min_threshold': '4', 'sstable_size_in_mb': '2'（fanout_size and > sstable_size_in_mb are too small just to make it easier to reproduce the > problem）; > # Insert data into the table by stress; > # Cassandra throw an ArrayIndexOutOfBoundsException when compact level9 > sstables(this level score bigger than 1.001) > ERROR [CompactionExecutor:4] 2020-03-28 08:59:00,990 CassandraDaemon.java:442 > - Exception in thread Thread[CompactionExecutor:4,1,main] > java.lang.ArrayIndexOutOfBoundsException: 9 > at > org.apache.cassandra.db.compaction.LeveledManifest.getLevel(LeveledManifest.java:814) > at > org.apache.cassandra.db.compaction.LeveledManifest.getCandidatesFor(LeveledManifest.java:746) > at > org.apache.cassandra.db.compaction.LeveledManifest.getCompactionCandidates(LeveledManifest.java:398) > at > org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getNextBackgroundTask(LeveledCompactionStrategy.java:131) > at > org.apache.cassandra.db.compaction.CompactionStrategyHolder.lambda$getBackgroundTaskSuppliers$0(CompactionStrategyHolder.java:109) > at > org.apache.cassandra.db.compaction.AbstractStrategyHolder$TaskSupplier.getTask(AbstractStrategyHolder.java:66) > at > org.apache.cassandra.db.compaction.CompactionStrategyManager.getNextBackgroundTask(CompactionStrategyManager.java:214) > at > org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:289) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266) > at java.util.concurrent.FutureTask.run(FutureTask.java) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at > io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) > at java.lang.Thread.run(Thread.java:748) > I tested it on cassandra version 3.11.3 & 4.0-alpha3. The exception all > happened. > once it triggers, level1- leveln compaction no longer works, level0 is still > valid > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16629) Refactor tests for NodeTool commands
[
https://issues.apache.org/jira/browse/CASSANDRA-16629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17340193#comment-17340193
]
Alexey Zotov commented on CASSANDRA-16629:
--
Thanks [~blerer]!
> Refactor tests for NodeTool commands
>
>
> Key: CASSANDRA-16629
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16629
> Project: Cassandra
> Issue Type: Task
> Components: Test/unit
>Reporter: Alexey Zotov
>Assignee: Alexey Zotov
>Priority: Low
> Fix For: 4.1
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> h3. Overview
> As a part of CASSANDRA-16404, I've been working on {{NodeTool}} commands for
> invalidation of the auth caches. While working on that, I've mentioned that
> the existing NodeTool unit tests are written in a different manner and reside
> in wrong packages. I've made the following changes to them:
> # moved them to the proper packages (that match original command class)
> # unified naming, formatting, imports
> # migrated from {{org.assertj}} to {{hamcrest}} for assertions (though it is
> not smth really required)
> # made a general cleanup
> PR: https://github.com/apache/cassandra/pull/982
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15669) LeveledCompactionStrategy compact last level throw an ArrayIndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/CASSANDRA-15669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17339393#comment-17339393
]
Alexey Zotov commented on CASSANDRA-15669:
--
Thanks [~yifanc]! Please, let me know if I need to make any other changes like
squashing or updating {{CHANGES.txt}}.
> LeveledCompactionStrategy compact last level throw an
> ArrayIndexOutOfBoundsException
>
>
> Key: CASSANDRA-15669
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15669
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Compaction/LCS
>Reporter: sunhaihong
>Assignee: Alexey Zotov
>Priority: Normal
> Fix For: 3.11.x, 4.0.x
>
> Attachments: cfs_compaction_info.png, error_info.png
>
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> Cassandra will throw an ArrayIndexOutOfBoundsException when compact last
> level.
> My test is as follows：
> # Create a table with LeveledCompactionStrategy and its params are
> 'enabled': 'true', 'fanout_size': '2', 'max_threshold': '32',
> 'min_threshold': '4', 'sstable_size_in_mb': '2'（fanout_size and
> sstable_size_in_mb are too small just to make it easier to reproduce the
> problem）;
> # Insert data into the table by stress;
> # Cassandra throw an ArrayIndexOutOfBoundsException when compact level9
> sstables(this level score bigger than 1.001)
> ERROR [CompactionExecutor:4] 2020-03-28 08:59:00,990 CassandraDaemon.java:442
> - Exception in thread Thread[CompactionExecutor:4,1,main]
> java.lang.ArrayIndexOutOfBoundsException: 9
> at
> org.apache.cassandra.db.compaction.LeveledManifest.getLevel(LeveledManifest.java:814)
> at
> org.apache.cassandra.db.compaction.LeveledManifest.getCandidatesFor(LeveledManifest.java:746)
> at
> org.apache.cassandra.db.compaction.LeveledManifest.getCompactionCandidates(LeveledManifest.java:398)
> at
> org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getNextBackgroundTask(LeveledCompactionStrategy.java:131)
> at
> org.apache.cassandra.db.compaction.CompactionStrategyHolder.lambda$getBackgroundTaskSuppliers$0(CompactionStrategyHolder.java:109)
> at
> org.apache.cassandra.db.compaction.AbstractStrategyHolder$TaskSupplier.getTask(AbstractStrategyHolder.java:66)
> at
> org.apache.cassandra.db.compaction.CompactionStrategyManager.getNextBackgroundTask(CompactionStrategyManager.java:214)
> at
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:289)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
> at java.util.concurrent.FutureTask.run(FutureTask.java)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> at java.lang.Thread.run(Thread.java:748)
> I tested it on cassandra version 3.11.3 & 4.0-alpha3. The exception all
> happened.
> once it triggers, level1- leveln compaction no longer works, level0 is still
> valid
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16629) Refactor tests for NodeTool commands
[
https://issues.apache.org/jira/browse/CASSANDRA-16629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17339106#comment-17339106
]
Alexey Zotov commented on CASSANDRA-16629:
--
# Done. I rebased and squashed all commits as per #5 of
https://cassandra.apache.org/doc/latest/development/patches.html#creating-a-patch.
# Thanks!
> Refactor tests for NodeTool commands
>
>
> Key: CASSANDRA-16629
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16629
> Project: Cassandra
> Issue Type: Task
> Components: Test/unit
>Reporter: Alexey Zotov
>Assignee: Alexey Zotov
>Priority: Low
> Fix For: 4.0.x
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> h3. Overview
> As a part of CASSANDRA-16404, I've been working on {{NodeTool}} commands for
> invalidation of the auth caches. While working on that, I've mentioned that
> the existing NodeTool unit tests are written in a different manner and reside
> in wrong packages. I've made the following changes to them:
> # moved them to the proper packages (that match original command class)
> # unified naming, formatting, imports
> # migrated from {{org.assertj}} to {{hamcrest}} for assertions (though it is
> not smth really required)
> # made a general cleanup
> PR: https://github.com/apache/cassandra/pull/982
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-16629) Refactor tests for NodeTool commands
[
https://issues.apache.org/jira/browse/CASSANDRA-16629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17338995#comment-17338995
]
Alexey Zotov edited comment on CASSANDRA-16629 at 5/4/21, 1:36 PM:
---
[~blerer]
A couple of questions:
# As far as I know 4.0 has been branched. Is it a good moment to rebase my
branch onto trunk?
# Are non-committers able to run CI? If yes, could you, please, let me know
the high-level steps on how to do it.
was (Author: azotcsit):
[~blerer]
A couple of questions:
# As far as I know 4.0 has been branched. Is it a good moment to rebase my
branch onto trunk?
# Are non-committers able to run CI? If yes, could you, please, let me know
the high-level steps on how to do it.
> Refactor tests for NodeTool commands
>
>
> Key: CASSANDRA-16629
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16629
> Project: Cassandra
> Issue Type: Task
> Components: Test/unit
>Reporter: Alexey Zotov
>Assignee: Alexey Zotov
>Priority: Low
> Fix For: 4.0.x
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> h3. Overview
> As a part of CASSANDRA-16404, I've been working on {{NodeTool}} commands for
> invalidation of the auth caches. While working on that, I've mentioned that
> the existing NodeTool unit tests are written in a different manner and reside
> in wrong packages. I've made the following changes to them:
> # moved them to the proper packages (that match original command class)
> # unified naming, formatting, imports
> # migrated from {{org.assertj}} to {{hamcrest}} for assertions (though it is
> not smth really required)
> # made a general cleanup
> PR: https://github.com/apache/cassandra/pull/982
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16629) Refactor tests for NodeTool commands
[
https://issues.apache.org/jira/browse/CASSANDRA-16629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17338995#comment-17338995
]
Alexey Zotov commented on CASSANDRA-16629:
--
[~blerer]
A couple of questions:
# As far as I know 4.0 has been branched. Is it a good moment to rebase my
branch onto trunk?
# Are non-committers able to run CI? If yes, could you, please, let me know
the high-level steps on how to do it.
> Refactor tests for NodeTool commands
>
>
> Key: CASSANDRA-16629
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16629
> Project: Cassandra
> Issue Type: Task
> Components: Test/unit
>Reporter: Alexey Zotov
>Assignee: Alexey Zotov
>Priority: Low
> Fix For: 4.0.x
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> h3. Overview
> As a part of CASSANDRA-16404, I've been working on {{NodeTool}} commands for
> invalidation of the auth caches. While working on that, I've mentioned that
> the existing NodeTool unit tests are written in a different manner and reside
> in wrong packages. I've made the following changes to them:
> # moved them to the proper packages (that match original command class)
> # unified naming, formatting, imports
> # migrated from {{org.assertj}} to {{hamcrest}} for assertions (though it is
> not smth really required)
> # made a general cleanup
> PR: https://github.com/apache/cassandra/pull/982
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16630) Migrate to JUnit5
[
https://issues.apache.org/jira/browse/CASSANDRA-16630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17338014#comment-17338014
]
Alexey Zotov commented on CASSANDRA-16630:
--
[~mck] [~benedict]
Thanks for bringing me back to the reality of the current project state. Even
though it is a bit frustrating to be unable to make cleanups and improvements
that are not related to a particular feature development, I completely
understand and admit your concerns.
It would be nice to come up with a new strategy on how we can maintain the
existing code (I'm talking not only about new style approaches, but also about
simple code cleanups) because the current one seems to be a kind of a stopper
for faster project development and evolving. In fact, I am not really sure what
could help here (I don't have a deep understanding of the current processes),
moreover, it is out of the scope for this particular discussion. I'll keep it
in mind and jump in if some ideas appear.
Now here is a question on the matter. I can see that {{cql-test-some}} and
{{cql-test}} targets (unlikely to the other) use {{junit}} instead of
{{junit-timeout}} (which has a support of {{jstack}} on timeout). Are there any
particular reasons for that? Or is it just a potential point for improvement?
> Migrate to JUnit5
> -
>
> Key: CASSANDRA-16630
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16630
> Project: Cassandra
> Issue Type: Improvement
> Components: Test/unit
>Reporter: Alexey Zotov
>Assignee: Alexey Zotov
>Priority: Low
>
> h3. Overview
> Currently C* uses JUnit4 (version 4.12) which is obsolete. There is a newer
> version 4.13.2 which we could update to. However, JUnit4 is generally
> considered to be outdated and it is reasonable to migrate to JUnit5.
> Despite of having a syntax sugar in JUnit5 (assertThrow, lamda's support,
> ect), there are no blockers that push us to move from JUnit4. The main
> motivation for this initiative is rule of thumb to use up-to-date versions of
> the dependencies.
> Obviously this change is not backward compatible with the open PRs and
> previous C* versions. Therefore, it will require an additional effort for
> backporting the changes and updating PRs that have tests. However, I believe
> it should not be a blocker for this initiative.
> h3. Scope (preliminary list)
> # change JUnit4 to JUnit5 dependencies and make necessary changes in ant
> tasks (https://ant.apache.org/manual/Tasks/junitlauncher.html)
> # update syntax in all tests (imports, Before/After annotations, etc)
> # update parameterized tests
> # create a new version of {{OrderedJUnit4ClassRunner}} and update
> corresponding tests
> # update tests that use {{BMUnitRunner}} (as per
> https://developer.jboss.org/docs/DOC-52953 it supports JUnit5)
> # update tests with {{@Rule}}
> # update tests with expected exceptions
> # update {{JStackJUnitTask}}
> # update formatters
> # create a separate ticket to migrate to {{ant-junitlauncher-1.10.11}} (once
> it is released) and simplify {{JStackJUnitTask}} after
> https://github.com/apache/ant/pull/147
> h3. Order of operations
> In order to make the transition more smooth we want to use a phased approach:
> # migrate to JUnit5 with [Vintage
> Engine|https://junit.org/junit5/docs/current/user-guide/#dependency-metadata-junit-vintage],
> so all JUnit4 tests work as is
> # update tests in a few bunches (to not have a huge single PR with numerous
> conflicts)
> # disable (remove dependency) Vintage Engine, so only JUnit5 tests work
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-16360) CRC32 is inefficient on x86
[ https://issues.apache.org/jira/browse/CASSANDRA-16360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17337324#comment-17337324 ] Alexey Zotov edited comment on CASSANDRA-16360 at 4/30/21, 5:03 PM: [~samt] That makes sense. I'll put this ticket on hold. However, in order to get the existing microbenchmark changes merged, I've raised CASSANDRA-16645. Please, take a look once you have a chance. was (Author: azotcsit): [~samt] That makes sense. I'll put this ticket on hold. However, in order to get the existing microbenchmark changes, I've raised CASSANDRA-16645. Please, take a look once you have a chance. > CRC32 is inefficient on x86 > --- > > Key: CASSANDRA-16360 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16360 > Project: Cassandra > Issue Type: Improvement > Components: Messaging/Client >Reporter: Avi Kivity >Priority: Normal > Labels: protocolv6 > Fix For: 4.x > > > The client/server protocol specifies CRC24 and CRC32 as the checksum > algorithm (cql_protocol_V5_framing.asc). Those however are expensive to > compute; this affects both the client and the server. > > A better checksum algorithm is CRC32C, which has hardware support on x86 (as > well as other modern architectures). -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-16630) Migrate to JUnit5
[
https://issues.apache.org/jira/browse/CASSANDRA-16630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexey Zotov updated CASSANDRA-16630:
-
Description:
h3. Overview
Currently C* uses JUnit4 (version 4.12) which is obsolete. There is a newer
version 4.13.2 which we could update to. However, JUnit4 is generally
considered to be outdated and it is reasonable to migrate to JUnit5.
Despite of having a syntax sugar in JUnit5 (assertThrow, lamda's support, ect),
there are no blockers that push us to move from JUnit4. The main motivation for
this initiative is rule of thumb to use up-to-date versions of the dependencies.
Obviously this change is not backward compatible with the open PRs and previous
C* versions. Therefore, it will require an additional effort for backporting
the changes and updating PRs that have tests. However, I believe it should not
be a blocker for this initiative.
h3. Scope (preliminary list)
# change JUnit4 to JUnit5 dependencies and make necessary changes in ant tasks
(https://ant.apache.org/manual/Tasks/junitlauncher.html)
# update syntax in all tests (imports, Before/After annotations, etc)
# update parameterized tests
# create a new version of {{OrderedJUnit4ClassRunner}} and update corresponding
tests
# update tests that use {{BMUnitRunner}} (as per
https://developer.jboss.org/docs/DOC-52953 it supports JUnit5)
# update tests with {{@Rule}}
# update tests with expected exceptions
# update {{JStackJUnitTask}}
# update formatters
# create a separate ticket to migrate to {{ant-junitlauncher-1.10.11}} (once it
is released) and simplify {{JStackJUnitTask}} after
https://github.com/apache/ant/pull/147
h3. Order of operations
In order to make the transition more smooth we want to use a phased approach:
# migrate to JUnit5 with [Vintage
Engine|https://junit.org/junit5/docs/current/user-guide/#dependency-metadata-junit-vintage],
so all JUnit4 tests work as is
# update tests in a few bunches (to not have a huge single PR with numerous
conflicts)
# disable (remove dependency) Vintage Engine, so only JUnit5 tests work
was:
h3. Overview
Currently C* uses JUnit4 (version 4.12) which is obsolete. There is a newer
version 4.13.2 which we could update to. However, JUnit4 is generally
considered to be outdated and it is reasonable to migrate to JUnit5.
Despite of having a syntax sugar in JUnit5 (assertThrow, lamda's support, ect),
there are no blockers that push us to move from JUnit4. The main motivation for
this initiative is rule of thumb to use up-to-date versions of the dependencies.
Obviously this change is not backward compatible with the open PRs and previous
C* versions. Therefore, it will require an additional effort for backporting
the changes and updating PRs that have tests. However, I believe it should not
be a blocker for this initiative.
h3. Scope (preliminary list)
# change JUnit4 to JUnit5 dependencies and make necessary changes in ant tasks
(https://ant.apache.org/manual/Tasks/junitlauncher.html)
# update syntax in all tests (imports, Before/After annotations, etc)
# update parameterized tests
# create a new version of {{OrderedJUnit4ClassRunner}} and update corresponding
tests
# update tests that use {{BMUnitRunner}} (as per
https://developer.jboss.org/docs/DOC-52953 it supports JUnit5)
# update tests with {{@Rule}}
# update tests with expected exceptions
h3. Order of operations
In order to make the transition more smooth we want to use a phased approach:
# migrate to JUnit5 with [Vintage
Engine|https://junit.org/junit5/docs/current/user-guide/#dependency-metadata-junit-vintage],
so all JUnit4 tests work as is
# update tests in a few bunches (to not have a huge single PR with numerous
conflicts)
# disable (remove dependency) Vintage Engine, so only JUnit5 tests work
> Migrate to JUnit5
> -
>
> Key: CASSANDRA-16630
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16630
> Project: Cassandra
> Issue Type: Improvement
> Components: Test/unit
>Reporter: Alexey Zotov
>Assignee: Alexey Zotov
>Priority: Low
>
> h3. Overview
> Currently C* uses JUnit4 (version 4.12) which is obsolete. There is a newer
> version 4.13.2 which we could update to. However, JUnit4 is generally
> considered to be outdated and it is reasonable to migrate to JUnit5.
> Despite of having a syntax sugar in JUnit5 (assertThrow, lamda's support,
> ect), there are no blockers that push us to move from JUnit4. The main
> motivation for this initiative is rule of thumb to use up-to-date versions of
> the dependencies.
> Obviously this change is not backward compatible with the open PRs and
> previous C* versions. Therefore, it will require an additional effort for
> backporting the changes and updating PRs that have tests. However, I believe
> it should not be a blocker for this
[jira] [Updated] (CASSANDRA-16645) Add a microbenchmark for checksum calculation
[
https://issues.apache.org/jira/browse/CASSANDRA-16645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexey Zotov updated CASSANDRA-16645:
-
Test and Documentation Plan: No tests or documentation are required.
Status: Patch Available (was: Open)
> Add a microbenchmark for checksum calculation
> -
>
> Key: CASSANDRA-16645
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16645
> Project: Cassandra
> Issue Type: Improvement
> Components: Test/benchmark
>Reporter: Alexey Zotov
>Assignee: Alexey Zotov
>Priority: Low
> Fix For: 4.0.x
>
>
> h2. Overview
> As a part CASSANDRA-16360 there was a need to change the checksum algo from
> _CRC32_ to _CRC32C_ for client/internode messaging. Before making that change
> it would be nice to have a way to measure performance for checksum
> calculation (similarly to {{HashingBench}} for hash calculation).
> In fact, the microbenchmark has been already created as a part of that
> ticket. However, after running it I saw a performance degradation for Java 8.
> After a discussion among the community, it was decided to wait with _CRC32C_
> until Java 8 support is abandoned. The purpose of this ticket is to merge the
> benchmark changes separately.
> h2. What needs to be done
> * Create a microbenchmark
> h2. Link to PR
> [https://github.com/apache/cassandra/pull/951/files]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-16645) Add a microbenchmark for checksum calculation
[
https://issues.apache.org/jira/browse/CASSANDRA-16645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexey Zotov updated CASSANDRA-16645:
-
Description:
h2. Overview
As a part CASSANDRA-16360 there was a need to change the checksum algo from
_CRC32_ to _CRC32C_ for client/internode messaging. Before making that change
it would be nice to have a way to measure performance for checksum calculation
(similarly to {{HashingBench}} for hash calculation).
In fact, the microbenchmark has been already created as a part of that ticket.
However, after running it I saw a performance degradation for Java 8. After a
discussion among the community, it was decided to wait with _CRC32C_ until Java
8 support is abandoned. The purpose of this ticket is to merge the benchmark
changes separately.
h2. What needs to be done
* Create a microbenchmark
h2. Link to PR
[https://github.com/apache/cassandra/pull/951/files]
was:
h2. Overview
As a part CASSANDRA-16360 there was a need to change the checksum algo from
_CRC32_ to _CRC32C_ for client/internode messaging. Before making that change
it would be nice to have a way to measure performance for checksum calculation
(similarly to {{HashingBench}} for hash calculation).
In fact, the microbenchmark has been already created as a part of that ticket.
However, after running it we saw a performance degradation for Java 8 and
decided to wait until Java 8 support is abandoned. The purpose of this ticket
is to merge the benchmark changes separately.
h2. What needs to be done
* Create a microbenchmark
h2. Link to PR
https://github.com/apache/cassandra/pull/951/files
> Add a microbenchmark for checksum calculation
> -
>
> Key: CASSANDRA-16645
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16645
> Project: Cassandra
> Issue Type: Improvement
> Components: Test/benchmark
>Reporter: Alexey Zotov
>Assignee: Alexey Zotov
>Priority: Low
> Fix For: 4.0.x
>
>
> h2. Overview
> As a part CASSANDRA-16360 there was a need to change the checksum algo from
> _CRC32_ to _CRC32C_ for client/internode messaging. Before making that change
> it would be nice to have a way to measure performance for checksum
> calculation (similarly to {{HashingBench}} for hash calculation).
> In fact, the microbenchmark has been already created as a part of that
> ticket. However, after running it I saw a performance degradation for Java 8.
> After a discussion among the community, it was decided to wait with _CRC32C_
> until Java 8 support is abandoned. The purpose of this ticket is to merge the
> benchmark changes separately.
> h2. What needs to be done
> * Create a microbenchmark
> h2. Link to PR
> [https://github.com/apache/cassandra/pull/951/files]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-16360) CRC32 is inefficient on x86
[ https://issues.apache.org/jira/browse/CASSANDRA-16360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17337324#comment-17337324 ] Alexey Zotov edited comment on CASSANDRA-16360 at 4/30/21, 11:59 AM: - [~samt] That makes sense. I'll put this ticket on hold. However, in order to get the existing microbenchmark changes, I've raised CASSANDRA-16645. Please, take a look once you have a chance. was (Author: azotcsit): [~beobal] That makes sense. I'll put this ticket on hold. However, in order to get the existing microbenchmark changes, I've raised CASSANDRA-16645. Please, take a look once you have a chance. > CRC32 is inefficient on x86 > --- > > Key: CASSANDRA-16360 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16360 > Project: Cassandra > Issue Type: Improvement > Components: Messaging/Client >Reporter: Avi Kivity >Priority: Normal > Labels: protocolv6 > Fix For: 4.x > > > The client/server protocol specifies CRC24 and CRC32 as the checksum > algorithm (cql_protocol_V5_framing.asc). Those however are expensive to > compute; this affects both the client and the server. > > A better checksum algorithm is CRC32C, which has hardware support on x86 (as > well as other modern architectures). -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16360) CRC32 is inefficient on x86
[ https://issues.apache.org/jira/browse/CASSANDRA-16360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17337324#comment-17337324 ] Alexey Zotov commented on CASSANDRA-16360: -- [~beobal] That makes sense. I'll put this ticket on hold. However, in order to get the existing microbenchmark changes, I've raised CASSANDRA-16645. Please, take a look once you have a chance. > CRC32 is inefficient on x86 > --- > > Key: CASSANDRA-16360 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16360 > Project: Cassandra > Issue Type: Improvement > Components: Messaging/Client >Reporter: Avi Kivity >Priority: Normal > Labels: protocolv6 > Fix For: 4.x > > > The client/server protocol specifies CRC24 and CRC32 as the checksum > algorithm (cql_protocol_V5_framing.asc). Those however are expensive to > compute; this affects both the client and the server. > > A better checksum algorithm is CRC32C, which has hardware support on x86 (as > well as other modern architectures). -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-16645) Add a microbenchmark for checksum calculation
[
https://issues.apache.org/jira/browse/CASSANDRA-16645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexey Zotov updated CASSANDRA-16645:
-
Description:
h2. Overview
As a part CASSANDRA-16360 there was a need to change the checksum algo from
_CRC32_ to _CRC32C_ for client/internode messaging. Before making that change
it would be nice to have a way to measure performance for checksum calculation
(similarly to {{HashingBench}} for hash calculation).
In fact, the microbenchmark has been already created as a part of that ticket.
However, after running it we saw a performance degradation for Java 8 and
decided to wait until Java 8 support is abandoned. The purpose of this ticket
is to merge the benchmark changes separately.
h2. What needs to be done
* Create a microbenchmark
h2. Link to PR
https://github.com/apache/cassandra/pull/951/files
was:
h2. Overview
As a part CASSANDRA-16360 there was a need to change the checksum algo from
_CRC32_ to _CRC32C_ for client/internode messaging. Before making that change
it would be nice to have a way to measure performance for checksum calculation
(similarly to {{HashingBench}} for hash calculation).
In fact, the microbenchmark has been already created as a part of that ticket.
However, after running it we saw a performance degradation for Java 8 and
decided to wait until Java 8 support is abandoned. The purpose of this ticket
is to merge the benchmark changes separately.
h2. What needs to be done
* Create a microbenchmark
> Add a microbenchmark for checksum calculation
> -
>
> Key: CASSANDRA-16645
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16645
> Project: Cassandra
> Issue Type: Improvement
> Components: Test/benchmark
>Reporter: Alexey Zotov
>Assignee: Alexey Zotov
>Priority: Low
> Fix For: 4.0.x
>
>
> h2. Overview
> As a part CASSANDRA-16360 there was a need to change the checksum algo from
> _CRC32_ to _CRC32C_ for client/internode messaging. Before making that change
> it would be nice to have a way to measure performance for checksum
> calculation (similarly to {{HashingBench}} for hash calculation).
> In fact, the microbenchmark has been already created as a part of that
> ticket. However, after running it we saw a performance degradation for Java 8
> and decided to wait until Java 8 support is abandoned. The purpose of this
> ticket is to merge the benchmark changes separately.
> h2. What needs to be done
> * Create a microbenchmark
> h2. Link to PR
> https://github.com/apache/cassandra/pull/951/files
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Assigned] (CASSANDRA-16360) CRC32 is inefficient on x86
[ https://issues.apache.org/jira/browse/CASSANDRA-16360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Zotov reassigned CASSANDRA-16360: Assignee: (was: Alexey Zotov) > CRC32 is inefficient on x86 > --- > > Key: CASSANDRA-16360 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16360 > Project: Cassandra > Issue Type: Improvement > Components: Messaging/Client >Reporter: Avi Kivity >Priority: Normal > Labels: protocolv6 > Fix For: 4.x > > > The client/server protocol specifies CRC24 and CRC32 as the checksum > algorithm (cql_protocol_V5_framing.asc). Those however are expensive to > compute; this affects both the client and the server. > > A better checksum algorithm is CRC32C, which has hardware support on x86 (as > well as other modern architectures). -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-16645) Add a microbenchmark for checksum calculation
[
https://issues.apache.org/jira/browse/CASSANDRA-16645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexey Zotov updated CASSANDRA-16645:
-
Priority: Low (was: Normal)
> Add a microbenchmark for checksum calculation
> -
>
> Key: CASSANDRA-16645
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16645
> Project: Cassandra
> Issue Type: Improvement
> Components: Test/benchmark
>Reporter: Alexey Zotov
>Assignee: Alexey Zotov
>Priority: Low
>
> h2. Overview
> As a part CASSANDRA-16360 there was a need to change the checksum algo from
> _CRC32_ to _CRC32C_ for client/internode messaging. Before making that change
> it would be nice to have a way to measure performance for checksum
> calculation (similarly to {{HashingBench}} for hash calculation).
> In fact, the microbenchmark has been already created as a part of that
> ticket. However, after running it we saw a performance degradation for Java 8
> and decided to wait until Java 8 support is abandoned. The purpose of this
> ticket is to merge the benchmark changes separately.
> h2. What needs to be done
> * Create a microbenchmark
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-16645) Add a microbenchmark for checksum calculation
[
https://issues.apache.org/jira/browse/CASSANDRA-16645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexey Zotov updated CASSANDRA-16645:
-
Fix Version/s: 4.0.x
> Add a microbenchmark for checksum calculation
> -
>
> Key: CASSANDRA-16645
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16645
> Project: Cassandra
> Issue Type: Improvement
> Components: Test/benchmark
>Reporter: Alexey Zotov
>Assignee: Alexey Zotov
>Priority: Low
> Fix For: 4.0.x
>
>
> h2. Overview
> As a part CASSANDRA-16360 there was a need to change the checksum algo from
> _CRC32_ to _CRC32C_ for client/internode messaging. Before making that change
> it would be nice to have a way to measure performance for checksum
> calculation (similarly to {{HashingBench}} for hash calculation).
> In fact, the microbenchmark has been already created as a part of that
> ticket. However, after running it we saw a performance degradation for Java 8
> and decided to wait until Java 8 support is abandoned. The purpose of this
> ticket is to merge the benchmark changes separately.
> h2. What needs to be done
> * Create a microbenchmark
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-16645) Add a microbenchmark for checksum calculation
[
https://issues.apache.org/jira/browse/CASSANDRA-16645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexey Zotov updated CASSANDRA-16645:
-
Change Category: Performance
Complexity: Low Hanging Fruit
Status: Open (was: Triage Needed)
> Add a microbenchmark for checksum calculation
> -
>
> Key: CASSANDRA-16645
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16645
> Project: Cassandra
> Issue Type: Improvement
> Components: Test/benchmark
>Reporter: Alexey Zotov
>Assignee: Alexey Zotov
>Priority: Low
> Fix For: 4.0.x
>
>
> h2. Overview
> As a part CASSANDRA-16360 there was a need to change the checksum algo from
> _CRC32_ to _CRC32C_ for client/internode messaging. Before making that change
> it would be nice to have a way to measure performance for checksum
> calculation (similarly to {{HashingBench}} for hash calculation).
> In fact, the microbenchmark has been already created as a part of that
> ticket. However, after running it we saw a performance degradation for Java 8
> and decided to wait until Java 8 support is abandoned. The purpose of this
> ticket is to merge the benchmark changes separately.
> h2. What needs to be done
> * Create a microbenchmark
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-16645) Add a microbenchmark for checksum calculation
Alexey Zotov created CASSANDRA-16645:
Summary: Add a microbenchmark for checksum calculation
Key: CASSANDRA-16645
URL: https://issues.apache.org/jira/browse/CASSANDRA-16645
Project: Cassandra
Issue Type: Improvement
Components: Test/benchmark
Reporter: Alexey Zotov
Assignee: Alexey Zotov
h2. Overview
As a part CASSANDRA-16360 there was a need to change the checksum algo from
_CRC32_ to _CRC32C_ for client/internode messaging. Before making that change
it would be nice to have a way to measure performance for checksum calculation
(similarly to {{HashingBench}} for hash calculation).
In fact, the microbenchmark has been already created as a part of that ticket.
However, after running it we saw a performance degradation for Java 8 and
decided to wait until Java 8 support is abandoned. The purpose of this ticket
is to merge the benchmark changes separately.
h2. What needs to be done
* Create a microbenchmark
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16630) Migrate to JUnit5
[
https://issues.apache.org/jira/browse/CASSANDRA-16630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17337186#comment-17337186
]
Alexey Zotov commented on CASSANDRA-16630:
--
These are details I was looking for. Thanks a lot!
> Migrate to JUnit5
> -
>
> Key: CASSANDRA-16630
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16630
> Project: Cassandra
> Issue Type: Improvement
> Components: Test/unit
>Reporter: Alexey Zotov
>Assignee: Alexey Zotov
>Priority: Low
>
> h3. Overview
> Currently C* uses JUnit4 (version 4.12) which is obsolete. There is a newer
> version 4.13.2 which we could update to. However, JUnit4 is generally
> considered to be outdated and it is reasonable to migrate to JUnit5.
> Despite of having a syntax sugar in JUnit5 (assertThrow, lamda's support,
> ect), there are no blockers that push us to move from JUnit4. The main
> motivation for this initiative is rule of thumb to use up-to-date versions of
> the dependencies.
> Obviously this change is not backward compatible with the open PRs and
> previous C* versions. Therefore, it will require an additional effort for
> backporting the changes and updating PRs that have tests. However, I believe
> it should not be a blocker for this initiative.
> h3. Scope (preliminary list)
> # change JUnit4 to JUnit5 dependencies and make necessary changes in ant
> tasks (https://ant.apache.org/manual/Tasks/junitlauncher.html)
> # update syntax in all tests (imports, Before/After annotations, etc)
> # update parameterized tests
> # create a new version of {{OrderedJUnit4ClassRunner}} and update
> corresponding tests
> # update tests that use {{BMUnitRunner}} (as per
> https://developer.jboss.org/docs/DOC-52953 it supports JUnit5)
> # update tests with {{@Rule}}
> # update tests with expected exceptions
> h3. Order of operations
> In order to make the transition more smooth we want to use a phased approach:
> # migrate to JUnit5 with [Vintage
> Engine|https://junit.org/junit5/docs/current/user-guide/#dependency-metadata-junit-vintage],
> so all JUnit4 tests work as is
> # update tests in a few bunches (to not have a huge single PR with numerous
> conflicts)
> # disable (remove dependency) Vintage Engine, so only JUnit5 tests work
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Assigned] (CASSANDRA-16630) Migrate to JUnit5
[
https://issues.apache.org/jira/browse/CASSANDRA-16630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexey Zotov reassigned CASSANDRA-16630:
Assignee: Alexey Zotov
> Migrate to JUnit5
> -
>
> Key: CASSANDRA-16630
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16630
> Project: Cassandra
> Issue Type: Improvement
> Components: Test/unit
>Reporter: Alexey Zotov
>Assignee: Alexey Zotov
>Priority: Normal
>
> h3. Overview
> Currently C* uses JUnit4 (version 4.12) which is obsolete. There is a newer
> version 4.13.2 which we could update to. However, JUnit4 is generally
> considered to be outdated and it is reasonable to migrate to JUnit5.
> Despite of having a syntax sugar in JUnit5 (assertThrow, lamda's support,
> ect), there are no blockers that push us to move from JUnit4. The main
> motivation for this initiative is rule of thumb to use up-to-date versions of
> the dependencies.
> Obviously this change is not backward compatible with the open PRs and
> previous C* versions. Therefore, it will require an additional effort for
> backporting the changes and updating PRs that have tests. However, I believe
> it should not be a blocker for this initiative.
> h3. Scope (preliminary list)
> # change JUnit4 to JUnit5 dependencies and make necessary changes in ant
> tasks (https://ant.apache.org/manual/Tasks/junitlauncher.html)
> # update syntax in all tests (imports, Before/After annotations, etc)
> # update parameterized tests
> # create a new version of {{OrderedJUnit4ClassRunner}} and update
> corresponding tests
> # update tests that use {{BMUnitRunner}} (as per
> https://developer.jboss.org/docs/DOC-52953 it supports JUnit5)
> # update tests with {{@Rule}}
> # update tests with expected exceptions
> h3. Order of operations
> In order to make the transition more smooth we want to use a phased approach:
> # migrate to JUnit5 with [Vintage
> Engine|https://junit.org/junit5/docs/current/user-guide/#dependency-metadata-junit-vintage],
> so all JUnit4 tests work as is
> # update tests in a few bunches (to not have a huge single PR with numerous
> conflicts)
> # disable (remove dependency) Vintage Engine, so only JUnit5 tests work
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-16630) Migrate to JUnit5
[
https://issues.apache.org/jira/browse/CASSANDRA-16630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexey Zotov updated CASSANDRA-16630:
-
Change Category: Quality Assurance
Complexity: Normal
Priority: Low (was: Normal)
Status: Open (was: Triage Needed)
> Migrate to JUnit5
> -
>
> Key: CASSANDRA-16630
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16630
> Project: Cassandra
> Issue Type: Improvement
> Components: Test/unit
>Reporter: Alexey Zotov
>Assignee: Alexey Zotov
>Priority: Low
>
> h3. Overview
> Currently C* uses JUnit4 (version 4.12) which is obsolete. There is a newer
> version 4.13.2 which we could update to. However, JUnit4 is generally
> considered to be outdated and it is reasonable to migrate to JUnit5.
> Despite of having a syntax sugar in JUnit5 (assertThrow, lamda's support,
> ect), there are no blockers that push us to move from JUnit4. The main
> motivation for this initiative is rule of thumb to use up-to-date versions of
> the dependencies.
> Obviously this change is not backward compatible with the open PRs and
> previous C* versions. Therefore, it will require an additional effort for
> backporting the changes and updating PRs that have tests. However, I believe
> it should not be a blocker for this initiative.
> h3. Scope (preliminary list)
> # change JUnit4 to JUnit5 dependencies and make necessary changes in ant
> tasks (https://ant.apache.org/manual/Tasks/junitlauncher.html)
> # update syntax in all tests (imports, Before/After annotations, etc)
> # update parameterized tests
> # create a new version of {{OrderedJUnit4ClassRunner}} and update
> corresponding tests
> # update tests that use {{BMUnitRunner}} (as per
> https://developer.jboss.org/docs/DOC-52953 it supports JUnit5)
> # update tests with {{@Rule}}
> # update tests with expected exceptions
> h3. Order of operations
> In order to make the transition more smooth we want to use a phased approach:
> # migrate to JUnit5 with [Vintage
> Engine|https://junit.org/junit5/docs/current/user-guide/#dependency-metadata-junit-vintage],
> so all JUnit4 tests work as is
> # update tests in a few bunches (to not have a huge single PR with numerous
> conflicts)
> # disable (remove dependency) Vintage Engine, so only JUnit5 tests work
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16630) Migrate to JUnit5
[
https://issues.apache.org/jira/browse/CASSANDRA-16630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17335842#comment-17335842
]
Alexey Zotov commented on CASSANDRA-16630:
--
Thanks [~wakingrufus]! I'll let you know once I have the changes for {{ant}}
part and the build works. So we can split the changes on tests.
[~blerer] [~mck]
I'm looking to the changes related to JUnit5 migration and looks like there are
a few large enough ({{jstack}} task wrapper and formatters). I have a question
on the formatters. Is it critical for us to save the existing output as is?
Basically I'd like to understand whether there is smth based on the format
(e.g. a component that parses the output). Definitely I will try to keep format
the same (I have not yet tried to update them), but I want to ensure it is
worth the effort.
> Migrate to JUnit5
> -
>
> Key: CASSANDRA-16630
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16630
> Project: Cassandra
> Issue Type: Improvement
> Components: Test/unit
>Reporter: Alexey Zotov
>Priority: Normal
>
> h3. Overview
> Currently C* uses JUnit4 (version 4.12) which is obsolete. There is a newer
> version 4.13.2 which we could update to. However, JUnit4 is generally
> considered to be outdated and it is reasonable to migrate to JUnit5.
> Despite of having a syntax sugar in JUnit5 (assertThrow, lamda's support,
> ect), there are no blockers that push us to move from JUnit4. The main
> motivation for this initiative is rule of thumb to use up-to-date versions of
> the dependencies.
> Obviously this change is not backward compatible with the open PRs and
> previous C* versions. Therefore, it will require an additional effort for
> backporting the changes and updating PRs that have tests. However, I believe
> it should not be a blocker for this initiative.
> h3. Scope (preliminary list)
> # change JUnit4 to JUnit5 dependencies and make necessary changes in ant
> tasks (https://ant.apache.org/manual/Tasks/junitlauncher.html)
> # update syntax in all tests (imports, Before/After annotations, etc)
> # update parameterized tests
> # create a new version of {{OrderedJUnit4ClassRunner}} and update
> corresponding tests
> # update tests that use {{BMUnitRunner}} (as per
> https://developer.jboss.org/docs/DOC-52953 it supports JUnit5)
> # update tests with {{@Rule}}
> # update tests with expected exceptions
> h3. Order of operations
> In order to make the transition more smooth we want to use a phased approach:
> # migrate to JUnit5 with [Vintage
> Engine|https://junit.org/junit5/docs/current/user-guide/#dependency-metadata-junit-vintage],
> so all JUnit4 tests work as is
> # update tests in a few bunches (to not have a huge single PR with numerous
> conflicts)
> # disable (remove dependency) Vintage Engine, so only JUnit5 tests work
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-16360) CRC32 is inefficient on x86
[ https://issues.apache.org/jira/browse/CASSANDRA-16360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Zotov updated CASSANDRA-16360: - Fix Version/s: (was: 4.0.x) 4.x > CRC32 is inefficient on x86 > --- > > Key: CASSANDRA-16360 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16360 > Project: Cassandra > Issue Type: Improvement > Components: Messaging/Client >Reporter: Avi Kivity >Assignee: Alexey Zotov >Priority: Normal > Labels: protocolv6 > Fix For: 4.x > > > The client/server protocol specifies CRC24 and CRC32 as the checksum > algorithm (cql_protocol_V5_framing.asc). Those however are expensive to > compute; this affects both the client and the server. > > A better checksum algorithm is CRC32C, which has hardware support on x86 (as > well as other modern architectures). -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-16360) CRC32 is inefficient on x86
[
https://issues.apache.org/jira/browse/CASSANDRA-16360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17335219#comment-17335219
]
Alexey Zotov edited comment on CASSANDRA-16360 at 4/29/21, 8:11 AM:
Oh ok, I got your point. You've been talking about lookup table caching rather
than data caching. That makes perfect sense to me now. I definitely agree with
your points regarding complexity of its testing and lack of the real necessity
to do it. Therefore:
* I'm parking the benchmark code as is (unless some changes are requested)
* I'm waiting for the decision whether I can currently proceed with step 7
from [~samt]
* I'll be waiting for 4.x where we have no Java 8 support
PS:
{quote}That said, you may well be benchmarking on a machine without crc32c
hardware support, anyway. Which would explain the very similar performance
profile.
{quote}
That's what I've checked on the early stage of the local benchmarking. I have
[i7-7500u|https://ark.intel.com/content/www/us/en/ark/products/95451/intel-core-i7-7500u-processor-4m-cache-up-to-3-50-ghz.html]
which supports SSE4.2 instructions set which in turn contains [CRC
operation|https://en.wikipedia.org/wiki/SSE4#SSE4.2] for _CRC32C_ calculation.
was (Author: azotcsit):
Oh ok, I got your point. You've been talking about lookup table caching rather
than data caching. That makes perfect sense to me now. I definitely agree with
your points regarding complexity of its testing and lack of the real necessity
to do it. Therefore:
* I'm parking the benchmark code as is (unless some changes are requested)
* I'm waiting for the decision whether I can currently proceed with step 7
from [~samt]
* I'll be waiting for 4.x where we have no Java 8 support
PS:
{quote}That said, you may well be benchmarking on a machine without crc32c
hardware support, anyway. Which would explain the very similar performance
profile.
{quote}
That's what I've checked on the early stage of the local benchmarking. I have
[i7-7500u|https://ark.intel.com/content/www/us/en/ark/products/95451/intel-core-i7-7500u-processor-4m-cache-up-to-3-50-ghz.html]
which support SSE4.2 instructions set which in turn contains [CRC
operation|https://en.wikipedia.org/wiki/SSE4#SSE4.2] for _CRC32C_ calculation.
> CRC32 is inefficient on x86
> ---
>
> Key: CASSANDRA-16360
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16360
> Project: Cassandra
> Issue Type: Improvement
> Components: Messaging/Client
>Reporter: Avi Kivity
>Assignee: Alexey Zotov
>Priority: Normal
> Labels: protocolv6
> Fix For: 4.0.x
>
>
> The client/server protocol specifies CRC24 and CRC32 as the checksum
> algorithm (cql_protocol_V5_framing.asc). Those however are expensive to
> compute; this affects both the client and the server.
>
> A better checksum algorithm is CRC32C, which has hardware support on x86 (as
> well as other modern architectures).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16360) CRC32 is inefficient on x86
[
https://issues.apache.org/jira/browse/CASSANDRA-16360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17335219#comment-17335219
]
Alexey Zotov commented on CASSANDRA-16360:
--
Oh ok, I got your point. You've been talking about lookup table caching rather
than data caching. That makes perfect sense to me now. I definitely agree with
your points regarding complexity of its testing and lack of the real necessity
to do it. Therefore:
* I'm parking the benchmark code as is (unless some changes are requested)
* I'm waiting for the decision whether I can currently proceed with step 7
from [~samt]
* I'll be waiting for 4.x where we have no Java 8 support
PS:
{quote}That said, you may well be benchmarking on a machine without crc32c
hardware support, anyway. Which would explain the very similar performance
profile.
{quote}
That's what I've checked on the early stage of the local benchmarking. I have
[i7-7500u|https://ark.intel.com/content/www/us/en/ark/products/95451/intel-core-i7-7500u-processor-4m-cache-up-to-3-50-ghz.html]
which support SSE4.2 instructions set which in turn contains [CRC
operation|https://en.wikipedia.org/wiki/SSE4#SSE4.2] for _CRC32C_ calculation.
> CRC32 is inefficient on x86
> ---
>
> Key: CASSANDRA-16360
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16360
> Project: Cassandra
> Issue Type: Improvement
> Components: Messaging/Client
>Reporter: Avi Kivity
>Assignee: Alexey Zotov
>Priority: Normal
> Labels: protocolv6
> Fix For: 4.0.x
>
>
> The client/server protocol specifies CRC24 and CRC32 as the checksum
> algorithm (cql_protocol_V5_framing.asc). Those however are expensive to
> compute; this affects both the client and the server.
>
> A better checksum algorithm is CRC32C, which has hardware support on x86 (as
> well as other modern architectures).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16360) CRC32 is inefficient on x86
[
https://issues.apache.org/jira/browse/CASSANDRA-16360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17335042#comment-17335042
]
Alexey Zotov commented on CASSANDRA-16360:
--
Thanks for the feedback!
[~benedict]
# I'm glad to hear that is smth useful. Though I'm not sure how to get the
existing PR merged. Please, let me know if I need to take any further actions
(e.g. update {{CHANGES.txt}}), otherwise I'll be just waiting.
# Ok, sounds like a good suggestion. I'll add it to the list of further action
items.
# That's what I thought too. Then I'll just update the fix version to 4.x and
put the work on hold for now.
{quote}That said, it looks like there might be some measurement error in your
results, and the crc32 intrinsic version may be benefitting from tight loop
benchmarking (so high cache occupancy). It certainly doesn't look like a slam
dunk for urgent migration anyway.{quote}
That's an interesting point. My feeling was that all the implementations
equally benefit from the cache occupancy. Therefore, it should not affect the
final results significantly (obviously it should affects the total numbers, but
not the relative numbers). At the end of the day, the origin of the
optimization is more about the number of CPU cycles required to calculate the
checksum which does not seem to be directly related to the cache occupancy.
I guess you expected to see a bigger improvement for _CRC32C_ (compared to
_CRC32_) and that's what causes the suspicion. I'm just curious, why do you
think there should be a bigger improvement and what kind of difference did you
expect?
In the meantime, I'll try to start generating a large byte array and calculate
hashes for slices (sub-arrays). By doing that, we'll ensure that data for hash
calculation is not cached because every time we calculate the hash for a new
slice. I'll raise one more experimental PR, so we can see the results and
re-visit the conclusion if needed.
[~samt]
I was going to start asking about clients compatibility a bit later, but since
you brought this point let me raise a couple of questions:
# Obviously there are multiple clients (java, python, etc). How should it be
approached? Do I just need to submit tickets to the corresponding projects?
Where can I find the whole list of clients? I just want to understand the high
level steps (no need to provide a detailed step-by-step instruction, it is
simply too early for that).
# Am I right that {{InboundConnectionInitiator / OutboundConnectionInitiator /
HandshakeProtocol}} are about internode communication, whereas
{{PipelineConfigurator / InitialConnectionHandler}} are about client
communication? Am I missing smth here?
{quote}If you implement step 7 so that the encoders/decoders support both CRC32
and CRC32C then this becomes two separate (possibly parallelisable) pieces of
work.{quote}
You suggestion makes perfect sense to me! Basically I can proceed with this
change only (it would be just a small refactoring/re-design without changing
the existing behavior). As a result of the change, {{ChecksumType.CRC32}} will
be used in decoders/encoders (as stated in step 7). By doing that, we will need
much less changes in the future when we actually start adding _CRC32C_ support.
Do you think it is worth to make this part now or we should wait and make the
full change later on?
PS:
Even though we're going to complete this ticket now and have to wait until Java
8 support in C* is abandoned, I'm still encouraged to get it done. If by some
reason I miss the moment when the full blown work can be started, please, ping
me :)
> CRC32 is inefficient on x86
> ---
>
> Key: CASSANDRA-16360
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16360
> Project: Cassandra
> Issue Type: Improvement
> Components: Messaging/Client
>Reporter: Avi Kivity
>Assignee: Alexey Zotov
>Priority: Normal
> Labels: protocolv6
> Fix For: 4.0.x
>
>
> The client/server protocol specifies CRC24 and CRC32 as the checksum
> algorithm (cql_protocol_V5_framing.asc). Those however are expensive to
> compute; this affects both the client and the server.
>
> A better checksum algorithm is CRC32C, which has hardware support on x86 (as
> well as other modern architectures).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16404) Provide a nodetool way of invalidating auth caches
[ https://issues.apache.org/jira/browse/CASSANDRA-16404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17332735#comment-17332735 ] Alexey Zotov commented on CASSANDRA-16404: -- It turned out that AssertJ is preferable library for test assertions. I've converted all assertions in the PR from Hamcrest to AssertJ. > Provide a nodetool way of invalidating auth caches > -- > > Key: CASSANDRA-16404 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16404 > Project: Cassandra > Issue Type: Improvement > Components: Feature/Authorization >Reporter: Sumanth Pasupuleti >Assignee: Alexey Zotov >Priority: Normal > Fix For: 4.x > > Time Spent: 0.5h > Remaining Estimate: 0h > > We currently have nodetool commands to invalidate certain caches like > KeyCache, RowCache and CounterCache. > Being able to invalidate auth caches as well can come in handy in situations > where, critical backend auth changes may need to be in effect right away for > all the connections, especially in configurations where cache validity is > chosen to be for a longer duration. An example can be that an authenticated > user "User1" is no longer authorized to access a table resource "table1" and > it is vital that this change is reflected right away, without having to wait > for cache expiry/refresh to trigger. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16629) Rafactor tests for NodeTool commands
[
https://issues.apache.org/jira/browse/CASSANDRA-16629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17332719#comment-17332719
]
Alexey Zotov commented on CASSANDRA-16629:
--
{quote}I do not know how Hamcrest was introduced.
{quote}
Not sure, but most of the {{NodeTool}} tests were Hamcrest-based. That's why
originally I converted the rest from AssertJ to Hamcrest. Since now it is clear
that AssertJ is preferable, I've converted all tests to AssertJ. Looking
forward for review comments.
Thanks for providing a great support!
> Rafactor tests for NodeTool commands
>
>
> Key: CASSANDRA-16629
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16629
> Project: Cassandra
> Issue Type: Task
> Components: Test/unit
>Reporter: Alexey Zotov
>Assignee: Alexey Zotov
>Priority: Low
> Fix For: 4.0.x
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> h3. Overview
> As a part of CASSANDRA-16404, I've been working on {{NodeTool}} commands for
> invalidation of the auth caches. While working on that, I've mentioned that
> the existing NodeTool unit tests are written in a different manner and reside
> in wrong packages. I've made the following changes to them:
> # moved them to the proper packages (that match original command class)
> # unified naming, formatting, imports
> # migrated from {{org.assertj}} to {{hamcrest}} for assertions (though it is
> not smth really required)
> # made a general cleanup
> PR: https://github.com/apache/cassandra/pull/982
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-16630) Migrate to JUnit5
[
https://issues.apache.org/jira/browse/CASSANDRA-16630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexey Zotov updated CASSANDRA-16630:
-
Description:
h3. Overview
Currently C* uses JUnit4 (version 4.12) which is obsolete. There is a newer
version 4.13.2 which we could update to. However, JUnit4 is generally
considered to be outdated and it is reasonable to migrate to JUnit5.
Despite of having a syntax sugar in JUnit5 (assertThrow, lamda's support, ect),
there are no blockers that push us to move from JUnit4. The main motivation for
this initiative is rule of thumb to use up-to-date versions of the dependencies.
Obviously this change is not backward compatible with the open PRs and previous
C* versions. Therefore, it will require an additional effort for backporting
the changes and updating PRs that have tests. However, I believe it should not
be a blocker for this initiative.
h3. Scope (preliminary list)
# change JUnit4 to JUnit5 dependencies and make necessary changes in ant tasks
(https://ant.apache.org/manual/Tasks/junitlauncher.html)
# update syntax in all tests (imports, Before/After annotations, etc)
# update parameterized tests
# create a new version of {{OrderedJUnit4ClassRunner}} and update corresponding
tests
# update tests that use {{BMUnitRunner}} (as per
https://developer.jboss.org/docs/DOC-52953 it supports JUnit5)
# update tests with {{@Rule}}
# update tests with expected exceptions
h3. Order of operations
In order to make the transition more smooth we want to use a phased approach:
# migrate to JUnit5 with [Vintage
Engine|https://junit.org/junit5/docs/current/user-guide/#dependency-metadata-junit-vintage],
so all JUnit4 tests work as is
# update tests in a few bunches (to not have a huge single PR with numerous
conflicts)
# disable (remove dependency) Vintage Engine, so only JUnit5 tests work
was:
h3. Overview
Currently C* uses JUnit4 (version 4.12) which is obsolete. There is a newer
version 4.13.2 which we could update to. However, JUnit4 is generally
considered to be outdated and it is reasonable to migrate to JUnit5.
Despite of having a syntax sugar in JUnit5 (assertThrow, lamda's support, ect),
there are no blockers that push us to move from JUnit4. The main motivation for
this initiative is rule of thumb to use up-to-date versions of the dependencies.
Obviously this change is not backward compatible with the open PRs and previous
C* versions. Therefore, it will require an additional effort for backporting
the changes and updating PRs that have tests. However, I believe it should not
be a blocker for this initiative.
h3. Scope (preliminary list)
# change JUnit4 to JUnit5 dependencies and make necessary changes in ant tasks
(https://ant.apache.org/manual/Tasks/junitlauncher.html)
# update syntax in all tests (imports, Before/After annotations, etc)
# update parameterized tests
# create a new version of {{OrderedJUnit4ClassRunner}} and update corresponding
tests
# update tests that use {{BMUnitRunner}} (as per
https://developer.jboss.org/docs/DOC-52953 it supports JUnit5)
# update tests with {{@Rule}}
# update tests with expected exceptions
h3. Order of operations
In order to make the transition more smooth we want to use a phased approach:
# migrate to JUnit5 with [Vintage
Engine|https://junit.org/junit5/docs/current/user-guide/#dependency-metadata-junit-vintage],
so all JUnit4 tests work as is
# update tests in a few bunches (to not have a huge single PR with a bunch of
conflicts)
# disable (remove dependency) Vintage Engine, so only JUnit5 tests work
> Migrate to JUnit5
> -
>
> Key: CASSANDRA-16630
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16630
> Project: Cassandra
> Issue Type: Improvement
> Components: Test/unit
>Reporter: Alexey Zotov
>Priority: Normal
>
> h3. Overview
> Currently C* uses JUnit4 (version 4.12) which is obsolete. There is a newer
> version 4.13.2 which we could update to. However, JUnit4 is generally
> considered to be outdated and it is reasonable to migrate to JUnit5.
> Despite of having a syntax sugar in JUnit5 (assertThrow, lamda's support,
> ect), there are no blockers that push us to move from JUnit4. The main
> motivation for this initiative is rule of thumb to use up-to-date versions of
> the dependencies.
> Obviously this change is not backward compatible with the open PRs and
> previous C* versions. Therefore, it will require an additional effort for
> backporting the changes and updating PRs that have tests. However, I believe
> it should not be a blocker for this initiative.
> h3. Scope (preliminary list)
> # change JUnit4 to JUnit5 dependencies and make necessary changes in ant
> tasks (https://ant.apache.org/manual/Tasks/junitlauncher.html)
> # update syntax in all tests (imports, Before/After annotations, etc)
>
[jira] [Updated] (CASSANDRA-16630) Migrate to JUnit5
[
https://issues.apache.org/jira/browse/CASSANDRA-16630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexey Zotov updated CASSANDRA-16630:
-
Description:
h3. Overview
Currently C* uses JUnit4 (version 4.12) which is obsolete. There is a newer
version 4.13.2 which we could update to. However, JUnit4 is generally
considered to be outdated and it is reasonable to migrate to JUnit5.
Despite of having a syntax sugar in JUnit5 (assertThrow, lamda's support, ect),
there are no blockers that push us to move from JUnit4. The main motivation for
this initiative is rule of thumb to use up-to-date versions of the dependencies.
Obviously this change is not backward compatible with the open PRs and previous
C* versions. Therefore, it will require an additional effort for backporting
the changes and updating PRs that have tests. However, I believe it should not
be a blocker for this initiative.
h3. Scope (preliminary list)
# change JUnit4 to JUnit5 dependencies and make necessary changes in ant tasks
(https://ant.apache.org/manual/Tasks/junitlauncher.html)
# update syntax in all tests (imports, Before/After annotations, etc)
# update parameterized tests
# create a new version of {{OrderedJUnit4ClassRunner}} and update corresponding
tests
# update tests that use {{BMUnitRunner}} (as per
https://developer.jboss.org/docs/DOC-52953 it supports JUnit5)
# update tests with {{@Rule}}
# update tests with expected exceptions
h3. Order of operations
In order to make the transition more smooth we want to use a phased approach:
# migrate to JUnit5 with [Vintage
Engine|https://junit.org/junit5/docs/current/user-guide/#dependency-metadata-junit-vintage],
so all JUnit4 tests work as is
# update tests in a few bunches (to not have a huge single PR with a bunch of
conflicts)
# disable (remove dependency) Vintage Engine, so only JUnit5 tests work
was:
h3. Overview
Currently C* uses JUnit4 (version 4.12) which is obsolete. There is a newer
version 4.13.2 which we could update to. However, JUnit4 is generally
considered to be outdated and it is reasonable to migrate to JUnit5.
Despite of having a syntax sugar in JUnit5 (assertThrow, lamda's support, ect),
there are no blockers that push us to move from JUnit4. The main motivation for
this initiative is rule of thumb to use up-to-date versions of the dependencies.
Obviously this change is not backward compatible with the open PRs and previous
C* versions. Therefore, it will require an additional effort for backporting
the changes and updating PRs that have tests. However, I believe it should not
be a blocker for this initiative.
h3. Scope (preliminary list)
# change JUnit4 to JUnit5 dependencies and make necessary changes in ant tasks
(https://ant.apache.org/manual/Tasks/junitlauncher.html)
# update syntax in all tests (imports, Before/After annotations, etc)
# update parameterized tests
# create a new version of {{OrderedJUnit4ClassRunner}} and update corresponding
tests
# update tests that use {{BMUnitRunner}} (as per
https://developer.jboss.org/docs/DOC-52953 it supports JUnit5)
# update tests with {{@Rule}}
# update tests with expected exceptions
h3. Order of operations
In order to make the transition more smooth we want to use a phased approach:
1. migrate to JUnit5 with [Vintage
Engine|https://junit.org/junit5/docs/current/user-guide/#dependency-metadata-junit-vintage],
so all JUnit4 tests work as is
2. update tests in a few bunches (to not have a huge single PR with a bunch of
conflicts)
3. disable (remove dependency) Vintage Engine, so only JUnit5 tests work
> Migrate to JUnit5
> -
>
> Key: CASSANDRA-16630
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16630
> Project: Cassandra
> Issue Type: Improvement
> Components: Test/unit
>Reporter: Alexey Zotov
>Priority: Normal
>
> h3. Overview
> Currently C* uses JUnit4 (version 4.12) which is obsolete. There is a newer
> version 4.13.2 which we could update to. However, JUnit4 is generally
> considered to be outdated and it is reasonable to migrate to JUnit5.
> Despite of having a syntax sugar in JUnit5 (assertThrow, lamda's support,
> ect), there are no blockers that push us to move from JUnit4. The main
> motivation for this initiative is rule of thumb to use up-to-date versions of
> the dependencies.
> Obviously this change is not backward compatible with the open PRs and
> previous C* versions. Therefore, it will require an additional effort for
> backporting the changes and updating PRs that have tests. However, I believe
> it should not be a blocker for this initiative.
> h3. Scope (preliminary list)
> # change JUnit4 to JUnit5 dependencies and make necessary changes in ant
> tasks (https://ant.apache.org/manual/Tasks/junitlauncher.html)
> # update syntax in all tests (imports, Before/After annotations,
[jira] [Updated] (CASSANDRA-16630) Migrate to JUnit5
[
https://issues.apache.org/jira/browse/CASSANDRA-16630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexey Zotov updated CASSANDRA-16630:
-
Description:
h3. Overview
Currently C* uses JUnit4 (version 4.12) which is obsolete. There is a newer
version 4.13.2 which we could update to. However, JUnit4 is generally
considered to be outdated and it is reasonable to migrate to JUnit5.
Despite of having a syntax sugar in JUnit5 (assertThrow, lamda's support, ect),
there are no blockers that push us to move from JUnit4. The main motivation for
this initiative is rule of thumb to use up-to-date versions of the dependencies.
Obviously this change is not backward compatible with the open PRs and previous
C* versions. Therefore, it will require an additional effort for backporting
the changes and updating PRs that have tests. However, I believe it should not
be a blocker for this initiative.
h3. Scope (preliminary list)
# change JUnit4 to JUnit5 dependencies and make necessary changes in ant tasks
(https://ant.apache.org/manual/Tasks/junitlauncher.html)
# update syntax in all tests (imports, Before/After annotations, etc)
# update parameterized tests
# create a new version of {{OrderedJUnit4ClassRunner}} and update corresponding
tests
# update tests that use {{BMUnitRunner}} (as per
https://developer.jboss.org/docs/DOC-52953 it supports JUnit5)
# update tests with {{@Rule}}
# update tests with expected exceptions
h3. Order of operations
In order to make the transition more smooth we want to use a phased approach:
1. migrate to JUnit5 with [Vintage
Engine|https://junit.org/junit5/docs/current/user-guide/#dependency-metadata-junit-vintage],
so all JUnit4 tests work as is
2. update tests in a few bunches (to not have a huge single PR with a bunch of
conflicts)
3. disable (remove dependency) Vintage Engine, so only JUnit5 tests work
was:
h3. Overview
Currently C* uses JUnit4 (version 4.12) which is obsolete. There is a newer
version 4.13.2 which we could update to. However, JUnit4 is generally
considered to be outdated and it is reasonable to migrate to JUnit5.
Despite of having a syntax sugar in JUnit5 (assertThrow, lamda's support, ect),
there are no blockers that push us to move from JUnit4. The main motivation for
this initiative is rule of thumb to use up-to-date versions of the dependencies.
Obviously this change is not backward compatible with the open PRs and previous
C* versions. Therefore, it will require an additional effort for backporting
the changes and updating PRs that have tests. However, I believe it should not
be a blocker for this initiative.
h3. Action items (preliminary list)
# change JUnit4 to JUnit5 dependencies and make necessary changes in ant tasks
(https://ant.apache.org/manual/Tasks/junitlauncher.html)
# update syntax in all tests (imports, Before/After annotations, etc)
# update parameterized tests
# create a new version of {{OrderedJUnit4ClassRunner}} and update corresponding
tests
# update tests that use {{BMUnitRunner}} (as per
https://developer.jboss.org/docs/DOC-52953 it supports JUnit5)
> Migrate to JUnit5
> -
>
> Key: CASSANDRA-16630
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16630
> Project: Cassandra
> Issue Type: Improvement
> Components: Test/unit
>Reporter: Alexey Zotov
>Priority: Normal
>
> h3. Overview
> Currently C* uses JUnit4 (version 4.12) which is obsolete. There is a newer
> version 4.13.2 which we could update to. However, JUnit4 is generally
> considered to be outdated and it is reasonable to migrate to JUnit5.
> Despite of having a syntax sugar in JUnit5 (assertThrow, lamda's support,
> ect), there are no blockers that push us to move from JUnit4. The main
> motivation for this initiative is rule of thumb to use up-to-date versions of
> the dependencies.
> Obviously this change is not backward compatible with the open PRs and
> previous C* versions. Therefore, it will require an additional effort for
> backporting the changes and updating PRs that have tests. However, I believe
> it should not be a blocker for this initiative.
> h3. Scope (preliminary list)
> # change JUnit4 to JUnit5 dependencies and make necessary changes in ant
> tasks (https://ant.apache.org/manual/Tasks/junitlauncher.html)
> # update syntax in all tests (imports, Before/After annotations, etc)
> # update parameterized tests
> # create a new version of {{OrderedJUnit4ClassRunner}} and update
> corresponding tests
> # update tests that use {{BMUnitRunner}} (as per
> https://developer.jboss.org/docs/DOC-52953 it supports JUnit5)
> # update tests with {{@Rule}}
> # update tests with expected exceptions
> h3. Order of operations
> In order to make the transition more smooth we want to use a phased approach:
> 1. migrate to JUnit5 with [Vintage
>
[jira] [Commented] (CASSANDRA-16630) Migrate to JUnit5
[
https://issues.apache.org/jira/browse/CASSANDRA-16630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17331625#comment-17331625
]
Alexey Zotov commented on CASSANDRA-16630:
--
[~slachiewicz]
Thanks for the info! I've also linked CASSANDRA-16629 to that ticket.
> Migrate to JUnit5
> -
>
> Key: CASSANDRA-16630
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16630
> Project: Cassandra
> Issue Type: Improvement
> Components: Test/unit
>Reporter: Alexey Zotov
>Priority: Normal
>
> h3. Overview
> Currently C* uses JUnit4 (version 4.12) which is obsolete. There is a newer
> version 4.13.2 which we could update to. However, JUnit4 is generally
> considered to be outdated and it is reasonable to migrate to JUnit5.
> Despite of having a syntax sugar in JUnit5 (assertThrow, lamda's support,
> ect), there are no blockers that push us to move from JUnit4. The main
> motivation for this initiative is rule of thumb to use up-to-date versions of
> the dependencies.
> Obviously this change is not backward compatible with the open PRs and
> previous C* versions. Therefore, it will require an additional effort for
> backporting the changes and updating PRs that have tests. However, I believe
> it should not be a blocker for this initiative.
> h3. Action items (preliminary list)
> # change JUnit4 to JUnit5 dependencies and make necessary changes in ant
> tasks (https://ant.apache.org/manual/Tasks/junitlauncher.html)
> # update syntax in all tests (imports, Before/After annotations, etc)
> # update parameterized tests
> # create a new version of {{OrderedJUnit4ClassRunner}} and update
> corresponding tests
> # update tests that use {{BMUnitRunner}} (as per
> https://developer.jboss.org/docs/DOC-52953 it supports JUnit5)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-16630) Migrate to JUnit5
[
https://issues.apache.org/jira/browse/CASSANDRA-16630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexey Zotov updated CASSANDRA-16630:
-
Description:
h3. Overview
Currently C* uses JUnit4 (version 4.12) which is obsolete. There is a newer
version 4.13.2 which we could update to. However, JUnit4 is generally
considered to be outdated and it is reasonable to migrate to JUnit5.
Despite of having a syntax sugar in JUnit5 (assertThrow, lamda's support, ect),
there are no blockers that push us to move from JUnit4. The main motivation for
this initiative is rule of thumb to use up-to-date versions of the dependencies.
Obviously this change is not backward compatible with the open PRs and previous
C* versions. Therefore, it will require an additional effort for backporting
the changes and updating PRs that have tests. However, I believe it should not
be a blocker for this initiative.
h3. Action items (preliminary list)
# change JUnit4 to JUnit5 dependencies and make necessary changes in ant tasks
(https://ant.apache.org/manual/Tasks/junitlauncher.html)
# update syntax in all tests (imports, Before/After annotations, etc)
# update parameterized tests
# create a new version of {{OrderedJUnit4ClassRunner}} and update corresponding
tests
# update tests that use {{BMUnitRunner}} (as per
https://developer.jboss.org/docs/DOC-52953 it supports JUnit5)
was:
h3. Overview
Currently C* uses JUnit4 (version 4.12) which is obsolete. There is a newer
version 4.13.2 which we could update to. However, JUnit4 is generally
considered to be outdated and it is reasonable to migrate to JUnit5.
Despite of having a syntax sugar in JUnit5 (assertThrow, lamda's support, ect),
there are no blockers that push us to move from JUnit4. The main motivation for
this initiative is rule of thumb to use up-to-date versions of the dependencies.
Obviously this change is not backward compatible with the open PRs and previous
C* versions. Therefore, it will require an additional effort for backporting
the changes and updating PRs that have tests. However, I believe it should not
be a blocker for this initiative.
h3. Action items (preliminary list)
# change JUnit4 to JUnit5 dependencies
# update syntax in all tests (imports, Before/After annotations, etc)
# update parameterized tests
# create a new version of {{OrderedJUnit4ClassRunner}} and update corresponding
tests
# update tests that use {{BMUnitRunner}} (as per
https://developer.jboss.org/docs/DOC-52953 it supports JUnit5)
> Migrate to JUnit5
> -
>
> Key: CASSANDRA-16630
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16630
> Project: Cassandra
> Issue Type: Improvement
> Components: Test/unit
>Reporter: Alexey Zotov
>Priority: Normal
>
> h3. Overview
> Currently C* uses JUnit4 (version 4.12) which is obsolete. There is a newer
> version 4.13.2 which we could update to. However, JUnit4 is generally
> considered to be outdated and it is reasonable to migrate to JUnit5.
> Despite of having a syntax sugar in JUnit5 (assertThrow, lamda's support,
> ect), there are no blockers that push us to move from JUnit4. The main
> motivation for this initiative is rule of thumb to use up-to-date versions of
> the dependencies.
> Obviously this change is not backward compatible with the open PRs and
> previous C* versions. Therefore, it will require an additional effort for
> backporting the changes and updating PRs that have tests. However, I believe
> it should not be a blocker for this initiative.
> h3. Action items (preliminary list)
> # change JUnit4 to JUnit5 dependencies and make necessary changes in ant
> tasks (https://ant.apache.org/manual/Tasks/junitlauncher.html)
> # update syntax in all tests (imports, Before/After annotations, etc)
> # update parameterized tests
> # create a new version of {{OrderedJUnit4ClassRunner}} and update
> corresponding tests
> # update tests that use {{BMUnitRunner}} (as per
> https://developer.jboss.org/docs/DOC-52953 it supports JUnit5)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Assigned] (CASSANDRA-16360) CRC32 is inefficient on x86
[ https://issues.apache.org/jira/browse/CASSANDRA-16360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Zotov reassigned CASSANDRA-16360: Assignee: Alexey Zotov > CRC32 is inefficient on x86 > --- > > Key: CASSANDRA-16360 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16360 > Project: Cassandra > Issue Type: Improvement > Components: Messaging/Client >Reporter: Avi Kivity >Assignee: Alexey Zotov >Priority: Normal > Labels: protocolv5 > Fix For: 4.0.x > > > The client/server protocol specifies CRC24 and CRC32 as the checksum > algorithm (cql_protocol_V5_framing.asc). Those however are expensive to > compute; this affects both the client and the server. > > A better checksum algorithm is CRC32C, which has hardware support on x86 (as > well as other modern architectures). -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-16404) Provide a nodetool way of invalidating auth caches
[
https://issues.apache.org/jira/browse/CASSANDRA-16404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexey Zotov updated CASSANDRA-16404:
-
Test and Documentation Plan: The changes are covered by unit tests. After
the patch is reviewed, I'll make one round of manual testing. As far as I
understand, [{{nodetool}}
documentation|https://cassandra.apache.org/doc/latest/tools/nodetool/nodetool.html]
is being generated automatically, so no separate steps with regard to
documentation.
Status: Patch Available (was: Open)
> Provide a nodetool way of invalidating auth caches
> --
>
> Key: CASSANDRA-16404
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16404
> Project: Cassandra
> Issue Type: Improvement
> Components: Feature/Authorization
>Reporter: Sumanth Pasupuleti
>Assignee: Alexey Zotov
>Priority: Normal
> Fix For: 4.x
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> We currently have nodetool commands to invalidate certain caches like
> KeyCache, RowCache and CounterCache.
> Being able to invalidate auth caches as well can come in handy in situations
> where, critical backend auth changes may need to be in effect right away for
> all the connections, especially in configurations where cache validity is
> chosen to be for a longer duration. An example can be that an authenticated
> user "User1" is no longer authorized to access a table resource "table1" and
> it is vital that this change is reflected right away, without having to wait
> for cache expiry/refresh to trigger.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-16629) Rafactor tests for NodeTool commands
[
https://issues.apache.org/jira/browse/CASSANDRA-16629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexey Zotov updated CASSANDRA-16629:
-
Test and Documentation Plan: These changes are purely cosmetic and affect
unit tests only. Hence, no test and documentation plans are required. (was:
These change are purely cosmetic and affect unit tests only. Hence, no test and
documentation plans are required.)
> Rafactor tests for NodeTool commands
>
>
> Key: CASSANDRA-16629
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16629
> Project: Cassandra
> Issue Type: Task
> Components: Test/unit
>Reporter: Alexey Zotov
>Assignee: Alexey Zotov
>Priority: Low
> Fix For: 4.0.x
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> h3. Overview
> As a part of CASSANDRA-16404, I've been working on {{NodeTool}} commands for
> invalidation of the auth caches. While working on that, I've mentioned that
> the existing NodeTool unit tests are written in a different manner and reside
> in wrong packages. I've made the following changes to them:
> # moved them to the proper packages (that match original command class)
> # unified naming, formatting, imports
> # migrated from {{org.assertj}} to {{hamcrest}} for assertions (though it is
> not smth really required)
> # made a general cleanup
> PR: https://github.com/apache/cassandra/pull/982
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-16629) Rafactor tests for NodeTool commands
[
https://issues.apache.org/jira/browse/CASSANDRA-16629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexey Zotov updated CASSANDRA-16629:
-
Test and Documentation Plan: These change are purely cosmetic and affect
unit tests only. Hence, no test and documentation plans are required.
Status: Patch Available (was: Open)
> Rafactor tests for NodeTool commands
>
>
> Key: CASSANDRA-16629
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16629
> Project: Cassandra
> Issue Type: Task
> Components: Test/unit
>Reporter: Alexey Zotov
>Assignee: Alexey Zotov
>Priority: Low
> Fix For: 4.0.x
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> h3. Overview
> As a part of CASSANDRA-16404, I've been working on {{NodeTool}} commands for
> invalidation of the auth caches. While working on that, I've mentioned that
> the existing NodeTool unit tests are written in a different manner and reside
> in wrong packages. I've made the following changes to them:
> # moved them to the proper packages (that match original command class)
> # unified naming, formatting, imports
> # migrated from {{org.assertj}} to {{hamcrest}} for assertions (though it is
> not smth really required)
> # made a general cleanup
> PR: https://github.com/apache/cassandra/pull/982
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-16629) Rafactor tests for NodeTool commands
[
https://issues.apache.org/jira/browse/CASSANDRA-16629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexey Zotov updated CASSANDRA-16629:
-
Change Category: Code Clarity
Complexity: Low Hanging Fruit
Priority: Low (was: Normal)
Status: Open (was: Triage Needed)
> Rafactor tests for NodeTool commands
>
>
> Key: CASSANDRA-16629
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16629
> Project: Cassandra
> Issue Type: Task
> Components: Test/unit
>Reporter: Alexey Zotov
>Assignee: Alexey Zotov
>Priority: Low
> Fix For: 4.0.x
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> h3. Overview
> As a part of CASSANDRA-16404, I've been working on {{NodeTool}} commands for
> invalidation of the auth caches. While working on that, I've mentioned that
> the existing NodeTool unit tests are written in a different manner and reside
> in wrong packages. I've made the following changes to them:
> # moved them to the proper packages (that match original command class)
> # unified naming, formatting, imports
> # migrated from {{org.assertj}} to {{hamcrest}} for assertions (though it is
> not smth really required)
> # made a general cleanup
> PR: https://github.com/apache/cassandra/pull/982
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-16629) Rafactor tests for NodeTool commands
[
https://issues.apache.org/jira/browse/CASSANDRA-16629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17331318#comment-17331318
]
Alexey Zotov edited comment on CASSANDRA-16629 at 4/24/21, 9:06 PM:
[~mck]
thanks for the quick response!
2,4. ok, I've rolled back some changes in a separate
[commit|https://github.com/apache/cassandra/pull/982/commits/655036bcc2b10a0133892a0a99f0e83cd3c23651].
I still feel they would be useful and safe, so if you check the commit and
find them good to go, please, let me know and I'll bring them back.
3. hmm.. actually that Slack communication was mine. And what I understood is
that there is no preference on one library over the other:
{quote}Thats test author dependent; some know different libs more than others.
The main question is how readable your tests are.{quote}
should I migrate tests to assertj? I have no problems with doing that since it
is not smth hard or time consuming.
{quote}I'm curious if we want to apply a patch to all branches. it's not much
fun on code that's unnecessarily changed between branches when
forward-merging.{quote}
I think it is makes sense to apply the code to all branches. I do not expect to
see conflicts with these changes, so it should be fairly easy. Anyway, I think
I need to wait until the current changes are reviewed/approved and only after
that start working on backporting. Here are a few general questions on the
process:
5. What branches should I backport these changes to? Should I create separate
PRs for every branch? Is there anything specific I should know?
was (Author: azotcsit):
[~mck]
thanks for the quick response!
2,4. ok, I've rolled back some changes in a separate
[commit|https://github.com/apache/cassandra/pull/982/commits/655036bcc2b10a0133892a0a99f0e83cd3c23651].
I still feel they would be useful and safe, so if you check the commit and
find them good to go, please, let me know and I'll bring them back.
3. hmm.. actually that Slack communication was mine. And what I understood is
that there is no preference on one library over the other:
{quote}Thats test author dependent; some know different libs more than others.
The main question is how readable your tests are.{quote}
should I migrate tests to assertj? I have no problems with doing that since it
is smth hard or time consuming.
{quote}I'm curious if we want to apply a patch to all branches. it's not much
fun on code that's unnecessarily changed between branches when
forward-merging.{quote}
I think it is makes sense to apply the code to all branches. I do not expect to
see conflicts with these changes, so it should be fairly easy. Anyway, I think
I need to wait until the current changes are reviewed/approved and only after
that start working on backporting. Here are a few general questions on the
process:
5. What branches should I backport these changes to? Should I create separate
PRs for every branch? Is there anything specific I should know?
> Rafactor tests for NodeTool commands
>
>
> Key: CASSANDRA-16629
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16629
> Project: Cassandra
> Issue Type: Task
> Components: Test/unit
>Reporter: Alexey Zotov
>Assignee: Alexey Zotov
>Priority: Normal
> Fix For: 4.0.x
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> h3. Overview
> As a part of CASSANDRA-16404, I've been working on {{NodeTool}} commands for
> invalidation of the auth caches. While working on that, I've mentioned that
> the existing NodeTool unit tests are written in a different manner and reside
> in wrong packages. I've made the following changes to them:
> # moved them to the proper packages (that match original command class)
> # unified naming, formatting, imports
> # migrated from {{org.assertj}} to {{hamcrest}} for assertions (though it is
> not smth really required)
> # made a general cleanup
> PR: https://github.com/apache/cassandra/pull/982
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16629) Rafactor tests for NodeTool commands
[
https://issues.apache.org/jira/browse/CASSANDRA-16629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17331318#comment-17331318
]
Alexey Zotov commented on CASSANDRA-16629:
--
[~mck]
thanks for the quick response!
2,4. ok, I've rolled back some changes in a separate
[commit|https://github.com/apache/cassandra/pull/982/commits/655036bcc2b10a0133892a0a99f0e83cd3c23651].
I still feel they would be useful and safe, so if you check the commit and
find them good to go, please, let me know and I'll bring them back.
3. hmm.. actually that Slack communication was mine. And what I understood is
that there is no preference on one library over the other:
{quote}Thats test author dependent; some know different libs more than others.
The main question is how readable your tests are.{quote}
should I migrate tests to assertj? I have no problems with doing that since it
is smth hard or time consuming.
{quote}I'm curious if we want to apply a patch to all branches. it's not much
fun on code that's unnecessarily changed between branches when
forward-merging.{quote}
I think it is makes sense to apply the code to all branches. I do not expect to
see conflicts with these changes, so it should be fairly easy. Anyway, I think
I need to wait until the current changes are reviewed/approved and only after
that start working on backporting. Here are a few general questions on the
process:
5. What branches should I backport these changes to? Should I create separate
PRs for every branch? Is there anything specific I should know?
> Rafactor tests for NodeTool commands
>
>
> Key: CASSANDRA-16629
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16629
> Project: Cassandra
> Issue Type: Task
> Components: Test/unit
>Reporter: Alexey Zotov
>Assignee: Alexey Zotov
>Priority: Normal
> Fix For: 4.0.x
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> h3. Overview
> As a part of CASSANDRA-16404, I've been working on {{NodeTool}} commands for
> invalidation of the auth caches. While working on that, I've mentioned that
> the existing NodeTool unit tests are written in a different manner and reside
> in wrong packages. I've made the following changes to them:
> # moved them to the proper packages (that match original command class)
> # unified naming, formatting, imports
> # migrated from {{org.assertj}} to {{hamcrest}} for assertions (though it is
> not smth really required)
> # made a general cleanup
> PR: https://github.com/apache/cassandra/pull/982
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16360) CRC32 is inefficient on x86
[ https://issues.apache.org/jira/browse/CASSANDRA-16360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17331222#comment-17331222 ] Alexey Zotov commented on CASSANDRA-16360: -- [~samt] Have you had chance to check the benchmark results? Sorry for being annoying, just want to ensure it is still on your radar. > CRC32 is inefficient on x86 > --- > > Key: CASSANDRA-16360 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16360 > Project: Cassandra > Issue Type: Improvement > Components: Messaging/Client >Reporter: Avi Kivity >Priority: Normal > Labels: protocolv5 > Fix For: 4.0.x > > > The client/server protocol specifies CRC24 and CRC32 as the checksum > algorithm (cql_protocol_V5_framing.asc). Those however are expensive to > compute; this affects both the client and the server. > > A better checksum algorithm is CRC32C, which has hardware support on x86 (as > well as other modern architectures). -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-16630) Migrate to JUnit5
[
https://issues.apache.org/jira/browse/CASSANDRA-16630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexey Zotov updated CASSANDRA-16630:
-
Description:
h3. Overview
Currently C* uses JUnit4 (version 4.12) which is obsolete. There is a newer
version 4.13.2 which we could update to. However, JUnit4 is generally
considered to be outdated and it is reasonable to migrate to JUnit5.
Despite of having a syntax sugar in JUnit5 (assertThrow, lamda's support, ect),
there are no blockers that push us to move from JUnit4. The main motivation for
this initiative is rule of thumb to use up-to-date versions of the dependencies.
Obviously this change is not backward compatible with the open PRs and previous
C* versions. Therefore, it will require an additional effort for backporting
the changes and updating PRs that have tests. However, I believe it should not
be a blocker for this initiative.
h3. Action items (preliminary list)
# change JUnit4 to JUnit5 dependencies
# update syntax in all tests (imports, Before/After annotations, etc)
# update parameterized tests
# create a new version of {{OrderedJUnit4ClassRunner}} and update corresponding
tests
# update tests that use {{BMUnitRunner}} (as per
https://developer.jboss.org/docs/DOC-52953 it supports JUnit5)
was:
h3. Overview
Currently C* uses JUnit4 (version 4.12) which is obsolete. There is a newer
version 4.13.2 which we could update to. However, JUnit4 is generally
considered to be outdated and it is reasonable to migrate to JUnit5.
Despite of having a syntax sugar in JUnit5 (assertThrow, lamda's support, ect),
there are no blockers that push us to move from JUnit4. The main motivation for
this initiative is rule of thumb to use up-to-date versions of the dependencies.
Obviously this change is not backward compatible with the open PRs and previous
C* versions. Therefore, it will require an additional effort for backporting
the changes and updating PRs that have tests. However, I believe it should not
be a blocker for this initiative.
h3. Action items (preliminary list)
# change JUnit4 to JUnit5 dependencies
# update all tests (imports, Before/After annotations, parametrized tests)
# create a new version of {{OrderedJUnit4ClassRunner}}
# update tests that use {{BMUnitRunner}} (as per
https://developer.jboss.org/docs/DOC-52953 it supports JUnit5)
> Migrate to JUnit5
> -
>
> Key: CASSANDRA-16630
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16630
> Project: Cassandra
> Issue Type: Improvement
> Components: Test/unit
>Reporter: Alexey Zotov
>Priority: Normal
>
> h3. Overview
> Currently C* uses JUnit4 (version 4.12) which is obsolete. There is a newer
> version 4.13.2 which we could update to. However, JUnit4 is generally
> considered to be outdated and it is reasonable to migrate to JUnit5.
> Despite of having a syntax sugar in JUnit5 (assertThrow, lamda's support,
> ect), there are no blockers that push us to move from JUnit4. The main
> motivation for this initiative is rule of thumb to use up-to-date versions of
> the dependencies.
> Obviously this change is not backward compatible with the open PRs and
> previous C* versions. Therefore, it will require an additional effort for
> backporting the changes and updating PRs that have tests. However, I believe
> it should not be a blocker for this initiative.
> h3. Action items (preliminary list)
> # change JUnit4 to JUnit5 dependencies
> # update syntax in all tests (imports, Before/After annotations, etc)
> # update parameterized tests
> # create a new version of {{OrderedJUnit4ClassRunner}} and update
> corresponding tests
> # update tests that use {{BMUnitRunner}} (as per
> https://developer.jboss.org/docs/DOC-52953 it supports JUnit5)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-16630) Migrate to JUnit5
[
https://issues.apache.org/jira/browse/CASSANDRA-16630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexey Zotov updated CASSANDRA-16630:
-
Description:
h3. Overview
Currently C* uses JUnit4 (version 4.12) which is obsolete. There is a newer
version 4.13.2 which we could update to. However, JUnit4 is generally
considered to be outdated and it is reasonable to migrate to JUnit5.
Despite of having a syntax sugar in JUnit5 (assertThrow, lamda's support, ect),
there are no blockers that push us to move from JUnit4. The main motivation for
this initiative is rule of thumb to use up-to-date versions of the dependencies.
Obviously this change is not backward compatible with the open PRs and previous
C* versions. Therefore, it will require an additional effort for backporting
the changes and updating PRs that have tests. However, I believe it should not
be a blocker for this initiative.
h3. Action items (preliminary list)
# change JUnit4 to JUnit5 dependencies
# update all tests (imports, Before/After annotations, parametrized tests)
# create a new version of {{OrderedJUnit4ClassRunner}}
# update tests that use {{BMUnitRunner}} (as per
https://developer.jboss.org/docs/DOC-52953 it supports JUnit5)
was:
h3. Overview
Currently C* uses JUnit4 (version 4.12) which is obsolete. There is a newer
version 4.13.2 which we could update to. However, JUnit4 is generally
considered to be outdated and it is reasonable to migrate to JUnit5.
Despite of having a syntax sugar in JUnit5 (assertThrow, lamda's support, ect),
there are no blockers that push us to move from JUnit4. The main motivation for
this initiative is rule of thumb to use up-to-date versions of the dependencies.
Obviously this change is not backward compatible with the open PRs and previous
C* versions. Therefore, it will require an additional effort for backporting
the changes and updating PRs that have tests. However, I believe it should not
be a blocker for this initiative.
h3. Action items (preliminary list)
# change JUnit4 to JUnit5 dependencies
# update all tests (imports, Before/After annotations, parametrized tests)
# create a new version of {{OrderedJUnit4ClassRunner}}
# updated tests that use {{BMUnitRunner}} (as per
https://developer.jboss.org/docs/DOC-52953 it supports JUnit5)
> Migrate to JUnit5
> -
>
> Key: CASSANDRA-16630
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16630
> Project: Cassandra
> Issue Type: Improvement
> Components: Test/unit
>Reporter: Alexey Zotov
>Priority: Normal
>
> h3. Overview
> Currently C* uses JUnit4 (version 4.12) which is obsolete. There is a newer
> version 4.13.2 which we could update to. However, JUnit4 is generally
> considered to be outdated and it is reasonable to migrate to JUnit5.
> Despite of having a syntax sugar in JUnit5 (assertThrow, lamda's support,
> ect), there are no blockers that push us to move from JUnit4. The main
> motivation for this initiative is rule of thumb to use up-to-date versions of
> the dependencies.
> Obviously this change is not backward compatible with the open PRs and
> previous C* versions. Therefore, it will require an additional effort for
> backporting the changes and updating PRs that have tests. However, I believe
> it should not be a blocker for this initiative.
> h3. Action items (preliminary list)
> # change JUnit4 to JUnit5 dependencies
> # update all tests (imports, Before/After annotations, parametrized tests)
> # create a new version of {{OrderedJUnit4ClassRunner}}
> # update tests that use {{BMUnitRunner}} (as per
> https://developer.jboss.org/docs/DOC-52953 it supports JUnit5)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-16630) Migrate to JUnit5
Alexey Zotov created CASSANDRA-16630:
Summary: Migrate to JUnit5
Key: CASSANDRA-16630
URL: https://issues.apache.org/jira/browse/CASSANDRA-16630
Project: Cassandra
Issue Type: Improvement
Components: Test/unit
Reporter: Alexey Zotov
h3. Overview
Currently C* uses JUnit4 (version 4.12) which is obsolete. There is a newer
version 4.13.2 which we could update to. However, JUnit4 is generally
considered to be outdated and it is reasonable to migrate to JUnit5.
Despite of having a syntax sugar in JUnit5 (assertThrow, lamda's support, ect),
there are no blockers that push us to move from JUnit4. The main motivation for
this initiative is rule of thumb to use up-to-date versions of the dependencies.
Obviously this change is not backward compatible with the open PRs and previous
C* versions. Therefore, it will require an additional effort for backporting
the changes and updating PRs that have tests. However, I believe it should not
be a blocker for this initiative.
h3. Action items (preliminary list)
# change JUnit4 to JUnit5 dependencies
# update all tests (imports, Before/After annotations, parametrized tests)
# create a new version of {{OrderedJUnit4ClassRunner}}
# updated tests that use {{BMUnitRunner}} (as per
https://developer.jboss.org/docs/DOC-52953 it supports JUnit5)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-16629) Rafactor tests for NodeTool commands
[
https://issues.apache.org/jira/browse/CASSANDRA-16629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexey Zotov updated CASSANDRA-16629:
-
Description:
h3. Overview
As a part of CASSANDRA-16404, I've been working on {{NodeTool}} commands for
invalidation of the auth caches. While working on that, I've mentioned that the
existing NodeTool unit tests are written in a different manner and reside in
wrong packages. I've made the following changes to them:
# moved them to the proper packages (that match original command class)
# unified naming, formatting, imports
# migrated from {{org.assertj}} to {{hamcrest}} for assertions (though it is
not smth really required)
# made a general cleanup
PR: https://github.com/apache/cassandra/pull/982
was:
h3. Overview
As a part of CASSANDRA-16404, I've been working on {{NodeTool}} commands for
invalidation of the auth caches. While working on that, I've mentioned that the
existing NodeTool unit tests are written in a different manner and reside in
wrong packages. I've made the following changes to them:
# moved them to the proper packages (that match original command class)
# unified naming, formatting, imports
# migrated from {{org.assertj}} to {{hamcrest}} for assertions (though it is
not smth really required)
# made a general cleanup
> Rafactor tests for NodeTool commands
>
>
> Key: CASSANDRA-16629
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16629
> Project: Cassandra
> Issue Type: Task
> Components: Test/unit
>Reporter: Alexey Zotov
>Assignee: Alexey Zotov
>Priority: Normal
> Fix For: 4.0.x
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> h3. Overview
> As a part of CASSANDRA-16404, I've been working on {{NodeTool}} commands for
> invalidation of the auth caches. While working on that, I've mentioned that
> the existing NodeTool unit tests are written in a different manner and reside
> in wrong packages. I've made the following changes to them:
> # moved them to the proper packages (that match original command class)
> # unified naming, formatting, imports
> # migrated from {{org.assertj}} to {{hamcrest}} for assertions (though it is
> not smth really required)
> # made a general cleanup
> PR: https://github.com/apache/cassandra/pull/982
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16629) Rafactor tests for NodeTool commands
[
https://issues.apache.org/jira/browse/CASSANDRA-16629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17331192#comment-17331192
]
Alexey Zotov commented on CASSANDRA-16629:
--
[~blerer] [~sumanth.pasupuleti]
Could you please review the changes and let me know if there are any concerns
on getting them merged.
> Rafactor tests for NodeTool commands
>
>
> Key: CASSANDRA-16629
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16629
> Project: Cassandra
> Issue Type: Task
> Components: Test/unit
>Reporter: Alexey Zotov
>Assignee: Alexey Zotov
>Priority: Normal
> Fix For: 4.0.x
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> h3. Overview
> As a part of CASSANDRA-16404, I've been working on {{NodeTool}} commands for
> invalidation of the auth caches. While working on that, I've mentioned that
> the existing NodeTool unit tests are written in a different manner and reside
> in wrong packages. I've made the following changes to them:
> # moved them to the proper packages (that match original command class)
> # unified naming, formatting, imports
> # migrated from {{org.assertj}} to {{hamcrest}} for assertions (though it is
> not smth really required)
> # made a general cleanup
> PR: https://github.com/apache/cassandra/pull/982
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-16629) Rafactor tests for NodeTool commands
[
https://issues.apache.org/jira/browse/CASSANDRA-16629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexey Zotov updated CASSANDRA-16629:
-
Fix Version/s: 4.0.x
> Rafactor tests for NodeTool commands
>
>
> Key: CASSANDRA-16629
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16629
> Project: Cassandra
> Issue Type: Task
> Components: Test/unit
>Reporter: Alexey Zotov
>Assignee: Alexey Zotov
>Priority: Normal
> Fix For: 4.0.x
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> h3. Overview
> As a part of CASSANDRA-16404, I've been working on {{NodeTool}} commands for
> invalidation of the auth caches. While working on that, I've mentioned that
> the existing NodeTool unit tests are written in a different manner and reside
> in wrong packages. I've made the following changes to them:
> # moved them to the proper packages (that match original command class)
> # unified naming, formatting, imports
> # migrated from {{org.assertj}} to {{hamcrest}} for assertions (though it is
> not smth really required)
> # made a general cleanup
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-16629) Rafactor tests for NodeTool commands
[
https://issues.apache.org/jira/browse/CASSANDRA-16629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexey Zotov updated CASSANDRA-16629:
-
Description:
h3. Overview
As a part of CASSANDRA-16404, I've been working on {{NodeTool}} commands for
invalidation of the auth caches. While working on that, I've mentioned that the
existing NodeTool unit tests are written in a different manner and reside in
wrong packages. I've made the following changes to them:
# moved them to the proper packages (that match original command class)
# unified naming, formatting, imports
# migrated from {{org.assertj}} to {{hamcrest}} for assertions (though it is
not smth really required)
# made a general cleanup
was:
h3. Overview
As a part of CASSANDRA-16404, I've been working on {{NodeTool}} commands for
invalidation of the auth caches. While working on that, I've mentioned that the
existing NodeTool unit tests are written in a different manner and reside in
wrong packages. I've made the following changes to them:
# moved them to the proper packages (that match original command class)
# unified naming, formatting, imports
# migrated from {{org.assertj}} to {{junit}} + {{hamcrest}} (though it is not
smth really required)
# made a general cleanup
> Rafactor tests for NodeTool commands
>
>
> Key: CASSANDRA-16629
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16629
> Project: Cassandra
> Issue Type: Task
> Components: Test/unit
>Reporter: Alexey Zotov
>Assignee: Alexey Zotov
>Priority: Normal
>
> h3. Overview
> As a part of CASSANDRA-16404, I've been working on {{NodeTool}} commands for
> invalidation of the auth caches. While working on that, I've mentioned that
> the existing NodeTool unit tests are written in a different manner and reside
> in wrong packages. I've made the following changes to them:
> # moved them to the proper packages (that match original command class)
> # unified naming, formatting, imports
> # migrated from {{org.assertj}} to {{hamcrest}} for assertions (though it is
> not smth really required)
> # made a general cleanup
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-16629) Rafactor tests for NodeTool commands
Alexey Zotov created CASSANDRA-16629:
Summary: Rafactor tests for NodeTool commands
Key: CASSANDRA-16629
URL: https://issues.apache.org/jira/browse/CASSANDRA-16629
Project: Cassandra
Issue Type: Task
Components: Test/unit
Reporter: Alexey Zotov
Assignee: Alexey Zotov
h3. Overview
As a part of CASSANDRA-16404, I've been working on {{NodeTool}} commands for
invalidation of the auth caches. While working on that, I've mentioned that the
existing NodeTool unit tests are written in a different manner and reside in
wrong packages. I've made the following changes to them:
# moved them to the proper packages (that match original command class)
# unified naming, formatting, imports
# migrated from {{org.assertj}} to {{junit}} + {{hamcrest}} (though it is not
smth really required)
# made a general cleanup
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-16404) Provide a nodetool way of invalidating auth caches
[
https://issues.apache.org/jira/browse/CASSANDRA-16404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17326113#comment-17326113
]
Alexey Zotov edited comment on CASSANDRA-16404 at 4/20/21, 9:37 PM:
[~blerer] [~sumanth.pasupuleti]
Thanks for the feedback! I've addressed your comments. I'll push refactoring
for {{NodeTool}} tests that are not related to the invalidation into a separate
PR.
Basically I'm done with the changes I wanted to make. Could you, please, review
the changes.
I read about [roles
hierarchy|https://www.datastax.com/blog/role-based-access-control-cassandra]
and I feel it would be complicated enough to support them. Basically the
problem is related to the fact that we cache permissions (permissions, network
permissions, jmx permissions) after we calculate them using roles hierarchy.
Permissions invalidation for a user (a role with "login" attribute = true)
works fine with the current changes. However, permissions invalidation for a
group role (a role that is granted to other roles) would require tracing the
hierarchy in a reverse direction and invalidating the whole affected hierarchy
from cache. It is theoretically possible, but in practice there is one
directional relation between roles ({{RolesCache extends
AuthCache>}}). I can take a look to this part further
if you believe that it needs to be address at this point, also I'd be glad to
hear a piece of advice on the best way to tackle the hierarchical invalidation.
was (Author: azotcsit):
[~blerer] [~sumanth.pasupuleti]
Thanks for the feedback! I've addressed your comments. I'll push tests
refactoring into a separate PR.
Basically I'm done with the changes I wanted to make. Could you, please, review
the changes.
I read about [roles
hierarchy|https://www.datastax.com/blog/role-based-access-control-cassandra]
and I feel it would be complicated enough to support them. Basically the
problem is related to the fact that we cache permissions (permissions, network
permissions, jmx permissions) after we calculate them using roles hierarchy.
Permissions invalidation for a user (a role with "login" attribute = true)
works fine with the current changes. However, permissions invalidation for a
group role (a role that is granted to other roles) would require tracing the
hierarchy in a reverse direction and invalidating the whole affected hierarchy
from cache. It is theoretically possible, but in practice there is one
directional relation between roles ({{RolesCache extends
AuthCache>}}). I can take a look to this part further
if you believe that it needs to be address at this point, also I'd be glad to
hear a piece of advice on the best way to tackle the hierarchical invalidation.
> Provide a nodetool way of invalidating auth caches
> --
>
> Key: CASSANDRA-16404
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16404
> Project: Cassandra
> Issue Type: Improvement
> Components: Feature/Authorization
>Reporter: Sumanth Pasupuleti
>Assignee: Alexey Zotov
>Priority: Normal
> Fix For: 4.x
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> We currently have nodetool commands to invalidate certain caches like
> KeyCache, RowCache and CounterCache.
> Being able to invalidate auth caches as well can come in handy in situations
> where, critical backend auth changes may need to be in effect right away for
> all the connections, especially in configurations where cache validity is
> chosen to be for a longer duration. An example can be that an authenticated
> user "User1" is no longer authorized to access a table resource "table1" and
> it is vital that this change is reflected right away, without having to wait
> for cache expiry/refresh to trigger.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16404) Provide a nodetool way of invalidating auth caches
[
https://issues.apache.org/jira/browse/CASSANDRA-16404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17326113#comment-17326113
]
Alexey Zotov commented on CASSANDRA-16404:
--
[~blerer] [~sumanth.pasupuleti]
Thanks for the feedback! I've addressed your comments. I'll push tests
refactoring into a separate PR.
Basically I'm done with the changes I wanted to make. Could you, please, review
the changes.
I read about [roles
hierarchy|https://www.datastax.com/blog/role-based-access-control-cassandra]
and I feel it would be complicated enough to support them. Basically the
problem is related to the fact that we cache permissions (permissions, network
permissions, jmx permissions) after we calculate them using roles hierarchy.
Permissions invalidation for a user (a role with "login" attribute = true)
works fine with the current changes. However, permissions invalidation for a
group role (a role that is granted to other roles) would require tracing the
hierarchy in a reverse direction and invalidating the whole affected hierarchy
from cache. It is theoretically possible, but in practice there is one
directional relation between roles ({{RolesCache extends
AuthCache>}}). I can take a look to this part further
if you believe that it needs to be address at this point, also I'd be glad to
hear a piece of advice on the best way to tackle the hierarchical invalidation.
> Provide a nodetool way of invalidating auth caches
> --
>
> Key: CASSANDRA-16404
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16404
> Project: Cassandra
> Issue Type: Improvement
> Components: Feature/Authorization
>Reporter: Sumanth Pasupuleti
>Assignee: Alexey Zotov
>Priority: Normal
> Fix For: 4.x
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> We currently have nodetool commands to invalidate certain caches like
> KeyCache, RowCache and CounterCache.
> Being able to invalidate auth caches as well can come in handy in situations
> where, critical backend auth changes may need to be in effect right away for
> all the connections, especially in configurations where cache validity is
> chosen to be for a longer duration. An example can be that an authenticated
> user "User1" is no longer authorized to access a table resource "table1" and
> it is vital that this change is reflected right away, without having to wait
> for cache expiry/refresh to trigger.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15669) LeveledCompactionStrategy compact last level throw an ArrayIndexOutOfBoundsException
[ https://issues.apache.org/jira/browse/CASSANDRA-15669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17324468#comment-17324468 ] Alexey Zotov commented on CASSANDRA-15669: -- Finally, it turned out that writing a test was quite fast and easy, so I raised a PR ([https://github.com/apache/cassandra/pull/971]) to illustrate my findings better. Please, review and let me know your thoughts. > LeveledCompactionStrategy compact last level throw an > ArrayIndexOutOfBoundsException > > > Key: CASSANDRA-15669 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15669 > Project: Cassandra > Issue Type: Bug >Reporter: sunhaihong >Assignee: sunhaihong >Priority: Normal > Attachments: cfs_compaction_info.png, error_info.png > > Time Spent: 50m > Remaining Estimate: 0h > > Cassandra will throw an ArrayIndexOutOfBoundsException when compact last > level. > My test is as follows： > # Create a table with LeveledCompactionStrategy and its params are > 'enabled': 'true', 'fanout_size': '2', 'max_threshold': '32', > 'min_threshold': '4', 'sstable_size_in_mb': '2'（fanout_size and > sstable_size_in_mb are too small just to make it easier to reproduce the > problem）; > # Insert data into the table by stress; > # Cassandra throw an ArrayIndexOutOfBoundsException when compact level9 > sstables(this level score bigger than 1.001) > ERROR [CompactionExecutor:4] 2020-03-28 08:59:00,990 CassandraDaemon.java:442 > - Exception in thread Thread[CompactionExecutor:4,1,main] > java.lang.ArrayIndexOutOfBoundsException: 9 > at > org.apache.cassandra.db.compaction.LeveledManifest.getLevel(LeveledManifest.java:814) > at > org.apache.cassandra.db.compaction.LeveledManifest.getCandidatesFor(LeveledManifest.java:746) > at > org.apache.cassandra.db.compaction.LeveledManifest.getCompactionCandidates(LeveledManifest.java:398) > at > org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getNextBackgroundTask(LeveledCompactionStrategy.java:131) > at > org.apache.cassandra.db.compaction.CompactionStrategyHolder.lambda$getBackgroundTaskSuppliers$0(CompactionStrategyHolder.java:109) > at > org.apache.cassandra.db.compaction.AbstractStrategyHolder$TaskSupplier.getTask(AbstractStrategyHolder.java:66) > at > org.apache.cassandra.db.compaction.CompactionStrategyManager.getNextBackgroundTask(CompactionStrategyManager.java:214) > at > org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:289) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266) > at java.util.concurrent.FutureTask.run(FutureTask.java) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at > io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) > at java.lang.Thread.run(Thread.java:748) > I tested it on cassandra version 3.11.3 & 4.0-alpha3. The exception all > happened. > once it triggers, level1- leveln compaction no longer works, level0 is still > valid > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-15669) LeveledCompactionStrategy compact last level throw an ArrayIndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/CASSANDRA-15669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17324337#comment-17324337
]
Alexey Zotov edited comment on CASSANDRA-15669 at 4/17/21, 8:51 PM:
I have checked this issue. I feel I have a kind of clear understanding of what
is going on.
[~sunhaihong]
Just a point of my curiosity - what values for {{sstable_size_in_mb}} and
{{fanout_size}} params do you use in prod and how much data do you have? I'm
just wondering how you were able to face this issue.
[~marcuse]
Looks like you are the best person to discuss this issue (as I can see, you
actively participated in LCS development).
First of all, I was able to reproduce this issue. I explored the code and
probably I found a couple of minor issues.
# *Wrong estimates calculation*
There is the following comment in the code:
{code:java}
// allocate enough generations for a PB of data, with a 1-MB sstable size.
(Note that if maxSSTableSize is
// updated, we will still have sstables of the older, potentially smaller size.
So don't make this
// dependent on maxSSTableSize.)
static final int MAX_LEVEL_COUNT = (int) Math.log10(1000 * 1000 * 1000);
{code}
It states about a PB of data for 1-MB sstable size configuration, but it does
not seem to be correct. It would be correct if 10 levels were supported.
However, 9 levels are currently supported. Here are my calculations (1-MB
sstable size and 10 fanout size):
{code:java}
L0:4 * 1 MB = 4 MB
L1: 10^1 * 1 MB = 10 MB
L2: 10^2 * 1 MB = 100 MB
L3: 10^3 * 1 MB = 1000 MB
L4: 10^4 * 1 MB = 1 MB = 9.76 GB
L5: 10^5 * 1 MB = 10 MB = 97.65 GB
L6: 10^6 * 1 MB = 100 MB= 976.56 GB
L7: 10^7 * 1 MB = 1000 MB = 9765.62 GB = 9.53 TB
L8: 10^8 * 1 MB = 1 MB = 97656.25 GB = 95.36 TB
L9: 10^9 * 1 MB = 10 MB = 976562.50 GB = 953.67 TB <-- this level is
not supported {code}
Here is the place where it is clearly shown that 9 levels (including L0) are
supported at the moment:
{code:java}
// note that since l0 is broken out, levels[0] represents L1:
private final TreeSet [] levels = new TreeSet[MAX_LEVEL_COUNT -
1];
{code}
Either the comment needs to be fixed or the number of levels needs to be
increased. I believe fixing the comment would be easier and amount of data
still would be enough for a regular C* setup.
# *There is no proper handling of a situation when there is more data than
supported*
The issue happens when compaction for L8 is going to be started. Here is the
flow: {{getCompactionCandidates}} --> {{getCandidatesFor\(i\)}} -->
{{generations.get(level + 1)}}. So while checking the compaction candidates for
L8, it tries to see what's going on L9 level and immediately fails. And that's
fair because we target to support a certain amount of data only.
Currently the above flow is triggered when {{score > 1.001}} (there is more
data than it should be on a level). In fact, we should not even try to check
candidates for compaction on the highest level, we should just fail fast since
it is an impossible situation for a properly configured C* cluster. I think a
clear error should be thrown when there is an attempt to handle more data than
expected on the highest level, smth like:
{code:java}
if (score > 1.001)
{
// the highest level should not ever exceed its maximum size
if (i == generations.levelCount() - 1)
throw new RuntimeException("Highest level (L" + i + ") should not exceed its
maximum size (" + maxBytesForLevel + "), but it has " + bytesForLevel + "
bytes");
// before proceeding with a higher level, let's see if L0 is far enough
behind to warrant STCS
if (l0Compaction != null)
return l0Compaction;
...
}
{code}
I'd be glad to hear you feedback on the points above. If you find the
suggestions reasonable, I'd like to come up with a patch (I have a draft, but
before polishing it I'd like to validate my understanding). Probably I'd also
update the documentation to clearly state number of levels supported and the
ways to estimate data size.
was (Author: azotcsit):
I have checked this issue. I feel I have a kind of clear understanding of what
is going on.
[~sunhaihong]
Just a point of my curiosity - what values for {{sstable_size_in_mb}} and
{{fanout_size}} params do you use in prod and how much data do you have? I'm
just wondering how you were able to face this issue.
[~marcuse]
Looks like you are the best person to discuss this issue (as I can see, you
actively participated in LCS development).
First of all, I was able to reproduce this issue. I explored the code and
probably I found a couple of minor issues.
# *Wrong estimates calculation*
There is the following comment in the code:
{code:java}
// allocate enough generations for a PB of data, with a 1-MB sstable size.
(Note that if maxSSTableSize is
// updated, we will still have
[jira] [Comment Edited] (CASSANDRA-15669) LeveledCompactionStrategy compact last level throw an ArrayIndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/CASSANDRA-15669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17324337#comment-17324337
]
Alexey Zotov edited comment on CASSANDRA-15669 at 4/17/21, 8:51 PM:
I have checked this issue. I feel I have a kind of clear understanding of what
is going on.
[~sunhaihong]
Just a point of my curiosity - what values for {{sstable_size_in_mb}} and
{{fanout_size}} params do you use in prod and how much data do you have? I'm
just wondering how you were able to face this issue.
[~marcuse]
Looks like you are the best person to discuss this issue (as I can see, you
actively participated in LCS development).
First of all, I was able to reproduce this issue. I explored the code and
probably I found a couple of minor issues.
# *Wrong estimates calculation*
There is the following comment in the code:
{code:java}
// allocate enough generations for a PB of data, with a 1-MB sstable size.
(Note that if maxSSTableSize is
// updated, we will still have sstables of the older, potentially smaller size.
So don't make this
// dependent on maxSSTableSize.)
static final int MAX_LEVEL_COUNT = (int) Math.log10(1000 * 1000 * 1000);
{code}
It states about a PB of data for 1-MB sstable size configuration, but it does
not seem to be correct. It would be correct if 10 levels were supported.
However, 9 levels are currently supported. Here are my calculations (1-MB
sstable size and 10 fanout size):
{code:java}
L0:4 * 1 MB = 4 MB
L1: 10^1 * 1 MB = 10 MB
L2: 10^2 * 1 MB = 100 MB
L3: 10^3 * 1 MB = 1000 MB
L4: 10^4 * 1 MB = 1 MB = 9.76 GB
L5: 10^5 * 1 MB = 10 MB = 97.65 GB
L6: 10^6 * 1 MB = 100 MB= 976.56 GB
L7: 10^7 * 1 MB = 1000 MB = 9765.62 GB = 9.53 TB
L8: 10^8 * 1 MB = 1 MB = 97656.25 GB = 95.36 TB
L9: 10^9 * 1 MB = 10 MB = 976562.50 GB = 953.67 TB <-- this level is
not supported {code}
Here is the place where it is clearly shown that 9 levels (including L0) are
supported at the moment:
{code:java}
// note that since l0 is broken out, levels[0] represents L1:
private final TreeSet [] levels = new TreeSet[MAX_LEVEL_COUNT -
1];
{code}
Either the comment needs to be fixed or the number of levels needs to be
increased. I believe fixing the comment would be easier and amount of data
still would be enough for a regular C* setup.
# *There is no proper handling of a situation when there is more data than
supported*
The issue happens when compaction for L8 is going to be started. Here is the
flow: {{getCompactionCandidates}} --> {{getCandidatesFor\(i\)}} -->
{{generations.get(level + 1)}}. So while checking the compaction candidates for
L8, it tries to see what's going on L9 level and immediately fails. And that's
fair because we target to support a certain amount of data only.
Currently the above flow is triggered when {{score > 1.001}} (there is more
data than it should be on a level). In fact, we should not even try to check
candidates for compaction on the highest level, we should just fail fast since
it is an impossible situation for a properly configured C* cluster. I think a
clear error should be thrown when there is an attempt to handle more data than
expected on the highest level, smth like:
{code:java}
if (score > 1.001)
{
// the highest level should not ever exceed its maximum size
if (i == generations.levelCount() - 1)
throw new RuntimeException("Highest level (L" + i + ") should not exceed its
maximum size (" + maxBytesForLevel + "), but it has " + bytesForLevel + "
bytes");
// before proceeding with a higher level, let's see if L0 is far enough
behind to warrant STCS
if (l0Compaction != null)
return l0Compaction;
...
}
{code}
I'd be glad to hear you feedback on the points above. If you find the
suggestions reasonable, I'd like to come up with a patch (I have a draft, but
before polishing it I'd like to validate my understanding). Probably I'd also
update the documentation to clearly state number of levels supported and the
ways to estimate data size.
was (Author: azotcsit):
I have checked this issue. I feel I have a kind of clear understanding of what
is going on.
[~sunhaihong]
Just a point of my curiosity - what values for {{sstable_size_in_mb}} and
{{fanout_size}} params do you use in prod and how much data do you have? I'm
just wondering how you were able to face this issue.
[~marcuse]
Looks like you are the best person to discuss this issue (as I can see, you
actively participated in LCS development).
First of all, I was able to reproduce this issue. I explored the code and
probably I found a few issues.
# Wrong estimates calculation
There is the following comment in the code:
{code:java}
// allocate enough generations for a PB of data, with a 1-MB sstable size.
(Note that if maxSSTableSize is
// updated, we will still have sstables of the
[jira] [Comment Edited] (CASSANDRA-15669) LeveledCompactionStrategy compact last level throw an ArrayIndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/CASSANDRA-15669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17324337#comment-17324337
]
Alexey Zotov edited comment on CASSANDRA-15669 at 4/17/21, 6:49 PM:
I have checked this issue. I feel I have a kind of clear understanding of what
is going on.
[~sunhaihong]
Just a point of my curiosity - what values for {{sstable_size_in_mb}} and
{{fanout_size}} params do you use in prod and how much data do you have? I'm
just wondering how you were able to face this issue.
[~marcuse]
Looks like you are the best person to discuss this issue (as I can see, you
actively participated in LCS development).
First of all, I was able to reproduce this issue. I explored the code and
probably I found a few issues.
# Wrong estimates calculation
There is the following comment in the code:
{code:java}
// allocate enough generations for a PB of data, with a 1-MB sstable size.
(Note that if maxSSTableSize is
// updated, we will still have sstables of the older, potentially smaller size.
So don't make this
// dependent on maxSSTableSize.)
static final int MAX_LEVEL_COUNT = (int) Math.log10(1000 * 1000 * 1000);
{code}
It states about a PB of data for 1-MB sstable size configuration, but it does
not seem to be correct. It would be correct if 10 levels were supported.
However, 9 levels are currently supported. Here are my calculations (1-MB
sstable size and 10 fanout size):
{code:java}
L0:4 * 1 MB = 4 MB
L1: 10^1 * 1 MB = 10 MB
L2: 10^2 * 1 MB = 100 MB
L3: 10^3 * 1 MB = 1000 MB
L4: 10^4 * 1 MB = 1 MB = 9.76 GB
L5: 10^5 * 1 MB = 10 MB = 97.65 GB
L6: 10^6 * 1 MB = 100 MB= 976.56 GB
L7: 10^7 * 1 MB = 1000 MB = 9765.62 GB = 9.53 TB
L8: 10^8 * 1 MB = 1 MB = 97656.25 GB = 95.36 TB
L9: 10^9 * 1 MB = 10 MB = 976562.50 GB = 953.67 TB <-- this level is
not supported {code}
Here is the place where it is clearly shown that 9 levels (including L0) are
supported at the moment:
{code:java}
// note that since l0 is broken out, levels[0] represents L1:
private final TreeSet [] levels = new TreeSet[MAX_LEVEL_COUNT -
1];
{code}
Either the comment needs to be fixed or the number of levels needs to be
increased. I believe fixing the comment would be easier and amount of data
still would be enough for a regular C* setup.
# L8 is not being handled properly
The issue happens when compaction for L8 is going to be started. Here is the
flow: {{getCompactionCandidates}} --> {{getCandidatesFor\(i\)}} -->
{{generations.get(level + 1)}}. So while checking the compaction candidates for
the L8, it tries to see what's going on L9 level and immediately fails. Which
means L8 is not ever being built/compacted. I think we need to handle the
highest level (8th) separately in {{getCandidatesFor\(i\)}} method as we handle
L0 (with a separate condition).
# There is no proper handling of a situation when there is more data than
supported
As it is stated in the first point, we target to support a certain amount of
data. I think a clear error should be thrown when there is an attempt to handle
more data than expected on the highest level, smth like:
{code:java}
long bytesForLevel = SSTableReader.getTotalBytes(sstablesInLevel);
long maxBytesForLevel = maxBytesForLevel(i, maxSSTableSizeInBytes);
if (i == generations.levelCount() - 1 && bytesForLevel > maxBytesForLevel)
throw new RuntimeException("Current size (" + bytesForLevel + ") exceeds
max size (" + maxBytesForLevel + ") for " + i + " level");
{code}
I'd be glad to hear you feedback on the points above. If you find the
suggestions reasonable, I'd like to come up with a patch (I have a draft, but
before polishing it I'd like to validate my understanding).
was (Author: azotcsit):
I have checked this issue. I feel I have a kind of clear understanding of what
is going on.
[~sunhaihong]
Just a point of my curiosity - what values for {{sstable_size_in_mb}} and
{{fanout_size }} params do you use in prod and how much data do you have? I'm
just wondering how you were able to face this issue.
[~marcuse]
Looks like you are the best person to discuss this issue (as I can see, you
actively participated in LCS development).
First of all, I was able to reproduce this issue. I explored the code and
probably I found a few issues.
# Wrong estimates calculation
There is the following comment in the code:
{code:java}
// allocate enough generations for a PB of data, with a 1-MB sstable size.
(Note that if maxSSTableSize is
// updated, we will still have sstables of the older, potentially smaller size.
So don't make this
// dependent on maxSSTableSize.)
static final int MAX_LEVEL_COUNT = (int) Math.log10(1000 * 1000 * 1000);
{code}
It states about a PB of data for 1-MB sstable size configuration, but it does
not seem to be correct. It would be correct if 10 levels were supported.
[jira] [Comment Edited] (CASSANDRA-15669) LeveledCompactionStrategy compact last level throw an ArrayIndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/CASSANDRA-15669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17324337#comment-17324337
]
Alexey Zotov edited comment on CASSANDRA-15669 at 4/17/21, 6:49 PM:
I have checked this issue. I feel I have a kind of clear understanding of what
is going on.
[~sunhaihong]
Just a point of my curiosity - what values for {{sstable_size_in_mb}} and
{{fanout_size}} params do you use in prod and how much data do you have? I'm
just wondering how you were able to face this issue.
[~marcuse]
Looks like you are the best person to discuss this issue (as I can see, you
actively participated in LCS development).
First of all, I was able to reproduce this issue. I explored the code and
probably I found a few issues.
# Wrong estimates calculation
There is the following comment in the code:
{code:java}
// allocate enough generations for a PB of data, with a 1-MB sstable size.
(Note that if maxSSTableSize is
// updated, we will still have sstables of the older, potentially smaller size.
So don't make this
// dependent on maxSSTableSize.)
static final int MAX_LEVEL_COUNT = (int) Math.log10(1000 * 1000 * 1000);
{code}
It states about a PB of data for 1-MB sstable size configuration, but it does
not seem to be correct. It would be correct if 10 levels were supported.
However, 9 levels are currently supported. Here are my calculations (1-MB
sstable size and 10 fanout size):
{code:java}
L0:4 * 1 MB = 4 MB
L1: 10^1 * 1 MB = 10 MB
L2: 10^2 * 1 MB = 100 MB
L3: 10^3 * 1 MB = 1000 MB
L4: 10^4 * 1 MB = 1 MB = 9.76 GB
L5: 10^5 * 1 MB = 10 MB = 97.65 GB
L6: 10^6 * 1 MB = 100 MB= 976.56 GB
L7: 10^7 * 1 MB = 1000 MB = 9765.62 GB = 9.53 TB
L8: 10^8 * 1 MB = 1 MB = 97656.25 GB = 95.36 TB
L9: 10^9 * 1 MB = 10 MB = 976562.50 GB = 953.67 TB <-- this level is
not supported {code}
Here is the place where it is clearly shown that 9 levels (including L0) are
supported at the moment:
{code:java}
// note that since l0 is broken out, levels[0] represents L1:
private final TreeSet [] levels = new TreeSet[MAX_LEVEL_COUNT -
1];
{code}
Either the comment needs to be fixed or the number of levels needs to be
increased. I believe fixing the comment would be easier and amount of data
still would be enough for a regular C* setup.
# L8 is not being handled properly
The issue happens when compaction for L8 is going to be started. Here is the
flow: {{getCompactionCandidates}} --> {{getCandidatesFor\(i\)}} -->
{{generations.get(level + 1)}}. So while checking the compaction candidates for
the L8, it tries to see what's going on L9 level and immediately fails. Which
means L8 is not ever being built/compacted. I think we need to handle the
highest level (8th) separately in {{getCandidatesFor\(i\)}} method as we handle
L0 (with a separate condition).
# There is no proper handling of a situation when there is more data than
supported
As it is stated in the first point, we target to support a certain amount of
data. I think a clear error should be thrown when there is an attempt to handle
more data than expected on the highest level, smth like:
{code:java}
long bytesForLevel = SSTableReader.getTotalBytes(sstablesInLevel);
long maxBytesForLevel = maxBytesForLevel(i, maxSSTableSizeInBytes);
if (i == generations.levelCount() - 1 && bytesForLevel > maxBytesForLevel)
throw new RuntimeException("Current size (" + bytesForLevel + ") exceeds
max size (" + maxBytesForLevel + ") for " + i + " level");
{code}
I'd be glad to hear you feedback on the points above. If you find the
suggestions reasonable, I'd like to come up with a patch (I have a draft, but
before polishing it I'd like to validate my understanding).
was (Author: azotcsit):
I have checked this issue. I feel I have a kind of clear understanding of what
is going on.
[~sunhaihong]
Just a point of my curiosity - what values for {{sstable_size_in_mb}} and
{{fanout_size}} params do you use in prod and how much data do you have? I'm
just wondering how you were able to face this issue.
[~marcuse]
Looks like you are the best person to discuss this issue (as I can see, you
actively participated in LCS development).
First of all, I was able to reproduce this issue. I explored the code and
probably I found a few issues.
# Wrong estimates calculation
There is the following comment in the code:
{code:java}
// allocate enough generations for a PB of data, with a 1-MB sstable size.
(Note that if maxSSTableSize is
// updated, we will still have sstables of the older, potentially smaller size.
So don't make this
// dependent on maxSSTableSize.)
static final int MAX_LEVEL_COUNT = (int) Math.log10(1000 * 1000 * 1000);
{code}
It states about a PB of data for 1-MB sstable size configuration, but it does
not seem to be correct. It would be correct if 10 levels were supported.
[jira] [Comment Edited] (CASSANDRA-15669) LeveledCompactionStrategy compact last level throw an ArrayIndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/CASSANDRA-15669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17324337#comment-17324337
]
Alexey Zotov edited comment on CASSANDRA-15669 at 4/17/21, 6:47 PM:
I have checked this issue. I feel I have a kind of clear understanding of what
is going on.
[~sunhaihong]
Just a point of my curiosity - what values for {{sstable_size_in_mb}} and
{{fanout_size }} params do you use in prod and how much data do you have? I'm
just wondering how you were able to face this issue.
[~marcuse]
Looks like you are the best person to discuss this issue (as I can see, you
actively participated in LCS development).
First of all, I was able to reproduce this issue. I explored the code and
probably I found a few issues.
# Wrong estimates calculation
There is the following comment in the code:
{code:java}
// allocate enough generations for a PB of data, with a 1-MB sstable size.
(Note that if maxSSTableSize is
// updated, we will still have sstables of the older, potentially smaller size.
So don't make this
// dependent on maxSSTableSize.)
static final int MAX_LEVEL_COUNT = (int) Math.log10(1000 * 1000 * 1000);
{code}
It states about a PB of data for 1-MB sstable size configuration, but it does
not seem to be correct. It would be correct if 10 levels were supported.
However, 9 levels are currently supported. Here are my calculations (1-MB
sstable size and 10 fanout size):
{code:java}
L0:4 * 1 MB = 4 MB
L1: 10^1 * 1 MB = 10 MB
L2: 10^2 * 1 MB = 100 MB
L3: 10^3 * 1 MB = 1000 MB
L4: 10^4 * 1 MB = 1 MB = 9.76 GB
L5: 10^5 * 1 MB = 10 MB = 97.65 GB
L6: 10^6 * 1 MB = 100 MB= 976.56 GB
L7: 10^7 * 1 MB = 1000 MB = 9765.62 GB = 9.53 TB
L8: 10^8 * 1 MB = 1 MB = 97656.25 GB = 95.36 TB
L9: 10^9 * 1 MB = 10 MB = 976562.50 GB = 953.67 TB <-- this level is
not supported {code}
Here is the place where it is clearly shown that 9 levels (including L0) are
supported at the moment:
{code:java}
// note that since l0 is broken out, levels[0] represents L1:
private final TreeSet [] levels = new TreeSet[MAX_LEVEL_COUNT -
1];
{code}
Either the comment needs to be fixed or the number of levels needs to be
increased. I believe fixing the comment would be easier and amount of data
still would be enough for a regular C* setup.
# L8 is not being handled properly
The issue happens when compaction for L8 is going to be started. Here is the
flow: {{getCompactionCandidates}} --> {{getCandidatesFor\(i\)}} -->
{{generations.get(level + 1)}} .So while checking the compaction candidates for
the L8, it tries to see what's going on L9 level and immediately fails. Which
means L8 is not ever being built/compacted. I think we need to handle the
highest level (8th) separately in {{getCandidatesFor\(i\)}} method as we handle
L0 (with a separate condition).
# There is no proper handling of a situation when there is more data than
supported
As it is stated in the first point, we target to support a certain amount of
data. I think a clear error should be thrown when there is an attempt to handle
more data than expected on the highest level, smth like:
{code:java}
long bytesForLevel = SSTableReader.getTotalBytes(sstablesInLevel);
long maxBytesForLevel = maxBytesForLevel(i, maxSSTableSizeInBytes);
if (i == generations.levelCount() - 1 && bytesForLevel > maxBytesForLevel)
throw new RuntimeException("Current size (" + bytesForLevel + ") exceeds
max size (" + maxBytesForLevel + ") for " + i + " level");
{code}
I'd be glad to hear you feedback on the points above. If you find the
suggestions reasonable, I'd like to come up with a patch (I have a draft, but
before polishing it I'd like to validate my understanding).
was (Author: azotcsit):
I have checked this issue. I feel I have a kind of clear understanding of what
is going on.
[~sunhaihong]
Just a point of my curiosity - what values for {{sstable_size_in_mb}} and
{{fanout_size }} params do you use in prod and how much data do you have? I'm
just wondering how you were able to face this issue.
[~marcuse]
Looks like you are the best person to discuss this issue (as I can see, you
actively participated in LCS development).
First of all, I was able to reproduce this issue. I explored the code and
probably I found a few issues.
# Wrong estimates calculation
There is the following comment in the code:
{code:java}
// allocate enough generations for a PB of data, with a 1-MB sstable size.
(Note that if maxSSTableSize is
// updated, we will still have sstables of the older, potentially smaller size.
So don't make this
// dependent on maxSSTableSize.)
static final int MAX_LEVEL_COUNT = (int) Math.log10(1000 * 1000 * 1000);
{code}
It states about a PB of data for 1-MB sstable size configuration, but it does
not seem to be correct. It would be correct if 10 levels were supported.
[jira] [Comment Edited] (CASSANDRA-15669) LeveledCompactionStrategy compact last level throw an ArrayIndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/CASSANDRA-15669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17324337#comment-17324337
]
Alexey Zotov edited comment on CASSANDRA-15669 at 4/17/21, 6:46 PM:
I have checked this issue. I feel I have a kind of clear understanding of what
is going on.
[~sunhaihong]
Just a point of my curiosity - what values for {{sstable_size_in_mb}} and
{{fanout_size }} params do you use in prod and how much data do you have? I'm
just wondering how you were able to face this issue.
[~marcuse]
Looks like you are the best person to discuss this issue (as I can see, you
actively participated in LCS development).
First of all, I was able to reproduce this issue. I explored the code and
probably I found a few issues.
# Wrong estimates calculation
There is the following comment in the code:
{code:java}
// allocate enough generations for a PB of data, with a 1-MB sstable size.
(Note that if maxSSTableSize is
// updated, we will still have sstables of the older, potentially smaller size.
So don't make this
// dependent on maxSSTableSize.)
static final int MAX_LEVEL_COUNT = (int) Math.log10(1000 * 1000 * 1000);
{code}
It states about a PB of data for 1-MB sstable size configuration, but it does
not seem to be correct. It would be correct if 10 levels were supported.
However, 9 levels are currently supported. Here are my calculations (1-MB
sstable size and 10 fanout size):
{code:java}
L0:4 * 1 MB = 4 MB
L1: 10^1 * 1 MB = 10 MB
L2: 10^2 * 1 MB = 100 MB
L3: 10^3 * 1 MB = 1000 MB
L4: 10^4 * 1 MB = 1 MB = 9.76 GB
L5: 10^5 * 1 MB = 10 MB = 97.65 GB
L6: 10^6 * 1 MB = 100 MB= 976.56 GB
L7: 10^7 * 1 MB = 1000 MB = 9765.62 GB = 9.53 TB
L8: 10^8 * 1 MB = 1 MB = 97656.25 GB = 95.36 TB
L9: 10^9 * 1 MB = 10 MB = 976562.50 GB = 953.67 TB <-- this level is
not supported {code}
Here is the place where it is clearly shown that 9 levels (including L0) are
supported at the moment:
{code:java}
// note that since l0 is broken out, levels[0] represents L1:
private final TreeSet [] levels = new TreeSet[MAX_LEVEL_COUNT -
1];
{code}
Either the comment needs to be fixed or the number of levels needs to be
increased. I believe fixing the comment would be easier and amount of data
still would be enough for a regular C* setup.
# L8 is not being handled properly
The issue happens when compaction for L8 is going to be started. Here is the
flow: {{getCompactionCandidates}} --> {{getCandidatesFor(i)}} -->
{{generations.get(level + 1)}} .So while checking the compaction candidates for
the L8, it tries to see what's going on L9 level and immediately fails. Which
means L8 is not ever being built/compacted. I think we need to handle the
highest level (8th) separately in {{getCandidatesFor(i)}} method as we handle
L0 (with a separate condition).
# There is no proper handling of a situation when there is more data than
supported
As it is stated in the first point, we target to support a certain amount of
data. I think a clear error should be thrown when there is an attempt to handle
more data than expected on the highest level, smth like:
{code:java}
long bytesForLevel = SSTableReader.getTotalBytes(sstablesInLevel);
long maxBytesForLevel = maxBytesForLevel(i, maxSSTableSizeInBytes);
if (i == generations.levelCount() - 1 && bytesForLevel > maxBytesForLevel)
throw new RuntimeException("Current size (" + bytesForLevel + ") exceeds
max size (" + maxBytesForLevel + ") for " + i + " level");
{code}
I'd be glad to hear you feedback on the points above. If you find the
suggestions reasonable, I'd like to come up with a patch (I have a draft, but
before polishing it I'd like to validate my understanding).
was (Author: azotcsit):
I have checked this issue. I feel I have a kind of clear understanding of what
is going on.
[~sunhaihong]
Just a point of my curiosity - what values for {{sstable_size_in_mb}} and
{{fanout_size }}params do you use in prod and how much data do you have? I'm
just wondering how you were able to face this issue.
[~marcuse]
Looks like you are the best person to discuss this issue (as I can see, you
actively participated in LCS development).
First of all, I was able to reproduce this issue. I explored the code and
probably I found a few issues.
# Wrong estimates calculation
There is the following comment in the code:
{code:java}
// allocate enough generations for a PB of data, with a 1-MB sstable size.
(Note that if maxSSTableSize is
// updated, we will still have sstables of the older, potentially smaller size.
So don't make this
// dependent on maxSSTableSize.)
static final int MAX_LEVEL_COUNT = (int) Math.log10(1000 * 1000 * 1000);
{code}
It states about a PB of data for 1-MB sstable size configuration, but it does
not seem to be correct. It would be correct if 10 levels were supported.
However,
[jira] [Commented] (CASSANDRA-15669) LeveledCompactionStrategy compact last level throw an ArrayIndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/CASSANDRA-15669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17324337#comment-17324337
]
Alexey Zotov commented on CASSANDRA-15669:
--
I have checked this issue. I feel I have a kind of clear understanding of what
is going on.
[~sunhaihong]
Just a point of my curiosity - what values for {{sstable_size_in_mb}} and
{{fanout_size }}params do you use in prod and how much data do you have? I'm
just wondering how you were able to face this issue.
[~marcuse]
Looks like you are the best person to discuss this issue (as I can see, you
actively participated in LCS development).
First of all, I was able to reproduce this issue. I explored the code and
probably I found a few issues.
# Wrong estimates calculation
There is the following comment in the code:
{code:java}
// allocate enough generations for a PB of data, with a 1-MB sstable size.
(Note that if maxSSTableSize is
// updated, we will still have sstables of the older, potentially smaller size.
So don't make this
// dependent on maxSSTableSize.)
static final int MAX_LEVEL_COUNT = (int) Math.log10(1000 * 1000 * 1000);
{code}
It states about a PB of data for 1-MB sstable size configuration, but it does
not seem to be correct. It would be correct if 10 levels were supported.
However, 9 levels are currently supported. Here are my calculations (1-MB
sstable size and 10 fanout size):
{code:java}
L0:4 * 1 MB = 4 MB
L1: 10^1 * 1 MB = 10 MB
L2: 10^2 * 1 MB = 100 MB
L3: 10^3 * 1 MB = 1000 MB
L4: 10^4 * 1 MB = 1 MB = 9.76 GB
L5: 10^5 * 1 MB = 10 MB = 97.65 GB
L6: 10^6 * 1 MB = 100 MB= 976.56 GB
L7: 10^7 * 1 MB = 1000 MB = 9765.62 GB = 9.53 TB
L8: 10^8 * 1 MB = 1 MB = 97656.25 GB = 95.36 TB
L9: 10^9 * 1 MB = 10 MB = 976562.50 GB = 953.67 TB <-- this level is
not supported {code}
Here is the place where it is clearly shown that 9 levels (including L0) are
supported at the moment:
{code:java}
// note that since l0 is broken out, levels[0] represents L1:
private final TreeSet [] levels = new TreeSet[MAX_LEVEL_COUNT -
1];
{code}
Either the comment needs to be fixed or the number of levels needs to be
increased. I believe fixing the comment would be easier and amount of data
still would be enough for a regular C* setup.
# L8 is not being handled properly
The issue happens when compaction for L8 is going to be started. Here is the
flow: {{getCompactionCandidates}} --> {{getCandidatesFor(i)}} -->
{{generations.get(level + 1)}} .So while checking the compaction candidates for
the L8, it tries to see what's going on L9 level and immediately fails. Which
means L8 is not ever being built/compacted. I think we need to handle the
highest level (8th) separately in {{getCandidatesFor(i)}} method as we handle
L0 (with a separate condition).
# There is no proper handling of a situation when there is more data than
supported
As it is stated in the first point, we target to support a certain amount of
data. I think a clear error should be thrown when there is an attempt to handle
more data than expected on the highest level, smth like:
{code:java}
long bytesForLevel = SSTableReader.getTotalBytes(sstablesInLevel);
long maxBytesForLevel = maxBytesForLevel(i, maxSSTableSizeInBytes);
if (i == generations.levelCount() - 1 && bytesForLevel > maxBytesForLevel)
throw new RuntimeException("Current size (" + bytesForLevel + ") exceeds
max size (" + maxBytesForLevel + ") for " + i + " level");
{code}
I'd be glad to hear you feedback on the points above. If you find the
suggestions reasonable, I'd like to come up with a patch (I have a draft, but
before polishing it I'd like to validate my understanding).
> LeveledCompactionStrategy compact last level throw an
> ArrayIndexOutOfBoundsException
>
>
> Key: CASSANDRA-15669
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15669
> Project: Cassandra
> Issue Type: Bug
>Reporter: sunhaihong
>Assignee: sunhaihong
>Priority: Normal
> Attachments: cfs_compaction_info.png, error_info.png
>
>
> Cassandra will throw an ArrayIndexOutOfBoundsException when compact last
> level.
> My test is as follows：
> # Create a table with LeveledCompactionStrategy and its params are
> 'enabled': 'true', 'fanout_size': '2', 'max_threshold': '32',
> 'min_threshold': '4', 'sstable_size_in_mb': '2'（fanout_size and
> sstable_size_in_mb are too small just to make it easier to reproduce the
> problem）;
> # Insert data into the table by stress;
> # Cassandra throw an ArrayIndexOutOfBoundsException when compact level9
> sstables(this level score bigger than 1.001)
> ERROR [CompactionExecutor:4] 2020-03-28 08:59:00,990 CassandraDaemon.java:442
> - Exception in thread
[jira] [Commented] (CASSANDRA-16360) CRC32 is inefficient on x86
[
https://issues.apache.org/jira/browse/CASSANDRA-16360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17324221#comment-17324221
]
Alexey Zotov commented on CASSANDRA-16360:
--
In the meantime I've explored the existing logic and the way to support this
new functionality in the next protocol version. Could you, please, validate my
understanding and let me know if I missed smth. Here are the changes I'm going
to do (if we finally decide to move forward):
1. {{ProtocolVersion}} already has "V6" defined, no changes are required here.
However, a new spec needs to be created: "native_protocol_v6_beta.spec".
2. Add a new version of {{MessagingService}}: "VERSION_401" and set it as a
current version (or not yet?).
3. Make changes in {{InboundConnectionInitiator}} /
{{OutboundConnectionInitiator}} to use _CRC32C_ for "VERSION_401".
4. Make changes in {{HandshakeProtocol}} to use _CRC32C_ for "VERSION_401".
5. Make changes in {{PipelineConfigurator}} to use _CRC32C_ for "V6".
6. Add "VERSION_401" case to all existing switch statements ({{Mutation}} /
{{CounterMutation}} / {{Message}} classes)
7. Make changes in {{Frame Decoder/Encoder Crc/LZ4}} to support both _CRC32_
and _CRC32C_. Now they must accept {{ChecksumType}} in constructor/"create"
method. I feel creating separate classes for _CRC32C_ would be an overhead.
Also I think using {{ChecksumType}} instead of {{Crc}} with static methods
makes a "structural" advantage. Thoughts/concerns?
8. Add _CRC32C_ support to {{ChecksumType}} and remove _CRC32_ support from
{{Crc}}. Going further {{Crc}} will be used for _CRC24_ calculation only. At
the moment, there is no intrinsic implementation for _CRC24_, hence re-writing
the implementation does not make any sense.
9. Tests, tests, tests..
> CRC32 is inefficient on x86
> ---
>
> Key: CASSANDRA-16360
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16360
> Project: Cassandra
> Issue Type: Improvement
> Components: Messaging/Client
>Reporter: Avi Kivity
>Priority: Normal
> Labels: protocolv5
> Fix For: 4.0.x
>
>
> The client/server protocol specifies CRC24 and CRC32 as the checksum
> algorithm (cql_protocol_V5_framing.asc). Those however are expensive to
> compute; this affects both the client and the server.
>
> A better checksum algorithm is CRC32C, which has hardware support on x86 (as
> well as other modern architectures).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-16360) CRC32 is inefficient on x86
[
https://issues.apache.org/jira/browse/CASSANDRA-16360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17314690#comment-17314690
]
Alexey Zotov edited comment on CASSANDRA-16360 at 4/13/21, 2:01 PM:
I've given a try to two more pure Java-based implementations for _CRC32C_
calculation:
- [Commons Codec
library|https://commons.apache.org/proper/commons-codec/apidocs/org/apache/commons/codec/digest/PureJavaCrc32C.html]
- Custom copy of CRC32C from Java9
Based on the benchmarking, I can see that they provide nearly the same results
as Snappy's implementation (the results are consistent with other runs, run was
made using Java 11.0.9):
{code:java}
[java] Benchmark (bufferSize) Mode Cnt Score
Error Units
[java] ChecksumBench.benchComCdcPureJavaCrc32c 31 avgt595.370 ±
4.438 ns/op
[java] ChecksumBench.benchComCdcPureJavaCrc32c 131 avgt5 212.066 ±
5.835 ns/op
[java] ChecksumBench.benchComCdcPureJavaCrc32c 517 avgt5 768.599 ±
12.188 ns/op
[java] ChecksumBench.benchComCdcPureJavaCrc32c2041 avgt5 2884.305 ±
86.143 ns/op
[java] ChecksumBench.benchJava9PortForCRC32C31 avgt589.839 ±
2.027 ns/op
[java] ChecksumBench.benchJava9PortForCRC32C 131 avgt5 233.196 ±
22.965 ns/op
[java] ChecksumBench.benchJava9PortForCRC32C 517 avgt5 738.161 ±
17.959 ns/op
[java] ChecksumBench.benchJava9PortForCRC32C 2041 avgt5 2561.263 ±
13.935 ns/op
[java] ChecksumBench.benchSnappyPureJavaCrc32c 31 avgt596.605 ±
4.376 ns/op
[java] ChecksumBench.benchSnappyPureJavaCrc32c 131 avgt5 239.617 ±
11.468 ns/op
[java] ChecksumBench.benchSnappyPureJavaCrc32c 517 avgt5 815.275 ±
8.615 ns/op
[java] ChecksumBench.benchSnappyPureJavaCrc32c2041 avgt5 2960.709 ±
68.225 ns/op
{code}
Looks like there is no easy way to improve pure Java implementations for
_CRC32C_ calculation. And we need to decide whether we're ok to move forward
with migration from _CRC32_ to _CRC32C_.
I'm waiting for some input.
I've run the same scenarios (obviously except of _benchCrc32c_ and
_benchCrc32cNoIntrinsic_) on Java 1.8.0_281 and results are consistent with
java 11.0.9.
The above changes are experimental. I'm not going to push them to the existing
PR, however, they are available for review at:
[https://github.com/alex-ninja/cassandra/pull/1/files].
was (Author: azotcsit):
I've given a try to two more pure Java-based implementations for _CRC32C_
calculation:
- [Common Codec
library|https://commons.apache.org/proper/commons-codec/apidocs/org/apache/commons/codec/digest/PureJavaCrc32C.html]
- Custom copy of CRC32C from Java9
Based on the benchmarking, I can see that they provide nearly the same results
as Snappy's implementation (the results are consistent with other runs, run was
made using Java 11.0.9):
{code:java}
[java] Benchmark (bufferSize) Mode Cnt Score
Error Units
[java] ChecksumBench.benchComCdcPureJavaCrc32c 31 avgt595.370 ±
4.438 ns/op
[java] ChecksumBench.benchComCdcPureJavaCrc32c 131 avgt5 212.066 ±
5.835 ns/op
[java] ChecksumBench.benchComCdcPureJavaCrc32c 517 avgt5 768.599 ±
12.188 ns/op
[java] ChecksumBench.benchComCdcPureJavaCrc32c2041 avgt5 2884.305 ±
86.143 ns/op
[java] ChecksumBench.benchJava9PortForCRC32C31 avgt589.839 ±
2.027 ns/op
[java] ChecksumBench.benchJava9PortForCRC32C 131 avgt5 233.196 ±
22.965 ns/op
[java] ChecksumBench.benchJava9PortForCRC32C 517 avgt5 738.161 ±
17.959 ns/op
[java] ChecksumBench.benchJava9PortForCRC32C 2041 avgt5 2561.263 ±
13.935 ns/op
[java] ChecksumBench.benchSnappyPureJavaCrc32c 31 avgt596.605 ±
4.376 ns/op
[java] ChecksumBench.benchSnappyPureJavaCrc32c 131 avgt5 239.617 ±
11.468 ns/op
[java] ChecksumBench.benchSnappyPureJavaCrc32c 517 avgt5 815.275 ±
8.615 ns/op
[java] ChecksumBench.benchSnappyPureJavaCrc32c2041 avgt5 2960.709 ±
68.225 ns/op
{code}
Looks like there is no easy way to improve pure Java implementations for
_CRC32C_ calculation. And we need to decide whether we're ok to move forward
with migration from _CRC32_ to _CRC32C_.
I'm waiting for some input.
I've run the same scenarios (obviously except of _benchCrc32c_ and
_benchCrc32cNoIntrinsic_) on Java 1.8.0_281 and results are consistent with
java 11.0.9.
The above changes are experimental. I'm not going to push them to the existing
PR, however, they are available for review at:
https://github.com/alex-ninja/cassandra/pull/1/files.
> CRC32 is inefficient on x86
> ---
>
> Key: CASSANDRA-16360
> URL:
[jira] [Comment Edited] (CASSANDRA-16360) CRC32 is inefficient on x86
[
https://issues.apache.org/jira/browse/CASSANDRA-16360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17314690#comment-17314690
]
Alexey Zotov edited comment on CASSANDRA-16360 at 4/13/21, 2:01 PM:
I've given a try to two more pure Java-based implementations for _CRC32C_
calculation:
- [Common Codec
library|https://commons.apache.org/proper/commons-codec/apidocs/org/apache/commons/codec/digest/PureJavaCrc32C.html]
- Custom copy of CRC32C from Java9
Based on the benchmarking, I can see that they provide nearly the same results
as Snappy's implementation (the results are consistent with other runs, run was
made using Java 11.0.9):
{code:java}
[java] Benchmark (bufferSize) Mode Cnt Score
Error Units
[java] ChecksumBench.benchComCdcPureJavaCrc32c 31 avgt595.370 ±
4.438 ns/op
[java] ChecksumBench.benchComCdcPureJavaCrc32c 131 avgt5 212.066 ±
5.835 ns/op
[java] ChecksumBench.benchComCdcPureJavaCrc32c 517 avgt5 768.599 ±
12.188 ns/op
[java] ChecksumBench.benchComCdcPureJavaCrc32c2041 avgt5 2884.305 ±
86.143 ns/op
[java] ChecksumBench.benchJava9PortForCRC32C31 avgt589.839 ±
2.027 ns/op
[java] ChecksumBench.benchJava9PortForCRC32C 131 avgt5 233.196 ±
22.965 ns/op
[java] ChecksumBench.benchJava9PortForCRC32C 517 avgt5 738.161 ±
17.959 ns/op
[java] ChecksumBench.benchJava9PortForCRC32C 2041 avgt5 2561.263 ±
13.935 ns/op
[java] ChecksumBench.benchSnappyPureJavaCrc32c 31 avgt596.605 ±
4.376 ns/op
[java] ChecksumBench.benchSnappyPureJavaCrc32c 131 avgt5 239.617 ±
11.468 ns/op
[java] ChecksumBench.benchSnappyPureJavaCrc32c 517 avgt5 815.275 ±
8.615 ns/op
[java] ChecksumBench.benchSnappyPureJavaCrc32c2041 avgt5 2960.709 ±
68.225 ns/op
{code}
Looks like there is no easy way to improve pure Java implementations for
_CRC32C_ calculation. And we need to decide whether we're ok to move forward
with migration from _CRC32_ to _CRC32C_.
I'm waiting for some input.
I've run the same scenarios (obviously except of _benchCrc32c_ and
_benchCrc32cNoIntrinsic_) on Java 1.8.0_281 and results are consistent with
java 11.0.9.
The above changes are experimental. I'm not going to push them to the existing
PR, however, they are available for review at:
https://github.com/alex-ninja/cassandra/pull/1/files.
was (Author: azotcsit):
Based on the benchmarking, I could see that native _CRC32C_ implementation
works really fast even without intrinsic. Here is one more test to highlight
that (the results are consistent with other runs, run was made using Java
11.0.9):
{code:java}
[java] Benchmark (bufferSize) Mode Cnt Score
Error Units
[java] ChecksumBench.benchCrc32 31 avgt5 107.191 ±
5.251 ns/op
[java] ChecksumBench.benchCrc32131 avgt583.716 ±
1.578 ns/op
[java] ChecksumBench.benchCrc32517 avgt5 123.176 ±
17.512 ns/op
[java] ChecksumBench.benchCrc32 2041 avgt5 273.591 ±
9.123 ns/op
[java] ChecksumBench.benchCrc32cNoIntrinsic 31 avgt552.850 ±
3.461 ns/op
[java] ChecksumBench.benchCrc32cNoIntrinsic131 avgt573.552 ±
1.624 ns/op
[java] ChecksumBench.benchCrc32cNoIntrinsic517 avgt5 196.009 ±
9.141 ns/op
[java] ChecksumBench.benchCrc32cNoIntrinsic 2041 avgt5 278.980 ±
7.515 ns/op
[java] ChecksumBench.benchPureJavaCrc32c31 avgt598.419 ±
5.221 ns/op
[java] ChecksumBench.benchPureJavaCrc32c 131 avgt5 239.515 ±
5.118 ns/op
[java] ChecksumBench.benchPureJavaCrc32c 517 avgt5 828.281 ±
107.874 ns/op
[java] ChecksumBench.benchPureJavaCrc32c 2041 avgt5 2941.934 ±
55.716 ns/op
{code}
I've checked the implementation and looks like the reason of such a great
performance of native _CRC32C_ implementation is that it heavily relies on
_Unsafe_ operations. Initially I though we can easily implement a custom
_CRC32C_ similar to the native one, however, now I do not think it is easy
enough and I have two concerns:
# there will be a need to use some libraries that wrap up work with Unsafe
# I'm not sure that from licensing perspective we are permitted to "re-work"
(copy-paste and adapt) the code from CRC32C
So I'm waiting for some input before moving forward in any direction.
> CRC32 is inefficient on x86
> ---
>
> Key: CASSANDRA-16360
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16360
> Project: Cassandra
> Issue Type: Improvement
> Components: Messaging/Client
>Reporter: Avi Kivity
>
[jira] [Comment Edited] (CASSANDRA-16360) CRC32 is inefficient on x86
[
https://issues.apache.org/jira/browse/CASSANDRA-16360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17314622#comment-17314622
]
Alexey Zotov edited comment on CASSANDRA-16360 at 4/13/21, 1:45 PM:
I've made an initial research on this task. Looks like the _CRC32C_ hardware
support is only available when intrinsic code is used. Here are a few reference
tickets mentioning that:
* [https://bugs.openjdk.java.net/browse/JDK-8189745]
* [https://bugs.java.com/bugdatabase/view_bug.do?bug_id=8191328]
* [https://bugs.java.com/bugdatabase/view_bug.do?bug_id=8073583]
And it looks like intrinsic code is only available for the [native _CRC32C_
class|https://docs.oracle.com/javase/9/docs/api/java/util/zip/CRC32C.html]
which comes with Java 9.
Here are the results of a microbenchmark test for native _CRC32_ and _CRC32C_
classes (run was made using Java 11.0.9):
{code:java}
[java] Benchmark (bufferSize) Mode Cnt Score
Error Units
[java] ChecksumBench.benchCrc32 31 avgt5 102.385 ±
2.628 ns/op
[java] ChecksumBench.benchCrc32131 avgt583.711 ±
2.624 ns/op
[java] ChecksumBench.benchCrc32517 avgt5 126.898 ±
44.337 ns/op
[java] ChecksumBench.benchCrc32 2041 avgt5 312.102 ±
202.493 ns/op
[java] ChecksumBench.benchCrc32NoIntrinsic 31 avgt5 208.107 ±
10.058 ns/op
[java] ChecksumBench.benchCrc32NoIntrinsic 131 avgt5 312.441 ±
34.561 ns/op
[java] ChecksumBench.benchCrc32NoIntrinsic 517 avgt5 674.960 ±
13.280 ns/op
[java] ChecksumBench.benchCrc32NoIntrinsic2041 avgt5 2225.843 ±
165.405 ns/op
[java] ChecksumBench.benchCrc32c31 avgt544.598 ±
1.540 ns/op
[java] ChecksumBench.benchCrc32c 131 avgt562.888 ±
9.017 ns/op
[java] ChecksumBench.benchCrc32c 517 avgt5 183.002 ±
6.637 ns/op
[java] ChecksumBench.benchCrc32c 2041 avgt5 264.787 ±
10.871 ns/op
[java] ChecksumBench.benchCrc32cNoIntrinsic 31 avgt589.640 ±
3.437 ns/op
[java] ChecksumBench.benchCrc32cNoIntrinsic131 avgt5 233.826 ±
7.405 ns/op
[java] ChecksumBench.benchCrc32cNoIntrinsic517 avgt5 742.341 ±
13.600 ns/op
[java] ChecksumBench.benchCrc32cNoIntrinsic 2041 avgt5 2566.061 ±
146.902 ns/op
{code}
Two obvious facts were reaffirmed by this run:
# intrinsic code works faster than non-intrinsic
# _CRC32C_ works faster than _CRC32_
As I mentioned before, _CRC32C_ class is unavailable in Java 8. That means we
cannot directly use that in the code base, otherwise it won't compile with Java
8. So there are two ways to proceed: 1) we wait until Java 8 support is
abandoned and start using _CRC32C_ 2) if Java 9+ is used then we use _CRC32C_ ,
otherwise we fallback to some Java-based implementation. The second approach
sounds reasonable to me since we need to target for newer versions and faster
solutions.
So I've given a try to two available Java-based implementations (Guava -
_hasherCrc32c_ and Snappy - _pureJavaCrc32c_) in another microbenchmark test
(run was made using Java 11.0.9):
{code:java}
[java] Benchmark (bufferSize) Mode Cnt Score
Error Units
[java] ChecksumBench.benchCrc32 31 avgt5 113.076 ±
33.568 ns/op
[java] ChecksumBench.benchCrc32131 avgt588.348 ±
11.433 ns/op
[java] ChecksumBench.benchCrc32517 avgt5 175.464 ±
56.718 ns/op
[java] ChecksumBench.benchCrc32 2041 avgt5 273.945 ±
16.331 ns/op
[java] ChecksumBench.benchHasherCrc32c 31 avgt 25 150.860 ±
2.883 ns/op
[java] ChecksumBench.benchHasherCrc32c 131 avgt 25 559.423 ±
141.932 ns/op
[java] ChecksumBench.benchHasherCrc32c 517 avgt 25 1599.504 ±
82.253 ns/op
[java] ChecksumBench.benchHasherCrc32c2041 avgt 25 5988.707 ±
103.558 ns/op
[java] ChecksumBench.benchPureJavaCrc32c31 avgt 2599.486 ±
1.464 ns/op
[java] ChecksumBench.benchPureJavaCrc32c 131 avgt 25 252.278 ±
20.278 ns/op
[java] ChecksumBench.benchPureJavaCrc32c 517 avgt 25 822.142 ±
38.860 ns/op
[java] ChecksumBench.benchPureJavaCrc32c 2041 avgt 25 3083.762 ±
200.493 ns/op
{code}
This test shows that:
# Snappy's implementation outperforms Guava's - which basically is not really
important, we can use either of them
# They both work slower than native _CRC32C_ implementation with intrinsic -
which is actually expected
# They both work slower than native _CRC32C_ implementation without intrinsic
- which is totally fine
#
[jira] [Comment Edited] (CASSANDRA-16360) CRC32 is inefficient on x86
[
https://issues.apache.org/jira/browse/CASSANDRA-16360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17314622#comment-17314622
]
Alexey Zotov edited comment on CASSANDRA-16360 at 4/13/21, 1:42 PM:
I've made an initial research on this task. Looks like the _CRC32C_ hardware
support is only available when intrinsic code is used. Here are a few reference
tickets mentioning that:
* [https://bugs.openjdk.java.net/browse/JDK-8189745]
* [https://bugs.java.com/bugdatabase/view_bug.do?bug_id=8191328]
* [https://bugs.java.com/bugdatabase/view_bug.do?bug_id=8073583]
And it looks like intrinsic code is only available for the [native _CRC32C_
class|[https://docs.oracle.com/javase/9/docs/api/java/util/zip/CRC32C.html]]
which comes with Java 9.
Here are the results of a microbenchmark test for native _CRC32_ and _CRC32C_
classes (run was made using Java 11.0.9):
{code:java}
[java] Benchmark (bufferSize) Mode Cnt Score
Error Units
[java] ChecksumBench.benchCrc32 31 avgt5 102.385 ±
2.628 ns/op
[java] ChecksumBench.benchCrc32131 avgt583.711 ±
2.624 ns/op
[java] ChecksumBench.benchCrc32517 avgt5 126.898 ±
44.337 ns/op
[java] ChecksumBench.benchCrc32 2041 avgt5 312.102 ±
202.493 ns/op
[java] ChecksumBench.benchCrc32NoIntrinsic 31 avgt5 208.107 ±
10.058 ns/op
[java] ChecksumBench.benchCrc32NoIntrinsic 131 avgt5 312.441 ±
34.561 ns/op
[java] ChecksumBench.benchCrc32NoIntrinsic 517 avgt5 674.960 ±
13.280 ns/op
[java] ChecksumBench.benchCrc32NoIntrinsic2041 avgt5 2225.843 ±
165.405 ns/op
[java] ChecksumBench.benchCrc32c31 avgt544.598 ±
1.540 ns/op
[java] ChecksumBench.benchCrc32c 131 avgt562.888 ±
9.017 ns/op
[java] ChecksumBench.benchCrc32c 517 avgt5 183.002 ±
6.637 ns/op
[java] ChecksumBench.benchCrc32c 2041 avgt5 264.787 ±
10.871 ns/op
[java] ChecksumBench.benchCrc32cNoIntrinsic 31 avgt589.640 ±
3.437 ns/op
[java] ChecksumBench.benchCrc32cNoIntrinsic131 avgt5 233.826 ±
7.405 ns/op
[java] ChecksumBench.benchCrc32cNoIntrinsic517 avgt5 742.341 ±
13.600 ns/op
[java] ChecksumBench.benchCrc32cNoIntrinsic 2041 avgt5 2566.061 ±
146.902 ns/op
{code}
Two obvious facts were reaffirmed by this run:
# intrinsic code works faster than non-intrinsic
# _CRC32C_ works faster than _CRC32_
As I mentioned before, _CRC32C_ class is unavailable in Java 8. That means we
cannot directly use that in the code base, otherwise it won't compile with Java
8. So there are two ways to proceed: 1) we wait until Java 8 support is
abandoned and start using _CRC32C_ 2) if Java 9+ is used then we use _CRC32C_ ,
otherwise we fallback to some Java-based implementation. The second approach
sounds reasonable to me since we need to target for newer versions and faster
solutions.
So I've given a try to two available Java-based implementations (Guava -
_hasherCrc32c_ and Snappy - _pureJavaCrc32c_) in another microbenchmark test
(run was made using Java 11.0.9):
{code:java}
[java] Benchmark (bufferSize) Mode Cnt Score
Error Units
[java] ChecksumBench.benchCrc32 31 avgt5 113.076 ±
33.568 ns/op
[java] ChecksumBench.benchCrc32131 avgt588.348 ±
11.433 ns/op
[java] ChecksumBench.benchCrc32517 avgt5 175.464 ±
56.718 ns/op
[java] ChecksumBench.benchCrc32 2041 avgt5 273.945 ±
16.331 ns/op
[java] ChecksumBench.benchHasherCrc32c 31 avgt 25 150.860 ±
2.883 ns/op
[java] ChecksumBench.benchHasherCrc32c 131 avgt 25 559.423 ±
141.932 ns/op
[java] ChecksumBench.benchHasherCrc32c 517 avgt 25 1599.504 ±
82.253 ns/op
[java] ChecksumBench.benchHasherCrc32c2041 avgt 25 5988.707 ±
103.558 ns/op
[java] ChecksumBench.benchPureJavaCrc32c31 avgt 2599.486 ±
1.464 ns/op
[java] ChecksumBench.benchPureJavaCrc32c 131 avgt 25 252.278 ±
20.278 ns/op
[java] ChecksumBench.benchPureJavaCrc32c 517 avgt 25 822.142 ±
38.860 ns/op
[java] ChecksumBench.benchPureJavaCrc32c 2041 avgt 25 3083.762 ±
200.493 ns/op
{code}
This test shows that:
# Snappy's implementation outperforms Guava's - which basically is not really
important, we can use either of them
# They both work slower than native _CRC32C_ implementation with intrinsic -
which is actually expected
# They both work slower than native _CRC32C_ implementation without intrinsic
- which is totally fine
#
[jira] [Comment Edited] (CASSANDRA-16360) CRC32 is inefficient on x86
[
https://issues.apache.org/jira/browse/CASSANDRA-16360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17314622#comment-17314622
]
Alexey Zotov edited comment on CASSANDRA-16360 at 4/13/21, 1:40 PM:
I've made an initial research on this task. Looks like the _CRC32C_ hardware
support is only available when intrinsic code is used. Here are a few reference
tickets mentioning that:
* [https://bugs.openjdk.java.net/browse/JDK-8189745]
* [https://bugs.java.com/bugdatabase/view_bug.do?bug_id=8191328]
* [https://bugs.java.com/bugdatabase/view_bug.do?bug_id=8073583]
And it looks like intrinsic code is only available for the [native _CRC32C_
class|[https://docs.oracle.com/javase/9/docs/api/java/util/zip/CRC32C.html]]
which comes with Java 9.
Here are the results of a microbenchmark test for native _CRC32_ and _CRC32C_
classes (run was made using Java 11.0.9):
{code:java}
[java] Benchmark (bufferSize) Mode Cnt
Score Error Units
[java] ChecksumBench.benchCrc32 31 avgt5
102.385 ± 2.628 ns/op
[java] ChecksumBench.benchCrc32131 avgt5
83.711 ± 2.624 ns/op
[java] ChecksumBench.benchCrc32517 avgt5
126.898 ± 44.337 ns/op
[java] ChecksumBench.benchCrc32 2041 avgt5
312.102 ± 202.493 ns/op
[java] ChecksumBench.benchCrc32NoIntrinsic 31 avgt5
208.107 ± 10.058 ns/op
[java] ChecksumBench.benchCrc32NoIntrinsic 131 avgt5
312.441 ± 34.561 ns/op
[java] ChecksumBench.benchCrc32NoIntrinsic 517 avgt5
674.960 ± 13.280 ns/op
[java] ChecksumBench.benchCrc32NoIntrinsic2041 avgt5
2225.843 ± 165.405 ns/op
[java] ChecksumBench.benchCrc32c31 avgt5
44.598 ± 1.540 ns/op
[java] ChecksumBench.benchCrc32c 131 avgt5
62.888 ± 9.017 ns/op
[java] ChecksumBench.benchCrc32c 517 avgt5
183.002 ± 6.637 ns/op
[java] ChecksumBench.benchCrc32c 2041 avgt5
264.787 ± 10.871 ns/op
[java] ChecksumBench.benchCrc32cNoIntrinsic 31 avgt5
89.640 ± 3.437 ns/op
[java] ChecksumBench.benchCrc32cNoIntrinsic131 avgt5
233.826 ± 7.405 ns/op
[java] ChecksumBench.benchCrc32cNoIntrinsic517 avgt5
742.341 ± 13.600 ns/op
[java] ChecksumBench.benchCrc32cNoIntrinsic 2041 avgt5
2566.061 ± 146.902 ns/op
{code}
Two obvious facts were reaffirmed by this run:
# intrinsic code works faster than non-intrinsic
# _CRC32C_ works faster than _CRC32_
As I mentioned before, _CRC32C_ class is unavailable in Java 8. That means we
cannot directly use that in the code base, otherwise it won't compile with Java
8. So there are two ways to proceed: 1) we wait until Java 8 support is
abandoned and start using _CRC32C_ 2) if Java 9+ is used then we use _CRC32C_ ,
otherwise we fallback to some Java-based implementation. The second approach
sounds reasonable to me since we need to target for newer versions and faster
solutions.
So I've given a try to two available Java-based implementations (Guava -
_hasherCrc32c_ and Snappy - _pureJavaCrc32c_) in another microbenchmark test
(run was made using Java 11.0.9):
{code:java}
[java] Benchmark (bufferSize) Mode Cnt Score
Error Units
[java] ChecksumBench.benchCrc32 31 avgt5 113.076 ±
33.568 ns/op
[java] ChecksumBench.benchCrc32131 avgt588.348 ±
11.433 ns/op
[java] ChecksumBench.benchCrc32517 avgt5 175.464 ±
56.718 ns/op
[java] ChecksumBench.benchCrc32 2041 avgt5 273.945 ±
16.331 ns/op
[java] ChecksumBench.benchHasherCrc32c 31 avgt 25 150.860 ±
2.883 ns/op
[java] ChecksumBench.benchHasherCrc32c 131 avgt 25 559.423 ±
141.932 ns/op
[java] ChecksumBench.benchHasherCrc32c 517 avgt 25 1599.504 ±
82.253 ns/op
[java] ChecksumBench.benchHasherCrc32c2041 avgt 25 5988.707 ±
103.558 ns/op
[java] ChecksumBench.benchPureJavaCrc32c31 avgt 2599.486 ±
1.464 ns/op
[java] ChecksumBench.benchPureJavaCrc32c 131 avgt 25 252.278 ±
20.278 ns/op
[java] ChecksumBench.benchPureJavaCrc32c 517 avgt 25 822.142 ±
38.860 ns/op
[java] ChecksumBench.benchPureJavaCrc32c 2041 avgt 25 3083.762 ±
200.493 ns/op
{code}
This test shows that:
# Snappy's implementation outperforms Guava's - which basically is not really
important, we can use either of them
# They
[jira] [Comment Edited] (CASSANDRA-16360) CRC32 is inefficient on x86
[
https://issues.apache.org/jira/browse/CASSANDRA-16360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17314622#comment-17314622
]
Alexey Zotov edited comment on CASSANDRA-16360 at 4/13/21, 1:39 PM:
I've made an initial research on this task. Looks like the _CRC32C_ hardware
support is only available when intrinsic code is used. Here are a few reference
tickets mentioning that:
* [https://bugs.openjdk.java.net/browse/JDK-8189745]
* [https://bugs.java.com/bugdatabase/view_bug.do?bug_id=8191328]
* [https://bugs.java.com/bugdatabase/view_bug.do?bug_id=8073583]
And it looks like intrinsic code is only available for the [native _CRC32C_
class|[https://docs.oracle.com/javase/9/docs/api/java/util/zip/CRC32C.html]]
which comes with Java 9.
Here are the results of a microbenchmark test for native _CRC32_ and _CRC32C_
classes (run was made using Java 11.0.9):
{code:java}
[java] ChecksumBench.benchCrc32 31 avgt5
102.385 ± 2.628 ns/op
[java] ChecksumBench.benchCrc32131 avgt5
83.711 ± 2.624 ns/op
[java] ChecksumBench.benchCrc32517 avgt5
126.898 ± 44.337 ns/op
[java] ChecksumBench.benchCrc32 2041 avgt5
312.102 ± 202.493 ns/op
[java] ChecksumBench.benchCrc32NoIntrinsic 31 avgt5
208.107 ± 10.058 ns/op
[java] ChecksumBench.benchCrc32NoIntrinsic 131 avgt5
312.441 ± 34.561 ns/op
[java] ChecksumBench.benchCrc32NoIntrinsic 517 avgt5
674.960 ± 13.280 ns/op
[java] ChecksumBench.benchCrc32NoIntrinsic2041 avgt5
2225.843 ± 165.405 ns/op
[java] ChecksumBench.benchCrc32c31 avgt5
44.598 ± 1.540 ns/op
[java] ChecksumBench.benchCrc32c 131 avgt5
62.888 ± 9.017 ns/op
[java] ChecksumBench.benchCrc32c 517 avgt5
183.002 ± 6.637 ns/op
[java] ChecksumBench.benchCrc32c 2041 avgt5
264.787 ± 10.871 ns/op
[java] ChecksumBench.benchCrc32cNoIntrinsic 31 avgt5
89.640 ± 3.437 ns/op
[java] ChecksumBench.benchCrc32cNoIntrinsic131 avgt5
233.826 ± 7.405 ns/op
[java] ChecksumBench.benchCrc32cNoIntrinsic517 avgt5
742.341 ± 13.600 ns/op
[java] ChecksumBench.benchCrc32cNoIntrinsic 2041 avgt5
2566.061 ± 146.902 ns/op
{code}
Two obvious facts were reaffirmed by this run:
# intrinsic code works faster than non-intrinsic
# _CRC32C_ works faster than _CRC32_
As I mentioned before, _CRC32C_ class is unavailable in Java 8. That means we
cannot directly use that in the code base, otherwise it won't compile with Java
8. So there are two ways to proceed: 1) we wait until Java 8 support is
abandoned and start using _CRC32C_ 2) if Java 9+ is used then we use _CRC32C_ ,
otherwise we fallback to some Java-based implementation. The second approach
sounds reasonable to me since we need to target for newer versions and faster
solutions.
So I've given a try to two available Java-based implementations (Guava -
_hasherCrc32c_ and Snappy - _pureJavaCrc32c_) in another microbenchmark test
(run was made using Java 11.0.9):
{code:java}
[java] Benchmark (bufferSize) Mode Cnt Score
Error Units
[java] ChecksumBench.benchCrc32 31 avgt5 113.076 ±
33.568 ns/op
[java] ChecksumBench.benchCrc32131 avgt588.348 ±
11.433 ns/op
[java] ChecksumBench.benchCrc32517 avgt5 175.464 ±
56.718 ns/op
[java] ChecksumBench.benchCrc32 2041 avgt5 273.945 ±
16.331 ns/op
[java] ChecksumBench.benchHasherCrc32c 31 avgt 25 150.860 ±
2.883 ns/op
[java] ChecksumBench.benchHasherCrc32c 131 avgt 25 559.423 ±
141.932 ns/op
[java] ChecksumBench.benchHasherCrc32c 517 avgt 25 1599.504 ±
82.253 ns/op
[java] ChecksumBench.benchHasherCrc32c2041 avgt 25 5988.707 ±
103.558 ns/op
[java] ChecksumBench.benchPureJavaCrc32c31 avgt 2599.486 ±
1.464 ns/op
[java] ChecksumBench.benchPureJavaCrc32c 131 avgt 25 252.278 ±
20.278 ns/op
[java] ChecksumBench.benchPureJavaCrc32c 517 avgt 25 822.142 ±
38.860 ns/op
[java] ChecksumBench.benchPureJavaCrc32c 2041 avgt 25 3083.762 ±
200.493 ns/op
{code}
This test shows that:
# Snappy's implementation outperforms Guava's - which basically is not really
important, we can use either of them
# They both work slower than native _CRC32C_ implementation with intrinsic -
which is actually expected
# They
[jira] [Commented] (CASSANDRA-16404) Provide a nodetool way of invalidating auth caches
[ https://issues.apache.org/jira/browse/CASSANDRA-16404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17315578#comment-17315578 ] Alexey Zotov commented on CASSANDRA-16404: -- [~blerer] Thanks for the quick response! It would be great if you could provide some input regarding other open questions in the PR description (unit tests was one of the questions). So I'll be able to address everything before your next review attempt. > Provide a nodetool way of invalidating auth caches > -- > > Key: CASSANDRA-16404 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16404 > Project: Cassandra > Issue Type: Improvement > Components: Feature/Authorization >Reporter: Sumanth Pasupuleti >Assignee: Alexey Zotov >Priority: Normal > Fix For: 4.x > > Time Spent: 20m > Remaining Estimate: 0h > > We currently have nodetool commands to invalidate certain caches like > KeyCache, RowCache and CounterCache. > Being able to invalidate auth caches as well can come in handy in situations > where, critical backend auth changes may need to be in effect right away for > all the connections, especially in configurations where cache validity is > chosen to be for a longer duration. An example can be that an authenticated > user "User1" is no longer authorized to access a table resource "table1" and > it is vital that this change is reflected right away, without having to wait > for cache expiry/refresh to trigger. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-16360) CRC32 is inefficient on x86
[
https://issues.apache.org/jira/browse/CASSANDRA-16360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17314690#comment-17314690
]
Alexey Zotov edited comment on CASSANDRA-16360 at 4/5/21, 8:43 AM:
---
Based on the benchmarking, I could see that native _CRC32C_ implementation
works really fast even without intrinsic. Here is one more test to highlight
that (the results are consistent with other runs, run was made using Java
11.0.9):
{code:java}
[java] Benchmark (bufferSize) Mode Cnt Score
Error Units
[java] ChecksumBench.benchCrc32 31 avgt5 107.191 ±
5.251 ns/op
[java] ChecksumBench.benchCrc32131 avgt583.716 ±
1.578 ns/op
[java] ChecksumBench.benchCrc32517 avgt5 123.176 ±
17.512 ns/op
[java] ChecksumBench.benchCrc32 2041 avgt5 273.591 ±
9.123 ns/op
[java] ChecksumBench.benchCrc32cNoIntrinsic 31 avgt552.850 ±
3.461 ns/op
[java] ChecksumBench.benchCrc32cNoIntrinsic131 avgt573.552 ±
1.624 ns/op
[java] ChecksumBench.benchCrc32cNoIntrinsic517 avgt5 196.009 ±
9.141 ns/op
[java] ChecksumBench.benchCrc32cNoIntrinsic 2041 avgt5 278.980 ±
7.515 ns/op
[java] ChecksumBench.benchPureJavaCrc32c31 avgt598.419 ±
5.221 ns/op
[java] ChecksumBench.benchPureJavaCrc32c 131 avgt5 239.515 ±
5.118 ns/op
[java] ChecksumBench.benchPureJavaCrc32c 517 avgt5 828.281 ±
107.874 ns/op
[java] ChecksumBench.benchPureJavaCrc32c 2041 avgt5 2941.934 ±
55.716 ns/op
{code}
I've checked the implementation and looks like the reason of such a great
performance of native _CRC32C_ implementation is that it heavily relies on
_Unsafe_ operations. Initially I though we can easily implement a custom
_CRC32C_ similar to the native one, however, now I do not think it is easy
enough and I have two concerns:
# there will be a need to use some libraries that wrap up work with Unsafe
# I'm not sure that from licensing perspective we are permitted to "re-work"
(copy-paste and adapt) the code from CRC32C
So I'm waiting for some input before moving forward in any direction.
was (Author: azotcsit):
Based on the benchmarking, I could see that native _CRC32C_ implementation
works really fast even without intrinsic. Here is one more test to highlight
that (the results are consistent with other runs, run was made using Java
11.0.9):
{code:java}
[java] Benchmark (bufferSize) Mode Cnt Score
Error Units
[java] ChecksumBench.benchCrc32 31 avgt5 107.191 ±
5.251 ns/op
[java] ChecksumBench.benchCrc32131 avgt583.716 ±
1.578 ns/op
[java] ChecksumBench.benchCrc32517 avgt5 123.176 ±
17.512 ns/op
[java] ChecksumBench.benchCrc32 2041 avgt5 273.591 ±
9.123 ns/op
[java] ChecksumBench.benchCrc32cNoIntrinsic 31 avgt552.850 ±
3.461 ns/op
[java] ChecksumBench.benchCrc32cNoIntrinsic131 avgt573.552 ±
1.624 ns/op
[java] ChecksumBench.benchCrc32cNoIntrinsic517 avgt5 196.009 ±
9.141 ns/op
[java] ChecksumBench.benchCrc32cNoIntrinsic 2041 avgt5 278.980 ±
7.515 ns/op
[java] ChecksumBench.benchPureJavaCrc32c31 avgt598.419 ±
5.221 ns/op
[java] ChecksumBench.benchPureJavaCrc32c 131 avgt5 239.515 ±
5.118 ns/op
[java] ChecksumBench.benchPureJavaCrc32c 517 avgt5 828.281 ±
107.874 ns/op
[java] ChecksumBench.benchPureJavaCrc32c 2041 avgt5 2941.934 ±
55.716 ns/op
{code}
I've checked the implementation and looks like the reason of such a great
performance of native _CRC32C_ implementation is that it heavily relies on
_Unsafe_ operations. Initially I though we can implement a custom _CRC32C_
similar to the native one, however, now I do not think it is possible because
we cannot use _Unsafe_ in the code.
> CRC32 is inefficient on x86
> ---
>
> Key: CASSANDRA-16360
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16360
> Project: Cassandra
> Issue Type: Improvement
> Components: Messaging/Client
>Reporter: Avi Kivity
>Priority: Normal
> Labels: protocolv5
> Fix For: 4.0.x
>
>
> The client/server protocol specifies CRC24 and CRC32 as the checksum
> algorithm (cql_protocol_V5_framing.asc). Those however are expensive to
> compute; this affects both the client and the server.
>
> A better checksum algorithm is CRC32C, which has hardware support on x86 (as
> well as other modern architectures).
--
This message was
[jira] [Comment Edited] (CASSANDRA-16360) CRC32 is inefficient on x86
[
https://issues.apache.org/jira/browse/CASSANDRA-16360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17314622#comment-17314622
]
Alexey Zotov edited comment on CASSANDRA-16360 at 4/5/21, 6:54 AM:
---
I've made an initial research on this task. Looks like the _CRC32C_ hardware
support is only available when intrinsic code is used. Here are a few reference
tickets mentioning that:
* [https://bugs.openjdk.java.net/browse/JDK-8189745]
* [https://bugs.java.com/bugdatabase/view_bug.do?bug_id=8191328]
* [https://bugs.java.com/bugdatabase/view_bug.do?bug_id=8073583]
And it looks like intrinsic code is only available for the [native _CRC32C_
class|[https://docs.oracle.com/javase/9/docs/api/java/util/zip/CRC32C.html]]
which comes with Java 9.
Here are the results of a microbenchmark test for native _CRC32_ and _CRC32C_
classes (run was made using Java 11.0.9):
{code:java}
[java] Benchmark (bufferSize) Mode Cnt Score
Error Units
[java] ChecksumBench.benchCrc3231 avgt598.823
± 2.667 ns/op
[java] ChecksumBench.benchCrc32 131 avgt5 133.014
± 5.831 ns/op
[java] ChecksumBench.benchCrc32 517 avgt5 173.939
± 14.456 ns/op
[java] ChecksumBench.benchCrc32 2041 avgt5 270.847
± 7.071 ns/op
[java] ChecksumBench.benchCrc32NoIntrinsic 31 avgt5 118.761
± 77.032 ns/op
[java] ChecksumBench.benchCrc32NoIntrinsic131 avgt5 157.799
± 377.481 ns/op
[java] ChecksumBench.benchCrc32NoIntrinsic517 avgt5 238.125
± 900.150 ns/op
[java] ChecksumBench.benchCrc32NoIntrinsic 2041 avgt5 276.828
± 7.814 ns/op
[java] ChecksumBench.benchCrc32c 31 avgt550.619
± 2.178 ns/op
[java] ChecksumBench.benchCrc32c 131 avgt569.229
± 2.186 ns/op
[java] ChecksumBench.benchCrc32c 517 avgt5 190.943
± 3.741 ns/op
[java] ChecksumBench.benchCrc32c 2041 avgt5 276.401
± 4.161 ns/op
[java] ChecksumBench.benchCrc32cNoIntrinsic31 avgt556.111
± 1.834 ns/op
[java] ChecksumBench.benchCrc32cNoIntrinsic 131 avgt575.475
± 1.912 ns/op
[java] ChecksumBench.benchCrc32cNoIntrinsic 517 avgt5 196.557
± 5.209 ns/op
[java] ChecksumBench.benchCrc32cNoIntrinsic 2041 avgt5 281.095
± 5.765 ns/op
{code}
Two obvious facts were reaffirmed by this run:
# intrinsic code works faster than non-intrinsic
# _CRC32C_ works faster than _CRC32_
As I mentioned before, _CRC32C_ class is unavailable in Java 8. That means we
cannot directly use that in the code base, otherwise it won't compile with Java
8. So there are two ways to proceed: 1) we wait until Java 8 support is
abandoned and start using _CRC32C_ 2) if Java 9+ is used then we use _CRC32C_ ,
otherwise we fallback to some Java-based implementation. The second approach
sounds reasonable to me since we need to target for newer versions and faster
solutions.
So I've given a try to two available Java-based implementations (Guava -
_hasherCrc32c_ and Snappy - _pureJavaCrc32c_) in another microbenchmark test
(run was made using Java 11.0.9):
{code:java}
[java] Benchmark (bufferSize) Mode Cnt Score
Error Units
[java] ChecksumBench.benchCrc32 31 avgt5 113.076 ±
33.568 ns/op
[java] ChecksumBench.benchCrc32131 avgt588.348 ±
11.433 ns/op
[java] ChecksumBench.benchCrc32517 avgt5 175.464 ±
56.718 ns/op
[java] ChecksumBench.benchCrc32 2041 avgt5 273.945 ±
16.331 ns/op
[java] ChecksumBench.benchHasherCrc32c 31 avgt 25 150.860 ±
2.883 ns/op
[java] ChecksumBench.benchHasherCrc32c 131 avgt 25 559.423 ±
141.932 ns/op
[java] ChecksumBench.benchHasherCrc32c 517 avgt 25 1599.504 ±
82.253 ns/op
[java] ChecksumBench.benchHasherCrc32c2041 avgt 25 5988.707 ±
103.558 ns/op
[java] ChecksumBench.benchPureJavaCrc32c31 avgt 2599.486 ±
1.464 ns/op
[java] ChecksumBench.benchPureJavaCrc32c 131 avgt 25 252.278 ±
20.278 ns/op
[java] ChecksumBench.benchPureJavaCrc32c 517 avgt 25 822.142 ±
38.860 ns/op
[java] ChecksumBench.benchPureJavaCrc32c 2041 avgt 25 3083.762 ±
200.493 ns/op
{code}
This test shows that:
# Snappy's implementation outperforms Guava's - which basically is not really
important, we can use either of them
# They both work slower than native _CRC32C_ implementation with intrinsic -
which is actually expected
# They both work slower than native _CRC32C_
[jira] [Commented] (CASSANDRA-16360) CRC32 is inefficient on x86
[
https://issues.apache.org/jira/browse/CASSANDRA-16360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17314690#comment-17314690
]
Alexey Zotov commented on CASSANDRA-16360:
--
Based on the benchmarking, I could see that native _CRC32C_ implementation
works really fast even without intrinsic. Here is one more test to highlight
that (the results are consistent with other runs, run was made using Java
11.0.9):
{code:java}
[java] Benchmark (bufferSize) Mode Cnt Score
Error Units
[java] ChecksumBench.benchCrc32 31 avgt5 107.191 ±
5.251 ns/op
[java] ChecksumBench.benchCrc32131 avgt583.716 ±
1.578 ns/op
[java] ChecksumBench.benchCrc32517 avgt5 123.176 ±
17.512 ns/op
[java] ChecksumBench.benchCrc32 2041 avgt5 273.591 ±
9.123 ns/op
[java] ChecksumBench.benchCrc32cNoIntrinsic 31 avgt552.850 ±
3.461 ns/op
[java] ChecksumBench.benchCrc32cNoIntrinsic131 avgt573.552 ±
1.624 ns/op
[java] ChecksumBench.benchCrc32cNoIntrinsic517 avgt5 196.009 ±
9.141 ns/op
[java] ChecksumBench.benchCrc32cNoIntrinsic 2041 avgt5 278.980 ±
7.515 ns/op
[java] ChecksumBench.benchPureJavaCrc32c31 avgt598.419 ±
5.221 ns/op
[java] ChecksumBench.benchPureJavaCrc32c 131 avgt5 239.515 ±
5.118 ns/op
[java] ChecksumBench.benchPureJavaCrc32c 517 avgt5 828.281 ±
107.874 ns/op
[java] ChecksumBench.benchPureJavaCrc32c 2041 avgt5 2941.934 ±
55.716 ns/op
{code}
I've checked the implementation and looks like the reason of such a great
performance of native _CRC32C_ implementation is that it heavily relies on
_Unsafe_ operations. Initially I though we can implement a custom _CRC32C_
similar to the native one, however, now I do not think it is possible because
we cannot use _Unsafe_ in the code.
> CRC32 is inefficient on x86
> ---
>
> Key: CASSANDRA-16360
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16360
> Project: Cassandra
> Issue Type: Improvement
> Components: Messaging/Client
>Reporter: Avi Kivity
>Priority: Normal
> Labels: protocolv5
> Fix For: 4.0.x
>
>
> The client/server protocol specifies CRC24 and CRC32 as the checksum
> algorithm (cql_protocol_V5_framing.asc). Those however are expensive to
> compute; this affects both the client and the server.
>
> A better checksum algorithm is CRC32C, which has hardware support on x86 (as
> well as other modern architectures).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-16360) CRC32 is inefficient on x86
[
https://issues.apache.org/jira/browse/CASSANDRA-16360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17314690#comment-17314690
]
Alexey Zotov edited comment on CASSANDRA-16360 at 4/5/21, 6:53 AM:
---
Based on the benchmarking, I could see that native _CRC32C_ implementation
works really fast even without intrinsic. Here is one more test to highlight
that (the results are consistent with other runs, run was made using Java
11.0.9):
{code:java}
[java] Benchmark (bufferSize) Mode Cnt Score
Error Units
[java] ChecksumBench.benchCrc32 31 avgt5 107.191 ±
5.251 ns/op
[java] ChecksumBench.benchCrc32131 avgt583.716 ±
1.578 ns/op
[java] ChecksumBench.benchCrc32517 avgt5 123.176 ±
17.512 ns/op
[java] ChecksumBench.benchCrc32 2041 avgt5 273.591 ±
9.123 ns/op
[java] ChecksumBench.benchCrc32cNoIntrinsic 31 avgt552.850 ±
3.461 ns/op
[java] ChecksumBench.benchCrc32cNoIntrinsic131 avgt573.552 ±
1.624 ns/op
[java] ChecksumBench.benchCrc32cNoIntrinsic517 avgt5 196.009 ±
9.141 ns/op
[java] ChecksumBench.benchCrc32cNoIntrinsic 2041 avgt5 278.980 ±
7.515 ns/op
[java] ChecksumBench.benchPureJavaCrc32c31 avgt598.419 ±
5.221 ns/op
[java] ChecksumBench.benchPureJavaCrc32c 131 avgt5 239.515 ±
5.118 ns/op
[java] ChecksumBench.benchPureJavaCrc32c 517 avgt5 828.281 ±
107.874 ns/op
[java] ChecksumBench.benchPureJavaCrc32c 2041 avgt5 2941.934 ±
55.716 ns/op
{code}
I've checked the implementation and looks like the reason of such a great
performance of native _CRC32C_ implementation is that it heavily relies on
_Unsafe_ operations. Initially I though we can implement a custom _CRC32C_
similar to the native one, however, now I do not think it is possible because
we cannot use _Unsafe_ in the code.
was (Author: azotcsit):
Based on the benchmarking, I could see that native _CRC32C_ implementation
works really fast even without intrinsic. Here is one more test to highlight
that (the results are consistent with other runs, run was made using Java
11.0.9):
{code:java}
[java] Benchmark (bufferSize) Mode Cnt Score
Error Units
[java] ChecksumBench.benchCrc32 31 avgt5 107.191 ±
5.251 ns/op
[java] ChecksumBench.benchCrc32131 avgt583.716 ±
1.578 ns/op
[java] ChecksumBench.benchCrc32517 avgt5 123.176 ±
17.512 ns/op
[java] ChecksumBench.benchCrc32 2041 avgt5 273.591 ±
9.123 ns/op
[java] ChecksumBench.benchCrc32cNoIntrinsic 31 avgt552.850 ±
3.461 ns/op
[java] ChecksumBench.benchCrc32cNoIntrinsic131 avgt573.552 ±
1.624 ns/op
[java] ChecksumBench.benchCrc32cNoIntrinsic517 avgt5 196.009 ±
9.141 ns/op
[java] ChecksumBench.benchCrc32cNoIntrinsic 2041 avgt5 278.980 ±
7.515 ns/op
[java] ChecksumBench.benchPureJavaCrc32c31 avgt598.419 ±
5.221 ns/op
[java] ChecksumBench.benchPureJavaCrc32c 131 avgt5 239.515 ±
5.118 ns/op
[java] ChecksumBench.benchPureJavaCrc32c 517 avgt5 828.281 ±
107.874 ns/op
[java] ChecksumBench.benchPureJavaCrc32c 2041 avgt5 2941.934 ±
55.716 ns/op
{code}
I've checked the implementation and looks like the reason of such a great
performance of native _CRC32C_ implementation is that it heavily relies on
_Unsafe_ operations. Initially I though we can implement a custom _CRC32C_
similar to the native one, however, now I do not think it is possible because
we cannot use _Unsafe_ in the code.
> CRC32 is inefficient on x86
> ---
>
> Key: CASSANDRA-16360
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16360
> Project: Cassandra
> Issue Type: Improvement
> Components: Messaging/Client
>Reporter: Avi Kivity
>Priority: Normal
> Labels: protocolv5
> Fix For: 4.0.x
>
>
> The client/server protocol specifies CRC24 and CRC32 as the checksum
> algorithm (cql_protocol_V5_framing.asc). Those however are expensive to
> compute; this affects both the client and the server.
>
> A better checksum algorithm is CRC32C, which has hardware support on x86 (as
> well as other modern architectures).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-16360) CRC32 is inefficient on x86
[
https://issues.apache.org/jira/browse/CASSANDRA-16360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17314622#comment-17314622
]
Alexey Zotov edited comment on CASSANDRA-16360 at 4/5/21, 6:14 AM:
---
I've made an initial research on this task. Looks like the _CRC32C_ hardware
support is only available when intrinsic code is used. Here are a few reference
tickets mentioning that:
* [https://bugs.openjdk.java.net/browse/JDK-8189745]
* [https://bugs.java.com/bugdatabase/view_bug.do?bug_id=8191328]
* [https://bugs.java.com/bugdatabase/view_bug.do?bug_id=8073583]
And it looks like intrinsic code is only available for the [native _CRC32C_
class|[https://docs.oracle.com/javase/9/docs/api/java/util/zip/CRC32C.html]]
which comes with Java 9.
Here are the results of a microbenchmark test for native _CRC32_ and _CRC32C_
classes (run was made using Java 11.0.9):
{code:java}
[java] Benchmark (bufferSize) Mode Cnt Score
Error Units
[java] ChecksumBench.benchCrc3231 avgt598.823
± 2.667 ns/op
[java] ChecksumBench.benchCrc32 131 avgt5 133.014
± 5.831 ns/op
[java] ChecksumBench.benchCrc32 517 avgt5 173.939
± 14.456 ns/op
[java] ChecksumBench.benchCrc32 2041 avgt5 270.847
± 7.071 ns/op
[java] ChecksumBench.benchCrc32NoIntrinsic 31 avgt5 118.761
± 77.032 ns/op
[java] ChecksumBench.benchCrc32NoIntrinsic131 avgt5 157.799
± 377.481 ns/op
[java] ChecksumBench.benchCrc32NoIntrinsic517 avgt5 238.125
± 900.150 ns/op
[java] ChecksumBench.benchCrc32NoIntrinsic 2041 avgt5 276.828
± 7.814 ns/op
[java] ChecksumBench.benchCrc32c 31 avgt550.619
± 2.178 ns/op
[java] ChecksumBench.benchCrc32c 131 avgt569.229
± 2.186 ns/op
[java] ChecksumBench.benchCrc32c 517 avgt5 190.943
± 3.741 ns/op
[java] ChecksumBench.benchCrc32c 2041 avgt5 276.401
± 4.161 ns/op
[java] ChecksumBench.benchCrc32cNoIntrinsic31 avgt556.111
± 1.834 ns/op
[java] ChecksumBench.benchCrc32cNoIntrinsic 131 avgt575.475
± 1.912 ns/op
[java] ChecksumBench.benchCrc32cNoIntrinsic 517 avgt5 196.557
± 5.209 ns/op
[java] ChecksumBench.benchCrc32cNoIntrinsic 2041 avgt5 281.095
± 5.765 ns/op
{code}
Two obvious facts were reaffirmed by this run:
# intrinsic code works faster than non-intrinsic
# _CRC32C_ works faster than _CRC32_
As I mentioned before, _CRC32C_ class is unavailable in Java 8. That means we
cannot directly use that in the code base, otherwise it won't compile with Java
8. So there are two ways to proceed: 1) we wait until Java 8 support is
abandoned and start using _CRC32C_ 2) if Java 9+ is used then we use _CRC32C_ ,
otherwise we fallback to some Java-based implementation. The second approach
sounds reasonable to me since we need to target for newer versions and faster
solutions.
So I've given a try to two available Java-based implementations (Guava -
_hasherCrc32c_ and Snappy - _pureJavaCrc32c_) in another microbenchmark test
(run was made using Java 11.0.9):
{code:java}
[java] Benchmark (bufferSize) Mode Cnt Score
Error Units
[java] ChecksumBench.benchCrc32 31 avgt5 113.076 ±
33.568 ns/op
[java] ChecksumBench.benchCrc32131 avgt588.348 ±
11.433 ns/op
[java] ChecksumBench.benchCrc32517 avgt5 175.464 ±
56.718 ns/op
[java] ChecksumBench.benchCrc32 2041 avgt5 273.945 ±
16.331 ns/op
[java] ChecksumBench.benchHasherCrc32c 31 avgt 25 150.860 ±
2.883 ns/op
[java] ChecksumBench.benchHasherCrc32c 131 avgt 25 559.423 ±
141.932 ns/op
[java] ChecksumBench.benchHasherCrc32c 517 avgt 25 1599.504 ±
82.253 ns/op
[java] ChecksumBench.benchHasherCrc32c2041 avgt 25 5988.707 ±
103.558 ns/op
[java] ChecksumBench.benchPureJavaCrc32c31 avgt 2599.486 ±
1.464 ns/op
[java] ChecksumBench.benchPureJavaCrc32c 131 avgt 25 252.278 ±
20.278 ns/op
[java] ChecksumBench.benchPureJavaCrc32c 517 avgt 25 822.142 ±
38.860 ns/op
[java] ChecksumBench.benchPureJavaCrc32c 2041 avgt 25 3083.762 ±
200.493 ns/op
{code}
This test shows that:
# Snappy's implementation outperforms Guava's - which basically is not really
important, we can use either of them
# They both work slower than native _CRC32C_ implementation with intrinsic -
which is actually expected
# They both work slower than native _CRC32C_
[jira] [Comment Edited] (CASSANDRA-16404) Provide a nodetool way of invalidating auth caches
[ https://issues.apache.org/jira/browse/CASSANDRA-16404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17314070#comment-17314070 ] Alexey Zotov edited comment on CASSANDRA-16404 at 4/4/21, 9:04 PM: --- [~blerer] Thanks for the details, they helped a lot! I raised a draft PR ([https://github.com/apache/cassandra/pull/950]) to validate the direction I follow, your feedback would be very helpful. Also, I posted a few questions related to the implementation. Once you confirm the direction, I'm going to make a round of local testing to ensure the functionality works as expected. was (Author: azotcsit): [~blerer] Thanks for the details, they helped a lot! I raised a draft PR to validate the direction I follow, your feedback would be very helpful. Also, I posted a few questions related to the implementation. Once you confirm the direction, I'm going to make a round of local testing to ensure the functionality works as expected. > Provide a nodetool way of invalidating auth caches > -- > > Key: CASSANDRA-16404 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16404 > Project: Cassandra > Issue Type: Improvement > Components: Feature/Authorization >Reporter: Sumanth Pasupuleti >Assignee: Sumanth Pasupuleti >Priority: Normal > Fix For: 4.x > > Time Spent: 20m > Remaining Estimate: 0h > > We currently have nodetool commands to invalidate certain caches like > KeyCache, RowCache and CounterCache. > Being able to invalidate auth caches as well can come in handy in situations > where, critical backend auth changes may need to be in effect right away for > all the connections, especially in configurations where cache validity is > chosen to be for a longer duration. An example can be that an authenticated > user "User1" is no longer authorized to access a table resource "table1" and > it is vital that this change is reflected right away, without having to wait > for cache expiry/refresh to trigger. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16360) CRC32 is inefficient on x86
[
https://issues.apache.org/jira/browse/CASSANDRA-16360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17314622#comment-17314622
]
Alexey Zotov commented on CASSANDRA-16360:
--
I've made an initial research on this task. Looks like the _CRC32C_ hardware
support is only available when intrinsic code is used. Here are a few reference
tickets mentioning that:
* [https://bugs.openjdk.java.net/browse/JDK-8189745]
* [https://bugs.java.com/bugdatabase/view_bug.do?bug_id=8191328]
* [https://bugs.java.com/bugdatabase/view_bug.do?bug_id=8073583]
And it looks like intrinsic code is only available for the [native _CRC32C_
class|[https://docs.oracle.com/javase/9/docs/api/java/util/zip/CRC32C.html]]
which comes with Java 9.
Here are the results of a microbenchmark test for native _CRC32_ and _CRC32C_
classes (run was made using Java 11.0.9):
{code:java}
[java] Benchmark (bufferSize) Mode Cnt Score
Error Units
[java] ChecksumBench.benchCrc3231 avgt598.823
± 2.667 ns/op
[java] ChecksumBench.benchCrc32 131 avgt5 133.014
± 5.831 ns/op
[java] ChecksumBench.benchCrc32 517 avgt5 173.939
± 14.456 ns/op
[java] ChecksumBench.benchCrc32 2041 avgt5 270.847
± 7.071 ns/op
[java] ChecksumBench.benchCrc32NoIntrinsic 31 avgt5 118.761
± 77.032 ns/op
[java] ChecksumBench.benchCrc32NoIntrinsic131 avgt5 157.799
± 377.481 ns/op
[java] ChecksumBench.benchCrc32NoIntrinsic517 avgt5 238.125
± 900.150 ns/op
[java] ChecksumBench.benchCrc32NoIntrinsic 2041 avgt5 276.828
± 7.814 ns/op
[java] ChecksumBench.benchCrc32c 31 avgt550.619
± 2.178 ns/op
[java] ChecksumBench.benchCrc32c 131 avgt569.229
± 2.186 ns/op
[java] ChecksumBench.benchCrc32c 517 avgt5 190.943
± 3.741 ns/op
[java] ChecksumBench.benchCrc32c 2041 avgt5 276.401
± 4.161 ns/op
[java] ChecksumBench.benchCrc32cNoIntrinsic31 avgt556.111
± 1.834 ns/op
[java] ChecksumBench.benchCrc32cNoIntrinsic 131 avgt575.475
± 1.912 ns/op
[java] ChecksumBench.benchCrc32cNoIntrinsic 517 avgt5 196.557
± 5.209 ns/op
[java] ChecksumBench.benchCrc32cNoIntrinsic 2041 avgt5 281.095
± 5.765 ns/op
{code}
Two obvious facts were reaffirmed by this run:
# intrinsic code works faster than non-intrinsic
# _CRC32C_ works faster than _CRC32_
As I mentioned before, _CRC32C_ class is unavailable in Java 8. That means we
cannot directly use that in the code base, otherwise it won't compile with Java
8. So there are two ways to proceed: 1) we wait until Java 8 support is
abandoned and start using _CRC32C_ 2) if Java 9+ is used then we use _CRC32C_ ,
otherwise we fallback to some Java-based implementation. The second approach
sounds reasonable to me since we need to target for newer versions and faster
solutions.
So I've given a try to two available Java-based implementations (Guava -
_hasherCrc32c_ and Snappy - _pureJavaCrc32c_) in another microbenchmark test
(run was made using Java 11.0.9):
{code:java}
[java] Benchmark (bufferSize) Mode Cnt Score
Error Units
[java] ChecksumBench.benchCrc32 31 avgt5 113.076 ±
33.568 ns/op
[java] ChecksumBench.benchCrc32131 avgt588.348 ±
11.433 ns/op
[java] ChecksumBench.benchCrc32517 avgt5 175.464 ±
56.718 ns/op
[java] ChecksumBench.benchCrc32 2041 avgt5 273.945 ±
16.331 ns/op
[java] ChecksumBench.benchHasherCrc32c 31 avgt 25 150.860 ±
2.883 ns/op
[java] ChecksumBench.benchHasherCrc32c 131 avgt 25 559.423 ±
141.932 ns/op
[java] ChecksumBench.benchHasherCrc32c 517 avgt 25 1599.504 ±
82.253 ns/op
[java] ChecksumBench.benchHasherCrc32c2041 avgt 25 5988.707 ±
103.558 ns/op
[java] ChecksumBench.benchPureJavaCrc32c31 avgt 2599.486 ±
1.464 ns/op
[java] ChecksumBench.benchPureJavaCrc32c 131 avgt 25 252.278 ±
20.278 ns/op
[java] ChecksumBench.benchPureJavaCrc32c 517 avgt 25 822.142 ±
38.860 ns/op
[java] ChecksumBench.benchPureJavaCrc32c 2041 avgt 25 3083.762 ±
200.493 ns/op
{code}
This test shows that:
# Snappy's implementation outperforms Guava's - which basically is not really
important, we can use either of them
# They both work slower than native _CRC32C_ implementation with intrinsic -
which is actually expected
# They both work slower than native _CRC32C_ implementation without intrinsic
- which is
[jira] [Comment Edited] (CASSANDRA-16360) CRC32 is inefficient on x86
[
https://issues.apache.org/jira/browse/CASSANDRA-16360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17314622#comment-17314622
]
Alexey Zotov edited comment on CASSANDRA-16360 at 4/4/21, 9:00 PM:
---
I've made an initial research on this task. Looks like the _CRC32C_ hardware
support is only available when intrinsic code is used. Here are a few reference
tickets mentioning that:
* [https://bugs.openjdk.java.net/browse/JDK-8189745]
* [https://bugs.java.com/bugdatabase/view_bug.do?bug_id=8191328]
* [https://bugs.java.com/bugdatabase/view_bug.do?bug_id=8073583]
And it looks like intrinsic code is only available for the [native _CRC32C_
class|[https://docs.oracle.com/javase/9/docs/api/java/util/zip/CRC32C.html]]
which comes with Java 9.
Here are the results of a microbenchmark test for native _CRC32_ and _CRC32C_
classes (run was made using Java 11.0.9):
{code:java}
[java] Benchmark (bufferSize) Mode Cnt Score
Error Units
[java] ChecksumBench.benchCrc3231 avgt598.823
± 2.667 ns/op
[java] ChecksumBench.benchCrc32 131 avgt5 133.014
± 5.831 ns/op
[java] ChecksumBench.benchCrc32 517 avgt5 173.939
± 14.456 ns/op
[java] ChecksumBench.benchCrc32 2041 avgt5 270.847
± 7.071 ns/op
[java] ChecksumBench.benchCrc32NoIntrinsic 31 avgt5 118.761
± 77.032 ns/op
[java] ChecksumBench.benchCrc32NoIntrinsic131 avgt5 157.799
± 377.481 ns/op
[java] ChecksumBench.benchCrc32NoIntrinsic517 avgt5 238.125
± 900.150 ns/op
[java] ChecksumBench.benchCrc32NoIntrinsic 2041 avgt5 276.828
± 7.814 ns/op
[java] ChecksumBench.benchCrc32c 31 avgt550.619
± 2.178 ns/op
[java] ChecksumBench.benchCrc32c 131 avgt569.229
± 2.186 ns/op
[java] ChecksumBench.benchCrc32c 517 avgt5 190.943
± 3.741 ns/op
[java] ChecksumBench.benchCrc32c 2041 avgt5 276.401
± 4.161 ns/op
[java] ChecksumBench.benchCrc32cNoIntrinsic31 avgt556.111
± 1.834 ns/op
[java] ChecksumBench.benchCrc32cNoIntrinsic 131 avgt575.475
± 1.912 ns/op
[java] ChecksumBench.benchCrc32cNoIntrinsic 517 avgt5 196.557
± 5.209 ns/op
[java] ChecksumBench.benchCrc32cNoIntrinsic 2041 avgt5 281.095
± 5.765 ns/op
{code}
Two obvious facts were reaffirmed by this run:
# intrinsic code works faster than non-intrinsic
# _CRC32C_ works faster than _CRC32_
As I mentioned before, _CRC32C_ class is unavailable in Java 8. That means we
cannot directly use that in the code base, otherwise it won't compile with Java
8. So there are two ways to proceed: 1) we wait until Java 8 support is
abandoned and start using _CRC32C_ 2) if Java 9+ is used then we use _CRC32C_ ,
otherwise we fallback to some Java-based implementation. The second approach
sounds reasonable to me since we need to target for newer versions and faster
solutions.
So I've given a try to two available Java-based implementations (Guava -
_hasherCrc32c_ and Snappy - _pureJavaCrc32c_) in another microbenchmark test
(run was made using Java 11.0.9):
{code:java}
[java] Benchmark (bufferSize) Mode Cnt Score
Error Units
[java] ChecksumBench.benchCrc32 31 avgt5 113.076 ±
33.568 ns/op
[java] ChecksumBench.benchCrc32131 avgt588.348 ±
11.433 ns/op
[java] ChecksumBench.benchCrc32517 avgt5 175.464 ±
56.718 ns/op
[java] ChecksumBench.benchCrc32 2041 avgt5 273.945 ±
16.331 ns/op
[java] ChecksumBench.benchHasherCrc32c 31 avgt 25 150.860 ±
2.883 ns/op
[java] ChecksumBench.benchHasherCrc32c 131 avgt 25 559.423 ±
141.932 ns/op
[java] ChecksumBench.benchHasherCrc32c 517 avgt 25 1599.504 ±
82.253 ns/op
[java] ChecksumBench.benchHasherCrc32c2041 avgt 25 5988.707 ±
103.558 ns/op
[java] ChecksumBench.benchPureJavaCrc32c31 avgt 2599.486 ±
1.464 ns/op
[java] ChecksumBench.benchPureJavaCrc32c 131 avgt 25 252.278 ±
20.278 ns/op
[java] ChecksumBench.benchPureJavaCrc32c 517 avgt 25 822.142 ±
38.860 ns/op
[java] ChecksumBench.benchPureJavaCrc32c 2041 avgt 25 3083.762 ±
200.493 ns/op
{code}
This test shows that:
# Snappy's implementation outperforms Guava's - which basically is not really
important, we can use either of them
# They both work slower than native _CRC32C_ implementation with intrinsic -
which is actually expected
# They both work slower than native _CRC32C_
[jira] [Commented] (CASSANDRA-16404) Provide a nodetool way of invalidating auth caches
[ https://issues.apache.org/jira/browse/CASSANDRA-16404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17314070#comment-17314070 ] Alexey Zotov commented on CASSANDRA-16404: -- [~blerer] Thanks for the details, they helped a lot! I raised a draft PR to validate the direction I follow, your feedback would be very helpful. Also, I posted a few questions related to the implementation. Once you confirm the direction, I'm going to make a round of local testing to ensure the functionality works as expected. > Provide a nodetool way of invalidating auth caches > -- > > Key: CASSANDRA-16404 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16404 > Project: Cassandra > Issue Type: Improvement > Components: Feature/Authorization >Reporter: Sumanth Pasupuleti >Assignee: Sumanth Pasupuleti >Priority: Normal > Fix For: 4.x > > Time Spent: 10m > Remaining Estimate: 0h > > We currently have nodetool commands to invalidate certain caches like > KeyCache, RowCache and CounterCache. > Being able to invalidate auth caches as well can come in handy in situations > where, critical backend auth changes may need to be in effect right away for > all the connections, especially in configurations where cache validity is > chosen to be for a longer duration. An example can be that an authenticated > user "User1" is no longer authorized to access a table resource "table1" and > it is vital that this change is reflected right away, without having to wait > for cache expiry/refresh to trigger. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16404) Provide a nodetool way of invalidating auth caches
[
https://issues.apache.org/jira/browse/CASSANDRA-16404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17310460#comment-17310460
]
Alexey Zotov commented on CASSANDRA-16404:
--
[~brandon.williams], [~blerer]
Could you please assist with triaging this issue. I went through the code and
in general this ticket seems to be clear from technical perspective, but there
are a few points to clarify:
# Despite the use case sounds reasonable to me, does the product ever need
this functionality?
# There are multiple caches (roles, permissions, network auth, credentials,
JMX permissions), but I feel it is enough to invalidate permissions cache only
to handle the use case. Could you please confirm?
# As far as I can see, the invalidation of permissions cache only does not
affect (need for re-login or similar side effects) logged in users. So it is
safe to invalidate the cache and does not bother all users of the system. Could
you please confirm?
# Even though permissions cache invalidation does not affect other users, we
need to provide an optional {{user}} parameter, so the administrator can
invalidate the cache for that particular user rather than for all users.
Thoughts?
# As far as I understand {{nodetool}} is executed on a single node, so the
auth cache invalidation needs to be triggered on all nodes manually. It seems
be slightly inconvenient for the administrator... However, it provides better
control and it is how it is done for all (? not sure) other commands. Is there
a way (concept) to execute such a command on all nodes?
> Provide a nodetool way of invalidating auth caches
> --
>
> Key: CASSANDRA-16404
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16404
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Sumanth Pasupuleti
>Assignee: Sumanth Pasupuleti
>Priority: Normal
> Fix For: 4.x
>
>
> We currently have nodetool commands to invalidate certain caches like
> KeyCache, RowCache and CounterCache.
> Being able to invalidate auth caches as well can come in handy in situations
> where, critical backend auth changes may need to be in effect right away for
> all the connections, especially in configurations where cache validity is
> chosen to be for a longer duration. An example can be that an authenticated
> user "User1" is no longer authorized to access a table resource "table1" and
> it is vital that this change is reflected right away, without having to wait
> for cache expiry/refresh to trigger.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-8181) Intermittent failure of SSTableImportTest unit test
[
https://issues.apache.org/jira/browse/CASSANDRA-8181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14222833#comment-14222833
]
Alexey Zotov commented on CASSANDRA-8181:
-
[~blerer] good catch, the fix LGTM. Thanks.
Intermittent failure of SSTableImportTest unit test
---
Key: CASSANDRA-8181
URL: https://issues.apache.org/jira/browse/CASSANDRA-8181
Project: Cassandra
Issue Type: Bug
Components: Tests
Reporter: Michael Shuler
Assignee: Benjamin Lerer
Fix For: 2.1.3
Attachments: CASSANDRA-8181.txt,
SSTableImportTest_failure_system.log.gz
{noformat}
[junit] Testsuite: org.apache.cassandra.tools.SSTableImportTest
[junit] Tests run: 8, Failures: 1, Errors: 0, Time elapsed: 6.712 sec
[junit]
[junit] - Standard Output ---
[junit] Counting keys to import, please wait... (NOTE: to skip this use
-n num_keys)
[junit] Importing 2 keys...
[junit] 2 keys imported successfully.
[junit] Counting keys to import, please wait... (NOTE: to skip this use
-n num_keys)
[junit] Importing 2 keys...
[junit] 2 keys imported successfully.
[junit] Counting keys to import, please wait... (NOTE: to skip this use
-n num_keys)
[junit] Importing 2 keys...
[junit] 2 keys imported successfully.
[junit] Counting keys to import, please wait... (NOTE: to skip this use
-n num_keys)
[junit] Importing 2 keys...
[junit] Importing 2 keys...
[junit] 2 keys imported successfully.
[junit] Counting keys to import, please wait... (NOTE: to skip this use
-n num_keys)
[junit] Importing 2 keys...
[junit] 2 keys imported successfully.
[junit] Counting keys to import, please wait... (NOTE: to skip this use
-n num_keys)
[junit] Importing 1 keys...
[junit] 1 keys imported successfully.
[junit] Counting keys to import, please wait... (NOTE: to skip this use
-n num_keys)
[junit] Importing 2 keys...
[junit] 2 keys imported successfully.
[junit] - ---
[junit] Testcase:
testImportWithAsciiKeyValidator(org.apache.cassandra.tools.SSTableImportTest):
FAILED
[junit] null
[junit] junit.framework.AssertionFailedError
[junit] at
org.apache.cassandra.tools.SSTableImportTest.testImportWithAsciiKeyValidator(SSTableImportTest.java:166)
[junit]
[junit]
[junit] Test org.apache.cassandra.tools.SSTableImportTest FAILED
{noformat}
testImportWithAsciiKeyValidator was added in CASSANDRA-7498 and fails as
above occasionally (~10-15% of runs) in CI. Attached is the system.log from
the failed test on 2.1 HEAD (8e5fdc2).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7493) Remove traces of rows_per_partition_to_cache
[ https://issues.apache.org/jira/browse/CASSANDRA-7493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14118101#comment-14118101 ] Alexey Zotov commented on CASSANDRA-7493: - Hmm.. why it was removed as a part of this cleanup? As for me _populate_io_cache_on_flush_ is irrelevant with caching mechanism because it uses internal OS capabilities for storing data in-memory. Also I think it is quite strange to remove this feature since it can be very useful if dataset fits memory. We used this option to have the whole dataset in memory and it gave us high performance with low latency. What do you think about reverting the changes and starting a discussion regarding this option? PS: I guess this option is not very popular because it does not have proper documentation. Maybe it would be better to write an article (+ some info in official documentation) with detailed explanation why and when it can be used than just remove it. If such approach works I can help with it. Remove traces of rows_per_partition_to_cache Key: CASSANDRA-7493 URL: https://issues.apache.org/jira/browse/CASSANDRA-7493 Project: Cassandra Issue Type: Bug Reporter: Marcus Eriksson Assignee: Marcus Eriksson Fix For: 2.1 rc3 Attachments: 0001-remove-rows_per_partition_to_cache.patch CASSANDRA-6745 changed the way we configure how many rows per partition we cache, some old stuff was left in, clear it out -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-7493) Remove traces of rows_per_partition_to_cache
[ https://issues.apache.org/jira/browse/CASSANDRA-7493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14118101#comment-14118101 ] Alexey Zotov edited comment on CASSANDRA-7493 at 9/2/14 11:18 AM: -- Hmm.. why it was removed as a part of this cleanup? As for me _populate_io_cache_on_flush_ is irrelevant with caching mechanism because it uses internal OS capabilities for storing data in-memory. If it was decided to remove it then should be created a separate ticket for that since it is isolated feature. Also I think it is quite strange to remove this feature since it can be very useful if dataset fits memory. We used this option to have the whole dataset in memory and it gave us high performance with low latency. What do you think about reverting the changes and starting a discussion regarding this option? PS: I guess this option is not very popular because it does not have proper documentation. Maybe it would be better to write an article (+ some info in official documentation) with detailed explanation why and when it can be used than just remove it. If such approach works I can help with it. was (Author: azotcsit): Hmm.. why it was removed as a part of this cleanup? As for me _populate_io_cache_on_flush_ is irrelevant with caching mechanism because it uses internal OS capabilities for storing data in-memory. Also I think it is quite strange to remove this feature since it can be very useful if dataset fits memory. We used this option to have the whole dataset in memory and it gave us high performance with low latency. What do you think about reverting the changes and starting a discussion regarding this option? PS: I guess this option is not very popular because it does not have proper documentation. Maybe it would be better to write an article (+ some info in official documentation) with detailed explanation why and when it can be used than just remove it. If such approach works I can help with it. Remove traces of rows_per_partition_to_cache Key: CASSANDRA-7493 URL: https://issues.apache.org/jira/browse/CASSANDRA-7493 Project: Cassandra Issue Type: Bug Reporter: Marcus Eriksson Assignee: Marcus Eriksson Fix For: 2.1 rc3 Attachments: 0001-remove-rows_per_partition_to_cache.patch CASSANDRA-6745 changed the way we configure how many rows per partition we cache, some old stuff was left in, clear it out -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7493) Remove traces of rows_per_partition_to_cache
[ https://issues.apache.org/jira/browse/CASSANDRA-7493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14118143#comment-14118143 ] Alexey Zotov commented on CASSANDRA-7493: - [~krummas], ohh.. That's a good new. Thanks for the pointing to the ticket. I've checked the code, it looks good to me. PS: Sorry for the false alarm :) Remove traces of rows_per_partition_to_cache Key: CASSANDRA-7493 URL: https://issues.apache.org/jira/browse/CASSANDRA-7493 Project: Cassandra Issue Type: Bug Reporter: Marcus Eriksson Assignee: Marcus Eriksson Fix For: 2.1 rc3 Attachments: 0001-remove-rows_per_partition_to_cache.patch CASSANDRA-6745 changed the way we configure how many rows per partition we cache, some old stuff was left in, clear it out -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7493) Remove traces of rows_per_partition_to_cache
[ https://issues.apache.org/jira/browse/CASSANDRA-7493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14116564#comment-14116564 ] Alexey Zotov commented on CASSANDRA-7493: - Am I right that _populate_io_cache_on_flush_ was removed for CQL only? Let's imagine I have just created a CQL table (i.e. column family) and I want to enable _populate_io_cache_on_flush_ for it. Is there any way to do it (cli or smth else)? Remove traces of rows_per_partition_to_cache Key: CASSANDRA-7493 URL: https://issues.apache.org/jira/browse/CASSANDRA-7493 Project: Cassandra Issue Type: Bug Reporter: Marcus Eriksson Assignee: Marcus Eriksson Fix For: 2.1 rc3 Attachments: 0001-remove-rows_per_partition_to_cache.patch CASSANDRA-6745 changed the way we configure how many rows per partition we cache, some old stuff was left in, clear it out -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-7498) sstable2json exports keys in a wrong format
[ https://issues.apache.org/jira/browse/CASSANDRA-7498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14089703#comment-14089703 ] Alexey Zotov commented on CASSANDRA-7498: - Thanks. sstable2json exports keys in a wrong format --- Key: CASSANDRA-7498 URL: https://issues.apache.org/jira/browse/CASSANDRA-7498 Project: Cassandra Issue Type: Bug Components: Tools Reporter: Alexey Zotov Assignee: Alexey Zotov Priority: Minor Labels: tools Fix For: 2.1.1 Attachments: CASSANDRA-2.1-7498.txt I examined code of _SSTableImport_ and _SSTableExport_ classes and found one thing that looks strange to me. Currently processing (encoding/decoding) of columns' names and values is performed based on CF Comparator and Value Validator correspondingly. But row keys are processed using _#bytesToHex_ and _#hexToBytes_ methods despite of there is Key Validator. Such approach works properly for bytes keys only. For other types it exports keys in a wrong format (i.e. they are exported in unreadable form). In the scope of this ticket we need to consider backward compatibility (import of previously exported sstables). For bytes keys it will work as previously since _#bytesToHex_ and _#hexToBytes_ methods are used in internals of _BytesType_ class. For other types we need to allow an ability to import data using bytes despite of configured validator. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-7498) sstable2json exports keys in a wrong format
[ https://issues.apache.org/jira/browse/CASSANDRA-7498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14055881#comment-14055881 ] Alexey Zotov commented on CASSANDRA-7498: - Jon, As I understand from CASSANDRA-7464's discussion new tools will be introduced and the current utilities will be deprecated (and after some time deleted). But since the final design is unclear we cannot say when they will be implemented (I suppose it will take about half a year since it is not minor improvement). Can we include this patch as a fix of the current issue? sstable2json exports keys in a wrong format --- Key: CASSANDRA-7498 URL: https://issues.apache.org/jira/browse/CASSANDRA-7498 Project: Cassandra Issue Type: Bug Components: Core Reporter: Alexey Zotov Assignee: Alexey Zotov Labels: tools Attachments: CASSANDRA-2.1-7498.txt I examined code of _SSTableImport_ and _SSTableExport_ classes and found one thing that looks strange to me. Currently processing (encoding/decoding) of columns' names and values is performed based on CF Comparator and Value Validator correspondingly. But row keys are processed using _#bytesToHex_ and _#hexToBytes_ methods despite of there is Key Validator. Such approach works properly for bytes keys only. For other types it exports keys in a wrong format (i.e. they are exported in unreadable form). In the scope of this ticket we need to consider backward compatibility (import of previously exported sstables). For bytes keys it will work as previously since _#bytesToHex_ and _#hexToBytes_ methods are used in internals of _BytesType_ class. For other types we need to allow an ability to import data using bytes despite of configured validator. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-7464) Retire/replace sstable2json and json2sstable
[ https://issues.apache.org/jira/browse/CASSANDRA-7464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14055883#comment-14055883 ] Alexey Zotov commented on CASSANDRA-7464: - Guys, as I understand you are trying to propose more cql-compatible output, am I rigth? I came to such conclusion because I do not now what is _partition_key_ and _static_block_ abstractions. I will read about them a bit later. Actually my question is where is the row key for above output? Is it just missed? Also I would like to try create a patch, but I'm confused by requirements for the output, for me they are quite unclear. Could you, please, give me more details. Retire/replace sstable2json and json2sstable Key: CASSANDRA-7464 URL: https://issues.apache.org/jira/browse/CASSANDRA-7464 Project: Cassandra Issue Type: Improvement Reporter: Sylvain Lebresne Priority: Minor Both tools are pretty awful. They are primarily meant for debugging (there is much more efficient and convenient ways to do import/export data), but their output manage to be hard to handle both for humans and for tools (especially as soon as you have modern stuff like composites). There is value to having tools to export sstable contents into a format that is easy to manipulate by human and tools for debugging, small hacks and general tinkering, but sstable2json and json2sstable are not that. So I propose that we deprecate those tools and consider writing better replacements. It shouldn't be too hard to come up with an output format that is more aware of modern concepts like composites, UDTs, -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (CASSANDRA-7498) sstable2json exports keys in a wrong format
[ https://issues.apache.org/jira/browse/CASSANDRA-7498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Zotov updated CASSANDRA-7498: Attachment: CASSANDRA-2.1-7498.txt I have already created a patch before your comment. The patch contains required changes for processing keys in sstable export/import utilities. Regarding backward compatibility, I added an ability to ignore the current key validator type to allow users import previously-generated json files for non-bytes keys. For that users need to add -Dskip.key.validator=true parameter during import. sstable2json exports keys in a wrong format --- Key: CASSANDRA-7498 URL: https://issues.apache.org/jira/browse/CASSANDRA-7498 Project: Cassandra Issue Type: Bug Components: Core Reporter: Alexey Zotov Assignee: Alexey Zotov Labels: tools Attachments: CASSANDRA-2.1-7498.txt I examined code of _SSTableImport_ and _SSTableExport_ classes and found one thing that looks strange to me. Currently processing (encoding/decoding) of columns' names and values is performed based on CF Comparator and Value Validator correspondingly. But row keys are processed using _#bytesToHex_ and _#hexToBytes_ methods despite of there is Key Validator. Such approach works properly for bytes keys only. For other types it exports keys in a wrong format (i.e. they are exported in unreadable form). In the scope of this ticket we need to consider backward compatibility (import of previously exported sstables). For bytes keys it will work as previously since _#bytesToHex_ and _#hexToBytes_ methods are used in internals of _BytesType_ class. For other types we need to allow an ability to import data using bytes despite of configured validator. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (CASSANDRA-7498) sstable2json exports keys in a wrong format
Alexey Zotov created CASSANDRA-7498: --- Summary: sstable2json exports keys in a wrong format Key: CASSANDRA-7498 URL: https://issues.apache.org/jira/browse/CASSANDRA-7498 Project: Cassandra Issue Type: Bug Components: Core Reporter: Alexey Zotov Assignee: Alexey Zotov I examined code of _SSTableImport_ and _SSTableExport_ classes and found one thing that looks strange to me. Currently processing (encoding/decoding) of columns' names and values is performed based on CF Comparator and Value Validator correspondingly. But row keys are processed using _#bytesToHex_ and _#hexToBytes_ methods despite of there is Key Validator. Such approach works properly for bytes keys only. For other types it exports keys in a wrong format (i.e. they are exported in unreadable form). In the scope of this ticket we need to consider backward compatibility (import of previously exported sstables). For bytes keys it will work as previously since _#bytesToHex_ and _#hexToBytes_ methods are used in internals of _BytesType_ class. For other types we need to allow an ability to import data using bytes despite of configured validator. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (CASSANDRA-6262) Nodetool compact throws an error after importing data with sstableloader
[
https://issues.apache.org/jira/browse/CASSANDRA-6262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexey Zotov updated CASSANDRA-6262:
Since Version: 1.2.4 (was: 1.2.6)
Nodetool compact throws an error after importing data with sstableloader
Key: CASSANDRA-6262
URL: https://issues.apache.org/jira/browse/CASSANDRA-6262
Project: Cassandra
Issue Type: Bug
Components: Core
Reporter: J.B. Langston
Assignee: Sylvain Lebresne
Fix For: 1.2.14
Attachments: 6262.patch
Exception when running nodetool compact:
{code}
Error occurred during compaction
java.util.concurrent.ExecutionException: java.lang.IndexOutOfBoundsException:
index (2) must be less than size (2)
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
at java.util.concurrent.FutureTask.get(FutureTask.java:111)
at
org.apache.cassandra.db.compaction.CompactionManager.performMaximal(CompactionManager.java:331)
at
org.apache.cassandra.db.ColumnFamilyStore.forceMajorCompaction(ColumnFamilyStore.java:1691)
at
org.apache.cassandra.service.StorageService.forceTableCompaction(StorageService.java:2198)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:75)
at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:279)
at
com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112)
at
com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46)
at
com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)
at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252)
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)
at
com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)
at
javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1487)
at
javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:97)
at
javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1328)
at
javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1420)
at
javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:848)
at sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
at sun.rmi.transport.Transport$1.run(Transport.java:177)
at sun.rmi.transport.Transport$1.run(Transport.java:174)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
at
sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
Caused by: java.lang.IndexOutOfBoundsException: index (2) must be less than
size (2)
at
com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:305)
at
com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:284)
at
com.google.common.collect.RegularImmutableList.get(RegularImmutableList.java:81)
at
org.apache.cassandra.db.marshal.CompositeType.getComparator(CompositeType.java:94)
at
org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractCompositeType.java:76)
at
org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractCompositeType.java:31)
at
[jira] [Commented] (CASSANDRA-6018) Add option to encrypt commitlog
[ https://issues.apache.org/jira/browse/CASSANDRA-6018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13858263#comment-13858263 ] Alexey Zotov commented on CASSANDRA-6018: - Jason, this is not my business, but I do not suggest you to use Cassandra for a billing system: - currently Cassandra has some issues with loosing data (at least one: CASSANDRA-5789) - there were fixed a lot of loosing data issues for the previous year - Cassandra is actively developed right now and this gives additional risks of newly-arrived issues. Moreover it's an open-source solution (despite of you are using Enterprise edition) and no-one guarantees nothing. But this is my vision and I hope everything will be fine. Add option to encrypt commitlog Key: CASSANDRA-6018 URL: https://issues.apache.org/jira/browse/CASSANDRA-6018 Project: Cassandra Issue Type: New Feature Components: Core Reporter: Jason Brown Assignee: Jason Brown Labels: commit_log, encryption, security Fix For: 2.1 We are going to start using cassandra for a billing system, and while I can encrypt sstables at rest (via Datastax Enterprise), commit logs are more or less plain text. Thus, an attacker would be able to easily read, for example, credit card numbers in the clear text commit log (if the calling app does not encrypt the data itself before sending it to cassandra). I want to allow the option of encrypting the commit logs, most likely controlled by a property in the yaml. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Assigned] (CASSANDRA-4655) Truncate operation doesn't delete rows from HintsColumnFamily.
[ https://issues.apache.org/jira/browse/CASSANDRA-4655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Zotov reassigned CASSANDRA-4655: --- Assignee: Alexey Zotov Truncate operation doesn't delete rows from HintsColumnFamily. -- Key: CASSANDRA-4655 URL: https://issues.apache.org/jira/browse/CASSANDRA-4655 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 0.7.0 Environment: Centos 6.2, Cassandra 1.1.4 (DataStax distribution), three-nodes cluster. Reporter: Alexey Zotov Assignee: Alexey Zotov Priority: Minor Labels: hintedhandoff, truncate Fix For: 1.2.3 Attachments: cassandra-1.2-4655-hints_truncation.txt, cassandra-1.2-4655-hints_truncation-v2.txt Steps to reproduce: 1. Start writing of data to some column family, let name it 'MyCF' 2. Stop 1 node 3. Wait some time (until some data will be collected in HintsColumnFamily) 4. Start node (HintedHandoff will be started automatically for 'MyCF') 5. Run 'truncate' command for 'MyCF' column family from command from cli 6. Wait until truncate will be finished 7. You will see that 'MyCF' is not empty because HintedHandoff is copying data So, I suggest to clean HintsColumnFamily (for truncated column family) before we had started to discard sstables. I think it should be done in CompactionManager#submitTrucate() method. I can try to create patch but I need to know right way of cleaning HintsColumnFamily. Could you clarify it? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4694) populate_io_cache_on_flush option should be configurable for each column family independently
[ https://issues.apache.org/jira/browse/CASSANDRA-4694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13568471#comment-13568471 ] Alexey Zotov commented on CASSANDRA-4694: - Thanks! populate_io_cache_on_flush option should be configurable for each column family independently - Key: CASSANDRA-4694 URL: https://issues.apache.org/jira/browse/CASSANDRA-4694 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Alexey Zotov Assignee: Alexey Zotov Priority: Minor Labels: cli, core, documentation Fix For: 1.2.2 Attachments: cassandra-1.1.2-4694-populate_io_cache_v3.txt, cassandra-1.1-4694-populate_io_cache.txt, cassandra-1.2-4694-populate_io_cache.txt, cassandra-1.2-4694-populate_io_cache_v2.txt, cassandra-1.2-populate_io_cache_v4.txt I suggest to configure populate_io_cache_on_flush option for each column family. It should be configurable from cassandra-cli and should be stored in System keyspace. That could be useful if you have a few column families inside single keyspace and you need to fit in memory only one of them. Patch has been attached. I've been testing it on pseudo-cluster using ccm. So I don't have fully confidence about lack of bugs. Please carefully review that code. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4694) populate_io_cache_on_flush option should be configurable for each column family independently
[ https://issues.apache.org/jira/browse/CASSANDRA-4694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Zotov updated CASSANDRA-4694: Attachment: cassandra-1.2-populate_io_cache_v4.txt Thanks. I attached a new version of the patch. populate_io_cache_on_flush option should be configurable for each column family independently - Key: CASSANDRA-4694 URL: https://issues.apache.org/jira/browse/CASSANDRA-4694 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Alexey Zotov Assignee: Alexey Zotov Priority: Minor Labels: cli, core, documentation Fix For: 1.2.1 Attachments: cassandra-1.1.2-4694-populate_io_cache_v3.txt, cassandra-1.1-4694-populate_io_cache.txt, cassandra-1.2-4694-populate_io_cache.txt, cassandra-1.2-4694-populate_io_cache_v2.txt, cassandra-1.2-populate_io_cache_v4.txt I suggest to configure populate_io_cache_on_flush option for each column family. It should be configurable from cassandra-cli and should be stored in System keyspace. That could be useful if you have a few column families inside single keyspace and you need to fit in memory only one of them. Patch has been attached. I've been testing it on pseudo-cluster using ccm. So I don't have fully confidence about lack of bugs. Please carefully review that code. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-5165) Need to trim DC and RACK names in cassandra-topology.properties file
[ https://issues.apache.org/jira/browse/CASSANDRA-5165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13557137#comment-13557137 ] Alexey Zotov commented on CASSANDRA-5165: - Thanks! Need to trim DC and RACK names in cassandra-topology.properties file Key: CASSANDRA-5165 URL: https://issues.apache.org/jira/browse/CASSANDRA-5165 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.1.6 Environment: cassandra-1.1.6 (DataStax distribution). Reporter: Alexey Zotov Assignee: Alexey Zotov Priority: Minor Labels: snitch Fix For: 1.2.1 Attachments: cassandra-1.2-5165_trim_spaces.txt Some misprints in cassandra-topology.properties file can be painful for debugging. For example extra symbols at the end of line like a space. So difference between DC1:RACK1 and DC1:RACK1 couldn't be detected using nodetool ring command. I think that symbols like a space shouldn't be allowed in DC/RACK names. So I suggest to trim them. The patch has been attached. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
