[jira] [Commented] (CASSANDRA-14162) Backport 7950

[Jeff Jirsa (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16321804#comment-16321804
 ] 

Jeff Jirsa commented on CASSANDRA-14162:


[~iamaleksey] / [~slebresne] / [~jjordan] - The only argument against such a 
backport seems like "it may break someone's tool parsing" (if someone's relying 
on some specifics of the old, awkward nodetool output). Is that something we 
typically protect against in minor versions?



> Backport 7950
> -
>
> Key: CASSANDRA-14162
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14162
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Kurt Greaves
>Assignee: Kurt Greaves
>Priority: Minor
> Fix For: 3.0.16
>
> Attachments: 14162-3.0.patch, Screenshot from 2018-01-11 
> 01-02-02.png, Screenshot from 2018-01-11 01-02-46.png, Screenshot from 
> 2018-01-11 01-02-51.png
>
>
> Colleagues have had issues with output of listsnapshots/compactionstats 
> because of things with really long names. Mostly cosmetic but I see no reason 
> we shouldn't backport CASSANDRA-7950 to 3.0. It's practically a bugfix. I've 
> attached a patch and a bunch of images to show the relevant commands working 
> as intended after applying the patch.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-14162) Backport 7950

[Kurt Greaves (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kurt Greaves updated CASSANDRA-14162:
-
Status: Patch Available  (was: Open)

> Backport 7950
> -
>
> Key: CASSANDRA-14162
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14162
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Kurt Greaves
>Assignee: Kurt Greaves
>Priority: Minor
> Fix For: 3.0.16
>
> Attachments: 14162-3.0.patch, Screenshot from 2018-01-11 
> 01-02-02.png, Screenshot from 2018-01-11 01-02-46.png, Screenshot from 
> 2018-01-11 01-02-51.png
>
>
> Colleagues have had issues with output of listsnapshots/compactionstats 
> because of things with really long names. Mostly cosmetic but I see no reason 
> we shouldn't backport CASSANDRA-7950 to 3.0. It's practically a bugfix. I've 
> attached a patch and a bunch of images to show the relevant commands working 
> as intended after applying the patch.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-14162) Backport 7950

[Kurt Greaves (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kurt Greaves updated CASSANDRA-14162:
-
Attachment: 14162-3.0.patch
Screenshot from 2018-01-11 01-02-51.png
Screenshot from 2018-01-11 01-02-46.png
Screenshot from 2018-01-11 01-02-02.png

> Backport 7950
> -
>
> Key: CASSANDRA-14162
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14162
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Kurt Greaves
>Assignee: Kurt Greaves
>Priority: Minor
> Fix For: 3.0.16
>
> Attachments: 14162-3.0.patch, Screenshot from 2018-01-11 
> 01-02-02.png, Screenshot from 2018-01-11 01-02-46.png, Screenshot from 
> 2018-01-11 01-02-51.png
>
>
> Colleagues have had issues with output of listsnapshots/compactionstats 
> because of things with really long names. Mostly cosmetic but I see no reason 
> we shouldn't backport CASSANDRA-7950 to 3.0. It's practically a bugfix. I've 
> attached a patch and a bunch of images to show the relevant commands working 
> as intended after applying the patch.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-14162) Backport 7950

[Kurt Greaves (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kurt Greaves updated CASSANDRA-14162:
-
Description: Colleagues have had issues with output of 
listsnapshots/compactionstats because of things with really long names. Mostly 
cosmetic but I see no reason we shouldn't backport CASSANDRA-7950 to 3.0. It's 
practically a bugfix. I've attached a patch and a bunch of images to show the 
relevant commands working as intended after applying the patch.  (was: 
Colleagues have had issues with output of listsnapshots/compactionstats because 
of things with really long names. Mostly cosmetic but I see no reason we 
shouldn't backport 7950 to 3.0. It's practically a bugfix. I've attached a 
patch and a bunch of images to show the relevant commands working as intended 
after applying the patch.)

> Backport 7950
> -
>
> Key: CASSANDRA-14162
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14162
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Kurt Greaves
>Assignee: Kurt Greaves
>Priority: Minor
> Fix For: 3.0.16
>
>
> Colleagues have had issues with output of listsnapshots/compactionstats 
> because of things with really long names. Mostly cosmetic but I see no reason 
> we shouldn't backport CASSANDRA-7950 to 3.0. It's practically a bugfix. I've 
> attached a patch and a bunch of images to show the relevant commands working 
> as intended after applying the patch.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Created] (CASSANDRA-14162) Backport 7950

[Kurt Greaves (JIRA)]

Kurt Greaves created CASSANDRA-14162:


 Summary: Backport 7950
 Key: CASSANDRA-14162
 URL: https://issues.apache.org/jira/browse/CASSANDRA-14162
 Project: Cassandra
  Issue Type: Bug
Reporter: Kurt Greaves
Assignee: Kurt Greaves
Priority: Minor
 Fix For: 3.0.16


Colleagues have had issues with output of listsnapshots/compactionstats because 
of things with really long names. Mostly cosmetic but I see no reason we 
shouldn't backport 7950 to 3.0. It's practically a bugfix. I've attached a 
patch and a bunch of images to show the relevant commands working as intended 
after applying the patch.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14134) Migrate dtests to use pytest and python3

[Michael Kjellman (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16321454#comment-16321454
 ] 

Michael Kjellman commented on CASSANDRA-14134:
--

Happy to remove any commits that added sleeps to attempt to work around flaky 
tests due to schema changes -- some of them I did with [~jjirsa] and some 
myself. We too already discussed that maybe these are fundamental things that 
ccm needs to handle so it's fixed everywhere. I don't really know the answer 
there ot be honest.

> Migrate dtests to use pytest and python3
> 
>
> Key: CASSANDRA-14134
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14134
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Testing
>Reporter: Michael Kjellman
>Assignee: Michael Kjellman
>
> h4. Get the C* dtests running on the pytest framework.
> C* DTests currently run using the python test framework nosetest. This 
> framework has been largely abandoned with no releases since 2015 and a 
> general strong consensus in the python community that pytest is the future.
> h4. Why should we do this.
> Currently (and historically) dtests have always been difficult to run, flaky 
> and unpredictable in CI environments, and almost impossible to debug.
> On November 28th, 2017, I proposed on the dev@ list that we move the dtests 
> from nosetests to pytests. I got replies from Jon Haddad, Philip Thompson, 
> and kurt greaves with really only "+1" like replies to the proposal.
> Since then I've been working pretty much non stop to complete the large 
> refactor of dtests to pytests. As part of this effort (and due to the 
> migration tools that exist require it) I also ported the code to python3 
> (from the current python 2.7 based code-base).
> h4. High-level summary of key changes, improvements, and new features.
> * Migrate dtests from executing using the nosetest framework to pytest
> * Port the entire code base from Python 2.7 to Python 3.6
> * Update run_dtests.py to work with pytest
> * Add --dtest-print-tests-only option to run_dtests.py to get easily parsable 
> list of all available collected tests
> * Update README.md for executing the dtests with pytest
> * Add new debugging tips section to README.md to help with some basics of 
> debugging python3 and pytest
> * Migrate all existing Enviornment Variable usage as a means to control dtest 
> operation modes to argparse command line options with documented help on each 
> toggles intended usage
> * Migration of old unitTest and nose based test structure to modern pytest 
> fixture approach
> * Automatic detection of physical system resources to automatically determine 
> if @pytest.mark.resource_intensive annotated tests should be collected and 
> run on the system where they are being executed
> * new pytest fixture replacements for @since and @pytest.mark.upgrade_test 
> annotations
> * Migration to python logging framework
> * Upgrade thrift bindings to latest version with full python3 compatibility
> * Remove deprecated cql and pycassa dependencies and migrate any remaining 
> tests to fully remove those dependencies
> * Fixed dozens of tests that would hang the pytest framework forever when run 
> in CI enviornments
> * Ran code nearly 300 times in CircleCI during the migration and to find, 
> identify, and fix any tests capable of hanging CI
> * Upgrade Tests do not yet run in CI and still need additional migration work 
> (although all upgrade test classes compile successfully)
> I started with the *nose2pytest* [https://github.com/pytest-dev/nose2pytest] 
> migration tool. As this required python 3 language support I found myself 
> down the 2to3 python migration path. While painful to do this, the benefits 
> of python3 over python2.7 are numerous and moving to python3 for the 
> additional debugging tools now available to use when fixing dtests makes the 
> effort worth it for that reason alone!
> After the automated tools did their thing I began what was a much longer and 
> tedious manual process than I ever could have expected due to the custom many 
> ways we did things in dtests (frequently to work around nosetest limitations 
> of missing features that thankfully are now all included with the pytest 
> framework). I've done nearly 300 test runs of my migration branch with 
> circleci.
> The latest CircleCI runs can be found at:
> (dtests without vnodes) [https://circleci.com/gh/mkjellman/cassandra/277]
> (dtests with vnodes) [https://circleci.com/gh/mkjellman/cassandra/278]
> With vnodes, there are currently only 6 remaining dtest test failures.
> Without vnodes, there are 12 remaining dtest failures.
> It turns out that after the dtests were moved to ASF Jenkins from cassci, the 
> jobs were misconfigured and we actually haven't been running the dtests in 
> the 

[jira] [Commented] (CASSANDRA-14134) Migrate dtests to use pytest and python3

[Ariel Weisberg (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16321420#comment-16321420
 ] 

Ariel Weisberg commented on CASSANDRA-14134:


So here are my ponies. I don't want to conflate the pytest changes with test 
fixes. The test fixes from Jan 4th and on I would really like to have broken 
out into individual tickets (per suite) and then we can run them to see which 
of these sleeps helps vs which doesn't. I just went and ran one test for 
instance and it's still flaky in multiple locations in the test including after 
a sleep was inserted.

I know it's annoying to break it out. The alternative is that we put in a bunch 
of changes some of which we don't know are effective and there some extra test 
code burden carrying around things that don't help. WDYT?

> Migrate dtests to use pytest and python3
> 
>
> Key: CASSANDRA-14134
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14134
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Testing
>Reporter: Michael Kjellman
>Assignee: Michael Kjellman
>
> h4. Get the C* dtests running on the pytest framework.
> C* DTests currently run using the python test framework nosetest. This 
> framework has been largely abandoned with no releases since 2015 and a 
> general strong consensus in the python community that pytest is the future.
> h4. Why should we do this.
> Currently (and historically) dtests have always been difficult to run, flaky 
> and unpredictable in CI environments, and almost impossible to debug.
> On November 28th, 2017, I proposed on the dev@ list that we move the dtests 
> from nosetests to pytests. I got replies from Jon Haddad, Philip Thompson, 
> and kurt greaves with really only "+1" like replies to the proposal.
> Since then I've been working pretty much non stop to complete the large 
> refactor of dtests to pytests. As part of this effort (and due to the 
> migration tools that exist require it) I also ported the code to python3 
> (from the current python 2.7 based code-base).
> h4. High-level summary of key changes, improvements, and new features.
> * Migrate dtests from executing using the nosetest framework to pytest
> * Port the entire code base from Python 2.7 to Python 3.6
> * Update run_dtests.py to work with pytest
> * Add --dtest-print-tests-only option to run_dtests.py to get easily parsable 
> list of all available collected tests
> * Update README.md for executing the dtests with pytest
> * Add new debugging tips section to README.md to help with some basics of 
> debugging python3 and pytest
> * Migrate all existing Enviornment Variable usage as a means to control dtest 
> operation modes to argparse command line options with documented help on each 
> toggles intended usage
> * Migration of old unitTest and nose based test structure to modern pytest 
> fixture approach
> * Automatic detection of physical system resources to automatically determine 
> if @pytest.mark.resource_intensive annotated tests should be collected and 
> run on the system where they are being executed
> * new pytest fixture replacements for @since and @pytest.mark.upgrade_test 
> annotations
> * Migration to python logging framework
> * Upgrade thrift bindings to latest version with full python3 compatibility
> * Remove deprecated cql and pycassa dependencies and migrate any remaining 
> tests to fully remove those dependencies
> * Fixed dozens of tests that would hang the pytest framework forever when run 
> in CI enviornments
> * Ran code nearly 300 times in CircleCI during the migration and to find, 
> identify, and fix any tests capable of hanging CI
> * Upgrade Tests do not yet run in CI and still need additional migration work 
> (although all upgrade test classes compile successfully)
> I started with the *nose2pytest* [https://github.com/pytest-dev/nose2pytest] 
> migration tool. As this required python 3 language support I found myself 
> down the 2to3 python migration path. While painful to do this, the benefits 
> of python3 over python2.7 are numerous and moving to python3 for the 
> additional debugging tools now available to use when fixing dtests makes the 
> effort worth it for that reason alone!
> After the automated tools did their thing I began what was a much longer and 
> tedious manual process than I ever could have expected due to the custom many 
> ways we did things in dtests (frequently to work around nosetest limitations 
> of missing features that thankfully are now all included with the pytest 
> framework). I've done nearly 300 test runs of my migration branch with 
> circleci.
> The latest CircleCI runs can be found at:
> (dtests without vnodes) [https://circleci.com/gh/mkjellman/cassandra/277]
> (dtests with vnodes) [https://circleci.com/gh/mkjellman/cassandra/278]
> 

[jira] [Commented] (CASSANDRA-14134) Migrate dtests to use pytest and python3

[Ariel Weisberg (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16321353#comment-16321353
 ] 

Ariel Weisberg commented on CASSANDRA-14134:


Reviewing the commits after the first review it seems like you have been adding 
a lot of sleeps to try and fix stuff and in at least the one [case I was 
looking at it doesn't fix the 
problem|https://github.com/apache/cassandra-dtest/commit/ab4b9b7f4b0f12fd667d16cc19d73efef99688a2].

The reason I push back on that is that future tests and modifications won't 
have the necessary sleeps. We want to push this into the infrastructure code as 
much as possible so we can "fix it once."

With several of them I need to review them on a case by case basis for 
appropriateness and does it fix the underlying problem. If a lot of these are 
schema change related I have to ask the question, why can't we change schema 
and know reliably when the schema is available to use? That seems like a pretty 
basic thing for the database to be able to do. If it doesn't work for us in 
tests I would say it's pretty broken for users. 

This also holds for nodetool. Nodetool shouldn't be generating errors some of 
the time spuriously.

[This shouldn't be necessary CCM has a log line it looks for to know when a 
node is up. If that's not working we need to fix 
it.|https://github.com/apache/cassandra-dtest/commit/bb1415597f80f2e4e25235e1ee8369b49561fc3f]

[This is another, wait for schema change 
stuff.|https://github.com/apache/cassandra-dtest/commit/c182a44b68a43dc0be50f1c3269ac220851010ac]
 If we really think this is necessary we should have them call a wrapper 
method. Really we shouldn't use sleeps we should do the schema change and then 
spin lightly waiting for the schema to be available everywhere it should be 
available by querying the nodes. Did this actually fix the test? 

[I really think we need to open up to a wider audience for discussing flaky 
annotations. We have already gone through this once and gone from using flaky 
to not using 
flaky.|https://github.com/apache/cassandra-dtest/commit/3ffa61717f760ef7bd6209ad1af9c7bba450492a]
 Flaky tests should be annoying and distracting.
Another usage of flakey 
https://github.com/apache/cassandra-dtest/commit/e2a3c5575155a45ae862400568be6463f47e1617
I am sort of warming up to using flaky because the alternative is that 
developers just don't use the tests and we are generally worse off.

[I couldn't figure out this fix for configuration 
test.|https://github.com/apache/cassandra-dtest/commit/aa225ab1c36959b3d19a607eddbae2b16f661194]
 Can you help out with an explanation for this? 
{{execute_concurrent_with_args}} doesn't document {{results_generator=True}}

I'm not sure if I should run the test fixes through CircleCI. Some of them have 
multiple rounds of changes and it's not clear which help and which are placebo 
or didn't end up contributing to reliability. Unless you know you only put in 
fixes that ended up clearing up a specific error?

> Migrate dtests to use pytest and python3
> 
>
> Key: CASSANDRA-14134
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14134
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Testing
>Reporter: Michael Kjellman
>Assignee: Michael Kjellman
>
> h4. Get the C* dtests running on the pytest framework.
> C* DTests currently run using the python test framework nosetest. This 
> framework has been largely abandoned with no releases since 2015 and a 
> general strong consensus in the python community that pytest is the future.
> h4. Why should we do this.
> Currently (and historically) dtests have always been difficult to run, flaky 
> and unpredictable in CI environments, and almost impossible to debug.
> On November 28th, 2017, I proposed on the dev@ list that we move the dtests 
> from nosetests to pytests. I got replies from Jon Haddad, Philip Thompson, 
> and kurt greaves with really only "+1" like replies to the proposal.
> Since then I've been working pretty much non stop to complete the large 
> refactor of dtests to pytests. As part of this effort (and due to the 
> migration tools that exist require it) I also ported the code to python3 
> (from the current python 2.7 based code-base).
> h4. High-level summary of key changes, improvements, and new features.
> * Migrate dtests from executing using the nosetest framework to pytest
> * Port the entire code base from Python 2.7 to Python 3.6
> * Update run_dtests.py to work with pytest
> * Add --dtest-print-tests-only option to run_dtests.py to get easily parsable 
> list of all available collected tests
> * Update README.md for executing the dtests with pytest
> * Add new debugging tips section to README.md to help with some basics of 
> debugging python3 and pytest
> * Migrate all existing 

[jira] [Commented] (CASSANDRA-8527) Account for range tombstones wherever we account for tombstones

[Jon Haddad (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-8527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16321328#comment-16321328
 ] 

Jon Haddad commented on CASSANDRA-8527:
---

Thanks [~adejanovski] I'll look at look at the patch this week, thanks for the 
revision.

> Account for range tombstones wherever we account for tombstones
> ---
>
> Key: CASSANDRA-8527
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8527
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Sylvain Lebresne
>Assignee: Alexander Dejanovski
> Fix For: 4.x
>
> Attachments: CASSANDRA-8527-4.x.diff
>
>
> As discussed in CASSANDRA-8477, we should make sure the tombstone thresholds 
> also apply to range tombstones, since they poses the same problems than cell 
> tombstones.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-14139) Acquire read lock before accessing CompactionStrategyManager fields

[Paulo Motta (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Paulo Motta updated CASSANDRA-14139:

   Resolution: Fixed
Fix Version/s: 4.0
   3.11.2
Reproduced In:   (was: 3.11.2)
   Status: Resolved  (was: Ready to Commit)

Committed as {{fe0ee85c71faada0acb48a65f249575c65bf0972}} to cassandra-3.11 and 
merged up to master. Thanks!

> Acquire read lock before accessing CompactionStrategyManager fields
> ---
>
> Key: CASSANDRA-14139
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14139
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Paulo Motta
>Assignee: Paulo Motta
> Fix For: 3.11.2, 4.0
>
> Attachments: 3.11-14139-dtest.png, 3.11-14139-testall.png, 
> trunk-14139-dtest.png, trunk-14139-testall.png
>
>
> There are a few methods in {{CompactionStrategyManager}} accessing the 
> repaired/unrepaired compaction strategy lists without using the read lock, 
> what could cause issues like the one below:
> {noformat}
> ERROR [CompactionExecutor:1] 2017-12-22 12:17:12,320 CassandraDaemon.java:141 
> - Exception in thread Thread[CompactionExecutor:1,5,main]
> java.lang.IndexOutOfBoundsException: Index: 0, Size: 1
> at java.util.ArrayList.rangeCheck(ArrayList.java:657)
> at java.util.ArrayList.get(ArrayList.java:433)
> at 
> org.apache.cassandra.db.compaction.CompactionStrategyManager.supportsEarlyOpen(CompactionStrategyManager.java:1262)
> at 
> org.apache.cassandra.db.ColumnFamilyStore.supportsEarlyOpen(ColumnFamilyStore.java:558)
> at 
> org.apache.cassandra.io.sstable.SSTableRewriter.construct(SSTableRewriter.java:119)
> at 
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.(CompactionAwareWriter.java:91)
> at 
> org.apache.cassandra.db.compaction.writers.DefaultCompactionWriter.(DefaultCompactionWriter.java:57)
> at 
> org.apache.cassandra.db.compaction.CompactionTask.getCompactionAwareWriter(CompactionTask.java:293)
> at 
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:200)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at 
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:90)
> at 
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:101)
> at 
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:310)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at 
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[3/3] cassandra git commit: Merge branch 'cassandra-3.11' into trunk

[paulo]

Merge branch 'cassandra-3.11' into trunk


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/7a29d22e
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/7a29d22e
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/7a29d22e

Branch: refs/heads/trunk
Commit: 7a29d22e60b52c38bb81a90b854017242850bef4
Parents: ee907a3 fe0ee85
Author: Paulo Motta 
Authored: Wed Jan 10 18:40:00 2018 -0200
Committer: Paulo Motta 
Committed: Wed Jan 10 18:42:54 2018 -0200

--
 CHANGES.txt |   1 +
 .../compaction/CompactionStrategyManager.java   | 123 ++-
 2 files changed, 92 insertions(+), 32 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/7a29d22e/CHANGES.txt
--
diff --cc CHANGES.txt
index 62559cb,b89ad99..d59fbe3
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,184 -1,9 +1,185 @@@
 +4.0
 + * Make PartitionUpdate and Mutation immutable (CASSANDRA-13867)
 + * Fix CommitLogReplayer exception for CDC data (CASSANDRA-14066)
 + * Fix cassandra-stress startup failure (CASSANDRA-14106)
 + * Remove initialDirectories from CFS (CASSANDRA-13928)
 + * Fix trivial log format error (CASSANDRA-14015)
 + * Allow sstabledump to do a json object per partition (CASSANDRA-13848)
 + * Add option to optimise merkle tree comparison across replicas 
(CASSANDRA-3200)
 + * Remove unused and deprecated methods from AbstractCompactionStrategy 
(CASSANDRA-14081)
 + * Fix Distribution.average in cassandra-stress (CASSANDRA-14090)
 + * Support a means of logging all queries as they were invoked 
(CASSANDRA-13983)
 + * Presize collections (CASSANDRA-13760)
 + * Add GroupCommitLogService (CASSANDRA-13530)
 + * Parallelize initial materialized view build (CASSANDRA-12245)
 + * Fix flaky SecondaryIndexManagerTest.assert[Not]MarkedAsBuilt 
(CASSANDRA-13965)
 + * Make LWTs send resultset metadata on every request (CASSANDRA-13992)
 + * Fix flaky indexWithFailedInitializationIsNotQueryableAfterPartialRebuild 
(CASSANDRA-13963)
 + * Introduce leaf-only iterator (CASSANDRA-9988)
 + * Upgrade Guava to 23.3 and Airline to 0.8 (CASSANDRA-13997)
 + * Allow only one concurrent call to StatusLogger (CASSANDRA-12182)
 + * Refactoring to specialised functional interfaces (CASSANDRA-13982)
 + * Speculative retry should allow more friendly params (CASSANDRA-13876)
 + * Throw exception if we send/receive repair messages to incompatible nodes 
(CASSANDRA-13944)
 + * Replace usages of MessageDigest with Guava's Hasher (CASSANDRA-13291)
 + * Add nodetool cmd to print hinted handoff window (CASSANDRA-13728)
 + * Fix some alerts raised by static analysis (CASSANDRA-13799)
 + * Checksum sstable metadata (CASSANDRA-13321, CASSANDRA-13593)
 + * Add result set metadata to prepared statement MD5 hash calculation 
(CASSANDRA-10786)
 + * Refactor GcCompactionTest to avoid boxing (CASSANDRA-13941)
 + * Expose recent histograms in JmxHistograms (CASSANDRA-13642)
 + * Fix buffer length comparison when decompressing in netty-based streaming 
(CASSANDRA-13899)
 + * Properly close StreamCompressionInputStream to release any ByteBuf 
(CASSANDRA-13906)
 + * Add SERIAL and LOCAL_SERIAL support for cassandra-stress (CASSANDRA-13925)
 + * LCS needlessly checks for L0 STCS candidates multiple times 
(CASSANDRA-12961)
 + * Correctly close netty channels when a stream session ends (CASSANDRA-13905)
 + * Update lz4 to 1.4.0 (CASSANDRA-13741)
 + * Optimize Paxos prepare and propose stage for local requests 
(CASSANDRA-13862)
 + * Throttle base partitions during MV repair streaming to prevent OOM 
(CASSANDRA-13299)
 + * Use compaction threshold for STCS in L0 (CASSANDRA-13861)
 + * Fix problem with min_compress_ratio: 1 and disallow ratio < 1 
(CASSANDRA-13703)
 + * Add extra information to SASI timeout exception (CASSANDRA-13677)
 + * Add incremental repair support for --hosts, --force, and subrange repair 
(CASSANDRA-13818)
 + * Rework CompactionStrategyManager.getScanners synchronization 
(CASSANDRA-13786)
 + * Add additional unit tests for batch behavior, TTLs, Timestamps 
(CASSANDRA-13846)
 + * Add keyspace and table name in schema validation exception 
(CASSANDRA-13845)
 + * Emit metrics whenever we hit tombstone failures and warn thresholds 
(CASSANDRA-13771)
 + * Make netty EventLoopGroups daemon threads (CASSANDRA-13837)
 + * Race condition when closing stream sessions (CASSANDRA-13852)
 + * NettyFactoryTest is failing in trunk on macOS (CASSANDRA-13831)
 + * Allow changing log levels via nodetool for related classes 
(CASSANDRA-12696)
 + * Add stress profile yaml with LWT (CASSANDRA-7960)
 + * Reduce memory copies and object creations when acting on ByteBufs 
(CASSANDRA-13789)
 + * Simplify mx4j 

[1/3] cassandra git commit: Acquire read lock before accessing CompactionStrategyManager fields

[paulo]

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-3.11 62e46f719 -> fe0ee85c7
  refs/heads/trunk ee907a321 -> 7a29d22e6


Acquire read lock before accessing CompactionStrategyManager fields

Patch by Paulo Motta; Reviewed by Marcus Eriksson for CASSANDRA-14139


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/fe0ee85c
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/fe0ee85c
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/fe0ee85c

Branch: refs/heads/cassandra-3.11
Commit: fe0ee85c71faada0acb48a65f249575c65bf0972
Parents: 62e46f7
Author: Paulo Motta 
Authored: Sat Dec 30 02:10:35 2017 +1100
Committer: Paulo Motta 
Committed: Wed Jan 10 18:39:12 2018 -0200

--
 CHANGES.txt |  1 +
 .../compaction/CompactionStrategyManager.java   | 47 ++--
 2 files changed, 34 insertions(+), 14 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/fe0ee85c/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index d6d8066..b89ad99 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.11.2
+ * Acquire read lock before accessing CompactionStrategyManager fields 
(CASSANDRA-14139)
  * Split CommitLogStressTest to avoid timeout (CASSANDRA-14143)
  * Avoid invalidating disk boundaries unnecessarily (CASSANDRA-14083)
  * Avoid exposing compaction strategy index externally (CASSANDRA-14082)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/fe0ee85c/src/java/org/apache/cassandra/db/compaction/CompactionStrategyManager.java
--
diff --git 
a/src/java/org/apache/cassandra/db/compaction/CompactionStrategyManager.java 
b/src/java/org/apache/cassandra/db/compaction/CompactionStrategyManager.java
index 39d253b..a50f428 100644
--- a/src/java/org/apache/cassandra/db/compaction/CompactionStrategyManager.java
+++ b/src/java/org/apache/cassandra/db/compaction/CompactionStrategyManager.java
@@ -105,6 +105,7 @@ public class CompactionStrategyManager implements 
INotificationConsumer
  **/
 private volatile CompactionParams schemaCompactionParams;
 private boolean shouldDefragment;
+private boolean supportsEarlyOpen;
 private int fanout;
 
 public CompactionStrategyManager(ColumnFamilyStore cfs)
@@ -216,6 +217,7 @@ public class CompactionStrategyManager implements 
INotificationConsumer
 repaired.forEach(AbstractCompactionStrategy::startup);
 unrepaired.forEach(AbstractCompactionStrategy::startup);
 shouldDefragment = repaired.get(0).shouldDefragment();
+supportsEarlyOpen = repaired.get(0).supportsEarlyOpen();
 fanout = (repaired.get(0) instanceof LeveledCompactionStrategy) ? 
((LeveledCompactionStrategy) repaired.get(0)).getLevelFanoutSize() : 
LeveledCompactionStrategy.DEFAULT_LEVEL_FANOUT_SIZE;
 }
 finally
@@ -1037,35 +1039,52 @@ public class CompactionStrategyManager implements 
INotificationConsumer
 
 public boolean isRepaired(AbstractCompactionStrategy strategy)
 {
-return repaired.contains(strategy);
+readLock.lock();
+try
+{
+return repaired.contains(strategy);
+}
+finally
+{
+readLock.unlock();
+}
 }
 
 public List getStrategyFolders(AbstractCompactionStrategy strategy)
 {
-List locations = 
currentBoundaries.directories;
-if (partitionSSTablesByTokenRange)
+readLock.lock();
+try
 {
-int unrepairedIndex = unrepaired.indexOf(strategy);
-if (unrepairedIndex > 0)
+List locations = 
currentBoundaries.directories;
+if (partitionSSTablesByTokenRange)
 {
-return 
Collections.singletonList(locations.get(unrepairedIndex).location.getAbsolutePath());
+int unrepairedIndex = unrepaired.indexOf(strategy);
+if (unrepairedIndex > 0)
+{
+return 
Collections.singletonList(locations.get(unrepairedIndex).location.getAbsolutePath());
+}
+int repairedIndex = repaired.indexOf(strategy);
+if (repairedIndex > 0)
+{
+return 
Collections.singletonList(locations.get(repairedIndex).location.getAbsolutePath());
+}
 }
-int repairedIndex = repaired.indexOf(strategy);
-if (repairedIndex > 0)
+List folders = new ArrayList<>(locations.size());
+for (Directories.DataDirectory location : locations)
 {
- 

[2/3] cassandra git commit: Acquire read lock before accessing CompactionStrategyManager fields

[paulo]

Acquire read lock before accessing CompactionStrategyManager fields

Patch by Paulo Motta; Reviewed by Marcus Eriksson for CASSANDRA-14139


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/fe0ee85c
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/fe0ee85c
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/fe0ee85c

Branch: refs/heads/trunk
Commit: fe0ee85c71faada0acb48a65f249575c65bf0972
Parents: 62e46f7
Author: Paulo Motta 
Authored: Sat Dec 30 02:10:35 2017 +1100
Committer: Paulo Motta 
Committed: Wed Jan 10 18:39:12 2018 -0200

--
 CHANGES.txt |  1 +
 .../compaction/CompactionStrategyManager.java   | 47 ++--
 2 files changed, 34 insertions(+), 14 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/fe0ee85c/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index d6d8066..b89ad99 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.11.2
+ * Acquire read lock before accessing CompactionStrategyManager fields 
(CASSANDRA-14139)
  * Split CommitLogStressTest to avoid timeout (CASSANDRA-14143)
  * Avoid invalidating disk boundaries unnecessarily (CASSANDRA-14083)
  * Avoid exposing compaction strategy index externally (CASSANDRA-14082)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/fe0ee85c/src/java/org/apache/cassandra/db/compaction/CompactionStrategyManager.java
--
diff --git 
a/src/java/org/apache/cassandra/db/compaction/CompactionStrategyManager.java 
b/src/java/org/apache/cassandra/db/compaction/CompactionStrategyManager.java
index 39d253b..a50f428 100644
--- a/src/java/org/apache/cassandra/db/compaction/CompactionStrategyManager.java
+++ b/src/java/org/apache/cassandra/db/compaction/CompactionStrategyManager.java
@@ -105,6 +105,7 @@ public class CompactionStrategyManager implements 
INotificationConsumer
  **/
 private volatile CompactionParams schemaCompactionParams;
 private boolean shouldDefragment;
+private boolean supportsEarlyOpen;
 private int fanout;
 
 public CompactionStrategyManager(ColumnFamilyStore cfs)
@@ -216,6 +217,7 @@ public class CompactionStrategyManager implements 
INotificationConsumer
 repaired.forEach(AbstractCompactionStrategy::startup);
 unrepaired.forEach(AbstractCompactionStrategy::startup);
 shouldDefragment = repaired.get(0).shouldDefragment();
+supportsEarlyOpen = repaired.get(0).supportsEarlyOpen();
 fanout = (repaired.get(0) instanceof LeveledCompactionStrategy) ? 
((LeveledCompactionStrategy) repaired.get(0)).getLevelFanoutSize() : 
LeveledCompactionStrategy.DEFAULT_LEVEL_FANOUT_SIZE;
 }
 finally
@@ -1037,35 +1039,52 @@ public class CompactionStrategyManager implements 
INotificationConsumer
 
 public boolean isRepaired(AbstractCompactionStrategy strategy)
 {
-return repaired.contains(strategy);
+readLock.lock();
+try
+{
+return repaired.contains(strategy);
+}
+finally
+{
+readLock.unlock();
+}
 }
 
 public List getStrategyFolders(AbstractCompactionStrategy strategy)
 {
-List locations = 
currentBoundaries.directories;
-if (partitionSSTablesByTokenRange)
+readLock.lock();
+try
 {
-int unrepairedIndex = unrepaired.indexOf(strategy);
-if (unrepairedIndex > 0)
+List locations = 
currentBoundaries.directories;
+if (partitionSSTablesByTokenRange)
 {
-return 
Collections.singletonList(locations.get(unrepairedIndex).location.getAbsolutePath());
+int unrepairedIndex = unrepaired.indexOf(strategy);
+if (unrepairedIndex > 0)
+{
+return 
Collections.singletonList(locations.get(unrepairedIndex).location.getAbsolutePath());
+}
+int repairedIndex = repaired.indexOf(strategy);
+if (repairedIndex > 0)
+{
+return 
Collections.singletonList(locations.get(repairedIndex).location.getAbsolutePath());
+}
 }
-int repairedIndex = repaired.indexOf(strategy);
-if (repairedIndex > 0)
+List folders = new ArrayList<>(locations.size());
+for (Directories.DataDirectory location : locations)
 {
-return 
Collections.singletonList(locations.get(repairedIndex).location.getAbsolutePath());
+

[jira] [Updated] (CASSANDRA-14161) node can't start, java.nio.file.AccessDeniedException to not existent file

[Jeff Jirsa (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeff Jirsa updated CASSANDRA-14161:
---
Description: 
this exception cuts the node starting:

{code}
java.lang.RuntimeException: java.nio.file.AccessDeniedException: 
/var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/mc_txn_flush_6579bf70-f625-11e7-8e56-71d63c551064.log

{code}

the offending file doesn't exists:

{code}
 ls -la /var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/
total 160
drwxr-xr-x  3 cassandra cassandra 4096 ene  7 11:48 .
drwxr-xr-x 26 cassandra cassandra 4096 dic 18 07:30 ..
drwxr-xr-x  2 cassandra cassandra 4096 dic 18 07:30 backups
-rw-r--r--  1 cassandra cassandra   43 ene  7 03:50 
mc-100-big-CompressionInfo.db
-rw-r--r--  1 cassandra cassandra   52 ene  7 03:50 mc-100-big-Data.db
-rw-r--r--  1 cassandra cassandra8 ene  7 03:50 mc-100-big-Digest.crc32
-rw-r--r--  1 cassandra cassandra   16 ene  7 03:50 mc-100-big-Filter.db
-rw-r--r--  1 cassandra cassandra9 ene  7 03:50 mc-100-big-Index.db
-rw-r--r--  1 cassandra cassandra 4615 ene  7 03:50 mc-100-big-Statistics.db
-rw-r--r--  1 cassandra cassandra   59 ene  7 03:50 mc-100-big-Summary.db
-rw-r--r--  1 cassandra cassandra   92 ene  7 03:50 mc-100-big-TOC.txt
-rw-r--r--  1 cassandra cassandra   43 ene  7 11:48 
mc-101-big-CompressionInfo.db
-rw-r--r--  1 cassandra cassandra   50 ene  7 11:48 mc-101-big-Data.db
-rw-r--r--  1 cassandra cassandra   10 ene  7 11:48 mc-101-big-Digest.crc32
-rw-r--r--  1 cassandra cassandra   16 ene  7 11:48 mc-101-big-Filter.db
-rw-r--r--  1 cassandra cassandra9 ene  7 11:48 mc-101-big-Index.db
-rw-r--r--  1 cassandra cassandra 4615 ene  7 11:48 mc-101-big-Statistics.db
-rw-r--r--  1 cassandra cassandra   59 ene  7 11:48 mc-101-big-Summary.db
-rw-r--r--  1 cassandra cassandra   92 ene  7 11:48 mc-101-big-TOC.txt
-rw-r--r--  1 cassandra cassandra   51 ene  6 17:50 mc-98-big-CompressionInfo.db
-rw-r--r--  1 cassandra cassandra 5226 ene  6 17:50 mc-98-big-Data.db
-rw-r--r--  1 cassandra cassandra   10 ene  6 17:50 mc-98-big-Digest.crc32
-rw-r--r--  1 cassandra cassandra   16 ene  6 17:50 mc-98-big-Filter.db
-rw-r--r--  1 cassandra cassandra9 ene  6 17:50 mc-98-big-Index.db
-rw-r--r--  1 cassandra cassandra 5654 ene  6 17:50 mc-98-big-Statistics.db
-rw-r--r--  1 cassandra cassandra   59 ene  6 17:50 mc-98-big-Summary.db
-rw-r--r--  1 cassandra cassandra   92 ene  6 17:50 mc-98-big-TOC.txt
-rw-r--r--  1 cassandra cassandra   43 ene  7 02:50 mc-99-big-CompressionInfo.db
-rw-r--r--  1 cassandra cassandra   50 ene  7 02:50 mc-99-big-Data.db
-rw-r--r--  1 cassandra cassandra   10 ene  7 02:50 mc-99-big-Digest.crc32
-rw-r--r--  1 cassandra cassandra   16 ene  7 02:50 mc-99-big-Filter.db
-rw-r--r--  1 cassandra cassandra9 ene  7 02:50 mc-99-big-Index.db
-rw-r--r--  1 cassandra cassandra 4615 ene  7 02:50 mc-99-big-Statistics.db
-rw-r--r--  1 cassandra cassandra   59 ene  7 02:50 mc-99-big-Summary.db
-rw-r--r--  1 cassandra cassandra   92 ene  7 02:50 mc-99-big-TOC.txt
{code}

permissions are ok, everything assigned to cassandra:cassandra

the system.log:

{code}
INFO  [main] 2018-01-10 10:43:29,367 YamlConfigurationLoader.java:89 - 
Configuration location: file:/etc/cassandra/cassandra.yaml
INFO  [main] 2018-01-10 10:43:29,887 Config.java:481 - Node 
configuration:[allocate_tokens_for_keyspace=null; 
authenticator=AllowAllAuthenticator; authorizer=AllowAllAuthorizer; 
auto_bootstrap=true; auto_snapshot=true; back_pressure_enabled=false; 
back_pressure_strategy=null; batch_size_fail_threshold_in_kb=500; 
batch_size_warn_threshold_in_kb=50; batchlog_replay_throttle_in_kb=1024; 
broadcast_address=null; broadcast_rpc_address=null; 
buffer_pool_use_heap_if_exhausted=true; cas_contention_timeout_in_ms=1; 
cdc_enabled=false; cdc_free_space_check_interval_ms=250; 
cdc_raw_directory=null; cdc_total_space_in_mb=0; 
client_encryption_options=; cluster_name=PAP Cluster; 
column_index_cache_size_in_kb=2; column_index_size_in_kb=64; 
commit_failure_policy=stop; commitlog_compression=null; 
commitlog_directory=/var/lib/cassandra/commitlog; 
commitlog_max_compression_buffers_in_pool=3; commitlog_periodic_queue_size=-1; 
commitlog_segment_size_in_mb=32; commitlog_sync=periodic; 
commitlog_sync_batch_window_in_ms=NaN; commitlog_sync_period_in_ms=1; 
commitlog_total_space_in_mb=null; 
compaction_large_partition_warning_threshold_mb=100; 
compaction_throughput_mb_per_sec=16; concurrent_compactors=null; 
concurrent_counter_writes=32; concurrent_materialized_view_writes=32; 
concurrent_reads=32; concurrent_replicates=null; concurrent_writes=32; 
counter_cache_keys_to_save=2147483647; counter_cache_save_period=7200; 
counter_cache_size_in_mb=null; counter_write_request_timeout_in_ms=5; 
credentials_cache_max_entries=1000; credentials_update_interval_in_ms=-1; 
credentials_validity_in_ms=2000; 

[jira] [Updated] (CASSANDRA-14161) node can't start, java.nio.file.AccessDeniedException to not existent file

[Jeff Jirsa (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeff Jirsa updated CASSANDRA-14161:
---
Description: 
this exception cuts the node starting:

{code}
java.lang.RuntimeException: java.nio.file.AccessDeniedException: 
/var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/mc_txn_flush_6579bf70-f625-11e7-8e56-71d63c551064.log
the offending file doesn't exists:
 ls -la /var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/
total 160
drwxr-xr-x  3 cassandra cassandra 4096 ene  7 11:48 .
drwxr-xr-x 26 cassandra cassandra 4096 dic 18 07:30 ..
drwxr-xr-x  2 cassandra cassandra 4096 dic 18 07:30 backups
-rw-r--r--  1 cassandra cassandra   43 ene  7 03:50 
mc-100-big-CompressionInfo.db
-rw-r--r--  1 cassandra cassandra   52 ene  7 03:50 mc-100-big-Data.db
-rw-r--r--  1 cassandra cassandra8 ene  7 03:50 mc-100-big-Digest.crc32
-rw-r--r--  1 cassandra cassandra   16 ene  7 03:50 mc-100-big-Filter.db
-rw-r--r--  1 cassandra cassandra9 ene  7 03:50 mc-100-big-Index.db
-rw-r--r--  1 cassandra cassandra 4615 ene  7 03:50 mc-100-big-Statistics.db
-rw-r--r--  1 cassandra cassandra   59 ene  7 03:50 mc-100-big-Summary.db
-rw-r--r--  1 cassandra cassandra   92 ene  7 03:50 mc-100-big-TOC.txt
-rw-r--r--  1 cassandra cassandra   43 ene  7 11:48 
mc-101-big-CompressionInfo.db
-rw-r--r--  1 cassandra cassandra   50 ene  7 11:48 mc-101-big-Data.db
-rw-r--r--  1 cassandra cassandra   10 ene  7 11:48 mc-101-big-Digest.crc32
-rw-r--r--  1 cassandra cassandra   16 ene  7 11:48 mc-101-big-Filter.db
-rw-r--r--  1 cassandra cassandra9 ene  7 11:48 mc-101-big-Index.db
-rw-r--r--  1 cassandra cassandra 4615 ene  7 11:48 mc-101-big-Statistics.db
-rw-r--r--  1 cassandra cassandra   59 ene  7 11:48 mc-101-big-Summary.db
-rw-r--r--  1 cassandra cassandra   92 ene  7 11:48 mc-101-big-TOC.txt
-rw-r--r--  1 cassandra cassandra   51 ene  6 17:50 mc-98-big-CompressionInfo.db
-rw-r--r--  1 cassandra cassandra 5226 ene  6 17:50 mc-98-big-Data.db
-rw-r--r--  1 cassandra cassandra   10 ene  6 17:50 mc-98-big-Digest.crc32
-rw-r--r--  1 cassandra cassandra   16 ene  6 17:50 mc-98-big-Filter.db
-rw-r--r--  1 cassandra cassandra9 ene  6 17:50 mc-98-big-Index.db
-rw-r--r--  1 cassandra cassandra 5654 ene  6 17:50 mc-98-big-Statistics.db
-rw-r--r--  1 cassandra cassandra   59 ene  6 17:50 mc-98-big-Summary.db
-rw-r--r--  1 cassandra cassandra   92 ene  6 17:50 mc-98-big-TOC.txt
-rw-r--r--  1 cassandra cassandra   43 ene  7 02:50 mc-99-big-CompressionInfo.db
-rw-r--r--  1 cassandra cassandra   50 ene  7 02:50 mc-99-big-Data.db
-rw-r--r--  1 cassandra cassandra   10 ene  7 02:50 mc-99-big-Digest.crc32
-rw-r--r--  1 cassandra cassandra   16 ene  7 02:50 mc-99-big-Filter.db
-rw-r--r--  1 cassandra cassandra9 ene  7 02:50 mc-99-big-Index.db
-rw-r--r--  1 cassandra cassandra 4615 ene  7 02:50 mc-99-big-Statistics.db
-rw-r--r--  1 cassandra cassandra   59 ene  7 02:50 mc-99-big-Summary.db
-rw-r--r--  1 cassandra cassandra   92 ene  7 02:50 mc-99-big-TOC.txt
{code}

permissions are ok, everything assigned to cassandra:cassandra

the system.log:

{code}
INFO  [main] 2018-01-10 10:43:29,367 YamlConfigurationLoader.java:89 - 
Configuration location: file:/etc/cassandra/cassandra.yaml
INFO  [main] 2018-01-10 10:43:29,887 Config.java:481 - Node 
configuration:[allocate_tokens_for_keyspace=null; 
authenticator=AllowAllAuthenticator; authorizer=AllowAllAuthorizer; 
auto_bootstrap=true; auto_snapshot=true; back_pressure_enabled=false; 
back_pressure_strategy=null; batch_size_fail_threshold_in_kb=500; 
batch_size_warn_threshold_in_kb=50; batchlog_replay_throttle_in_kb=1024; 
broadcast_address=null; broadcast_rpc_address=null; 
buffer_pool_use_heap_if_exhausted=true; cas_contention_timeout_in_ms=1; 
cdc_enabled=false; cdc_free_space_check_interval_ms=250; 
cdc_raw_directory=null; cdc_total_space_in_mb=0; 
client_encryption_options=; cluster_name=PAP Cluster; 
column_index_cache_size_in_kb=2; column_index_size_in_kb=64; 
commit_failure_policy=stop; commitlog_compression=null; 
commitlog_directory=/var/lib/cassandra/commitlog; 
commitlog_max_compression_buffers_in_pool=3; commitlog_periodic_queue_size=-1; 
commitlog_segment_size_in_mb=32; commitlog_sync=periodic; 
commitlog_sync_batch_window_in_ms=NaN; commitlog_sync_period_in_ms=1; 
commitlog_total_space_in_mb=null; 
compaction_large_partition_warning_threshold_mb=100; 
compaction_throughput_mb_per_sec=16; concurrent_compactors=null; 
concurrent_counter_writes=32; concurrent_materialized_view_writes=32; 
concurrent_reads=32; concurrent_replicates=null; concurrent_writes=32; 
counter_cache_keys_to_save=2147483647; counter_cache_save_period=7200; 
counter_cache_size_in_mb=null; counter_write_request_timeout_in_ms=5; 
credentials_cache_max_entries=1000; credentials_update_interval_in_ms=-1; 
credentials_validity_in_ms=2000; cross_node_timeout=false; 

[jira] [Updated] (CASSANDRA-9625) GraphiteReporter not reporting

[Jeff Jirsa (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-9625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeff Jirsa updated CASSANDRA-9625:
--
Reproduced In: 3.11.1, 2.1.6  (was: 2.1.6)

> GraphiteReporter not reporting
> --
>
> Key: CASSANDRA-9625
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9625
> Project: Cassandra
>  Issue Type: Bug
>  Components: Metrics
> Environment: Debian Jessie, 7u79-2.5.5-1~deb8u1, Cassandra 2.1.3
>Reporter: Eric Evans
> Fix For: 3.0.x, 3.11.x, 4.x
>
> Attachments: Screen Shot 2016-04-13 at 10.40.58 AM.png, metrics.yaml, 
> thread-dump.log, thread-dump2.log
>
>
> When upgrading from 2.1.3 to 2.1.6, the Graphite metrics reporter stops 
> working.  The usual startup is logged, and one batch of samples is sent, but 
> the reporting interval comes and goes, and no other samples are ever sent.  
> The logs are free from errors.
> Frustratingly, metrics reporting works in our smaller (staging) environment 
> on 2.1.6; We are able to reproduce this on all 6 of production nodes, but not 
> on a 3 node (otherwise identical) staging cluster (maybe it takes a certain 
> level of concurrency?).
> Attached is a thread dump, and our metrics.yaml.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Reopened] (CASSANDRA-9625) GraphiteReporter not reporting

[Jeff Jirsa (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-9625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeff Jirsa reopened CASSANDRA-9625:
---

Re-opened. [~scott.hunter.iii] please do help with any info you can provide. I 
imagine many of the developers dont really run graphite, so concrete steps to 
repro would be useful (unless/until one of the committers spins up graphite to 
prove it works/doesnt work). 



> GraphiteReporter not reporting
> --
>
> Key: CASSANDRA-9625
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9625
> Project: Cassandra
>  Issue Type: Bug
>  Components: Metrics
> Environment: Debian Jessie, 7u79-2.5.5-1~deb8u1, Cassandra 2.1.3
>Reporter: Eric Evans
> Fix For: 3.0.x, 3.11.x, 4.x
>
> Attachments: Screen Shot 2016-04-13 at 10.40.58 AM.png, metrics.yaml, 
> thread-dump.log, thread-dump2.log
>
>
> When upgrading from 2.1.3 to 2.1.6, the Graphite metrics reporter stops 
> working.  The usual startup is logged, and one batch of samples is sent, but 
> the reporting interval comes and goes, and no other samples are ever sent.  
> The logs are free from errors.
> Frustratingly, metrics reporting works in our smaller (staging) environment 
> on 2.1.6; We are able to reproduce this on all 6 of production nodes, but not 
> on a 3 node (otherwise identical) staging cluster (maybe it takes a certain 
> level of concurrency?).
> Attached is a thread dump, and our metrics.yaml.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-9625) GraphiteReporter not reporting

[Jeff Jirsa (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-9625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeff Jirsa updated CASSANDRA-9625:
--
Component/s: Metrics

> GraphiteReporter not reporting
> --
>
> Key: CASSANDRA-9625
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9625
> Project: Cassandra
>  Issue Type: Bug
>  Components: Metrics
> Environment: Debian Jessie, 7u79-2.5.5-1~deb8u1, Cassandra 2.1.3
>Reporter: Eric Evans
> Fix For: 3.0.x, 3.11.x, 4.x
>
> Attachments: Screen Shot 2016-04-13 at 10.40.58 AM.png, metrics.yaml, 
> thread-dump.log, thread-dump2.log
>
>
> When upgrading from 2.1.3 to 2.1.6, the Graphite metrics reporter stops 
> working.  The usual startup is logged, and one batch of samples is sent, but 
> the reporting interval comes and goes, and no other samples are ever sent.  
> The logs are free from errors.
> Frustratingly, metrics reporting works in our smaller (staging) environment 
> on 2.1.6; We are able to reproduce this on all 6 of production nodes, but not 
> on a 3 node (otherwise identical) staging cluster (maybe it takes a certain 
> level of concurrency?).
> Attached is a thread dump, and our metrics.yaml.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-9625) GraphiteReporter not reporting

[Jeff Jirsa (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-9625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeff Jirsa updated CASSANDRA-9625:
--
Fix Version/s: 4.x
   3.11.x
   3.0.x

> GraphiteReporter not reporting
> --
>
> Key: CASSANDRA-9625
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9625
> Project: Cassandra
>  Issue Type: Bug
>  Components: Metrics
> Environment: Debian Jessie, 7u79-2.5.5-1~deb8u1, Cassandra 2.1.3
>Reporter: Eric Evans
> Fix For: 3.0.x, 3.11.x, 4.x
>
> Attachments: Screen Shot 2016-04-13 at 10.40.58 AM.png, metrics.yaml, 
> thread-dump.log, thread-dump2.log
>
>
> When upgrading from 2.1.3 to 2.1.6, the Graphite metrics reporter stops 
> working.  The usual startup is logged, and one batch of samples is sent, but 
> the reporting interval comes and goes, and no other samples are ever sent.  
> The logs are free from errors.
> Frustratingly, metrics reporting works in our smaller (staging) environment 
> on 2.1.6; We are able to reproduce this on all 6 of production nodes, but not 
> on a 3 node (otherwise identical) staging cluster (maybe it takes a certain 
> level of concurrency?).
> Attached is a thread dump, and our metrics.yaml.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-9625) GraphiteReporter not reporting

[Scott Hunter (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-9625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16320971#comment-16320971
 ] 

Scott Hunter commented on CASSANDRA-9625:
-

Please re-open this issue as it is still a problem for us in both our 2.1.18 
and 3.11.1 clusters.  The metrics reporter thread will randomly stop reporting 
metrics on nodes in the cluster.  Restarting cassandra fixes it, but we have 
dense nodes which require several minutes to start up, so would prefer not to 
have to do that.

> GraphiteReporter not reporting
> --
>
> Key: CASSANDRA-9625
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9625
> Project: Cassandra
>  Issue Type: Bug
> Environment: Debian Jessie, 7u79-2.5.5-1~deb8u1, Cassandra 2.1.3
>Reporter: Eric Evans
> Attachments: Screen Shot 2016-04-13 at 10.40.58 AM.png, metrics.yaml, 
> thread-dump.log, thread-dump2.log
>
>
> When upgrading from 2.1.3 to 2.1.6, the Graphite metrics reporter stops 
> working.  The usual startup is logged, and one batch of samples is sent, but 
> the reporting interval comes and goes, and no other samples are ever sent.  
> The logs are free from errors.
> Frustratingly, metrics reporting works in our smaller (staging) environment 
> on 2.1.6; We are able to reproduce this on all 6 of production nodes, but not 
> on a 3 node (otherwise identical) staging cluster (maybe it takes a certain 
> level of concurrency?).
> Attached is a thread dump, and our metrics.yaml.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Created] (CASSANDRA-14161) node can't start, java.nio.file.AccessDeniedException to not existent file

[JIRA]

Rogelio Triviño González created CASSANDRA-14161:


 Summary: node can't start, java.nio.file.AccessDeniedException to 
not existent file
 Key: CASSANDRA-14161
 URL: https://issues.apache.org/jira/browse/CASSANDRA-14161
 Project: Cassandra
  Issue Type: Bug
 Environment: ubuntu +cassandra 3.11.1
Reporter: Rogelio Triviño González
 Fix For: 3.11.1


this exception cuts the node starting:
java.lang.RuntimeException: java.nio.file.AccessDeniedException: 
/var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/mc_txn_flush_6579bf70-f625-11e7-8e56-71d63c551064.log
the offending file doesn't exists:
 ls -la /var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/
total 160
drwxr-xr-x  3 cassandra cassandra 4096 ene  7 11:48 .
drwxr-xr-x 26 cassandra cassandra 4096 dic 18 07:30 ..
drwxr-xr-x  2 cassandra cassandra 4096 dic 18 07:30 backups
-rw-r--r--  1 cassandra cassandra   43 ene  7 03:50 
mc-100-big-CompressionInfo.db
-rw-r--r--  1 cassandra cassandra   52 ene  7 03:50 mc-100-big-Data.db
-rw-r--r--  1 cassandra cassandra8 ene  7 03:50 mc-100-big-Digest.crc32
-rw-r--r--  1 cassandra cassandra   16 ene  7 03:50 mc-100-big-Filter.db
-rw-r--r--  1 cassandra cassandra9 ene  7 03:50 mc-100-big-Index.db
-rw-r--r--  1 cassandra cassandra 4615 ene  7 03:50 mc-100-big-Statistics.db
-rw-r--r--  1 cassandra cassandra   59 ene  7 03:50 mc-100-big-Summary.db
-rw-r--r--  1 cassandra cassandra   92 ene  7 03:50 mc-100-big-TOC.txt
-rw-r--r--  1 cassandra cassandra   43 ene  7 11:48 
mc-101-big-CompressionInfo.db
-rw-r--r--  1 cassandra cassandra   50 ene  7 11:48 mc-101-big-Data.db
-rw-r--r--  1 cassandra cassandra   10 ene  7 11:48 mc-101-big-Digest.crc32
-rw-r--r--  1 cassandra cassandra   16 ene  7 11:48 mc-101-big-Filter.db
-rw-r--r--  1 cassandra cassandra9 ene  7 11:48 mc-101-big-Index.db
-rw-r--r--  1 cassandra cassandra 4615 ene  7 11:48 mc-101-big-Statistics.db
-rw-r--r--  1 cassandra cassandra   59 ene  7 11:48 mc-101-big-Summary.db
-rw-r--r--  1 cassandra cassandra   92 ene  7 11:48 mc-101-big-TOC.txt
-rw-r--r--  1 cassandra cassandra   51 ene  6 17:50 mc-98-big-CompressionInfo.db
-rw-r--r--  1 cassandra cassandra 5226 ene  6 17:50 mc-98-big-Data.db
-rw-r--r--  1 cassandra cassandra   10 ene  6 17:50 mc-98-big-Digest.crc32
-rw-r--r--  1 cassandra cassandra   16 ene  6 17:50 mc-98-big-Filter.db
-rw-r--r--  1 cassandra cassandra9 ene  6 17:50 mc-98-big-Index.db
-rw-r--r--  1 cassandra cassandra 5654 ene  6 17:50 mc-98-big-Statistics.db
-rw-r--r--  1 cassandra cassandra   59 ene  6 17:50 mc-98-big-Summary.db
-rw-r--r--  1 cassandra cassandra   92 ene  6 17:50 mc-98-big-TOC.txt
-rw-r--r--  1 cassandra cassandra   43 ene  7 02:50 mc-99-big-CompressionInfo.db
-rw-r--r--  1 cassandra cassandra   50 ene  7 02:50 mc-99-big-Data.db
-rw-r--r--  1 cassandra cassandra   10 ene  7 02:50 mc-99-big-Digest.crc32
-rw-r--r--  1 cassandra cassandra   16 ene  7 02:50 mc-99-big-Filter.db
-rw-r--r--  1 cassandra cassandra9 ene  7 02:50 mc-99-big-Index.db
-rw-r--r--  1 cassandra cassandra 4615 ene  7 02:50 mc-99-big-Statistics.db
-rw-r--r--  1 cassandra cassandra   59 ene  7 02:50 mc-99-big-Summary.db
-rw-r--r--  1 cassandra cassandra   92 ene  7 02:50 mc-99-big-TOC.txt

permissions are ok, everything assigned to cassandra:cassandra

the system.log:

INFO  [main] 2018-01-10 10:43:29,367 YamlConfigurationLoader.java:89 - 
Configuration location: file:/etc/cassandra/cassandra.yaml
INFO  [main] 2018-01-10 10:43:29,887 Config.java:481 - Node 
configuration:[allocate_tokens_for_keyspace=null; 
authenticator=AllowAllAuthenticator; authorizer=AllowAllAuthorizer; 
auto_bootstrap=true; auto_snapshot=true; back_pressure_enabled=false; 
back_pressure_strategy=null; batch_size_fail_threshold_in_kb=500; 
batch_size_warn_threshold_in_kb=50; batchlog_replay_throttle_in_kb=1024; 
broadcast_address=null; broadcast_rpc_address=null; 
buffer_pool_use_heap_if_exhausted=true; cas_contention_timeout_in_ms=1; 
cdc_enabled=false; cdc_free_space_check_interval_ms=250; 
cdc_raw_directory=null; cdc_total_space_in_mb=0; 
client_encryption_options=; cluster_name=PAP Cluster; 
column_index_cache_size_in_kb=2; column_index_size_in_kb=64; 
commit_failure_policy=stop; commitlog_compression=null; 
commitlog_directory=/var/lib/cassandra/commitlog; 
commitlog_max_compression_buffers_in_pool=3; commitlog_periodic_queue_size=-1; 
commitlog_segment_size_in_mb=32; commitlog_sync=periodic; 
commitlog_sync_batch_window_in_ms=NaN; commitlog_sync_period_in_ms=1; 
commitlog_total_space_in_mb=null; 
compaction_large_partition_warning_threshold_mb=100; 
compaction_throughput_mb_per_sec=16; concurrent_compactors=null; 
concurrent_counter_writes=32; concurrent_materialized_view_writes=32; 
concurrent_reads=32; concurrent_replicates=null; concurrent_writes=32; 

[jira] [Updated] (CASSANDRA-14160) maxPurgeableTimestamp should traverse tables in order of minTimestamp

[Jeff Jirsa (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeff Jirsa updated CASSANDRA-14160:
---
Status: Patch Available  (was: Open)

Thanks for the patch! Marking as patch-available so available reviewers know 
it's here. 

Typically for new features, we like to see unit tests demonstrating coverage. 
Any chance you could throw together a quick JUnit test to demonstrate that this 
works as intended (in particular, that the Comparator is sorting timestamps in 
the proper order)? 

> maxPurgeableTimestamp should traverse tables in order of minTimestamp
> -
>
> Key: CASSANDRA-14160
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14160
> Project: Cassandra
>  Issue Type: Bug
>  Components: Compaction
>Reporter: Josh Snyder
>Assignee: Josh Snyder
>  Labels: performance
> Fix For: 4.x
>
>
> In maxPurgeableTimestamp, we iterate over the bloom filters of each 
> overlapping SSTable. Of the bloom filter hits, we take the SSTable with the 
> lowest minTimestamp. If we kept the SSTables in sorted order of minTimestamp, 
> then we could short-circuit the operation at the first bloom filter hit, 
> reducing cache pressure (or worse, I/O) and CPU time.
> I've written (but not yet benchmarked) [some 
> code|https://github.com/hashbrowncipher/cassandra/commit/29859a4a2e617f6775be49448858bc59fdafab44]
>  to demonstrate this possibility.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Comment Edited] (CASSANDRA-14160) maxPurgeableTimestamp should traverse tables in order of minTimestamp

[Jeff Jirsa (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16320908#comment-16320908
 ] 

Jeff Jirsa edited comment on CASSANDRA-14160 at 1/10/18 7:22 PM:
-

Thanks for the patch! Marking as patch-available so available reviewers know 
it's here. 

Typically for new features, we like to see unit tests demonstrating 
correctness. Any chance you could throw together a quick JUnit test to 
demonstrate that this works as intended (in particular, that the Comparator is 
sorting timestamps in the proper order)? 


was (Author: jjirsa):
Thanks for the patch! Marking as patch-available so available reviewers know 
it's here. 

Typically for new features, we like to see unit tests demonstrating coverage. 
Any chance you could throw together a quick JUnit test to demonstrate that this 
works as intended (in particular, that the Comparator is sorting timestamps in 
the proper order)? 

> maxPurgeableTimestamp should traverse tables in order of minTimestamp
> -
>
> Key: CASSANDRA-14160
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14160
> Project: Cassandra
>  Issue Type: Bug
>  Components: Compaction
>Reporter: Josh Snyder
>Assignee: Josh Snyder
>  Labels: performance
> Fix For: 4.x
>
>
> In maxPurgeableTimestamp, we iterate over the bloom filters of each 
> overlapping SSTable. Of the bloom filter hits, we take the SSTable with the 
> lowest minTimestamp. If we kept the SSTables in sorted order of minTimestamp, 
> then we could short-circuit the operation at the first bloom filter hit, 
> reducing cache pressure (or worse, I/O) and CPU time.
> I've written (but not yet benchmarked) [some 
> code|https://github.com/hashbrowncipher/cassandra/commit/29859a4a2e617f6775be49448858bc59fdafab44]
>  to demonstrate this possibility.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Assigned] (CASSANDRA-14160) maxPurgeableTimestamp should traverse tables in order of minTimestamp

[Jeff Jirsa (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeff Jirsa reassigned CASSANDRA-14160:
--

Assignee: Josh Snyder

> maxPurgeableTimestamp should traverse tables in order of minTimestamp
> -
>
> Key: CASSANDRA-14160
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14160
> Project: Cassandra
>  Issue Type: Bug
>  Components: Compaction
>Reporter: Josh Snyder
>Assignee: Josh Snyder
>  Labels: performance
> Fix For: 4.x
>
>
> In maxPurgeableTimestamp, we iterate over the bloom filters of each 
> overlapping SSTable. Of the bloom filter hits, we take the SSTable with the 
> lowest minTimestamp. If we kept the SSTables in sorted order of minTimestamp, 
> then we could short-circuit the operation at the first bloom filter hit, 
> reducing cache pressure (or worse, I/O) and CPU time.
> I've written (but not yet benchmarked) [some 
> code|https://github.com/hashbrowncipher/cassandra/commit/29859a4a2e617f6775be49448858bc59fdafab44]
>  to demonstrate this possibility.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-14160) maxPurgeableTimestamp should traverse tables in order of minTimestamp

[Jeff Jirsa (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeff Jirsa updated CASSANDRA-14160:
---
Fix Version/s: 4.x

> maxPurgeableTimestamp should traverse tables in order of minTimestamp
> -
>
> Key: CASSANDRA-14160
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14160
> Project: Cassandra
>  Issue Type: Bug
>  Components: Compaction
>Reporter: Josh Snyder
>  Labels: performance
> Fix For: 4.x
>
>
> In maxPurgeableTimestamp, we iterate over the bloom filters of each 
> overlapping SSTable. Of the bloom filter hits, we take the SSTable with the 
> lowest minTimestamp. If we kept the SSTables in sorted order of minTimestamp, 
> then we could short-circuit the operation at the first bloom filter hit, 
> reducing cache pressure (or worse, I/O) and CPU time.
> I've written (but not yet benchmarked) [some 
> code|https://github.com/hashbrowncipher/cassandra/commit/29859a4a2e617f6775be49448858bc59fdafab44]
>  to demonstrate this possibility.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Created] (CASSANDRA-14160) maxPurgeableTimestamp should traverse tables in order of minTimestamp

[Josh Snyder (JIRA)]

Josh Snyder created CASSANDRA-14160:
---

 Summary: maxPurgeableTimestamp should traverse tables in order of 
minTimestamp
 Key: CASSANDRA-14160
 URL: https://issues.apache.org/jira/browse/CASSANDRA-14160
 Project: Cassandra
  Issue Type: Bug
  Components: Compaction
Reporter: Josh Snyder


In maxPurgeableTimestamp, we iterate over the bloom filters of each overlapping 
SSTable. Of the bloom filter hits, we take the SSTable with the lowest 
minTimestamp. If we kept the SSTables in sorted order of minTimestamp, then we 
could short-circuit the operation at the first bloom filter hit, reducing cache 
pressure (or worse, I/O) and CPU time.

I've written (but not yet benchmarked) [some 
code|https://github.com/hashbrowncipher/cassandra/commit/29859a4a2e617f6775be49448858bc59fdafab44]
 to demonstrate this possibility.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-9630) Killing cassandra process results in unclosed connections

[Paulo Motta (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-9630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Paulo Motta updated CASSANDRA-9630:
---
Status: Patch Available  (was: Open)

> Killing cassandra process results in unclosed connections
> -
>
> Key: CASSANDRA-9630
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9630
> Project: Cassandra
>  Issue Type: Bug
>  Components: Distributed Metadata, Streaming and Messaging
>Reporter: Paulo Motta
>Assignee: Paulo Motta
>Priority: Minor
> Fix For: 3.11.x
>
> Attachments: apache-cassandra-3.0.8-SNAPSHOT.jar
>
>
> After upgrading from Cassandra from 2.0.12 to 2.0.15, whenever we killed a 
> cassandra process (with SIGTERM), some other nodes maintained a connection 
> with the killed node in the CLOSE_WAIT state on port 7000 for about 5-20 
> minutes.
> So, when we started the killed node again, other nodes could not establish a 
> handshake because of the connections on the CLOSE_WAIT state, so they 
> remained on the DOWN state to each other until the initial connection expired.
> The problem did not happen if I ran a nodetool disablegossip before killing 
> the node.
> I was able to fix this issue by reverting the CASSANDRA-8336 commits 
> (including CASSANDRA-9238). After reverting this, cassandra now closes 
> connection correctly when killed with -TERM, but leaves connections on 
> CLOSE_WAIT state if I run nodetool disablethrift before killing the nodes.
> I did not try to reproduce the problem in a clean environment.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-9630) Killing cassandra process results in unclosed connections

[Paulo Motta (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-9630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16320733#comment-16320733
 ] 

Paulo Motta commented on CASSANDRA-9630:


Even though we didn't hear back from someone who tested the patch, I'm quite 
confident this will fix the hanging sockets problem, so I will set this to 
patch available.

Would you mind having a look [~snazy]? Patch 
[here|https://github.com/pauloricardomg/cassandra/tree/3.0-9630]. Submitted CI, 
will update after results.

> Killing cassandra process results in unclosed connections
> -
>
> Key: CASSANDRA-9630
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9630
> Project: Cassandra
>  Issue Type: Bug
>  Components: Distributed Metadata, Streaming and Messaging
>Reporter: Paulo Motta
>Assignee: Paulo Motta
>Priority: Minor
> Fix For: 3.11.x
>
> Attachments: apache-cassandra-3.0.8-SNAPSHOT.jar
>
>
> After upgrading from Cassandra from 2.0.12 to 2.0.15, whenever we killed a 
> cassandra process (with SIGTERM), some other nodes maintained a connection 
> with the killed node in the CLOSE_WAIT state on port 7000 for about 5-20 
> minutes.
> So, when we started the killed node again, other nodes could not establish a 
> handshake because of the connections on the CLOSE_WAIT state, so they 
> remained on the DOWN state to each other until the initial connection expired.
> The problem did not happen if I ran a nodetool disablegossip before killing 
> the node.
> I was able to fix this issue by reverting the CASSANDRA-8336 commits 
> (including CASSANDRA-9238). After reverting this, cassandra now closes 
> connection correctly when killed with -TERM, but leaves connections on 
> CLOSE_WAIT state if I run nodetool disablethrift before killing the nodes.
> I did not try to reproduce the problem in a clean environment.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-14069) Node stopped serving write requests when a table has a lot of sstables

[Jeff Jirsa (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeff Jirsa updated CASSANDRA-14069:
---
Assignee: Sergey Lapukhov
  Status: Patch Available  (was: Open)

> Node stopped serving write requests when a table has a lot of sstables
> --
>
> Key: CASSANDRA-14069
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14069
> Project: Cassandra
>  Issue Type: Bug
>  Components: Compaction
>Reporter: Sergey Lapukhov
>Assignee: Sergey Lapukhov
> Attachments: CAS-14069.patch, afterPatch.svg, beforePatch.svg, 
> create.cql, stack.txt
>
>
> Cluster was flooded with SSTables. A table had ~2 sstables. Write 
> requests started failing. 
> Steps to reproduce:
> * Create cluster with 3 nodes
> * Specify 
>{noformat}
> memtable_heap_space_in_mb: 10
> {noformat}
>  in cassandra.yaml
> * Create table standard1 in keyspace1 (for the cassandra-stress tool) with 
> the script [^create.cql]. Please note
> {noformat} compaction = {'class': 'SizeTieredCompactionStrategy', 'enabled': 
> 'false'} {noformat}
>   i.e. compaction will be turned off for now.
> *  Populate node with data:
>   {noformat} cassandra-stress write n=10 -node 127.0.0.1 {noformat}
> * After node was populated, put both read and write pressure on it:
>   {noformat}  cassandra-stress read n=10 -node 127.0.0.1 
>   cassandra-stress write n=10 -node 127.0.0.1 {noformat}
> * While still under pressure, enable LeveledCompactionStrategy
> {code:java}  echo "ALTER TABLE keyspace1.standard1 WITH compaction = { 
> 'class' : 'LeveledCompactionStrategy', 'sstable_size_in_mb' : 1 }; DESC 
> keyspace1.standard1; exit" | bin/cqlsh; {code}
> *Results:*
> Write requests failing.
> 'bin/nodetool cfstats' and 'bin/nodetool compactionstats' commands hanging, 
> if issued from the node running cassandra-stress tool.
> If issued from another node:
> {noformat}
>  $ bin/nodetool cfstats
> ...
> Table: standard1
> SSTable count: 22637
> SSTables in each level: [22651/4, 0, 0, 0, 0, 0, 0, 0, 0] 
> ...
> {noformat}
> {noformat}
> $ bin/nodetool compactionstats
> pending tasks: 12656
>  id   compaction typekeyspace   
> table   completed   totalunit   progress
>935bbc00-d03b-11e7-a47d-2b44293495b8Compaction   keyspace1   
> standard15955601459557860   bytes100.00%
>a29ee660-d03b-11e7-a47d-2b44293495b8Compaction   keyspace1   
> standard180432114   742151655   bytes 10.84%
>9766e400-d03b-11e7-a47d-2b44293495b8Compaction   keyspace1   
> standard15889160458893215   bytes100.00%
>9cdc9880-d03b-11e7-a47d-2b44293495b8Compaction   keyspace1   
> standard12028944920290800   bytes 99.99%
>90f98910-d03b-11e7-a47d-2b44293495b8Compaction   keyspace1   
> standard15968982459695545   bytes 99.99%
>986ede20-d03b-11e7-a47d-2b44293495b8Compaction   keyspace1   
> standard14059859440598820   bytes100.00%
>9cd322a0-d03b-11e7-a47d-2b44293495b8Compaction   keyspace1   
> standard1607567396070   bytes 99.98% 
> {noformat}
> Special note about 'bin/nodetool compactionstats' - picture above is quite 
> typical for this issue. I.e. compaction tasks manage to make it through, but 
> hinder near the full completion (around 99.9 %).
> Maybe the root of the problem is in this thread (see [^stack.txt]):
> {noformat}
> "CompactionExecutor:1748" #4649 daemon prio=1 os_prio=4 
> tid=0x7f35a0096100 nid=0x65f6 runnable [0x7f3228bce000]
>java.lang.Thread.State: RUNNABLE
>   at org.apache.cassandra.dht.AbstractBounds.(AbstractBounds.java:53)
>   at org.apache.cassandra.dht.Bounds.(Bounds.java:42)
>   at 
> org.apache.cassandra.db.compaction.LeveledManifest.overlapping(LeveledManifest.java:562)
>   at 
> org.apache.cassandra.db.compaction.LeveledManifest.overlapping(LeveledManifest.java:549)
>   at 
> org.apache.cassandra.db.compaction.LeveledManifest.getCandidatesFor(LeveledManifest.java:624)
>   at 
> org.apache.cassandra.db.compaction.LeveledManifest.getCompactionCandidates(LeveledManifest.java:378)
>   - locked <0x00070d4c3bc8> (a 
> org.apache.cassandra.db.compaction.LeveledManifest)
>   at 
> org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getNextBackgroundTask(LeveledCompactionStrategy.java:105)
>   - locked <0x00070d6cb2c8> (a 
> org.apache.cassandra.db.compaction.LeveledCompactionStrategy)
>   at 
> org.apache.cassandra.db.compaction.CompactionStrategyManager.getNextBackgroundTask(CompactionStrategyManager.java:102)
>   - locked <0x0006467268b8> (a 
> org.apache.cassandra.db.compaction.CompactionStrategyManager)
>   at 

[jira] [Commented] (CASSANDRA-14069) Node stopped serving write requests when a table has a lot of sstables

[Jeff Jirsa (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16320465#comment-16320465
 ] 

Jeff Jirsa commented on CASSANDRA-14069:


[~krummas] have time for another review? 


> Node stopped serving write requests when a table has a lot of sstables
> --
>
> Key: CASSANDRA-14069
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14069
> Project: Cassandra
>  Issue Type: Bug
>  Components: Compaction
>Reporter: Sergey Lapukhov
>Assignee: Sergey Lapukhov
> Attachments: CAS-14069.patch, afterPatch.svg, beforePatch.svg, 
> create.cql, stack.txt
>
>
> Cluster was flooded with SSTables. A table had ~2 sstables. Write 
> requests started failing. 
> Steps to reproduce:
> * Create cluster with 3 nodes
> * Specify 
>{noformat}
> memtable_heap_space_in_mb: 10
> {noformat}
>  in cassandra.yaml
> * Create table standard1 in keyspace1 (for the cassandra-stress tool) with 
> the script [^create.cql]. Please note
> {noformat} compaction = {'class': 'SizeTieredCompactionStrategy', 'enabled': 
> 'false'} {noformat}
>   i.e. compaction will be turned off for now.
> *  Populate node with data:
>   {noformat} cassandra-stress write n=10 -node 127.0.0.1 {noformat}
> * After node was populated, put both read and write pressure on it:
>   {noformat}  cassandra-stress read n=10 -node 127.0.0.1 
>   cassandra-stress write n=10 -node 127.0.0.1 {noformat}
> * While still under pressure, enable LeveledCompactionStrategy
> {code:java}  echo "ALTER TABLE keyspace1.standard1 WITH compaction = { 
> 'class' : 'LeveledCompactionStrategy', 'sstable_size_in_mb' : 1 }; DESC 
> keyspace1.standard1; exit" | bin/cqlsh; {code}
> *Results:*
> Write requests failing.
> 'bin/nodetool cfstats' and 'bin/nodetool compactionstats' commands hanging, 
> if issued from the node running cassandra-stress tool.
> If issued from another node:
> {noformat}
>  $ bin/nodetool cfstats
> ...
> Table: standard1
> SSTable count: 22637
> SSTables in each level: [22651/4, 0, 0, 0, 0, 0, 0, 0, 0] 
> ...
> {noformat}
> {noformat}
> $ bin/nodetool compactionstats
> pending tasks: 12656
>  id   compaction typekeyspace   
> table   completed   totalunit   progress
>935bbc00-d03b-11e7-a47d-2b44293495b8Compaction   keyspace1   
> standard15955601459557860   bytes100.00%
>a29ee660-d03b-11e7-a47d-2b44293495b8Compaction   keyspace1   
> standard180432114   742151655   bytes 10.84%
>9766e400-d03b-11e7-a47d-2b44293495b8Compaction   keyspace1   
> standard15889160458893215   bytes100.00%
>9cdc9880-d03b-11e7-a47d-2b44293495b8Compaction   keyspace1   
> standard12028944920290800   bytes 99.99%
>90f98910-d03b-11e7-a47d-2b44293495b8Compaction   keyspace1   
> standard15968982459695545   bytes 99.99%
>986ede20-d03b-11e7-a47d-2b44293495b8Compaction   keyspace1   
> standard14059859440598820   bytes100.00%
>9cd322a0-d03b-11e7-a47d-2b44293495b8Compaction   keyspace1   
> standard1607567396070   bytes 99.98% 
> {noformat}
> Special note about 'bin/nodetool compactionstats' - picture above is quite 
> typical for this issue. I.e. compaction tasks manage to make it through, but 
> hinder near the full completion (around 99.9 %).
> Maybe the root of the problem is in this thread (see [^stack.txt]):
> {noformat}
> "CompactionExecutor:1748" #4649 daemon prio=1 os_prio=4 
> tid=0x7f35a0096100 nid=0x65f6 runnable [0x7f3228bce000]
>java.lang.Thread.State: RUNNABLE
>   at org.apache.cassandra.dht.AbstractBounds.(AbstractBounds.java:53)
>   at org.apache.cassandra.dht.Bounds.(Bounds.java:42)
>   at 
> org.apache.cassandra.db.compaction.LeveledManifest.overlapping(LeveledManifest.java:562)
>   at 
> org.apache.cassandra.db.compaction.LeveledManifest.overlapping(LeveledManifest.java:549)
>   at 
> org.apache.cassandra.db.compaction.LeveledManifest.getCandidatesFor(LeveledManifest.java:624)
>   at 
> org.apache.cassandra.db.compaction.LeveledManifest.getCompactionCandidates(LeveledManifest.java:378)
>   - locked <0x00070d4c3bc8> (a 
> org.apache.cassandra.db.compaction.LeveledManifest)
>   at 
> org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getNextBackgroundTask(LeveledCompactionStrategy.java:105)
>   - locked <0x00070d6cb2c8> (a 
> org.apache.cassandra.db.compaction.LeveledCompactionStrategy)
>   at 
> org.apache.cassandra.db.compaction.CompactionStrategyManager.getNextBackgroundTask(CompactionStrategyManager.java:102)
>   - locked <0x0006467268b8> (a 
> 

[jira] [Commented] (CASSANDRA-14102) Vault support for transparent data encryption

[Romain Hardouin (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16320368#comment-16320368
 ] 

Romain Hardouin commented on CASSANDRA-14102:
-

I meant EncryptionContext use. Thanks for your feedback! 

> Vault support for transparent data encryption
> -
>
> Key: CASSANDRA-14102
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14102
> Project: Cassandra
>  Issue Type: New Feature
>Reporter: Stefan Podkowinski
>Assignee: Stefan Podkowinski
>  Labels: encryption
> Fix For: 4.x
>
>
> Transparent data encryption provided by CASSANDRA-9945 can currently be used 
> for commitlog and hints. The default {{KeyProvider}} implementation that we 
> ship allows to use a local keystore for storing and retrieving keys. Thanks 
> to the pluggable handling of the {{KeyStore}} provider and basic Vault 
> related classes introduced in CASSANDRA-13971, a Vault based implementation 
> can be provided as {{KeyProvider}} as well. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-14143) CommitLogStressTest timeout in 3.11

[Jason Brown (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Brown updated CASSANDRA-14143:

   Resolution: Fixed
Fix Version/s: 3.11.2
   Status: Resolved  (was: Patch Available)

+1, and committed as sha {{62e46f71903b339d962c4dcb3d2c04991c391a68}} to 3.11 
only

Thanks!

> CommitLogStressTest timeout in 3.11
> ---
>
> Key: CASSANDRA-14143
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14143
> Project: Cassandra
>  Issue Type: Bug
>  Components: Testing
>Reporter: Jay Zhuang
>Assignee: Jay Zhuang
>  Labels: testing
> Fix For: 3.11.2
>
>
> [~jasobrown] fixed the CommitLogStressTest timeout issue as part of 
> CASSANDRA-13530, but it's only in trunk, it would be better to backport the 
> unittest change to 3.11.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[2/3] cassandra git commit: Split CommitLogStressTest to avoid timeout

[jasobrown]

Split CommitLogStressTest to avoid timeout

patch by Jay Zhuang; reviewed by jasobrown for CASSANDRA-14143


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/62e46f71
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/62e46f71
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/62e46f71

Branch: refs/heads/trunk
Commit: 62e46f71903b339d962c4dcb3d2c04991c391a68
Parents: 4faa6e1
Author: Jay Zhuang 
Authored: Thu Dec 28 11:56:09 2017 -0800
Committer: Jason Brown 
Committed: Wed Jan 10 06:44:26 2018 -0800

--
 CHANGES.txt |   1 +
 .../db/commitlog/BatchCommitLogStressTest.java  |  37 ++
 .../db/commitlog/CommitLogStressTest.java   | 128 ++-
 .../commitlog/PeriodicCommitLogStressTest.java  |  39 ++
 4 files changed, 117 insertions(+), 88 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/62e46f71/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 1b252f5..d6d8066 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.11.2
+ * Split CommitLogStressTest to avoid timeout (CASSANDRA-14143)
  * Avoid invalidating disk boundaries unnecessarily (CASSANDRA-14083)
  * Avoid exposing compaction strategy index externally (CASSANDRA-14082)
  * Prevent continuous schema exchange between 3.0 and 3.11 nodes 
(CASSANDRA-14109)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/62e46f71/test/long/org/apache/cassandra/db/commitlog/BatchCommitLogStressTest.java
--
diff --git 
a/test/long/org/apache/cassandra/db/commitlog/BatchCommitLogStressTest.java 
b/test/long/org/apache/cassandra/db/commitlog/BatchCommitLogStressTest.java
new file mode 100644
index 000..3665882
--- /dev/null
+++ b/test/long/org/apache/cassandra/db/commitlog/BatchCommitLogStressTest.java
@@ -0,0 +1,37 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.commitlog;
+
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import org.apache.cassandra.config.Config;
+import org.apache.cassandra.config.DatabaseDescriptor;
+import org.apache.cassandra.config.ParameterizedClass;
+import org.apache.cassandra.security.EncryptionContext;
+
+@RunWith(Parameterized.class)
+public class BatchCommitLogStressTest extends CommitLogStressTest
+{
+public BatchCommitLogStressTest(ParameterizedClass commitLogCompression, 
EncryptionContext encryptionContext)
+{
+super(commitLogCompression, encryptionContext);
+DatabaseDescriptor.setCommitLogSync(Config.CommitLogSync.batch);
+}
+}

http://git-wip-us.apache.org/repos/asf/cassandra/blob/62e46f71/test/long/org/apache/cassandra/db/commitlog/CommitLogStressTest.java
--
diff --git 
a/test/long/org/apache/cassandra/db/commitlog/CommitLogStressTest.java 
b/test/long/org/apache/cassandra/db/commitlog/CommitLogStressTest.java
index 3f5be03..2162d85 100644
--- a/test/long/org/apache/cassandra/db/commitlog/CommitLogStressTest.java
+++ b/test/long/org/apache/cassandra/db/commitlog/CommitLogStressTest.java
@@ -33,25 +33,30 @@ import com.google.common.util.concurrent.RateLimiter;
 import org.junit.Assert;
 import org.junit.Before;
 import org.junit.BeforeClass;
+import org.junit.Ignore;
 import org.junit.Test;
+import org.junit.runners.Parameterized.Parameters;
 
 import io.netty.util.concurrent.FastThreadLocalThread;
 import org.apache.cassandra.SchemaLoader;
 import org.apache.cassandra.Util;
 import org.apache.cassandra.UpdateBuilder;
-import org.apache.cassandra.config.Config.CommitLogSync;
 import org.apache.cassandra.config.*;
 import org.apache.cassandra.db.Mutation;
 import org.apache.cassandra.db.marshal.UTF8Type;
 import org.apache.cassandra.db.partitions.PartitionUpdate;
 import org.apache.cassandra.db.rows.*;

[1/3] cassandra git commit: Split CommitLogStressTest to avoid timeout

[jasobrown]

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-3.11 4faa6e12f -> 62e46f719
  refs/heads/trunk 39837d479 -> ee907a321


Split CommitLogStressTest to avoid timeout

patch by Jay Zhuang; reviewed by jasobrown for CASSANDRA-14143


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/62e46f71
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/62e46f71
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/62e46f71

Branch: refs/heads/cassandra-3.11
Commit: 62e46f71903b339d962c4dcb3d2c04991c391a68
Parents: 4faa6e1
Author: Jay Zhuang 
Authored: Thu Dec 28 11:56:09 2017 -0800
Committer: Jason Brown 
Committed: Wed Jan 10 06:44:26 2018 -0800

--
 CHANGES.txt |   1 +
 .../db/commitlog/BatchCommitLogStressTest.java  |  37 ++
 .../db/commitlog/CommitLogStressTest.java   | 128 ++-
 .../commitlog/PeriodicCommitLogStressTest.java  |  39 ++
 4 files changed, 117 insertions(+), 88 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/62e46f71/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 1b252f5..d6d8066 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.11.2
+ * Split CommitLogStressTest to avoid timeout (CASSANDRA-14143)
  * Avoid invalidating disk boundaries unnecessarily (CASSANDRA-14083)
  * Avoid exposing compaction strategy index externally (CASSANDRA-14082)
  * Prevent continuous schema exchange between 3.0 and 3.11 nodes 
(CASSANDRA-14109)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/62e46f71/test/long/org/apache/cassandra/db/commitlog/BatchCommitLogStressTest.java
--
diff --git 
a/test/long/org/apache/cassandra/db/commitlog/BatchCommitLogStressTest.java 
b/test/long/org/apache/cassandra/db/commitlog/BatchCommitLogStressTest.java
new file mode 100644
index 000..3665882
--- /dev/null
+++ b/test/long/org/apache/cassandra/db/commitlog/BatchCommitLogStressTest.java
@@ -0,0 +1,37 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.commitlog;
+
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import org.apache.cassandra.config.Config;
+import org.apache.cassandra.config.DatabaseDescriptor;
+import org.apache.cassandra.config.ParameterizedClass;
+import org.apache.cassandra.security.EncryptionContext;
+
+@RunWith(Parameterized.class)
+public class BatchCommitLogStressTest extends CommitLogStressTest
+{
+public BatchCommitLogStressTest(ParameterizedClass commitLogCompression, 
EncryptionContext encryptionContext)
+{
+super(commitLogCompression, encryptionContext);
+DatabaseDescriptor.setCommitLogSync(Config.CommitLogSync.batch);
+}
+}

http://git-wip-us.apache.org/repos/asf/cassandra/blob/62e46f71/test/long/org/apache/cassandra/db/commitlog/CommitLogStressTest.java
--
diff --git 
a/test/long/org/apache/cassandra/db/commitlog/CommitLogStressTest.java 
b/test/long/org/apache/cassandra/db/commitlog/CommitLogStressTest.java
index 3f5be03..2162d85 100644
--- a/test/long/org/apache/cassandra/db/commitlog/CommitLogStressTest.java
+++ b/test/long/org/apache/cassandra/db/commitlog/CommitLogStressTest.java
@@ -33,25 +33,30 @@ import com.google.common.util.concurrent.RateLimiter;
 import org.junit.Assert;
 import org.junit.Before;
 import org.junit.BeforeClass;
+import org.junit.Ignore;
 import org.junit.Test;
+import org.junit.runners.Parameterized.Parameters;
 
 import io.netty.util.concurrent.FastThreadLocalThread;
 import org.apache.cassandra.SchemaLoader;
 import org.apache.cassandra.Util;
 import org.apache.cassandra.UpdateBuilder;
-import org.apache.cassandra.config.Config.CommitLogSync;
 import org.apache.cassandra.config.*;
 import org.apache.cassandra.db.Mutation;
 import 

[3/3] cassandra git commit: Merge branch 'cassandra-3.11' into trunk

[jasobrown]

Merge branch 'cassandra-3.11' into trunk


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/ee907a32
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/ee907a32
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/ee907a32

Branch: refs/heads/trunk
Commit: ee907a3219b3f725b6f969d3a84afdeb11e1f397
Parents: 39837d4 62e46f7
Author: Jason Brown 
Authored: Wed Jan 10 06:47:16 2018 -0800
Committer: Jason Brown 
Committed: Wed Jan 10 06:47:16 2018 -0800

--

--



-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-14143) CommitLogStressTest timeout in 3.11

[Jason Brown (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Brown updated CASSANDRA-14143:

Reviewer: Jason Brown

> CommitLogStressTest timeout in 3.11
> ---
>
> Key: CASSANDRA-14143
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14143
> Project: Cassandra
>  Issue Type: Bug
>  Components: Testing
>Reporter: Jay Zhuang
>Assignee: Jay Zhuang
>  Labels: testing
>
> [~jasobrown] fixed the CommitLogStressTest timeout issue as part of 
> CASSANDRA-13530, but it's only in trunk, it would be better to backport the 
> unittest change to 3.11.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-9067) BloomFilter serialization format should not change byte ordering

[Jason Brown (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-9067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16320304#comment-16320304
 ] 

Jason Brown commented on CASSANDRA-9067:


bq. But as the BloomFilter file doesn't have version support, how could we 
migrate the existing format to the new one?

Because the bloom filter is one of the sstable components, a specific bloom 
filter file is attached to a given sstable. And as each sstable has a version 
(ka, jb, mc, ...), you can add version-specific behavior to 
{{org.apache.cassandra.io.sstable.format.Version}}. 

In {{BigTableVersion}}, say we create a new version ({{na}}) for 4.0. You can 
know that any bloom filter file with anything less than {{na}} will use the old 
serialization format, and anything greater than or equal to {{na}} uses the 
optimized version.  You'll need to keep both serialization paths around in the 
code base, but as new sstables get written out and as old sstables get 
compacted away, eventually users will be on the optimized version.

For an example of how to do this, check out 
[{{BigTableVersion#hasMetadataChecksum}}|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/io/sstable/format/big/BigFormat.java#L133]
 (CASSANDRA-13321)


> BloomFilter serialization format should not change byte ordering
> 
>
> Key: CASSANDRA-9067
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9067
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Benedict
>Assignee: Jay Zhuang
>Priority: Minor
> Fix For: 4.x
>
>
> As a follow-up to CASSANDRA-9066 and CASSANDRA-9060, it appears we do some 
> unnecessary byte swapping during the serialization of bloom filters, which 
> makes the logic slower and harder to follow. We should either perform them 
> more efficiently (using Long.reverseBytes) or, preferably, eliminate the 
> conversion altogether since it does not appear to serve any purpose.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14102) Vault support for transparent data encryption

[Stefan Podkowinski (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16320298#comment-16320298
 ] 

Stefan Podkowinski commented on CASSANDRA-14102:


Performance impact by using the Vault key provider instead of a local keystore? 
Or the optimization of EncryptionContext use? 

Each key for any key alias is immutable and will be cached infinitely. The 
interaction with Vault should thus be minimal and in most cases we will only 
call Vault once on startup, to retrieve the key. With CASSANDRA-14107 we also 
call Vault to check for new keys and to fetch them. But also only sporadically. 

As for the optimization of  EncryptionContext use it probably depends on how 
many encrypted committlog segments and hint files are being generated. Not so 
trivial to create a good benchmark for that.

> Vault support for transparent data encryption
> -
>
> Key: CASSANDRA-14102
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14102
> Project: Cassandra
>  Issue Type: New Feature
>Reporter: Stefan Podkowinski
>Assignee: Stefan Podkowinski
>  Labels: encryption
> Fix For: 4.x
>
>
> Transparent data encryption provided by CASSANDRA-9945 can currently be used 
> for commitlog and hints. The default {{KeyProvider}} implementation that we 
> ship allows to use a local keystore for storing and retrieving keys. Thanks 
> to the pluggable handling of the {{KeyStore}} provider and basic Vault 
> related classes introduced in CASSANDRA-13971, a Vault based implementation 
> can be provided as {{KeyProvider}} as well. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14152) Remove unused on-heap BloomFilter implementation

[Jason Brown (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16320263#comment-16320263
 ] 

Jason Brown commented on CASSANDRA-14152:
-

I don't have a strong opinion, but since at least 3.0 we've hard coded to using 
the off-heap bloom filters. As any legacy sstables should have been updated by 
now, I think it's fine to go ahead and drop the on-heap filters. Note: this 
should go on trunk only (4.0)

[~jay.zhuang] I can review if you create a patch.

> Remove unused on-heap BloomFilter implementation
> 
>
> Key: CASSANDRA-14152
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14152
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Jay Zhuang
>Priority: Minor
>
> Seems like it's just dead code, should that be removed?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14153) BloomFilterTest generates un-deleted test file

[Jason Brown (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16320256#comment-16320256
 ] 

Jason Brown commented on CASSANDRA-14153:
-

As an alternative to calling {{file#delete()}} and using a {{try/catch}} block, 
I created a new function {{FileUtils#createDeletableTempFile()}}, which calls 
{{File#deleteOnExit}} on the temp file. This way puts the cleanup logic in a 
one place. wdyt?

||14153||
|[branch|https://github.com/jasobrown/cassandra/tree/14153]|
|[utests|https://circleci.com/gh/jasobrown/workflows/cassandra/tree/14153]|


> BloomFilterTest generates un-deleted test file
> --
>
> Key: CASSANDRA-14153
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14153
> Project: Cassandra
>  Issue Type: Test
>  Components: Testing
>Reporter: Jay Zhuang
>Assignee: Jay Zhuang
>Priority: Minor
>  Labels: testing
>
> https://github.com/apache/cassandra/blob/trunk/test/unit/org/apache/cassandra/utils/BloomFilterTest.java#L196



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-14153) BloomFilterTest generates un-deleted test file

[Jason Brown (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Brown updated CASSANDRA-14153:

Reviewer: Jason Brown

> BloomFilterTest generates un-deleted test file
> --
>
> Key: CASSANDRA-14153
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14153
> Project: Cassandra
>  Issue Type: Test
>  Components: Testing
>Reporter: Jay Zhuang
>Assignee: Jay Zhuang
>Priority: Minor
>  Labels: testing
>
> https://github.com/apache/cassandra/blob/trunk/test/unit/org/apache/cassandra/utils/BloomFilterTest.java#L196



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-14139) Acquire read lock before accessing CompactionStrategyManager fields

[Marcus Eriksson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Marcus Eriksson updated CASSANDRA-14139:

Status: Ready to Commit  (was: Patch Available)

> Acquire read lock before accessing CompactionStrategyManager fields
> ---
>
> Key: CASSANDRA-14139
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14139
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Paulo Motta
>Assignee: Paulo Motta
> Attachments: 3.11-14139-dtest.png, 3.11-14139-testall.png, 
> trunk-14139-dtest.png, trunk-14139-testall.png
>
>
> There are a few methods in {{CompactionStrategyManager}} accessing the 
> repaired/unrepaired compaction strategy lists without using the read lock, 
> what could cause issues like the one below:
> {noformat}
> ERROR [CompactionExecutor:1] 2017-12-22 12:17:12,320 CassandraDaemon.java:141 
> - Exception in thread Thread[CompactionExecutor:1,5,main]
> java.lang.IndexOutOfBoundsException: Index: 0, Size: 1
> at java.util.ArrayList.rangeCheck(ArrayList.java:657)
> at java.util.ArrayList.get(ArrayList.java:433)
> at 
> org.apache.cassandra.db.compaction.CompactionStrategyManager.supportsEarlyOpen(CompactionStrategyManager.java:1262)
> at 
> org.apache.cassandra.db.ColumnFamilyStore.supportsEarlyOpen(ColumnFamilyStore.java:558)
> at 
> org.apache.cassandra.io.sstable.SSTableRewriter.construct(SSTableRewriter.java:119)
> at 
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.(CompactionAwareWriter.java:91)
> at 
> org.apache.cassandra.db.compaction.writers.DefaultCompactionWriter.(DefaultCompactionWriter.java:57)
> at 
> org.apache.cassandra.db.compaction.CompactionTask.getCompactionAwareWriter(CompactionTask.java:293)
> at 
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:200)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at 
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:90)
> at 
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:101)
> at 
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:310)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at 
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14139) Acquire read lock before accessing CompactionStrategyManager fields

[Marcus Eriksson (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16320246#comment-16320246
 ] 

Marcus Eriksson commented on CASSANDRA-14139:
-

lgtm, +1

> Acquire read lock before accessing CompactionStrategyManager fields
> ---
>
> Key: CASSANDRA-14139
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14139
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Paulo Motta
>Assignee: Paulo Motta
> Attachments: 3.11-14139-dtest.png, 3.11-14139-testall.png, 
> trunk-14139-dtest.png, trunk-14139-testall.png
>
>
> There are a few methods in {{CompactionStrategyManager}} accessing the 
> repaired/unrepaired compaction strategy lists without using the read lock, 
> what could cause issues like the one below:
> {noformat}
> ERROR [CompactionExecutor:1] 2017-12-22 12:17:12,320 CassandraDaemon.java:141 
> - Exception in thread Thread[CompactionExecutor:1,5,main]
> java.lang.IndexOutOfBoundsException: Index: 0, Size: 1
> at java.util.ArrayList.rangeCheck(ArrayList.java:657)
> at java.util.ArrayList.get(ArrayList.java:433)
> at 
> org.apache.cassandra.db.compaction.CompactionStrategyManager.supportsEarlyOpen(CompactionStrategyManager.java:1262)
> at 
> org.apache.cassandra.db.ColumnFamilyStore.supportsEarlyOpen(ColumnFamilyStore.java:558)
> at 
> org.apache.cassandra.io.sstable.SSTableRewriter.construct(SSTableRewriter.java:119)
> at 
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.(CompactionAwareWriter.java:91)
> at 
> org.apache.cassandra.db.compaction.writers.DefaultCompactionWriter.(DefaultCompactionWriter.java:57)
> at 
> org.apache.cassandra.db.compaction.CompactionTask.getCompactionAwareWriter(CompactionTask.java:293)
> at 
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:200)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at 
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:90)
> at 
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:101)
> at 
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:310)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at 
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-14139) Acquire read lock before accessing CompactionStrategyManager fields

[Paulo Motta (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Paulo Motta updated CASSANDRA-14139:

Reviewer: Marcus Eriksson

> Acquire read lock before accessing CompactionStrategyManager fields
> ---
>
> Key: CASSANDRA-14139
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14139
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Paulo Motta
>Assignee: Paulo Motta
> Attachments: 3.11-14139-dtest.png, 3.11-14139-testall.png, 
> trunk-14139-dtest.png, trunk-14139-testall.png
>
>
> There are a few methods in {{CompactionStrategyManager}} accessing the 
> repaired/unrepaired compaction strategy lists without using the read lock, 
> what could cause issues like the one below:
> {noformat}
> ERROR [CompactionExecutor:1] 2017-12-22 12:17:12,320 CassandraDaemon.java:141 
> - Exception in thread Thread[CompactionExecutor:1,5,main]
> java.lang.IndexOutOfBoundsException: Index: 0, Size: 1
> at java.util.ArrayList.rangeCheck(ArrayList.java:657)
> at java.util.ArrayList.get(ArrayList.java:433)
> at 
> org.apache.cassandra.db.compaction.CompactionStrategyManager.supportsEarlyOpen(CompactionStrategyManager.java:1262)
> at 
> org.apache.cassandra.db.ColumnFamilyStore.supportsEarlyOpen(ColumnFamilyStore.java:558)
> at 
> org.apache.cassandra.io.sstable.SSTableRewriter.construct(SSTableRewriter.java:119)
> at 
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.(CompactionAwareWriter.java:91)
> at 
> org.apache.cassandra.db.compaction.writers.DefaultCompactionWriter.(DefaultCompactionWriter.java:57)
> at 
> org.apache.cassandra.db.compaction.CompactionTask.getCompactionAwareWriter(CompactionTask.java:293)
> at 
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:200)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at 
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:90)
> at 
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:101)
> at 
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:310)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at 
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Resolved] (CASSANDRA-14154) `ant javadoc` task broken due to UTF-8 characters in multiple source files

[Jason Brown (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Brown resolved CASSANDRA-14154.
-
   Resolution: Fixed
Fix Version/s: 4.0
   3.11.2
   3.0.16

Nice find. +1 and committed as sha {{fde05f4f1b4ad814acf79bed61500aaf2ebe39d6}}.

Thanks!

> `ant javadoc` task broken due to UTF-8 characters in multiple source files
> --
>
> Key: CASSANDRA-14154
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14154
> Project: Cassandra
>  Issue Type: Bug
>  Components: Build
> Environment: Built on OpenSUSE Tumbleweed. I used the following java 
> packages when building:
> {quote}
> titan% rpm -qa | grep java
> javapackages-local-4.7.0+git20170331.ef4057e7-1.5.x86_64
> java-1_8_0-openjdk-devel-1.8.0.151-1.3.x86_64
> python3-javapackages-4.7.0+git20170331.ef4057e7-1.5.x86_64
> java-1_8_0-openjdk-headless-1.8.0.151-1.3.x86_64
> timezone-java-2017c-1.3.noarch
> java-1_8_0-openjdk-1.8.0.151-1.3.x86_64
> libjavascriptcoregtk-1_0-0-2.4.11-7.6.x86_64
> libjavascriptcoregtk-4_0-18-2.18.4-1.1.x86_64
> javapackages-tools-4.7.0+git20170331.ef4057e7-1.5.x86_64
> {quote}
>Reporter: Johannes Grassler
>Assignee: Johannes Grassler
>Priority: Minor
> Fix For: 3.0.16, 3.11.2, 4.0
>
> Attachments: build.log, javadoc-encoding.patch
>
>
> Several source files contain UTF-8 characters in String literals. When 
> building the {{javadoc}} target with ant ({{ant javadoc}}), these will trip 
> up javadoc, which defaults to ASCII encoding. See the {{build.log}} for what 
> I did and the resulting output.
> I created a patch that will fix the problem ({{javadoc-encoding.patch}}), 
> which is attached.
> I encountered this problem in 3.11.1, but I haven't checked whether other 
> versions are affected as well.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[3/6] cassandra git commit: Set encoding for javadoc generation

[jasobrown]

Set encoding for javadoc generation

patch by Johannes Grassler; reviewed by jasobrown for CASSANDRA-14154


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/fde05f4f
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/fde05f4f
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/fde05f4f

Branch: refs/heads/trunk
Commit: fde05f4f1b4ad814acf79bed61500aaf2ebe39d6
Parents: 5b4485b
Author: Johannes Grassler 
Authored: Wed Jan 10 05:07:48 2018 -0800
Committer: Jason Brown 
Committed: Wed Jan 10 05:07:48 2018 -0800

--
 CHANGES.txt | 1 +
 build.xml   | 2 +-
 2 files changed, 2 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/fde05f4f/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 7746c73..c32e56a 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.16
+ * Set encoding for javadoc generation (CASSANDRA-14154)
  * Fix index target computation for dense composite tables with dropped 
compact storage (CASSANDRA-14104)
  * Improve commit log chain marker updating (CASSANDRA-14108)
  * Extra range tombstone bound creates double rows (CASSANDRA-14008)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/fde05f4f/build.xml
--
diff --git a/build.xml b/build.xml
index 386bb4dc..6f98242 100644
--- a/build.xml
+++ b/build.xml
@@ -170,7 +170,7 @@
   
 
   


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[2/6] cassandra git commit: Set encoding for javadoc generation

[jasobrown]

Set encoding for javadoc generation

patch by Johannes Grassler; reviewed by jasobrown for CASSANDRA-14154


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/fde05f4f
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/fde05f4f
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/fde05f4f

Branch: refs/heads/cassandra-3.11
Commit: fde05f4f1b4ad814acf79bed61500aaf2ebe39d6
Parents: 5b4485b
Author: Johannes Grassler 
Authored: Wed Jan 10 05:07:48 2018 -0800
Committer: Jason Brown 
Committed: Wed Jan 10 05:07:48 2018 -0800

--
 CHANGES.txt | 1 +
 build.xml   | 2 +-
 2 files changed, 2 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/fde05f4f/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 7746c73..c32e56a 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.16
+ * Set encoding for javadoc generation (CASSANDRA-14154)
  * Fix index target computation for dense composite tables with dropped 
compact storage (CASSANDRA-14104)
  * Improve commit log chain marker updating (CASSANDRA-14108)
  * Extra range tombstone bound creates double rows (CASSANDRA-14008)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/fde05f4f/build.xml
--
diff --git a/build.xml b/build.xml
index 386bb4dc..6f98242 100644
--- a/build.xml
+++ b/build.xml
@@ -170,7 +170,7 @@
   
 
   


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[6/6] cassandra git commit: Merge branch 'cassandra-3.11' into trunk

[jasobrown]

Merge branch 'cassandra-3.11' into trunk


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/39837d47
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/39837d47
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/39837d47

Branch: refs/heads/trunk
Commit: 39837d4793292b2378baa30eb9704124950363db
Parents: de7c24b 4faa6e1
Author: Jason Brown 
Authored: Wed Jan 10 05:15:45 2018 -0800
Committer: Jason Brown 
Committed: Wed Jan 10 05:17:06 2018 -0800

--
 CHANGES.txt | 1 +
 build.xml   | 2 +-
 2 files changed, 2 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/39837d47/CHANGES.txt
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/39837d47/build.xml
--


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[1/6] cassandra git commit: Set encoding for javadoc generation

[jasobrown]

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-3.0 5b4485b29 -> fde05f4f1
  refs/heads/cassandra-3.11 9337ca316 -> 4faa6e12f
  refs/heads/trunk de7c24b39 -> 39837d479


Set encoding for javadoc generation

patch by Johannes Grassler; reviewed by jasobrown for CASSANDRA-14154


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/fde05f4f
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/fde05f4f
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/fde05f4f

Branch: refs/heads/cassandra-3.0
Commit: fde05f4f1b4ad814acf79bed61500aaf2ebe39d6
Parents: 5b4485b
Author: Johannes Grassler 
Authored: Wed Jan 10 05:07:48 2018 -0800
Committer: Jason Brown 
Committed: Wed Jan 10 05:07:48 2018 -0800

--
 CHANGES.txt | 1 +
 build.xml   | 2 +-
 2 files changed, 2 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/fde05f4f/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 7746c73..c32e56a 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.16
+ * Set encoding for javadoc generation (CASSANDRA-14154)
  * Fix index target computation for dense composite tables with dropped 
compact storage (CASSANDRA-14104)
  * Improve commit log chain marker updating (CASSANDRA-14108)
  * Extra range tombstone bound creates double rows (CASSANDRA-14008)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/fde05f4f/build.xml
--
diff --git a/build.xml b/build.xml
index 386bb4dc..6f98242 100644
--- a/build.xml
+++ b/build.xml
@@ -170,7 +170,7 @@
   
 
   


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[4/6] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11

[jasobrown]

Merge branch 'cassandra-3.0' into cassandra-3.11


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/4faa6e12
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/4faa6e12
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/4faa6e12

Branch: refs/heads/trunk
Commit: 4faa6e12ff22daa2c6330993825c28b433175df6
Parents: 9337ca3 fde05f4
Author: Jason Brown 
Authored: Wed Jan 10 05:09:45 2018 -0800
Committer: Jason Brown 
Committed: Wed Jan 10 05:10:54 2018 -0800

--
 CHANGES.txt | 1 +
 build.xml   | 2 +-
 2 files changed, 2 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/4faa6e12/CHANGES.txt
--
diff --cc CHANGES.txt
index 7ac528c,c32e56a..1b252f5
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,17 -1,5 +1,18 @@@
 -3.0.16
 +3.11.2
 + * Avoid invalidating disk boundaries unnecessarily (CASSANDRA-14083)
 + * Avoid exposing compaction strategy index externally (CASSANDRA-14082)
 + * Prevent continuous schema exchange between 3.0 and 3.11 nodes 
(CASSANDRA-14109)
 + * Fix imbalanced disks when replacing node with same address with JBOD 
(CASSANDRA-14084)
 + * Reload compaction strategies when disk boundaries are invalidated 
(CASSANDRA-13948)
 + * Remove OpenJDK log warning (CASSANDRA-13916)
 + * Prevent compaction strategies from looping indefinitely (CASSANDRA-14079)
 + * Cache disk boundaries (CASSANDRA-13215)
 + * Add asm jar to build.xml for maven builds (CASSANDRA-11193)
 + * Round buffer size to powers of 2 for the chunk cache (CASSANDRA-13897)
 + * Update jackson JSON jars (CASSANDRA-13949)
 + * Avoid locks when checking LCS fanout and if we should defrag 
(CASSANDRA-13930)
 +Merged from 3.0:
+  * Set encoding for javadoc generation (CASSANDRA-14154)
   * Fix index target computation for dense composite tables with dropped 
compact storage (CASSANDRA-14104)
   * Improve commit log chain marker updating (CASSANDRA-14108)
   * Extra range tombstone bound creates double rows (CASSANDRA-14008)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/4faa6e12/build.xml
--


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[5/6] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11

[jasobrown]

Merge branch 'cassandra-3.0' into cassandra-3.11


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/4faa6e12
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/4faa6e12
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/4faa6e12

Branch: refs/heads/cassandra-3.11
Commit: 4faa6e12ff22daa2c6330993825c28b433175df6
Parents: 9337ca3 fde05f4
Author: Jason Brown 
Authored: Wed Jan 10 05:09:45 2018 -0800
Committer: Jason Brown 
Committed: Wed Jan 10 05:10:54 2018 -0800

--
 CHANGES.txt | 1 +
 build.xml   | 2 +-
 2 files changed, 2 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/4faa6e12/CHANGES.txt
--
diff --cc CHANGES.txt
index 7ac528c,c32e56a..1b252f5
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,17 -1,5 +1,18 @@@
 -3.0.16
 +3.11.2
 + * Avoid invalidating disk boundaries unnecessarily (CASSANDRA-14083)
 + * Avoid exposing compaction strategy index externally (CASSANDRA-14082)
 + * Prevent continuous schema exchange between 3.0 and 3.11 nodes 
(CASSANDRA-14109)
 + * Fix imbalanced disks when replacing node with same address with JBOD 
(CASSANDRA-14084)
 + * Reload compaction strategies when disk boundaries are invalidated 
(CASSANDRA-13948)
 + * Remove OpenJDK log warning (CASSANDRA-13916)
 + * Prevent compaction strategies from looping indefinitely (CASSANDRA-14079)
 + * Cache disk boundaries (CASSANDRA-13215)
 + * Add asm jar to build.xml for maven builds (CASSANDRA-11193)
 + * Round buffer size to powers of 2 for the chunk cache (CASSANDRA-13897)
 + * Update jackson JSON jars (CASSANDRA-13949)
 + * Avoid locks when checking LCS fanout and if we should defrag 
(CASSANDRA-13930)
 +Merged from 3.0:
+  * Set encoding for javadoc generation (CASSANDRA-14154)
   * Fix index target computation for dense composite tables with dropped 
compact storage (CASSANDRA-14104)
   * Improve commit log chain marker updating (CASSANDRA-14108)
   * Extra range tombstone bound creates double rows (CASSANDRA-14008)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/4faa6e12/build.xml
--


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14139) Acquire read lock before accessing CompactionStrategyManager fields

[Paulo Motta (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16320204#comment-16320204
 ] 

Paulo Motta commented on CASSANDRA-14139:
-

Thanks for the review! See follow-up below:

bq. for the supportsEarlyOpen method we can instead store the boolean in 
startup like we do with fanout and shouldDefragment

good catch, fixed!

bq. in trunk, for getRepaired, getUnrepaired and getPendingRepairManagers we 
don't need the readlock - we return the list directly (and when refreshing we 
clear the same list) - to make this safe we would need to copy the list and 
return the copy, but those methods are only used in tests so I doubt we need it 
(maybe add a comment though)

even though this is only used in test, for consistency I opted for copying the 
list and returning a copying, so it is safe in case anybody uses it outside 
testing.

Updated patch with fixes above, CI looks good. Please let me know what do you 
think.

> Acquire read lock before accessing CompactionStrategyManager fields
> ---
>
> Key: CASSANDRA-14139
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14139
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Paulo Motta
>Assignee: Paulo Motta
> Attachments: 3.11-14139-dtest.png, 3.11-14139-testall.png, 
> trunk-14139-dtest.png, trunk-14139-testall.png
>
>
> There are a few methods in {{CompactionStrategyManager}} accessing the 
> repaired/unrepaired compaction strategy lists without using the read lock, 
> what could cause issues like the one below:
> {noformat}
> ERROR [CompactionExecutor:1] 2017-12-22 12:17:12,320 CassandraDaemon.java:141 
> - Exception in thread Thread[CompactionExecutor:1,5,main]
> java.lang.IndexOutOfBoundsException: Index: 0, Size: 1
> at java.util.ArrayList.rangeCheck(ArrayList.java:657)
> at java.util.ArrayList.get(ArrayList.java:433)
> at 
> org.apache.cassandra.db.compaction.CompactionStrategyManager.supportsEarlyOpen(CompactionStrategyManager.java:1262)
> at 
> org.apache.cassandra.db.ColumnFamilyStore.supportsEarlyOpen(ColumnFamilyStore.java:558)
> at 
> org.apache.cassandra.io.sstable.SSTableRewriter.construct(SSTableRewriter.java:119)
> at 
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.(CompactionAwareWriter.java:91)
> at 
> org.apache.cassandra.db.compaction.writers.DefaultCompactionWriter.(DefaultCompactionWriter.java:57)
> at 
> org.apache.cassandra.db.compaction.CompactionTask.getCompactionAwareWriter(CompactionTask.java:293)
> at 
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:200)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at 
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:90)
> at 
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:101)
> at 
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:310)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at 
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Assigned] (CASSANDRA-14154) `ant javadoc` task broken due to UTF-8 characters in multiple source files

[Jason Brown (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Brown reassigned CASSANDRA-14154:
---

Assignee: Johannes Grassler
Reviewer: Jason Brown

> `ant javadoc` task broken due to UTF-8 characters in multiple source files
> --
>
> Key: CASSANDRA-14154
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14154
> Project: Cassandra
>  Issue Type: Bug
>  Components: Build
> Environment: Built on OpenSUSE Tumbleweed. I used the following java 
> packages when building:
> {quote}
> titan% rpm -qa | grep java
> javapackages-local-4.7.0+git20170331.ef4057e7-1.5.x86_64
> java-1_8_0-openjdk-devel-1.8.0.151-1.3.x86_64
> python3-javapackages-4.7.0+git20170331.ef4057e7-1.5.x86_64
> java-1_8_0-openjdk-headless-1.8.0.151-1.3.x86_64
> timezone-java-2017c-1.3.noarch
> java-1_8_0-openjdk-1.8.0.151-1.3.x86_64
> libjavascriptcoregtk-1_0-0-2.4.11-7.6.x86_64
> libjavascriptcoregtk-4_0-18-2.18.4-1.1.x86_64
> javapackages-tools-4.7.0+git20170331.ef4057e7-1.5.x86_64
> {quote}
>Reporter: Johannes Grassler
>Assignee: Johannes Grassler
>Priority: Minor
> Attachments: build.log, javadoc-encoding.patch
>
>
> Several source files contain UTF-8 characters in String literals. When 
> building the {{javadoc}} target with ant ({{ant javadoc}}), these will trip 
> up javadoc, which defaults to ASCII encoding. See the {{build.log}} for what 
> I did and the resulting output.
> I created a patch that will fix the problem ({{javadoc-encoding.patch}}), 
> which is attached.
> I encountered this problem in 3.11.1, but I haven't checked whether other 
> versions are affected as well.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-14107) Dynamic key rotation support for transparent data encryption

[Stefan Podkowinski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stefan Podkowinski updated CASSANDRA-14107:
---
Summary: Dynamic key rotation support for transparent data encryption  
(was: Introduce simple key alias versioning scheme for TDE)

> Dynamic key rotation support for transparent data encryption
> 
>
> Key: CASSANDRA-14107
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14107
> Project: Cassandra
>  Issue Type: New Feature
>Reporter: Stefan Podkowinski
>Assignee: Stefan Podkowinski
>Priority: Minor
>  Labels: encryption
> Fix For: 4.x
>
>
> Handling of encryption keys as introduced in CASSANDRA-9945 takes place by 
> referencing a key alias in either cassandra.yaml, or the header of the 
> (commitlog/hints) file that has been encrypted. Using the alias as literal 
> value will work, but requires some attention when rotating keys.
> Currently each time a key is rotated (i.e. adding a new key to the keystore 
> while preserving the previous version), the alias in cassandra.yaml has to be 
> update as well and the node needs to be restarted. It would be more 
> convenient to use a symbolic reference instead. My suggestion here would be 
> to use ":latest" for referring to the latest version. In this case 
> Cassandra always picks the key with the highest version in 
> ":".
> The non-trivial part of this suggestion is how the "latest" key is referenced 
> in the file header. If we use "latest", e.g. for the commit log header, and 
> the key gets rotated, we'd now try do decrypt the file with the new key, 
> instead of the key it has been created with. Therefor we'd have to introduce 
> an extra step that will resolve the canonical version for "latest" and refer 
> to that one during any encrypt operation. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14107) Introduce simple key alias versioning scheme for TDE

[Stefan Podkowinski (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16320193#comment-16320193
 ] 

Stefan Podkowinski commented on CASSANDRA-14107:


Yes. The basic idea is that you just need to add a new key with an incremented 
version number (could also be a timestamp I guess) and Cassandra will pick it 
up automatically and uses it from that point. Any data written to disk will 
still refer to the original key alias (e.g. mykey:1) that has been used for 
encryption. 

I've also added a periodical job that will every 10 mins reload the local 
JKCKeyStore and check for new keys, so you wouldn't have to restart or touch 
the config after adding a new key. The Vault implementation supports this as 
well of course.

> Introduce simple key alias versioning scheme for TDE
> 
>
> Key: CASSANDRA-14107
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14107
> Project: Cassandra
>  Issue Type: New Feature
>Reporter: Stefan Podkowinski
>Assignee: Stefan Podkowinski
>Priority: Minor
>  Labels: encryption
> Fix For: 4.x
>
>
> Handling of encryption keys as introduced in CASSANDRA-9945 takes place by 
> referencing a key alias in either cassandra.yaml, or the header of the 
> (commitlog/hints) file that has been encrypted. Using the alias as literal 
> value will work, but requires some attention when rotating keys.
> Currently each time a key is rotated (i.e. adding a new key to the keystore 
> while preserving the previous version), the alias in cassandra.yaml has to be 
> update as well and the node needs to be restarted. It would be more 
> convenient to use a symbolic reference instead. My suggestion here would be 
> to use ":latest" for referring to the latest version. In this case 
> Cassandra always picks the key with the highest version in 
> ":".
> The non-trivial part of this suggestion is how the "latest" key is referenced 
> in the file header. If we use "latest", e.g. for the commit log header, and 
> the key gets rotated, we'd now try do decrypt the file with the new key, 
> instead of the key it has been created with. Therefor we'd have to introduce 
> an extra step that will resolve the canonical version for "latest" and refer 
> to that one during any encrypt operation. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14107) Introduce simple key alias versioning scheme for TDE

[Jason Brown (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16320174#comment-16320174
 ] 

Jason Brown commented on CASSANDRA-14107:
-

Does this mean that users would need to add keys to a keystore (or whatever) in 
a named manner like this; day intervals are just for example:

- mykey:1 (on day 1)
- mykey:2 (on day 100)
- mykey:3 (on day 300)
- mykey:4 (on day 500)
- ... and so on

If so, then we would need to load all available keys at startup, sort by index, 
and choose the highest indexed value for the current encryption key. We then 
store the proper name of the key ("mykey:3", for example) in the file to know 
which specific key to use for decryption. Is this a reasonable understanding of 
what you are proposing?

> Introduce simple key alias versioning scheme for TDE
> 
>
> Key: CASSANDRA-14107
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14107
> Project: Cassandra
>  Issue Type: New Feature
>Reporter: Stefan Podkowinski
>Assignee: Stefan Podkowinski
>Priority: Minor
>  Labels: encryption
> Fix For: 4.x
>
>
> Handling of encryption keys as introduced in CASSANDRA-9945 takes place by 
> referencing a key alias in either cassandra.yaml, or the header of the 
> (commitlog/hints) file that has been encrypted. Using the alias as literal 
> value will work, but requires some attention when rotating keys.
> Currently each time a key is rotated (i.e. adding a new key to the keystore 
> while preserving the previous version), the alias in cassandra.yaml has to be 
> update as well and the node needs to be restarted. It would be more 
> convenient to use a symbolic reference instead. My suggestion here would be 
> to use ":latest" for referring to the latest version. In this case 
> Cassandra always picks the key with the highest version in 
> ":".
> The non-trivial part of this suggestion is how the "latest" key is referenced 
> in the file header. If we use "latest", e.g. for the commit log header, and 
> the key gets rotated, we'd now try do decrypt the file with the new key, 
> instead of the key it has been created with. Therefor we'd have to introduce 
> an extra step that will resolve the canonical version for "latest" and refer 
> to that one during any encrypt operation. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Created] (CASSANDRA-14159) Fix flaky test_drop_with_stopped_build

[Paulo Motta (JIRA)]

Paulo Motta created CASSANDRA-14159:
---

 Summary: Fix flaky test_drop_with_stopped_build
 Key: CASSANDRA-14159
 URL: https://issues.apache.org/jira/browse/CASSANDRA-14159
 Project: Cassandra
  Issue Type: Test
Reporter: Paulo Motta
Assignee: Paulo Motta


Test is failing with
{noformat}
  File "/usr/lib/python2.7/unittest/case.py", line 329, in run
testMethod()
  File "/home/automaton/cassandra-dtest/tools/decorators.py", line 48, in 
wrapped
f(obj)
  File "/home/automaton/cassandra-dtest/materialized_views_test.py", line 1129, 
in test_drop_with_stopped_build
self._wait_for_view('ks', 't_by_v')
  File "/home/automaton/cassandra-dtest/materialized_views_test.py", line 130, 
in _wait_for_view
{noformat}

test_drop_while_building is also failing with a similar stack trace:
{noformat}
  File "/usr/lib/python2.7/unittest/case.py", line 329, in run
testMethod()
  File "/home/automaton/cassandra-dtest/tools/decorators.py", line 48, in 
wrapped
f(obj)
  File "/home/automaton/cassandra-dtest/materialized_views_test.py", line 1073, 
in test_drop_while_building
self._wait_for_view('ks', 't_by_v')
  File "/home/automaton/cassandra-dtest/materialized_views_test.py", line 130, 
in _wait_for_view
raise RuntimeError("View {}.{} build not finished after 50 
seconds.".format(ks, view))
{noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14069) Node stopped serving write requests when a table has a lot of sstables

[Sergey Lapukhov (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16320093#comment-16320093
 ] 

Sergey Lapukhov commented on CASSANDRA-14069:
-

I profiled Cassandra (using [Honest 
profiler|https://github.com/jvm-profiling-tools/honest-profiler]). Please see 
[^beforePatch.svg] flame graph.

Definitely, 
[CASSANDRA-12763|https://issues.apache.org/jira/browse/CASSANDRA-12763] has the 
main influence on performance, with java.io.UnixFileSystem.list taking about 
40% of time on cluster flooded with SSTables. 

However, 
org.apache.cassandra.db.compaction.LeveledManifest.overlapping(LeveledManifest.java:549)
 is having second largest impact on performance, taking about 20% of time.

If search overlapping sstables using interval tree (see CAS-14069.patch) 
instead of just O(n^2) iteration this 20% hotspot goes away, (see 
[^afterPatch.svg]).

> Node stopped serving write requests when a table has a lot of sstables
> --
>
> Key: CASSANDRA-14069
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14069
> Project: Cassandra
>  Issue Type: Bug
>  Components: Compaction
>Reporter: Sergey Lapukhov
> Attachments: CAS-14069.patch, afterPatch.svg, beforePatch.svg, 
> create.cql, stack.txt
>
>
> Cluster was flooded with SSTables. A table had ~2 sstables. Write 
> requests started failing. 
> Steps to reproduce:
> * Create cluster with 3 nodes
> * Specify 
>{noformat}
> memtable_heap_space_in_mb: 10
> {noformat}
>  in cassandra.yaml
> * Create table standard1 in keyspace1 (for the cassandra-stress tool) with 
> the script [^create.cql]. Please note
> {noformat} compaction = {'class': 'SizeTieredCompactionStrategy', 'enabled': 
> 'false'} {noformat}
>   i.e. compaction will be turned off for now.
> *  Populate node with data:
>   {noformat} cassandra-stress write n=10 -node 127.0.0.1 {noformat}
> * After node was populated, put both read and write pressure on it:
>   {noformat}  cassandra-stress read n=10 -node 127.0.0.1 
>   cassandra-stress write n=10 -node 127.0.0.1 {noformat}
> * While still under pressure, enable LeveledCompactionStrategy
> {code:java}  echo "ALTER TABLE keyspace1.standard1 WITH compaction = { 
> 'class' : 'LeveledCompactionStrategy', 'sstable_size_in_mb' : 1 }; DESC 
> keyspace1.standard1; exit" | bin/cqlsh; {code}
> *Results:*
> Write requests failing.
> 'bin/nodetool cfstats' and 'bin/nodetool compactionstats' commands hanging, 
> if issued from the node running cassandra-stress tool.
> If issued from another node:
> {noformat}
>  $ bin/nodetool cfstats
> ...
> Table: standard1
> SSTable count: 22637
> SSTables in each level: [22651/4, 0, 0, 0, 0, 0, 0, 0, 0] 
> ...
> {noformat}
> {noformat}
> $ bin/nodetool compactionstats
> pending tasks: 12656
>  id   compaction typekeyspace   
> table   completed   totalunit   progress
>935bbc00-d03b-11e7-a47d-2b44293495b8Compaction   keyspace1   
> standard15955601459557860   bytes100.00%
>a29ee660-d03b-11e7-a47d-2b44293495b8Compaction   keyspace1   
> standard180432114   742151655   bytes 10.84%
>9766e400-d03b-11e7-a47d-2b44293495b8Compaction   keyspace1   
> standard15889160458893215   bytes100.00%
>9cdc9880-d03b-11e7-a47d-2b44293495b8Compaction   keyspace1   
> standard12028944920290800   bytes 99.99%
>90f98910-d03b-11e7-a47d-2b44293495b8Compaction   keyspace1   
> standard15968982459695545   bytes 99.99%
>986ede20-d03b-11e7-a47d-2b44293495b8Compaction   keyspace1   
> standard14059859440598820   bytes100.00%
>9cd322a0-d03b-11e7-a47d-2b44293495b8Compaction   keyspace1   
> standard1607567396070   bytes 99.98% 
> {noformat}
> Special note about 'bin/nodetool compactionstats' - picture above is quite 
> typical for this issue. I.e. compaction tasks manage to make it through, but 
> hinder near the full completion (around 99.9 %).
> Maybe the root of the problem is in this thread (see [^stack.txt]):
> {noformat}
> "CompactionExecutor:1748" #4649 daemon prio=1 os_prio=4 
> tid=0x7f35a0096100 nid=0x65f6 runnable [0x7f3228bce000]
>java.lang.Thread.State: RUNNABLE
>   at org.apache.cassandra.dht.AbstractBounds.(AbstractBounds.java:53)
>   at org.apache.cassandra.dht.Bounds.(Bounds.java:42)
>   at 
> org.apache.cassandra.db.compaction.LeveledManifest.overlapping(LeveledManifest.java:562)
>   at 
> org.apache.cassandra.db.compaction.LeveledManifest.overlapping(LeveledManifest.java:549)
>   at 
> org.apache.cassandra.db.compaction.LeveledManifest.getCandidatesFor(LeveledManifest.java:624)
>   at 
> 

[jira] [Issue Comment Deleted] (CASSANDRA-14069) Node stopped serving write requests when a table has a lot of sstables

[Sergey Lapukhov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Lapukhov updated CASSANDRA-14069:

Comment: was deleted

(was: Performance flamegraphs before and after applying CAS-14069 patch)

> Node stopped serving write requests when a table has a lot of sstables
> --
>
> Key: CASSANDRA-14069
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14069
> Project: Cassandra
>  Issue Type: Bug
>  Components: Compaction
>Reporter: Sergey Lapukhov
> Attachments: CAS-14069.patch, afterPatch.svg, beforePatch.svg, 
> create.cql, stack.txt
>
>
> Cluster was flooded with SSTables. A table had ~2 sstables. Write 
> requests started failing. 
> Steps to reproduce:
> * Create cluster with 3 nodes
> * Specify 
>{noformat}
> memtable_heap_space_in_mb: 10
> {noformat}
>  in cassandra.yaml
> * Create table standard1 in keyspace1 (for the cassandra-stress tool) with 
> the script [^create.cql]. Please note
> {noformat} compaction = {'class': 'SizeTieredCompactionStrategy', 'enabled': 
> 'false'} {noformat}
>   i.e. compaction will be turned off for now.
> *  Populate node with data:
>   {noformat} cassandra-stress write n=10 -node 127.0.0.1 {noformat}
> * After node was populated, put both read and write pressure on it:
>   {noformat}  cassandra-stress read n=10 -node 127.0.0.1 
>   cassandra-stress write n=10 -node 127.0.0.1 {noformat}
> * While still under pressure, enable LeveledCompactionStrategy
> {code:java}  echo "ALTER TABLE keyspace1.standard1 WITH compaction = { 
> 'class' : 'LeveledCompactionStrategy', 'sstable_size_in_mb' : 1 }; DESC 
> keyspace1.standard1; exit" | bin/cqlsh; {code}
> *Results:*
> Write requests failing.
> 'bin/nodetool cfstats' and 'bin/nodetool compactionstats' commands hanging, 
> if issued from the node running cassandra-stress tool.
> If issued from another node:
> {noformat}
>  $ bin/nodetool cfstats
> ...
> Table: standard1
> SSTable count: 22637
> SSTables in each level: [22651/4, 0, 0, 0, 0, 0, 0, 0, 0] 
> ...
> {noformat}
> {noformat}
> $ bin/nodetool compactionstats
> pending tasks: 12656
>  id   compaction typekeyspace   
> table   completed   totalunit   progress
>935bbc00-d03b-11e7-a47d-2b44293495b8Compaction   keyspace1   
> standard15955601459557860   bytes100.00%
>a29ee660-d03b-11e7-a47d-2b44293495b8Compaction   keyspace1   
> standard180432114   742151655   bytes 10.84%
>9766e400-d03b-11e7-a47d-2b44293495b8Compaction   keyspace1   
> standard15889160458893215   bytes100.00%
>9cdc9880-d03b-11e7-a47d-2b44293495b8Compaction   keyspace1   
> standard12028944920290800   bytes 99.99%
>90f98910-d03b-11e7-a47d-2b44293495b8Compaction   keyspace1   
> standard15968982459695545   bytes 99.99%
>986ede20-d03b-11e7-a47d-2b44293495b8Compaction   keyspace1   
> standard14059859440598820   bytes100.00%
>9cd322a0-d03b-11e7-a47d-2b44293495b8Compaction   keyspace1   
> standard1607567396070   bytes 99.98% 
> {noformat}
> Special note about 'bin/nodetool compactionstats' - picture above is quite 
> typical for this issue. I.e. compaction tasks manage to make it through, but 
> hinder near the full completion (around 99.9 %).
> Maybe the root of the problem is in this thread (see [^stack.txt]):
> {noformat}
> "CompactionExecutor:1748" #4649 daemon prio=1 os_prio=4 
> tid=0x7f35a0096100 nid=0x65f6 runnable [0x7f3228bce000]
>java.lang.Thread.State: RUNNABLE
>   at org.apache.cassandra.dht.AbstractBounds.(AbstractBounds.java:53)
>   at org.apache.cassandra.dht.Bounds.(Bounds.java:42)
>   at 
> org.apache.cassandra.db.compaction.LeveledManifest.overlapping(LeveledManifest.java:562)
>   at 
> org.apache.cassandra.db.compaction.LeveledManifest.overlapping(LeveledManifest.java:549)
>   at 
> org.apache.cassandra.db.compaction.LeveledManifest.getCandidatesFor(LeveledManifest.java:624)
>   at 
> org.apache.cassandra.db.compaction.LeveledManifest.getCompactionCandidates(LeveledManifest.java:378)
>   - locked <0x00070d4c3bc8> (a 
> org.apache.cassandra.db.compaction.LeveledManifest)
>   at 
> org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getNextBackgroundTask(LeveledCompactionStrategy.java:105)
>   - locked <0x00070d6cb2c8> (a 
> org.apache.cassandra.db.compaction.LeveledCompactionStrategy)
>   at 
> org.apache.cassandra.db.compaction.CompactionStrategyManager.getNextBackgroundTask(CompactionStrategyManager.java:102)
>   - locked <0x0006467268b8> (a 
> org.apache.cassandra.db.compaction.CompactionStrategyManager)
>   at 
> 

[jira] [Updated] (CASSANDRA-14069) Node stopped serving write requests when a table has a lot of sstables

[Sergey Lapukhov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Lapukhov updated CASSANDRA-14069:

Attachment: beforePatch.svg
afterPatch.svg

Performance flamegraphs before and after applying CAS-14069 patch

> Node stopped serving write requests when a table has a lot of sstables
> --
>
> Key: CASSANDRA-14069
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14069
> Project: Cassandra
>  Issue Type: Bug
>  Components: Compaction
>Reporter: Sergey Lapukhov
> Attachments: CAS-14069.patch, afterPatch.svg, beforePatch.svg, 
> create.cql, stack.txt
>
>
> Cluster was flooded with SSTables. A table had ~2 sstables. Write 
> requests started failing. 
> Steps to reproduce:
> * Create cluster with 3 nodes
> * Specify 
>{noformat}
> memtable_heap_space_in_mb: 10
> {noformat}
>  in cassandra.yaml
> * Create table standard1 in keyspace1 (for the cassandra-stress tool) with 
> the script [^create.cql]. Please note
> {noformat} compaction = {'class': 'SizeTieredCompactionStrategy', 'enabled': 
> 'false'} {noformat}
>   i.e. compaction will be turned off for now.
> *  Populate node with data:
>   {noformat} cassandra-stress write n=10 -node 127.0.0.1 {noformat}
> * After node was populated, put both read and write pressure on it:
>   {noformat}  cassandra-stress read n=10 -node 127.0.0.1 
>   cassandra-stress write n=10 -node 127.0.0.1 {noformat}
> * While still under pressure, enable LeveledCompactionStrategy
> {code:java}  echo "ALTER TABLE keyspace1.standard1 WITH compaction = { 
> 'class' : 'LeveledCompactionStrategy', 'sstable_size_in_mb' : 1 }; DESC 
> keyspace1.standard1; exit" | bin/cqlsh; {code}
> *Results:*
> Write requests failing.
> 'bin/nodetool cfstats' and 'bin/nodetool compactionstats' commands hanging, 
> if issued from the node running cassandra-stress tool.
> If issued from another node:
> {noformat}
>  $ bin/nodetool cfstats
> ...
> Table: standard1
> SSTable count: 22637
> SSTables in each level: [22651/4, 0, 0, 0, 0, 0, 0, 0, 0] 
> ...
> {noformat}
> {noformat}
> $ bin/nodetool compactionstats
> pending tasks: 12656
>  id   compaction typekeyspace   
> table   completed   totalunit   progress
>935bbc00-d03b-11e7-a47d-2b44293495b8Compaction   keyspace1   
> standard15955601459557860   bytes100.00%
>a29ee660-d03b-11e7-a47d-2b44293495b8Compaction   keyspace1   
> standard180432114   742151655   bytes 10.84%
>9766e400-d03b-11e7-a47d-2b44293495b8Compaction   keyspace1   
> standard15889160458893215   bytes100.00%
>9cdc9880-d03b-11e7-a47d-2b44293495b8Compaction   keyspace1   
> standard12028944920290800   bytes 99.99%
>90f98910-d03b-11e7-a47d-2b44293495b8Compaction   keyspace1   
> standard15968982459695545   bytes 99.99%
>986ede20-d03b-11e7-a47d-2b44293495b8Compaction   keyspace1   
> standard14059859440598820   bytes100.00%
>9cd322a0-d03b-11e7-a47d-2b44293495b8Compaction   keyspace1   
> standard1607567396070   bytes 99.98% 
> {noformat}
> Special note about 'bin/nodetool compactionstats' - picture above is quite 
> typical for this issue. I.e. compaction tasks manage to make it through, but 
> hinder near the full completion (around 99.9 %).
> Maybe the root of the problem is in this thread (see [^stack.txt]):
> {noformat}
> "CompactionExecutor:1748" #4649 daemon prio=1 os_prio=4 
> tid=0x7f35a0096100 nid=0x65f6 runnable [0x7f3228bce000]
>java.lang.Thread.State: RUNNABLE
>   at org.apache.cassandra.dht.AbstractBounds.(AbstractBounds.java:53)
>   at org.apache.cassandra.dht.Bounds.(Bounds.java:42)
>   at 
> org.apache.cassandra.db.compaction.LeveledManifest.overlapping(LeveledManifest.java:562)
>   at 
> org.apache.cassandra.db.compaction.LeveledManifest.overlapping(LeveledManifest.java:549)
>   at 
> org.apache.cassandra.db.compaction.LeveledManifest.getCandidatesFor(LeveledManifest.java:624)
>   at 
> org.apache.cassandra.db.compaction.LeveledManifest.getCompactionCandidates(LeveledManifest.java:378)
>   - locked <0x00070d4c3bc8> (a 
> org.apache.cassandra.db.compaction.LeveledManifest)
>   at 
> org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getNextBackgroundTask(LeveledCompactionStrategy.java:105)
>   - locked <0x00070d6cb2c8> (a 
> org.apache.cassandra.db.compaction.LeveledCompactionStrategy)
>   at 
> org.apache.cassandra.db.compaction.CompactionStrategyManager.getNextBackgroundTask(CompactionStrategyManager.java:102)
>   - locked <0x0006467268b8> (a 
> 

[jira] [Issue Comment Deleted] (CASSANDRA-14069) Node stopped serving write requests when a table has a lot of sstables

[Sergey Lapukhov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Lapukhov updated CASSANDRA-14069:

Comment: was deleted

(was: Use interval tree to cool down O(n^2) overlapping sstables search)

> Node stopped serving write requests when a table has a lot of sstables
> --
>
> Key: CASSANDRA-14069
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14069
> Project: Cassandra
>  Issue Type: Bug
>  Components: Compaction
>Reporter: Sergey Lapukhov
> Attachments: CAS-14069.patch, afterPatch.svg, beforePatch.svg, 
> create.cql, stack.txt
>
>
> Cluster was flooded with SSTables. A table had ~2 sstables. Write 
> requests started failing. 
> Steps to reproduce:
> * Create cluster with 3 nodes
> * Specify 
>{noformat}
> memtable_heap_space_in_mb: 10
> {noformat}
>  in cassandra.yaml
> * Create table standard1 in keyspace1 (for the cassandra-stress tool) with 
> the script [^create.cql]. Please note
> {noformat} compaction = {'class': 'SizeTieredCompactionStrategy', 'enabled': 
> 'false'} {noformat}
>   i.e. compaction will be turned off for now.
> *  Populate node with data:
>   {noformat} cassandra-stress write n=10 -node 127.0.0.1 {noformat}
> * After node was populated, put both read and write pressure on it:
>   {noformat}  cassandra-stress read n=10 -node 127.0.0.1 
>   cassandra-stress write n=10 -node 127.0.0.1 {noformat}
> * While still under pressure, enable LeveledCompactionStrategy
> {code:java}  echo "ALTER TABLE keyspace1.standard1 WITH compaction = { 
> 'class' : 'LeveledCompactionStrategy', 'sstable_size_in_mb' : 1 }; DESC 
> keyspace1.standard1; exit" | bin/cqlsh; {code}
> *Results:*
> Write requests failing.
> 'bin/nodetool cfstats' and 'bin/nodetool compactionstats' commands hanging, 
> if issued from the node running cassandra-stress tool.
> If issued from another node:
> {noformat}
>  $ bin/nodetool cfstats
> ...
> Table: standard1
> SSTable count: 22637
> SSTables in each level: [22651/4, 0, 0, 0, 0, 0, 0, 0, 0] 
> ...
> {noformat}
> {noformat}
> $ bin/nodetool compactionstats
> pending tasks: 12656
>  id   compaction typekeyspace   
> table   completed   totalunit   progress
>935bbc00-d03b-11e7-a47d-2b44293495b8Compaction   keyspace1   
> standard15955601459557860   bytes100.00%
>a29ee660-d03b-11e7-a47d-2b44293495b8Compaction   keyspace1   
> standard180432114   742151655   bytes 10.84%
>9766e400-d03b-11e7-a47d-2b44293495b8Compaction   keyspace1   
> standard15889160458893215   bytes100.00%
>9cdc9880-d03b-11e7-a47d-2b44293495b8Compaction   keyspace1   
> standard12028944920290800   bytes 99.99%
>90f98910-d03b-11e7-a47d-2b44293495b8Compaction   keyspace1   
> standard15968982459695545   bytes 99.99%
>986ede20-d03b-11e7-a47d-2b44293495b8Compaction   keyspace1   
> standard14059859440598820   bytes100.00%
>9cd322a0-d03b-11e7-a47d-2b44293495b8Compaction   keyspace1   
> standard1607567396070   bytes 99.98% 
> {noformat}
> Special note about 'bin/nodetool compactionstats' - picture above is quite 
> typical for this issue. I.e. compaction tasks manage to make it through, but 
> hinder near the full completion (around 99.9 %).
> Maybe the root of the problem is in this thread (see [^stack.txt]):
> {noformat}
> "CompactionExecutor:1748" #4649 daemon prio=1 os_prio=4 
> tid=0x7f35a0096100 nid=0x65f6 runnable [0x7f3228bce000]
>java.lang.Thread.State: RUNNABLE
>   at org.apache.cassandra.dht.AbstractBounds.(AbstractBounds.java:53)
>   at org.apache.cassandra.dht.Bounds.(Bounds.java:42)
>   at 
> org.apache.cassandra.db.compaction.LeveledManifest.overlapping(LeveledManifest.java:562)
>   at 
> org.apache.cassandra.db.compaction.LeveledManifest.overlapping(LeveledManifest.java:549)
>   at 
> org.apache.cassandra.db.compaction.LeveledManifest.getCandidatesFor(LeveledManifest.java:624)
>   at 
> org.apache.cassandra.db.compaction.LeveledManifest.getCompactionCandidates(LeveledManifest.java:378)
>   - locked <0x00070d4c3bc8> (a 
> org.apache.cassandra.db.compaction.LeveledManifest)
>   at 
> org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getNextBackgroundTask(LeveledCompactionStrategy.java:105)
>   - locked <0x00070d6cb2c8> (a 
> org.apache.cassandra.db.compaction.LeveledCompactionStrategy)
>   at 
> org.apache.cassandra.db.compaction.CompactionStrategyManager.getNextBackgroundTask(CompactionStrategyManager.java:102)
>   - locked <0x0006467268b8> (a 
> org.apache.cassandra.db.compaction.CompactionStrategyManager)
>   at 
> 

[jira] [Updated] (CASSANDRA-14069) Node stopped serving write requests when a table has a lot of sstables

[Sergey Lapukhov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Lapukhov updated CASSANDRA-14069:

Attachment: CAS-14069.patch

Use interval tree to cool down O(n^2) overlapping sstables search

> Node stopped serving write requests when a table has a lot of sstables
> --
>
> Key: CASSANDRA-14069
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14069
> Project: Cassandra
>  Issue Type: Bug
>  Components: Compaction
>Reporter: Sergey Lapukhov
> Attachments: CAS-14069.patch, create.cql, stack.txt
>
>
> Cluster was flooded with SSTables. A table had ~2 sstables. Write 
> requests started failing. 
> Steps to reproduce:
> * Create cluster with 3 nodes
> * Specify 
>{noformat}
> memtable_heap_space_in_mb: 10
> {noformat}
>  in cassandra.yaml
> * Create table standard1 in keyspace1 (for the cassandra-stress tool) with 
> the script [^create.cql]. Please note
> {noformat} compaction = {'class': 'SizeTieredCompactionStrategy', 'enabled': 
> 'false'} {noformat}
>   i.e. compaction will be turned off for now.
> *  Populate node with data:
>   {noformat} cassandra-stress write n=10 -node 127.0.0.1 {noformat}
> * After node was populated, put both read and write pressure on it:
>   {noformat}  cassandra-stress read n=10 -node 127.0.0.1 
>   cassandra-stress write n=10 -node 127.0.0.1 {noformat}
> * While still under pressure, enable LeveledCompactionStrategy
> {code:java}  echo "ALTER TABLE keyspace1.standard1 WITH compaction = { 
> 'class' : 'LeveledCompactionStrategy', 'sstable_size_in_mb' : 1 }; DESC 
> keyspace1.standard1; exit" | bin/cqlsh; {code}
> *Results:*
> Write requests failing.
> 'bin/nodetool cfstats' and 'bin/nodetool compactionstats' commands hanging, 
> if issued from the node running cassandra-stress tool.
> If issued from another node:
> {noformat}
>  $ bin/nodetool cfstats
> ...
> Table: standard1
> SSTable count: 22637
> SSTables in each level: [22651/4, 0, 0, 0, 0, 0, 0, 0, 0] 
> ...
> {noformat}
> {noformat}
> $ bin/nodetool compactionstats
> pending tasks: 12656
>  id   compaction typekeyspace   
> table   completed   totalunit   progress
>935bbc00-d03b-11e7-a47d-2b44293495b8Compaction   keyspace1   
> standard15955601459557860   bytes100.00%
>a29ee660-d03b-11e7-a47d-2b44293495b8Compaction   keyspace1   
> standard180432114   742151655   bytes 10.84%
>9766e400-d03b-11e7-a47d-2b44293495b8Compaction   keyspace1   
> standard15889160458893215   bytes100.00%
>9cdc9880-d03b-11e7-a47d-2b44293495b8Compaction   keyspace1   
> standard12028944920290800   bytes 99.99%
>90f98910-d03b-11e7-a47d-2b44293495b8Compaction   keyspace1   
> standard15968982459695545   bytes 99.99%
>986ede20-d03b-11e7-a47d-2b44293495b8Compaction   keyspace1   
> standard14059859440598820   bytes100.00%
>9cd322a0-d03b-11e7-a47d-2b44293495b8Compaction   keyspace1   
> standard1607567396070   bytes 99.98% 
> {noformat}
> Special note about 'bin/nodetool compactionstats' - picture above is quite 
> typical for this issue. I.e. compaction tasks manage to make it through, but 
> hinder near the full completion (around 99.9 %).
> Maybe the root of the problem is in this thread (see [^stack.txt]):
> {noformat}
> "CompactionExecutor:1748" #4649 daemon prio=1 os_prio=4 
> tid=0x7f35a0096100 nid=0x65f6 runnable [0x7f3228bce000]
>java.lang.Thread.State: RUNNABLE
>   at org.apache.cassandra.dht.AbstractBounds.(AbstractBounds.java:53)
>   at org.apache.cassandra.dht.Bounds.(Bounds.java:42)
>   at 
> org.apache.cassandra.db.compaction.LeveledManifest.overlapping(LeveledManifest.java:562)
>   at 
> org.apache.cassandra.db.compaction.LeveledManifest.overlapping(LeveledManifest.java:549)
>   at 
> org.apache.cassandra.db.compaction.LeveledManifest.getCandidatesFor(LeveledManifest.java:624)
>   at 
> org.apache.cassandra.db.compaction.LeveledManifest.getCompactionCandidates(LeveledManifest.java:378)
>   - locked <0x00070d4c3bc8> (a 
> org.apache.cassandra.db.compaction.LeveledManifest)
>   at 
> org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getNextBackgroundTask(LeveledCompactionStrategy.java:105)
>   - locked <0x00070d6cb2c8> (a 
> org.apache.cassandra.db.compaction.LeveledCompactionStrategy)
>   at 
> org.apache.cassandra.db.compaction.CompactionStrategyManager.getNextBackgroundTask(CompactionStrategyManager.java:102)
>   - locked <0x0006467268b8> (a 
> org.apache.cassandra.db.compaction.CompactionStrategyManager)
>   at 
> 

[jira] [Commented] (CASSANDRA-10968) When taking snapshot, manifest.json contains incorrect or no files when column family has secondary indexes

[Fred A (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16320007#comment-16320007
 ] 

Fred A commented on CASSANDRA-10968:


When and to which versions will this fix be comitted?

> When taking snapshot, manifest.json contains incorrect or no files when 
> column family has secondary indexes
> ---
>
> Key: CASSANDRA-10968
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10968
> Project: Cassandra
>  Issue Type: Bug
>  Components: Secondary Indexes
>Reporter: Fred A
>Assignee: Aleksandr Sorokoumov
>  Labels: lhf
> Fix For: 2.2.x, 3.0.x, 3.11.x
>
>
> xNoticed indeterminate behaviour when taking snapshot on column families that 
> has secondary indexes setup. The created manifest.json created when doing 
> snapshot, sometimes contains no file names at all and sometimes some file 
> names. 
> I don't know if this post is related but that was the only thing I could find:
> http://www.mail-archive.com/user%40cassandra.apache.org/msg42019.html



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14102) Vault support for transparent data encryption

[Romain Hardouin (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16319903#comment-16319903
 ] 

Romain Hardouin commented on CASSANDRA-14102:
-

It's a nice feature. Out of curiosity, did you make few benchmarks to measure 
impacts on performances?

> Vault support for transparent data encryption
> -
>
> Key: CASSANDRA-14102
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14102
> Project: Cassandra
>  Issue Type: New Feature
>Reporter: Stefan Podkowinski
>Assignee: Stefan Podkowinski
>  Labels: encryption
> Fix For: 4.x
>
>
> Transparent data encryption provided by CASSANDRA-9945 can currently be used 
> for commitlog and hints. The default {{KeyProvider}} implementation that we 
> ship allows to use a local keystore for storing and retrieving keys. Thanks 
> to the pluggable handling of the {{KeyStore}} provider and basic Vault 
> related classes introduced in CASSANDRA-13971, a Vault based implementation 
> can be provided as {{KeyProvider}} as well. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org