[jira] [Issue Comment Deleted] (CASSANDRA-12888) Incremental repairs broken for MVs and CDC

[Jeff Jirsa (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-12888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeff Jirsa updated CASSANDRA-12888:
---
Comment: was deleted

(was: {quote}
What effect does the repairedAt flag have for future repairs except that a 
non-zero value means, that a table has been repaired at some time?
{quote}

It's used (heavily) for incremental repairs. CASSANDRA-9143 may be a starting 
point reading there (potentially).)

> Incremental repairs broken for MVs and CDC
> --
>
> Key: CASSANDRA-12888
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12888
> Project: Cassandra
>  Issue Type: Bug
>  Components: Streaming and Messaging
>Reporter: Stefan Podkowinski
>Assignee: Benjamin Roth
>Priority: Critical
> Fix For: 3.0.x, 3.11.x
>
>
> SSTables streamed during the repair process will first be written locally and 
> afterwards either simply added to the pool of existing sstables or, in case 
> of existing MVs or active CDC, replayed on mutation basis:
> As described in {{StreamReceiveTask.OnCompletionRunnable}}:
> {quote}
> We have a special path for views and for CDC.
> For views, since the view requires cleaning up any pre-existing state, we 
> must put all partitions through the same write path as normal mutations. This 
> also ensures any 2is are also updated.
> For CDC-enabled tables, we want to ensure that the mutations are run through 
> the CommitLog so they can be archived by the CDC process on discard.
> {quote}
> Using the regular write path turns out to be an issue for incremental 
> repairs, as we loose the {{repaired_at}} state in the process. Eventually the 
> streamed rows will end up in the unrepaired set, in contrast to the rows on 
> the sender site moved to the repaired set. The next repair run will stream 
> the same data back again, causing rows to bounce on and on between nodes on 
> each repair.
> See linked dtest on steps to reproduce. An example for reproducing this 
> manually using ccm can be found 
> [here|https://gist.github.com/spodkowinski/2d8e0408516609c7ae701f2bf1e515e8]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (CASSANDRA-12888) Incremental repairs broken for MVs and CDC

[Jeff Jirsa (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15895457#comment-15895457
 ] 

Jeff Jirsa commented on CASSANDRA-12888:


{quote}
What effect does the repairedAt flag have for future repairs except that a 
non-zero value means, that a table has been repaired at some time?
{quote}

It's used (heavily) for incremental repairs. CASSANDRA-9143 may be a starting 
point reading there (potentially).

> Incremental repairs broken for MVs and CDC
> --
>
> Key: CASSANDRA-12888
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12888
> Project: Cassandra
>  Issue Type: Bug
>  Components: Streaming and Messaging
>Reporter: Stefan Podkowinski
>Assignee: Benjamin Roth
>Priority: Critical
> Fix For: 3.0.x, 3.11.x
>
>
> SSTables streamed during the repair process will first be written locally and 
> afterwards either simply added to the pool of existing sstables or, in case 
> of existing MVs or active CDC, replayed on mutation basis:
> As described in {{StreamReceiveTask.OnCompletionRunnable}}:
> {quote}
> We have a special path for views and for CDC.
> For views, since the view requires cleaning up any pre-existing state, we 
> must put all partitions through the same write path as normal mutations. This 
> also ensures any 2is are also updated.
> For CDC-enabled tables, we want to ensure that the mutations are run through 
> the CommitLog so they can be archived by the CDC process on discard.
> {quote}
> Using the regular write path turns out to be an issue for incremental 
> repairs, as we loose the {{repaired_at}} state in the process. Eventually the 
> streamed rows will end up in the unrepaired set, in contrast to the rows on 
> the sender site moved to the repaired set. The next repair run will stream 
> the same data back again, causing rows to bounce on and on between nodes on 
> each repair.
> See linked dtest on steps to reproduce. An example for reproducing this 
> manually using ccm can be found 
> [here|https://gist.github.com/spodkowinski/2d8e0408516609c7ae701f2bf1e515e8]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (CASSANDRA-13294) Possible data loss on upgrade 2.1 - 3.0

[Wei Deng (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15895295#comment-15895295
 ] 

Wei Deng commented on CASSANDRA-13294:
--

If this problem is caused by the file name change, would we need to port to 2.2 
as well, because the file name change was introduced in 2.2?

> Possible data loss on upgrade 2.1 - 3.0
> ---
>
> Key: CASSANDRA-13294
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13294
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Marcus Eriksson
>Assignee: Marcus Eriksson
>Priority: Blocker
> Fix For: 3.0.x
>
>
> After finishing a compaction we delete the compacted away files. This is done 
> [here|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/db/lifecycle/LogFile.java#L328-L337]
>  which uses 
> [this|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/db/lifecycle/LogRecord.java#L265-L271]
>  to get the files - we get all files starting with {{absoluteFilePath}}. 
> Absolute file path is generated 
> [here|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/io/sstable/Descriptor.java#L142-L153].
>  For 3.0 version files the filename looks like this: 
> {{/blabla/keyspace1/standard1-bdb031c0ff7b11e6940fdd0479dd8912/mc-1332-big}} 
> but for 2.1 version files, they look like this: 
> {{/blabla/keyspace1/standard1-bdb031c0ff7b11e6940fdd0479dd8912/keyspace1-standard1-ka-2}}.
> The problem is then that if we were to finish a compaction including the 
> legacy file, we would actually delete all legacy files having a generation 
> starting with '2'



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Comment Edited] (CASSANDRA-12588) Cannot find column durable_wrıtes

[Jeff Jirsa (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15895222#comment-15895222
 ] 

Jeff Jirsa edited comment on CASSANDRA-12588 at 3/3/17 11:17 PM:
-

When creating the system keyspace, it checks 
{{KeyspaceParams.Option.DURABLE_WRITES.toString()}} , - there's probably a wide 
class of bugs in cassandra where we assume everything is {{Locale.ENGLISH}}, 
but we break in awful ways in Turkish (and likely other locales).



was (Author: jjirsa):
When creating the system keyspace, it checks 
{{KeyspaceParams.Option.DURABLE_WRITES.toString()}}, but without a locale - 
there's probably a wide class of bugs in cassandra where we assume everything 
is {{Locale.ENGLISH}}, but we break in awful ways in Turkish (and likely other 
locales).


> Cannot find column durable_wrıtes
> -
>
> Key: CASSANDRA-12588
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12588
> Project: Cassandra
>  Issue Type: Bug
>Reporter: LLc.
>
> help please
> run :
> cassandra -f
> ERROR 17:00:16 Exception encountered during startup
> java.lang.AssertionError: Cannot find column durable_wrıtes



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (CASSANDRA-12588) Cannot find column durable_wrıtes

[Jeff Jirsa (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15895222#comment-15895222
 ] 

Jeff Jirsa commented on CASSANDRA-12588:


When creating the system keyspace, it checks 
{{KeyspaceParams.Option.DURABLE_WRITES.toString()}}, but without a locale - 
there's probably a wide class of bugs in cassandra where we assume everything 
is {{Locale.ENGLISH}}, but we break in awful ways in Turkish (and likely other 
locales).


> Cannot find column durable_wrıtes
> -
>
> Key: CASSANDRA-12588
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12588
> Project: Cassandra
>  Issue Type: Bug
>Reporter: LLc.
>
> help please
> run :
> cassandra -f
> ERROR 17:00:16 Exception encountered during startup
> java.lang.AssertionError: Cannot find column durable_wrıtes



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (CASSANDRA-13297) Cleanup naming around native transport version parsing

[Michael Kjellman (JIRA)]

Michael Kjellman created CASSANDRA-13297:


 Summary: Cleanup naming around native transport version parsing
 Key: CASSANDRA-13297
 URL: https://issues.apache.org/jira/browse/CASSANDRA-13297
 Project: Cassandra
  Issue Type: Bug
Reporter: Michael Kjellman


While [~jasobrown] and myself were chasing down how features and versions are 
parsed in the binary protocol, it took us a while to realize that the following 
check in StartupMessage that creates a {{CassandraVersion}} object is actually 
parsing the *CQL* Version (but again.. with a class named 
{{CassandraVersion}}...)

{code}
if (new CassandraVersion(cqlVersion).compareTo(new CassandraVersion("2.99.0")) 
< 0)
{code}

It then turns out there is already a {{CqlVersion}} class that has some of what 
we want but only appears to be being used by Stress.

This isn't very clear so renaming the classes to make it obvious that we're 
parsing a CQL Language version vs. the Cassandra Server version vs. the Binary 
Native Protocol version will make this situation slightly better for people 
trying to contribute to the codebase.




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (CASSANDRA-12888) Incremental repairs broken for MVs and CDC

[Benjamin Roth (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15895140#comment-15895140
 ] 

Benjamin Roth commented on CASSANDRA-12888:
---

Maybe my former question perished:

What effect does the repairedAt flag have for future repairs except that a 
non-zero value means, that a table has been repaired at some time?
I am happy about any code references.

> Incremental repairs broken for MVs and CDC
> --
>
> Key: CASSANDRA-12888
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12888
> Project: Cassandra
>  Issue Type: Bug
>  Components: Streaming and Messaging
>Reporter: Stefan Podkowinski
>Assignee: Benjamin Roth
>Priority: Critical
> Fix For: 3.0.x, 3.11.x
>
>
> SSTables streamed during the repair process will first be written locally and 
> afterwards either simply added to the pool of existing sstables or, in case 
> of existing MVs or active CDC, replayed on mutation basis:
> As described in {{StreamReceiveTask.OnCompletionRunnable}}:
> {quote}
> We have a special path for views and for CDC.
> For views, since the view requires cleaning up any pre-existing state, we 
> must put all partitions through the same write path as normal mutations. This 
> also ensures any 2is are also updated.
> For CDC-enabled tables, we want to ensure that the mutations are run through 
> the CommitLog so they can be archived by the CDC process on discard.
> {quote}
> Using the regular write path turns out to be an issue for incremental 
> repairs, as we loose the {{repaired_at}} state in the process. Eventually the 
> streamed rows will end up in the unrepaired set, in contrast to the rows on 
> the sender site moved to the repaired set. The next repair run will stream 
> the same data back again, causing rows to bounce on and on between nodes on 
> each repair.
> See linked dtest on steps to reproduce. An example for reproducing this 
> manually using ccm can be found 
> [here|https://gist.github.com/spodkowinski/2d8e0408516609c7ae701f2bf1e515e8]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (CASSANDRA-13090) Coalescing strategy sleeps too much

[Michael Shuler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Shuler updated CASSANDRA-13090:
---
Fix Version/s: (was: 3.0.11)
   3.0.12

> Coalescing strategy sleeps too much
> ---
>
> Key: CASSANDRA-13090
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13090
> Project: Cassandra
>  Issue Type: Bug
>  Components: Streaming and Messaging
>Reporter: Corentin Chary
>Assignee: Corentin Chary
> Fix For: 2.2.10, 3.0.12, 3.11.0, 4.0
>
> Attachments: 0001-Fix-wait-time-coalescing-CASSANDRA-13090-2.patch, 
> 0001-Fix-wait-time-coalescing-CASSANDRA-13090.patch
>
>
> With the current code maybeSleep is called even if we managed to take 
> maxItems out of the backlog. In this case we should really avoid sleeping 
> because it means that backlog is building up.
> I'll send a patch shortly.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

cassandra-builds git commit: Add cassandra-cqlsh-tests build script

[mshuler]

Repository: cassandra-builds
Updated Branches:
  refs/heads/master 0c4fc4713 -> 08f76054c


Add cassandra-cqlsh-tests build script


Project: http://git-wip-us.apache.org/repos/asf/cassandra-builds/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra-builds/commit/08f76054
Tree: http://git-wip-us.apache.org/repos/asf/cassandra-builds/tree/08f76054
Diff: http://git-wip-us.apache.org/repos/asf/cassandra-builds/diff/08f76054

Branch: refs/heads/master
Commit: 08f76054c5adc4d5bae010e041f5b204b0880fba
Parents: 0c4fc47
Author: Michael Shuler 
Authored: Fri Mar 3 15:59:46 2017 -0600
Committer: Michael Shuler 
Committed: Fri Mar 3 15:59:46 2017 -0600

--
 build-scripts/cassandra-cqlsh-tests.sh | 99 +
 1 file changed, 99 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/cassandra-builds/blob/08f76054/build-scripts/cassandra-cqlsh-tests.sh
--
diff --git a/build-scripts/cassandra-cqlsh-tests.sh 
b/build-scripts/cassandra-cqlsh-tests.sh
new file mode 100755
index 000..8cd867d
--- /dev/null
+++ b/build-scripts/cassandra-cqlsh-tests.sh
@@ -0,0 +1,99 @@
+#!/bin/bash -x
+
+
+#
+# Prep
+#
+
+
+export PYTHONIOENCODING="utf-8"
+export PYTHONUNBUFFERED=true
+export CCM_MAX_HEAP_SIZE="2048M"
+export CCM_HEAP_NEWSIZE="200M"
+export NUM_TOKENS="32"
+export CASSANDRA_DIR=${WORKSPACE}
+
+# Loop to prevent failure due to maven-ant-tasks not downloading a jar..
+for x in $(seq 1 3); do
+ant clean jar
+RETURN="$?"
+if [ "${RETURN}" -eq "0" ]; then
+break
+fi
+done
+# Exit, if we didn't build successfully
+if [ "${RETURN}" -ne "0" ]; then
+echo "Build failed with exit code: ${RETURN}"
+exit ${RETURN}
+fi
+
+
+#
+# Main
+#
+
+
+if [ "$cython" = "yes" ]; then
+virtualenv --python=python2 --no-site-packages venv
+source venv/bin/activate
+pip install "Cython>=0.20,<0.25"
+pip freeze
+cd pylib/; python setup.py build_ext --inplace
+cd ${WORKSPACE}
+fi
+
+ccm create test -n 1
+ccm updateconf "enable_user_defined_functions: true"
+
+version_from_build=$(ccm node1 versionfrombuild)
+export pre_or_post_cdc=$(python -c """from distutils.version import 
LooseVersion
+print \"postcdc\" if LooseVersion(\"${version_from_build}\") >= \"3.8\" else 
\"precdc\"
+""")
+case "${pre_or_post_cdc}" in
+postcdc)
+ccm updateconf "cdc_enabled: true"
+;;
+precdc)
+:
+;;
+*)
+echo "${pre_or_post_cdc}" is an invalid value.
+exit 1
+;;
+esac
+
+ccm start --wait-for-binary-proto
+
+cd pylib/cqlshlib/
+cat << 'EOF' > setup.cfg
+[nosetests]
+verbosity=3
+detailed-errors=1
+with-xunit=1
+EOF
+
+nosetests
+
+ccm remove
+mv nosetests.xml ${WORKSPACE}/cqlshlib.xml
+if [ "$cython" = "yes" ]; then
+deactivate  # venv
+fi
+cd ${WORKSPACE}
+
+
+# run dtest cqlsh suite
+cd cassandra-dtest/
+if [ "$cython" = "no" ]; then
+export CASS_DRIVER_NO_EXTENSIONS=true
+export CASS_DRIVER_NO_CYTHON=true
+fi
+virtualenv --python=python2 --no-site-packages venv
+source venv/bin/activate
+pip install -r requirements.txt
+pip freeze
+
+nosetests --verbosity=3 --with-xunit --nocapture cqlsh_tests/
+
+mv nosetests.xml ${WORKSPACE}/
+deactivate  # venv

[jira] [Updated] (CASSANDRA-13090) Coalescing strategy sleeps too much

[Brandon Williams (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Brandon Williams updated CASSANDRA-13090:
-
Fix Version/s: (was: 2.2.9)
   2.2.10

> Coalescing strategy sleeps too much
> ---
>
> Key: CASSANDRA-13090
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13090
> Project: Cassandra
>  Issue Type: Bug
>  Components: Streaming and Messaging
>Reporter: Corentin Chary
>Assignee: Corentin Chary
> Fix For: 2.2.10, 3.0.11, 3.11.0, 4.0
>
> Attachments: 0001-Fix-wait-time-coalescing-CASSANDRA-13090-2.patch, 
> 0001-Fix-wait-time-coalescing-CASSANDRA-13090.patch
>
>
> With the current code maybeSleep is called even if we managed to take 
> maxItems out of the backlog. In this case we should really avoid sleeping 
> because it means that backlog is building up.
> I'll send a patch shortly.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (CASSANDRA-12588) Cannot find column durable_wrıtes

[Gokhan Mansuroglu (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15895030#comment-15895030
 ] 

Gokhan Mansuroglu commented on CASSANDRA-12588:
---

The same problem is troubling me. I came across a question on stackoverflow 
that emphasise that Turkish language could be the problem. In fact the "ı" 
character gives a clue. Here is the link :

http://stackoverflow.com/questions/38113849/starting-cassandra-assertionerror

> Cannot find column durable_wrıtes
> -
>
> Key: CASSANDRA-12588
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12588
> Project: Cassandra
>  Issue Type: Bug
>Reporter: LLc.
>
> help please
> run :
> cassandra -f
> ERROR 17:00:16 Exception encountered during startup
> java.lang.AssertionError: Cannot find column durable_wrıtes



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (CASSANDRA-13233) Improve testing on macOS by eliminating sigar logging

[Jason Brown (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Brown updated CASSANDRA-13233:

   Resolution: Fixed
Fix Version/s: 4.0
   3.11.0
   3.0.12
   Status: Resolved  (was: Patch Available)

Committed as sha {{e3968cfd1e786f260029cf8cde4164719ae56c53}}

[~mkjellman] feel free to change to subject of this ticket to something more 
apropos. It'll be the current name in the git history and CHANGES.txt, but the 
jira could have more more descriptive if you think it'll help. naming is hard ;)

> Improve testing on macOS by eliminating sigar logging
> -
>
> Key: CASSANDRA-13233
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13233
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Michael Kjellman
>Assignee: Michael Kjellman
> Fix For: 3.0.12, 3.11.0, 4.0
>
> Attachments: 28827709.diff, CASSANDRA-13233-trunk-v2.diff
>
>
> The changes introduced in CASSANDRA-7838 (Resolved; Fixed; 2.2.0 beta 1): 
> "Warn user when OS settings are poor / integrate sigar" are not Mac friendly.
> {code}
> INFO  [main] 2016-10-18T11:20:10,330 SigarLibrary.java:44 - Initializing 
> SIGAR library
> DEBUG [main] 2016-10-18T11:20:10,342 SigarLog.java:60 - no 
> libsigar-universal64-macosx.dylib in java.library.path
> org.hyperic.sigar.SigarException: no libsigar-universal64-macosx.dylib in 
> java.library.path
> at org.hyperic.sigar.Sigar.loadLibrary(Sigar.java:172) 
> ~[sigar-1.6.4.jar:?]
> at org.hyperic.sigar.Sigar.(Sigar.java:100) 
> [sigar-1.6.4.jar:?]
> at 
> org.apache.cassandra.utils.SigarLibrary.(SigarLibrary.java:47) [main/:?]
> at 
> org.apache.cassandra.utils.SigarLibrary.(SigarLibrary.java:28) 
> [main/:?]
> at org.apache.cassandra.utils.UUIDGen.hash(UUIDGen.java:363) [main/:?]
> at org.apache.cassandra.utils.UUIDGen.makeNode(UUIDGen.java:342) 
> [main/:?]
> at 
> org.apache.cassandra.utils.UUIDGen.makeClockSeqAndNode(UUIDGen.java:291) 
> [main/:?]
> at org.apache.cassandra.utils.UUIDGen.(UUIDGen.java:42) 
> [main/:?]
> at 
> org.apache.cassandra.config.CFMetaData$Builder.build(CFMetaData.java:1278) 
> [main/:?]
> at 
> org.apache.cassandra.SchemaLoader.standardCFMD(SchemaLoader.java:369) 
> [classes/:?]
> at 
> org.apache.cassandra.SchemaLoader.standardCFMD(SchemaLoader.java:356) 
> [classes/:?]
> at 
> org.apache.cassandra.SchemaLoader.standardCFMD(SchemaLoader.java:351) 
> [classes/:?]
> at 
> org.apache.cassandra.batchlog.BatchTest.defineSchema(BatchTest.java:59) 
> [classes/:?]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
> ~[?:1.8.0_66]
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
> ~[?:1.8.0_66]
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  ~[?:1.8.0_66]
> at java.lang.reflect.Method.invoke(Method.java:497) ~[?:1.8.0_66]
> at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
>  [junit-4.6.jar:?]
> at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
>  [junit-4.6.jar:?]
> at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
>  [junit-4.6.jar:?]
> at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:27) 
> [junit-4.6.jar:?]
> at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31) 
> [junit-4.6.jar:?]
> at org.junit.runners.ParentRunner.run(ParentRunner.java:220) 
> [junit-4.6.jar:?]
> at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:39) 
> [junit-4.6.jar:?]
> at 
> org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:535)
>  [ant-junit.jar:?]
> at 
> org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1182)
>  [ant-junit.jar:?]
> at 
> org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:1033)
>  [ant-junit.jar:?]
> INFO  [main] 2016-10-18T11:20:10,350 SigarLibrary.java:57 - Could not 
> initialize SIGAR library 
> org.hyperic.sigar.Sigar.getFileSystemListNative()[Lorg/hyperic/sigar/FileSystem;
> {code}
> There are 2 issues addressed by the attached patch:
> # Create platform aware (windows, Darwin, linux) implementations of CLibrary 
> (for instance CLibrary today assumes all platforms have support for 
> posix_fadvise but this doesn't exist in the Darwin kernel). If methods are 
> defined with the "native" JNI keyword in java when the class is loaded it 
> will cause our jna check to fail 

[5/6] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11

[jasobrown]

Merge branch 'cassandra-3.0' into cassandra-3.11


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/92b7322b
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/92b7322b
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/92b7322b

Branch: refs/heads/cassandra-3.11
Commit: 92b7322bbf0b53203adcfb6c4479022f975d9785
Parents: a830cf5 e3968cf
Author: Jason Brown 
Authored: Fri Mar 3 11:17:17 2017 -0800
Committer: Jason Brown 
Committed: Fri Mar 3 11:19:30 2017 -0800

--
 CHANGES.txt |   1 +
 .../org/apache/cassandra/utils/CLibrary.java| 135 +++
 .../apache/cassandra/utils/CLibraryDarwin.java  | 131 ++
 .../apache/cassandra/utils/CLibraryLinux.java   | 131 ++
 .../apache/cassandra/utils/CLibraryWindows.java | 103 ++
 .../apache/cassandra/utils/CLibraryWrapper.java |  40 ++
 .../org/apache/cassandra/utils/HeapUtils.java   |   6 +-
 .../org/apache/cassandra/utils/UUIDGen.java |   2 +-
 .../apache/cassandra/utils/CLibraryTest.java|   8 ++
 9 files changed, 495 insertions(+), 62 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/92b7322b/CHANGES.txt
--
diff --cc CHANGES.txt
index 5de9ece,076b337..b91edb4
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,17 -1,7 +1,18 @@@
 -3.0.12
 +3.11.0
 + * Fix equality comparisons of columns using the duration type 
(CASSANDRA-13174)
 + * Obfuscate password in stress-graphs (CASSANDRA-12233)
 + * Move to FastThreadLocalThread and FastThreadLocal (CASSANDRA-13034)
 + * nodetool stopdaemon errors out (CASSANDRA-13030)
 + * Tables in system_distributed should not use gcgs of 0 (CASSANDRA-12954)
 + * Fix primary index calculation for SASI (CASSANDRA-12910)
 + * More fixes to the TokenAllocator (CASSANDRA-12990)
 + * NoReplicationTokenAllocator should work with zero replication factor 
(CASSANDRA-12983)
 + * Address message coalescing regression (CASSANDRA-12676)
 +Merged from 3.0:
+  * Improve testing on macOS by eliminating sigar logging (CASSANDRA-13233)
   * Cqlsh copy-from should error out when csv contains invalid data for 
collections (CASSANDRA-13071)
 - * Update c.yaml doc for offheap memtables (CASSANDRA-13179)
 + * Fix "multiple versions of ant detected..." when running ant test 
(CASSANDRA-13232)
 + * Coalescing strategy sleeps too much (CASSANDRA-13090)
   * Faster StreamingHistogram (CASSANDRA-13038)
   * Legacy deserializer can create unexpected boundary range tombstones 
(CASSANDRA-13237)
   * Remove unnecessary assertion from AntiCompactionTest (CASSANDRA-13070)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/92b7322b/src/java/org/apache/cassandra/utils/CLibrary.java
--
diff --cc src/java/org/apache/cassandra/utils/CLibrary.java
index 937d662,71f4148..ae3ce63
--- a/src/java/org/apache/cassandra/utils/CLibrary.java
+++ b/src/java/org/apache/cassandra/utils/CLibrary.java
@@@ -29,10 -29,12 +29,13 @@@ import org.slf4j.Logger
  import org.slf4j.LoggerFactory;
  
  import com.sun.jna.LastErrorException;
- import com.sun.jna.Native;
- import com.sun.jna.Pointer;
 +import sun.nio.ch.FileChannelImpl;
  
+ import static org.apache.cassandra.utils.CLibrary.OSType.LINUX;
+ import static org.apache.cassandra.utils.CLibrary.OSType.MAC;
+ import static org.apache.cassandra.utils.CLibrary.OSType.WINDOWS;
+ import static org.apache.cassandra.utils.CLibrary.OSType.AIX;
+ 
  public final class CLibrary
  {
  private static final Logger logger = 
LoggerFactory.getLogger(CLibrary.class);
@@@ -55,45 -68,22 +69,28 @@@
  private static final int POSIX_FADV_DONTNEED   = 4; /* fadvise.h */
  private static final int POSIX_FADV_NOREUSE= 5; /* fadvise.h */
  
- static boolean jnaAvailable = true;
- static boolean jnaLockable = false;
- private static boolean fsyncUnavailable = false;
- 
- private static final boolean OS_LINUX;
- private static final boolean OS_AIX;
- private static final boolean OS_MAC;
+ private static final CLibraryWrapper wrappedCLibrary;
+ private static boolean jnaLockable = false;
  
 +private static final Field FILE_DESCRIPTOR_FD_FIELD;
 +private static final Field FILE_CHANNEL_FD_FIELD;
 +
  static
  {
- String os = System.getProperty("os.name").toLowerCase();
- OS_LINUX = os.contains("linux");
- OS_AIX = os.contains("aix");
- OS_MAC = os.contains("mac");
- 
 +FILE_DESCRIPTOR_FD_FIELD = 
FBUtilities.getProtectedField(FileDescriptor.class, "fd");
 +FILE_CHANNEL_FD_FIELD = 
FBUtilities.getProtectedField(FileChannelImpl.class, 

[1/6] cassandra git commit: Improve testing on macOS by eliminating sigar logging

[jasobrown]

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-3.0 244f887f4 -> e3968cfd1
  refs/heads/cassandra-3.11 a830cf528 -> 92b7322bb
  refs/heads/trunk 643234664 -> 979b19b7c


Improve testing on macOS by eliminating sigar logging

patch by mkjellman, reviewed by jasobrown for CASSANDRA-13233


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/e3968cfd
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/e3968cfd
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/e3968cfd

Branch: refs/heads/cassandra-3.0
Commit: e3968cfd1e786f260029cf8cde4164719ae56c53
Parents: 244f887
Author: Michael Kjellman 
Authored: Thu Mar 2 14:03:25 2017 -0800
Committer: Jason Brown 
Committed: Fri Mar 3 11:15:52 2017 -0800

--
 CHANGES.txt |   1 +
 .../org/apache/cassandra/utils/CLibrary.java| 124 +++---
 .../apache/cassandra/utils/CLibraryDarwin.java  | 131 +++
 .../apache/cassandra/utils/CLibraryLinux.java   | 131 +++
 .../apache/cassandra/utils/CLibraryWindows.java | 103 +++
 .../apache/cassandra/utils/CLibraryWrapper.java |  40 ++
 .../org/apache/cassandra/utils/HeapUtils.java   |   6 +-
 .../org/apache/cassandra/utils/UUIDGen.java |   2 +-
 .../apache/cassandra/utils/CLibraryTest.java|   8 ++
 9 files changed, 494 insertions(+), 52 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/e3968cfd/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 1c3869f..076b337 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.12
+ * Improve testing on macOS by eliminating sigar logging (CASSANDRA-13233)
  * Cqlsh copy-from should error out when csv contains invalid data for 
collections (CASSANDRA-13071)
  * Update c.yaml doc for offheap memtables (CASSANDRA-13179)
  * Faster StreamingHistogram (CASSANDRA-13038)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/e3968cfd/src/java/org/apache/cassandra/utils/CLibrary.java
--
diff --git a/src/java/org/apache/cassandra/utils/CLibrary.java 
b/src/java/org/apache/cassandra/utils/CLibrary.java
index 347b0df..71f4148 100644
--- a/src/java/org/apache/cassandra/utils/CLibrary.java
+++ b/src/java/org/apache/cassandra/utils/CLibrary.java
@@ -23,21 +23,33 @@ import java.io.FileInputStream;
 import java.io.IOException;
 import java.lang.reflect.Field;
 import java.nio.channels.FileChannel;
-import java.nio.file.Paths;
-import java.nio.file.StandardOpenOption;
 import java.util.concurrent.TimeUnit;
 
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
 import com.sun.jna.LastErrorException;
-import com.sun.jna.Native;
-import com.sun.jna.Pointer;
+
+import static org.apache.cassandra.utils.CLibrary.OSType.LINUX;
+import static org.apache.cassandra.utils.CLibrary.OSType.MAC;
+import static org.apache.cassandra.utils.CLibrary.OSType.WINDOWS;
+import static org.apache.cassandra.utils.CLibrary.OSType.AIX;
 
 public final class CLibrary
 {
 private static final Logger logger = 
LoggerFactory.getLogger(CLibrary.class);
 
+public enum OSType
+{
+LINUX,
+MAC,
+WINDOWS,
+AIX,
+OTHER;
+}
+
+private static final OSType osType;
+
 private static final int MCL_CURRENT;
 private static final int MCL_FUTURE;
 
@@ -56,39 +68,32 @@ public final class CLibrary
 private static final int POSIX_FADV_DONTNEED   = 4; /* fadvise.h */
 private static final int POSIX_FADV_NOREUSE= 5; /* fadvise.h */
 
-static boolean jnaAvailable = true;
-static boolean jnaLockable = false;
+private static final CLibraryWrapper wrappedCLibrary;
+private static boolean jnaLockable = false;
 
 static
 {
-try
-{
-Native.register("c");
-}
-catch (NoClassDefFoundError e)
-{
-logger.warn("JNA not found. Native methods will be disabled.");
-jnaAvailable = false;
-}
-catch (UnsatisfiedLinkError e)
+// detect the OS type the JVM is running on and then set the 
CLibraryWrapper
+// instance to a compatable implementation of CLibraryWrapper for that 
OS type
+osType = getOsType();
+switch (osType)
 {
-logger.warn("JNA link failure, one or more native method will be 
unavailable.");
-logger.trace("JNA link failure details: {}", e.getMessage());
-}
-catch (NoSuchMethodError e)
-{
-logger.warn("Obsolete version of JNA present; unable to register C 
library. Upgrade to JNA 3.2.7 or later");
-

[3/6] cassandra git commit: Improve testing on macOS by eliminating sigar logging

[jasobrown]

Improve testing on macOS by eliminating sigar logging

patch by mkjellman, reviewed by jasobrown for CASSANDRA-13233


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/e3968cfd
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/e3968cfd
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/e3968cfd

Branch: refs/heads/trunk
Commit: e3968cfd1e786f260029cf8cde4164719ae56c53
Parents: 244f887
Author: Michael Kjellman 
Authored: Thu Mar 2 14:03:25 2017 -0800
Committer: Jason Brown 
Committed: Fri Mar 3 11:15:52 2017 -0800

--
 CHANGES.txt |   1 +
 .../org/apache/cassandra/utils/CLibrary.java| 124 +++---
 .../apache/cassandra/utils/CLibraryDarwin.java  | 131 +++
 .../apache/cassandra/utils/CLibraryLinux.java   | 131 +++
 .../apache/cassandra/utils/CLibraryWindows.java | 103 +++
 .../apache/cassandra/utils/CLibraryWrapper.java |  40 ++
 .../org/apache/cassandra/utils/HeapUtils.java   |   6 +-
 .../org/apache/cassandra/utils/UUIDGen.java |   2 +-
 .../apache/cassandra/utils/CLibraryTest.java|   8 ++
 9 files changed, 494 insertions(+), 52 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/e3968cfd/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 1c3869f..076b337 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.12
+ * Improve testing on macOS by eliminating sigar logging (CASSANDRA-13233)
  * Cqlsh copy-from should error out when csv contains invalid data for 
collections (CASSANDRA-13071)
  * Update c.yaml doc for offheap memtables (CASSANDRA-13179)
  * Faster StreamingHistogram (CASSANDRA-13038)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/e3968cfd/src/java/org/apache/cassandra/utils/CLibrary.java
--
diff --git a/src/java/org/apache/cassandra/utils/CLibrary.java 
b/src/java/org/apache/cassandra/utils/CLibrary.java
index 347b0df..71f4148 100644
--- a/src/java/org/apache/cassandra/utils/CLibrary.java
+++ b/src/java/org/apache/cassandra/utils/CLibrary.java
@@ -23,21 +23,33 @@ import java.io.FileInputStream;
 import java.io.IOException;
 import java.lang.reflect.Field;
 import java.nio.channels.FileChannel;
-import java.nio.file.Paths;
-import java.nio.file.StandardOpenOption;
 import java.util.concurrent.TimeUnit;
 
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
 import com.sun.jna.LastErrorException;
-import com.sun.jna.Native;
-import com.sun.jna.Pointer;
+
+import static org.apache.cassandra.utils.CLibrary.OSType.LINUX;
+import static org.apache.cassandra.utils.CLibrary.OSType.MAC;
+import static org.apache.cassandra.utils.CLibrary.OSType.WINDOWS;
+import static org.apache.cassandra.utils.CLibrary.OSType.AIX;
 
 public final class CLibrary
 {
 private static final Logger logger = 
LoggerFactory.getLogger(CLibrary.class);
 
+public enum OSType
+{
+LINUX,
+MAC,
+WINDOWS,
+AIX,
+OTHER;
+}
+
+private static final OSType osType;
+
 private static final int MCL_CURRENT;
 private static final int MCL_FUTURE;
 
@@ -56,39 +68,32 @@ public final class CLibrary
 private static final int POSIX_FADV_DONTNEED   = 4; /* fadvise.h */
 private static final int POSIX_FADV_NOREUSE= 5; /* fadvise.h */
 
-static boolean jnaAvailable = true;
-static boolean jnaLockable = false;
+private static final CLibraryWrapper wrappedCLibrary;
+private static boolean jnaLockable = false;
 
 static
 {
-try
-{
-Native.register("c");
-}
-catch (NoClassDefFoundError e)
-{
-logger.warn("JNA not found. Native methods will be disabled.");
-jnaAvailable = false;
-}
-catch (UnsatisfiedLinkError e)
+// detect the OS type the JVM is running on and then set the 
CLibraryWrapper
+// instance to a compatable implementation of CLibraryWrapper for that 
OS type
+osType = getOsType();
+switch (osType)
 {
-logger.warn("JNA link failure, one or more native method will be 
unavailable.");
-logger.trace("JNA link failure details: {}", e.getMessage());
-}
-catch (NoSuchMethodError e)
-{
-logger.warn("Obsolete version of JNA present; unable to register C 
library. Upgrade to JNA 3.2.7 or later");
-jnaAvailable = false;
+case MAC: wrappedCLibrary = new CLibraryDarwin(); break;
+case WINDOWS: wrappedCLibrary = new CLibraryWindows(); break;
+case LINUX:
+ 

[6/6] cassandra git commit: Merge branch 'cassandra-3.11' into trunk

[jasobrown]

Merge branch 'cassandra-3.11' into trunk


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/979b19b7
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/979b19b7
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/979b19b7

Branch: refs/heads/trunk
Commit: 979b19b7c0ac64b14b51e8b92b00f50d0eed9be7
Parents: 6432346 92b7322
Author: Jason Brown 
Authored: Fri Mar 3 11:20:16 2017 -0800
Committer: Jason Brown 
Committed: Fri Mar 3 11:20:36 2017 -0800

--
 CHANGES.txt |   1 +
 .../org/apache/cassandra/utils/CLibrary.java| 135 +++
 .../apache/cassandra/utils/CLibraryDarwin.java  | 131 ++
 .../apache/cassandra/utils/CLibraryLinux.java   | 131 ++
 .../apache/cassandra/utils/CLibraryWindows.java | 103 ++
 .../apache/cassandra/utils/CLibraryWrapper.java |  40 ++
 .../org/apache/cassandra/utils/HeapUtils.java   |   6 +-
 .../org/apache/cassandra/utils/UUIDGen.java |   2 +-
 .../apache/cassandra/utils/CLibraryTest.java|   8 ++
 9 files changed, 495 insertions(+), 62 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/979b19b7/CHANGES.txt
--

[2/6] cassandra git commit: Improve testing on macOS by eliminating sigar logging

[jasobrown]

Improve testing on macOS by eliminating sigar logging

patch by mkjellman, reviewed by jasobrown for CASSANDRA-13233


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/e3968cfd
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/e3968cfd
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/e3968cfd

Branch: refs/heads/cassandra-3.11
Commit: e3968cfd1e786f260029cf8cde4164719ae56c53
Parents: 244f887
Author: Michael Kjellman 
Authored: Thu Mar 2 14:03:25 2017 -0800
Committer: Jason Brown 
Committed: Fri Mar 3 11:15:52 2017 -0800

--
 CHANGES.txt |   1 +
 .../org/apache/cassandra/utils/CLibrary.java| 124 +++---
 .../apache/cassandra/utils/CLibraryDarwin.java  | 131 +++
 .../apache/cassandra/utils/CLibraryLinux.java   | 131 +++
 .../apache/cassandra/utils/CLibraryWindows.java | 103 +++
 .../apache/cassandra/utils/CLibraryWrapper.java |  40 ++
 .../org/apache/cassandra/utils/HeapUtils.java   |   6 +-
 .../org/apache/cassandra/utils/UUIDGen.java |   2 +-
 .../apache/cassandra/utils/CLibraryTest.java|   8 ++
 9 files changed, 494 insertions(+), 52 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/e3968cfd/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 1c3869f..076b337 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.12
+ * Improve testing on macOS by eliminating sigar logging (CASSANDRA-13233)
  * Cqlsh copy-from should error out when csv contains invalid data for 
collections (CASSANDRA-13071)
  * Update c.yaml doc for offheap memtables (CASSANDRA-13179)
  * Faster StreamingHistogram (CASSANDRA-13038)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/e3968cfd/src/java/org/apache/cassandra/utils/CLibrary.java
--
diff --git a/src/java/org/apache/cassandra/utils/CLibrary.java 
b/src/java/org/apache/cassandra/utils/CLibrary.java
index 347b0df..71f4148 100644
--- a/src/java/org/apache/cassandra/utils/CLibrary.java
+++ b/src/java/org/apache/cassandra/utils/CLibrary.java
@@ -23,21 +23,33 @@ import java.io.FileInputStream;
 import java.io.IOException;
 import java.lang.reflect.Field;
 import java.nio.channels.FileChannel;
-import java.nio.file.Paths;
-import java.nio.file.StandardOpenOption;
 import java.util.concurrent.TimeUnit;
 
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
 import com.sun.jna.LastErrorException;
-import com.sun.jna.Native;
-import com.sun.jna.Pointer;
+
+import static org.apache.cassandra.utils.CLibrary.OSType.LINUX;
+import static org.apache.cassandra.utils.CLibrary.OSType.MAC;
+import static org.apache.cassandra.utils.CLibrary.OSType.WINDOWS;
+import static org.apache.cassandra.utils.CLibrary.OSType.AIX;
 
 public final class CLibrary
 {
 private static final Logger logger = 
LoggerFactory.getLogger(CLibrary.class);
 
+public enum OSType
+{
+LINUX,
+MAC,
+WINDOWS,
+AIX,
+OTHER;
+}
+
+private static final OSType osType;
+
 private static final int MCL_CURRENT;
 private static final int MCL_FUTURE;
 
@@ -56,39 +68,32 @@ public final class CLibrary
 private static final int POSIX_FADV_DONTNEED   = 4; /* fadvise.h */
 private static final int POSIX_FADV_NOREUSE= 5; /* fadvise.h */
 
-static boolean jnaAvailable = true;
-static boolean jnaLockable = false;
+private static final CLibraryWrapper wrappedCLibrary;
+private static boolean jnaLockable = false;
 
 static
 {
-try
-{
-Native.register("c");
-}
-catch (NoClassDefFoundError e)
-{
-logger.warn("JNA not found. Native methods will be disabled.");
-jnaAvailable = false;
-}
-catch (UnsatisfiedLinkError e)
+// detect the OS type the JVM is running on and then set the 
CLibraryWrapper
+// instance to a compatable implementation of CLibraryWrapper for that 
OS type
+osType = getOsType();
+switch (osType)
 {
-logger.warn("JNA link failure, one or more native method will be 
unavailable.");
-logger.trace("JNA link failure details: {}", e.getMessage());
-}
-catch (NoSuchMethodError e)
-{
-logger.warn("Obsolete version of JNA present; unable to register C 
library. Upgrade to JNA 3.2.7 or later");
-jnaAvailable = false;
+case MAC: wrappedCLibrary = new CLibraryDarwin(); break;
+case WINDOWS: wrappedCLibrary = new CLibraryWindows(); break;
+case 

[4/6] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11

[jasobrown]

Merge branch 'cassandra-3.0' into cassandra-3.11


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/92b7322b
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/92b7322b
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/92b7322b

Branch: refs/heads/trunk
Commit: 92b7322bbf0b53203adcfb6c4479022f975d9785
Parents: a830cf5 e3968cf
Author: Jason Brown 
Authored: Fri Mar 3 11:17:17 2017 -0800
Committer: Jason Brown 
Committed: Fri Mar 3 11:19:30 2017 -0800

--
 CHANGES.txt |   1 +
 .../org/apache/cassandra/utils/CLibrary.java| 135 +++
 .../apache/cassandra/utils/CLibraryDarwin.java  | 131 ++
 .../apache/cassandra/utils/CLibraryLinux.java   | 131 ++
 .../apache/cassandra/utils/CLibraryWindows.java | 103 ++
 .../apache/cassandra/utils/CLibraryWrapper.java |  40 ++
 .../org/apache/cassandra/utils/HeapUtils.java   |   6 +-
 .../org/apache/cassandra/utils/UUIDGen.java |   2 +-
 .../apache/cassandra/utils/CLibraryTest.java|   8 ++
 9 files changed, 495 insertions(+), 62 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/92b7322b/CHANGES.txt
--
diff --cc CHANGES.txt
index 5de9ece,076b337..b91edb4
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,17 -1,7 +1,18 @@@
 -3.0.12
 +3.11.0
 + * Fix equality comparisons of columns using the duration type 
(CASSANDRA-13174)
 + * Obfuscate password in stress-graphs (CASSANDRA-12233)
 + * Move to FastThreadLocalThread and FastThreadLocal (CASSANDRA-13034)
 + * nodetool stopdaemon errors out (CASSANDRA-13030)
 + * Tables in system_distributed should not use gcgs of 0 (CASSANDRA-12954)
 + * Fix primary index calculation for SASI (CASSANDRA-12910)
 + * More fixes to the TokenAllocator (CASSANDRA-12990)
 + * NoReplicationTokenAllocator should work with zero replication factor 
(CASSANDRA-12983)
 + * Address message coalescing regression (CASSANDRA-12676)
 +Merged from 3.0:
+  * Improve testing on macOS by eliminating sigar logging (CASSANDRA-13233)
   * Cqlsh copy-from should error out when csv contains invalid data for 
collections (CASSANDRA-13071)
 - * Update c.yaml doc for offheap memtables (CASSANDRA-13179)
 + * Fix "multiple versions of ant detected..." when running ant test 
(CASSANDRA-13232)
 + * Coalescing strategy sleeps too much (CASSANDRA-13090)
   * Faster StreamingHistogram (CASSANDRA-13038)
   * Legacy deserializer can create unexpected boundary range tombstones 
(CASSANDRA-13237)
   * Remove unnecessary assertion from AntiCompactionTest (CASSANDRA-13070)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/92b7322b/src/java/org/apache/cassandra/utils/CLibrary.java
--
diff --cc src/java/org/apache/cassandra/utils/CLibrary.java
index 937d662,71f4148..ae3ce63
--- a/src/java/org/apache/cassandra/utils/CLibrary.java
+++ b/src/java/org/apache/cassandra/utils/CLibrary.java
@@@ -29,10 -29,12 +29,13 @@@ import org.slf4j.Logger
  import org.slf4j.LoggerFactory;
  
  import com.sun.jna.LastErrorException;
- import com.sun.jna.Native;
- import com.sun.jna.Pointer;
 +import sun.nio.ch.FileChannelImpl;
  
+ import static org.apache.cassandra.utils.CLibrary.OSType.LINUX;
+ import static org.apache.cassandra.utils.CLibrary.OSType.MAC;
+ import static org.apache.cassandra.utils.CLibrary.OSType.WINDOWS;
+ import static org.apache.cassandra.utils.CLibrary.OSType.AIX;
+ 
  public final class CLibrary
  {
  private static final Logger logger = 
LoggerFactory.getLogger(CLibrary.class);
@@@ -55,45 -68,22 +69,28 @@@
  private static final int POSIX_FADV_DONTNEED   = 4; /* fadvise.h */
  private static final int POSIX_FADV_NOREUSE= 5; /* fadvise.h */
  
- static boolean jnaAvailable = true;
- static boolean jnaLockable = false;
- private static boolean fsyncUnavailable = false;
- 
- private static final boolean OS_LINUX;
- private static final boolean OS_AIX;
- private static final boolean OS_MAC;
+ private static final CLibraryWrapper wrappedCLibrary;
+ private static boolean jnaLockable = false;
  
 +private static final Field FILE_DESCRIPTOR_FD_FIELD;
 +private static final Field FILE_CHANNEL_FD_FIELD;
 +
  static
  {
- String os = System.getProperty("os.name").toLowerCase();
- OS_LINUX = os.contains("linux");
- OS_AIX = os.contains("aix");
- OS_MAC = os.contains("mac");
- 
 +FILE_DESCRIPTOR_FD_FIELD = 
FBUtilities.getProtectedField(FileDescriptor.class, "fd");
 +FILE_CHANNEL_FD_FIELD = 
FBUtilities.getProtectedField(FileChannelImpl.class, "fd");
 +
- 

[09/10] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11

[mshuler]

Merge branch 'cassandra-3.0' into cassandra-3.11


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/a830cf52
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/a830cf52
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/a830cf52

Branch: refs/heads/trunk
Commit: a830cf528ea2ab85288bf7503b282507346c71db
Parents: ff170af 244f887
Author: Michael Shuler 
Authored: Fri Mar 3 12:42:19 2017 -0600
Committer: Michael Shuler 
Committed: Fri Mar 3 12:42:19 2017 -0600

--
 debian/control| 2 +-
 redhat/cassandra.spec | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/a830cf52/redhat/cassandra.spec
--

[01/10] cassandra git commit: Update ant build dependency to >= 1.9 in deb/rpm

[mshuler]

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-2.2 c0f99c4e6 -> 288d243d8
  refs/heads/cassandra-3.0 e11f75081 -> 244f887f4
  refs/heads/cassandra-3.11 ff170af2a -> a830cf528
  refs/heads/trunk b9b63925f -> 643234664


Update ant build dependency to >= 1.9 in deb/rpm


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/288d243d
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/288d243d
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/288d243d

Branch: refs/heads/cassandra-2.2
Commit: 288d243d806c4b5143c38875515776dae2fc99ae
Parents: c0f99c4
Author: Michael Shuler 
Authored: Fri Mar 3 12:39:37 2017 -0600
Committer: Michael Shuler 
Committed: Fri Mar 3 12:39:37 2017 -0600

--
 debian/control| 2 +-
 redhat/cassandra.spec | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/288d243d/debian/control
--
diff --git a/debian/control b/debian/control
index cc5928c..d45a875 100644
--- a/debian/control
+++ b/debian/control
@@ -3,7 +3,7 @@ Section: misc
 Priority: extra
 Maintainer: Eric Evans 
 Uploaders: Sylvain Lebresne 
-Build-Depends: debhelper (>= 5), openjdk-7-jdk | java7-jdk, ant (>= 1.7), 
ant-optional (>= 1.7), dh-python, python-dev (>= 2.7), dpatch, bash-completion
+Build-Depends: debhelper (>= 5), openjdk-7-jdk | java7-jdk, ant (>= 1.9), 
ant-optional (>= 1.9), dh-python, python-dev (>= 2.7), dpatch, bash-completion
 Homepage: http://cassandra.apache.org
 Vcs-Git: http://git-wip-us.apache.org/repos/asf/cassandra.git
 Vcs-Browser: https://git-wip-us.apache.org/repos/asf?p=cassandra.git

http://git-wip-us.apache.org/repos/asf/cassandra/blob/288d243d/redhat/cassandra.spec
--
diff --git a/redhat/cassandra.spec b/redhat/cassandra.spec
index b40a706..c3c22d0 100644
--- a/redhat/cassandra.spec
+++ b/redhat/cassandra.spec
@@ -18,7 +18,7 @@ URL:   http://cassandra.apache.org/
 Source0:   %{relname}-src.tar.gz
 BuildRoot: %{_tmppath}/%{relname}root-%(%{__id_u} -n)
 
-BuildRequires: ant
+BuildRequires: ant >= 1.9
 
 Requires:  jre >= 1.7.0
 Requires:  python(abi) >= 2.7

[08/10] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11

[mshuler]

Merge branch 'cassandra-3.0' into cassandra-3.11


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/a830cf52
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/a830cf52
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/a830cf52

Branch: refs/heads/cassandra-3.11
Commit: a830cf528ea2ab85288bf7503b282507346c71db
Parents: ff170af 244f887
Author: Michael Shuler 
Authored: Fri Mar 3 12:42:19 2017 -0600
Committer: Michael Shuler 
Committed: Fri Mar 3 12:42:19 2017 -0600

--
 debian/control| 2 +-
 redhat/cassandra.spec | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/a830cf52/redhat/cassandra.spec
--

[03/10] cassandra git commit: Update ant build dependency to >= 1.9 in deb/rpm

[mshuler]

Update ant build dependency to >= 1.9 in deb/rpm


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/288d243d
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/288d243d
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/288d243d

Branch: refs/heads/cassandra-3.11
Commit: 288d243d806c4b5143c38875515776dae2fc99ae
Parents: c0f99c4
Author: Michael Shuler 
Authored: Fri Mar 3 12:39:37 2017 -0600
Committer: Michael Shuler 
Committed: Fri Mar 3 12:39:37 2017 -0600

--
 debian/control| 2 +-
 redhat/cassandra.spec | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/288d243d/debian/control
--
diff --git a/debian/control b/debian/control
index cc5928c..d45a875 100644
--- a/debian/control
+++ b/debian/control
@@ -3,7 +3,7 @@ Section: misc
 Priority: extra
 Maintainer: Eric Evans 
 Uploaders: Sylvain Lebresne 
-Build-Depends: debhelper (>= 5), openjdk-7-jdk | java7-jdk, ant (>= 1.7), 
ant-optional (>= 1.7), dh-python, python-dev (>= 2.7), dpatch, bash-completion
+Build-Depends: debhelper (>= 5), openjdk-7-jdk | java7-jdk, ant (>= 1.9), 
ant-optional (>= 1.9), dh-python, python-dev (>= 2.7), dpatch, bash-completion
 Homepage: http://cassandra.apache.org
 Vcs-Git: http://git-wip-us.apache.org/repos/asf/cassandra.git
 Vcs-Browser: https://git-wip-us.apache.org/repos/asf?p=cassandra.git

http://git-wip-us.apache.org/repos/asf/cassandra/blob/288d243d/redhat/cassandra.spec
--
diff --git a/redhat/cassandra.spec b/redhat/cassandra.spec
index b40a706..c3c22d0 100644
--- a/redhat/cassandra.spec
+++ b/redhat/cassandra.spec
@@ -18,7 +18,7 @@ URL:   http://cassandra.apache.org/
 Source0:   %{relname}-src.tar.gz
 BuildRoot: %{_tmppath}/%{relname}root-%(%{__id_u} -n)
 
-BuildRequires: ant
+BuildRequires: ant >= 1.9
 
 Requires:  jre >= 1.7.0
 Requires:  python(abi) >= 2.7

[04/10] cassandra git commit: Update ant build dependency to >= 1.9 in deb/rpm

[mshuler]

Update ant build dependency to >= 1.9 in deb/rpm


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/288d243d
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/288d243d
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/288d243d

Branch: refs/heads/trunk
Commit: 288d243d806c4b5143c38875515776dae2fc99ae
Parents: c0f99c4
Author: Michael Shuler 
Authored: Fri Mar 3 12:39:37 2017 -0600
Committer: Michael Shuler 
Committed: Fri Mar 3 12:39:37 2017 -0600

--
 debian/control| 2 +-
 redhat/cassandra.spec | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/288d243d/debian/control
--
diff --git a/debian/control b/debian/control
index cc5928c..d45a875 100644
--- a/debian/control
+++ b/debian/control
@@ -3,7 +3,7 @@ Section: misc
 Priority: extra
 Maintainer: Eric Evans 
 Uploaders: Sylvain Lebresne 
-Build-Depends: debhelper (>= 5), openjdk-7-jdk | java7-jdk, ant (>= 1.7), 
ant-optional (>= 1.7), dh-python, python-dev (>= 2.7), dpatch, bash-completion
+Build-Depends: debhelper (>= 5), openjdk-7-jdk | java7-jdk, ant (>= 1.9), 
ant-optional (>= 1.9), dh-python, python-dev (>= 2.7), dpatch, bash-completion
 Homepage: http://cassandra.apache.org
 Vcs-Git: http://git-wip-us.apache.org/repos/asf/cassandra.git
 Vcs-Browser: https://git-wip-us.apache.org/repos/asf?p=cassandra.git

http://git-wip-us.apache.org/repos/asf/cassandra/blob/288d243d/redhat/cassandra.spec
--
diff --git a/redhat/cassandra.spec b/redhat/cassandra.spec
index b40a706..c3c22d0 100644
--- a/redhat/cassandra.spec
+++ b/redhat/cassandra.spec
@@ -18,7 +18,7 @@ URL:   http://cassandra.apache.org/
 Source0:   %{relname}-src.tar.gz
 BuildRoot: %{_tmppath}/%{relname}root-%(%{__id_u} -n)
 
-BuildRequires: ant
+BuildRequires: ant >= 1.9
 
 Requires:  jre >= 1.7.0
 Requires:  python(abi) >= 2.7

[10/10] cassandra git commit: Merge branch 'cassandra-3.11' into trunk

[mshuler]

Merge branch 'cassandra-3.11' into trunk


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/64323466
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/64323466
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/64323466

Branch: refs/heads/trunk
Commit: 64323466402f33536ba0318f02e7829eafcbd711
Parents: b9b6392 a830cf5
Author: Michael Shuler 
Authored: Fri Mar 3 12:42:40 2017 -0600
Committer: Michael Shuler 
Committed: Fri Mar 3 12:42:40 2017 -0600

--
 debian/control| 2 +-
 redhat/cassandra.spec | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/64323466/redhat/cassandra.spec
--

[06/10] cassandra git commit: Merge branch 'cassandra-2.2' into cassandra-3.0

[mshuler]

Merge branch 'cassandra-2.2' into cassandra-3.0


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/244f887f
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/244f887f
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/244f887f

Branch: refs/heads/cassandra-3.0
Commit: 244f887f4c61ed75cc7250386b4ae5371e79f48f
Parents: e11f750 288d243
Author: Michael Shuler 
Authored: Fri Mar 3 12:42:00 2017 -0600
Committer: Michael Shuler 
Committed: Fri Mar 3 12:42:00 2017 -0600

--
 debian/control| 2 +-
 redhat/cassandra.spec | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/244f887f/debian/control
--
diff --cc debian/control
index 2d30077,d45a875..6a0128e
--- a/debian/control
+++ b/debian/control
@@@ -3,7 -3,7 +3,7 @@@ Section: mis
  Priority: extra
  Maintainer: Eric Evans 
  Uploaders: Sylvain Lebresne 
- Build-Depends: debhelper (>= 5), openjdk-8-jdk | java8-jdk, ant (>= 1.7), 
ant-optional (>= 1.7), dh-python, python-dev (>= 2.7), dpatch, bash-completion
 -Build-Depends: debhelper (>= 5), openjdk-7-jdk | java7-jdk, ant (>= 1.9), 
ant-optional (>= 1.9), dh-python, python-dev (>= 2.7), dpatch, bash-completion
++Build-Depends: debhelper (>= 5), openjdk-8-jdk | java8-jdk, ant (>= 1.9), 
ant-optional (>= 1.9), dh-python, python-dev (>= 2.7), dpatch, bash-completion
  Homepage: http://cassandra.apache.org
  Vcs-Git: http://git-wip-us.apache.org/repos/asf/cassandra.git
  Vcs-Browser: https://git-wip-us.apache.org/repos/asf?p=cassandra.git

http://git-wip-us.apache.org/repos/asf/cassandra/blob/244f887f/redhat/cassandra.spec
--
diff --cc redhat/cassandra.spec
index a1bfa5d,c3c22d0..d5e743d
--- a/redhat/cassandra.spec
+++ b/redhat/cassandra.spec
@@@ -18,9 -18,9 +18,9 @@@ URL:   http://cassandra.apache.
  Source0:   %{relname}-src.tar.gz
  BuildRoot: %{_tmppath}/%{relname}root-%(%{__id_u} -n)
  
- BuildRequires: ant
+ BuildRequires: ant >= 1.9
  
 -Requires:  jre >= 1.7.0
 +Requires:  jre >= 1.8.0
  Requires:  python(abi) >= 2.7
  Requires(pre): user(cassandra)
  Requires(pre): group(cassandra)

[02/10] cassandra git commit: Update ant build dependency to >= 1.9 in deb/rpm

[mshuler]

Update ant build dependency to >= 1.9 in deb/rpm


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/288d243d
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/288d243d
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/288d243d

Branch: refs/heads/cassandra-3.0
Commit: 288d243d806c4b5143c38875515776dae2fc99ae
Parents: c0f99c4
Author: Michael Shuler 
Authored: Fri Mar 3 12:39:37 2017 -0600
Committer: Michael Shuler 
Committed: Fri Mar 3 12:39:37 2017 -0600

--
 debian/control| 2 +-
 redhat/cassandra.spec | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/288d243d/debian/control
--
diff --git a/debian/control b/debian/control
index cc5928c..d45a875 100644
--- a/debian/control
+++ b/debian/control
@@ -3,7 +3,7 @@ Section: misc
 Priority: extra
 Maintainer: Eric Evans 
 Uploaders: Sylvain Lebresne 
-Build-Depends: debhelper (>= 5), openjdk-7-jdk | java7-jdk, ant (>= 1.7), 
ant-optional (>= 1.7), dh-python, python-dev (>= 2.7), dpatch, bash-completion
+Build-Depends: debhelper (>= 5), openjdk-7-jdk | java7-jdk, ant (>= 1.9), 
ant-optional (>= 1.9), dh-python, python-dev (>= 2.7), dpatch, bash-completion
 Homepage: http://cassandra.apache.org
 Vcs-Git: http://git-wip-us.apache.org/repos/asf/cassandra.git
 Vcs-Browser: https://git-wip-us.apache.org/repos/asf?p=cassandra.git

http://git-wip-us.apache.org/repos/asf/cassandra/blob/288d243d/redhat/cassandra.spec
--
diff --git a/redhat/cassandra.spec b/redhat/cassandra.spec
index b40a706..c3c22d0 100644
--- a/redhat/cassandra.spec
+++ b/redhat/cassandra.spec
@@ -18,7 +18,7 @@ URL:   http://cassandra.apache.org/
 Source0:   %{relname}-src.tar.gz
 BuildRoot: %{_tmppath}/%{relname}root-%(%{__id_u} -n)
 
-BuildRequires: ant
+BuildRequires: ant >= 1.9
 
 Requires:  jre >= 1.7.0
 Requires:  python(abi) >= 2.7

[07/10] cassandra git commit: Merge branch 'cassandra-2.2' into cassandra-3.0

[mshuler]

Merge branch 'cassandra-2.2' into cassandra-3.0


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/244f887f
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/244f887f
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/244f887f

Branch: refs/heads/trunk
Commit: 244f887f4c61ed75cc7250386b4ae5371e79f48f
Parents: e11f750 288d243
Author: Michael Shuler 
Authored: Fri Mar 3 12:42:00 2017 -0600
Committer: Michael Shuler 
Committed: Fri Mar 3 12:42:00 2017 -0600

--
 debian/control| 2 +-
 redhat/cassandra.spec | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/244f887f/debian/control
--
diff --cc debian/control
index 2d30077,d45a875..6a0128e
--- a/debian/control
+++ b/debian/control
@@@ -3,7 -3,7 +3,7 @@@ Section: mis
  Priority: extra
  Maintainer: Eric Evans 
  Uploaders: Sylvain Lebresne 
- Build-Depends: debhelper (>= 5), openjdk-8-jdk | java8-jdk, ant (>= 1.7), 
ant-optional (>= 1.7), dh-python, python-dev (>= 2.7), dpatch, bash-completion
 -Build-Depends: debhelper (>= 5), openjdk-7-jdk | java7-jdk, ant (>= 1.9), 
ant-optional (>= 1.9), dh-python, python-dev (>= 2.7), dpatch, bash-completion
++Build-Depends: debhelper (>= 5), openjdk-8-jdk | java8-jdk, ant (>= 1.9), 
ant-optional (>= 1.9), dh-python, python-dev (>= 2.7), dpatch, bash-completion
  Homepage: http://cassandra.apache.org
  Vcs-Git: http://git-wip-us.apache.org/repos/asf/cassandra.git
  Vcs-Browser: https://git-wip-us.apache.org/repos/asf?p=cassandra.git

http://git-wip-us.apache.org/repos/asf/cassandra/blob/244f887f/redhat/cassandra.spec
--
diff --cc redhat/cassandra.spec
index a1bfa5d,c3c22d0..d5e743d
--- a/redhat/cassandra.spec
+++ b/redhat/cassandra.spec
@@@ -18,9 -18,9 +18,9 @@@ URL:   http://cassandra.apache.
  Source0:   %{relname}-src.tar.gz
  BuildRoot: %{_tmppath}/%{relname}root-%(%{__id_u} -n)
  
- BuildRequires: ant
+ BuildRequires: ant >= 1.9
  
 -Requires:  jre >= 1.7.0
 +Requires:  jre >= 1.8.0
  Requires:  python(abi) >= 2.7
  Requires(pre): user(cassandra)
  Requires(pre): group(cassandra)

[05/10] cassandra git commit: Merge branch 'cassandra-2.2' into cassandra-3.0

[mshuler]

Merge branch 'cassandra-2.2' into cassandra-3.0


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/244f887f
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/244f887f
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/244f887f

Branch: refs/heads/cassandra-3.11
Commit: 244f887f4c61ed75cc7250386b4ae5371e79f48f
Parents: e11f750 288d243
Author: Michael Shuler 
Authored: Fri Mar 3 12:42:00 2017 -0600
Committer: Michael Shuler 
Committed: Fri Mar 3 12:42:00 2017 -0600

--
 debian/control| 2 +-
 redhat/cassandra.spec | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/244f887f/debian/control
--
diff --cc debian/control
index 2d30077,d45a875..6a0128e
--- a/debian/control
+++ b/debian/control
@@@ -3,7 -3,7 +3,7 @@@ Section: mis
  Priority: extra
  Maintainer: Eric Evans 
  Uploaders: Sylvain Lebresne 
- Build-Depends: debhelper (>= 5), openjdk-8-jdk | java8-jdk, ant (>= 1.7), 
ant-optional (>= 1.7), dh-python, python-dev (>= 2.7), dpatch, bash-completion
 -Build-Depends: debhelper (>= 5), openjdk-7-jdk | java7-jdk, ant (>= 1.9), 
ant-optional (>= 1.9), dh-python, python-dev (>= 2.7), dpatch, bash-completion
++Build-Depends: debhelper (>= 5), openjdk-8-jdk | java8-jdk, ant (>= 1.9), 
ant-optional (>= 1.9), dh-python, python-dev (>= 2.7), dpatch, bash-completion
  Homepage: http://cassandra.apache.org
  Vcs-Git: http://git-wip-us.apache.org/repos/asf/cassandra.git
  Vcs-Browser: https://git-wip-us.apache.org/repos/asf?p=cassandra.git

http://git-wip-us.apache.org/repos/asf/cassandra/blob/244f887f/redhat/cassandra.spec
--
diff --cc redhat/cassandra.spec
index a1bfa5d,c3c22d0..d5e743d
--- a/redhat/cassandra.spec
+++ b/redhat/cassandra.spec
@@@ -18,9 -18,9 +18,9 @@@ URL:   http://cassandra.apache.
  Source0:   %{relname}-src.tar.gz
  BuildRoot: %{_tmppath}/%{relname}root-%(%{__id_u} -n)
  
- BuildRequires: ant
+ BuildRequires: ant >= 1.9
  
 -Requires:  jre >= 1.7.0
 +Requires:  jre >= 1.8.0
  Requires:  python(abi) >= 2.7
  Requires(pre): user(cassandra)
  Requires(pre): group(cassandra)

[jira] [Commented] (CASSANDRA-13265) Expiration in OutboundTcpConnection can block the reader Thread

[Jason Brown (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15894783#comment-15894783
 ] 

Jason Brown commented on CASSANDRA-13265:
-

I'm +1 on the ticket, but [~aweisberg] should also weigh in.

As a note, we have {{ApproximateTime}} for an estimated time, but that works 
off of {{System.currentTimeMillis()}} rather than {{System.nanoTime()}}, so 
monotonicity.

> Expiration in OutboundTcpConnection can block the reader Thread
> ---
>
> Key: CASSANDRA-13265
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13265
> Project: Cassandra
>  Issue Type: Bug
> Environment: Cassandra 3.0.9
> Java HotSpot(TM) 64-Bit Server VM version 25.112-b15 (Java version 
> 1.8.0_112-b15)
> Linux 3.16
>Reporter: Christian Esken
>Assignee: Christian Esken
> Attachments: cassandra.pb-cache4-dus.2017-02-17-19-36-26.chist.xz, 
> cassandra.pb-cache4-dus.2017-02-17-19-36-26.td.xz
>
>
> I observed that sometimes a single node in a Cassandra cluster fails to 
> communicate to the other nodes. This can happen at any time, during peak load 
> or low load. Restarting that single node from the cluster fixes the issue.
> Before going in to details, I want to state that I have analyzed the 
> situation and am already developing a possible fix. Here is the analysis so 
> far:
> - A Threaddump in this situation showed  324 Threads in the 
> OutboundTcpConnection class that want to lock the backlog queue for doing 
> expiration.
> - A class histogram shows 262508 instances of 
> OutboundTcpConnection$QueuedMessage.
> What is the effect of it? As soon as the Cassandra node has reached a certain 
> amount of queued messages, it starts thrashing itself to death. Each of the 
> Thread fully locks the Queue for reading and writing by calling 
> iterator.next(), making the situation worse and worse.
> - Writing: Only after 262508 locking operation it can progress with actually 
> writing to the Queue.
> - Reading: Is also blocked, as 324 Threads try to do iterator.next(), and 
> fully lock the Queue
> This means: Writing blocks the Queue for reading, and readers might even be 
> starved which makes the situation even worse.
> -
> The setup is:
>  - 3-node cluster
>  - replication factor 2
>  - Consistency LOCAL_ONE
>  - No remote DC's
>  - high write throughput (10 INSERT statements per second and more during 
> peak times).
>  



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (CASSANDRA-13271) Reduce lock contention for collection types and serializers

[Robert Stupp (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Stupp resolved CASSANDRA-13271.
--
   Resolution: Fixed
Fix Version/s: (was: 4.x)
   4.0

Thanks!

Committed as 
[b9b63925f9b818637b986a300aee658df03a5500|https://github.com/apache/cassandra/commit/b9b63925f9b818637b986a300aee658df03a5500]
 to [trunk|https://github.com/apache/cassandra/tree/trunk]


> Reduce lock contention for collection types and serializers
> ---
>
> Key: CASSANDRA-13271
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13271
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Core
>Reporter: vincent royer
>Priority: Minor
>  Labels: performance
> Fix For: 4.0
>
> Attachments: 0001-CASSANDRA-13271-computeIfAbsent.patch, 
> 0001-CASSANDRA-13271-singleton-factory-concurrency-opimiz.patch
>
>
> By doing some performance tests, i noticed that getInstance() in 
> org.apache.cassandra.db.marshal.ListType and SetType could suffer from lock 
> contention on the singleton factory getInstance(). Here is a proposal to 
> reduce lock contention by using a ConcurrentMap and the putIfAbsent method 
> rather than a synchronized method.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

cassandra git commit: Reduce lock contention for collection types and serializers

[snazy]

Repository: cassandra
Updated Branches:
  refs/heads/trunk a49cf2c6c -> b9b63925f


Reduce lock contention for collection types and serializers

patch by vincent royer; reviewed by Robert Stupp for CASSANDRA-13271


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/b9b63925
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/b9b63925
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/b9b63925

Branch: refs/heads/trunk
Commit: b9b63925f9b818637b986a300aee658df03a5500
Parents: a49cf2c
Author: vincent royer 
Authored: Fri Mar 3 19:02:25 2017 +0100
Committer: Robert Stupp 
Committed: Fri Mar 3 19:02:25 2017 +0100

--
 CHANGES.txt  |  1 +
 .../cassandra/db/marshal/DynamicCompositeType.java   | 11 +--
 .../org/apache/cassandra/db/marshal/ListType.java| 15 +++
 .../org/apache/cassandra/db/marshal/MapType.java | 15 +++
 .../org/apache/cassandra/db/marshal/SetType.java | 15 +++
 .../apache/cassandra/serializers/ListSerializer.java | 11 +--
 .../apache/cassandra/serializers/MapSerializer.java  | 11 +--
 .../apache/cassandra/serializers/SetSerializer.java  | 11 +--
 8 files changed, 42 insertions(+), 48 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/b9b63925/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 9d07956..0ac9d50 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 4.0
+ * Reduce lock contention for collection types and serializers 
(CASSANDRA-13271)
  * Make it possible to override MessagingService.Verb ids (CASSANDRA-13283)
  * Avoid synchronized on prepareForRepair in ActiveRepairService 
(CASSANDRA-9292)
  * Adds the ability to use uncompressed chunks in compressed files 
(CASSANDRA-10520)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/b9b63925/src/java/org/apache/cassandra/db/marshal/DynamicCompositeType.java
--
diff --git a/src/java/org/apache/cassandra/db/marshal/DynamicCompositeType.java 
b/src/java/org/apache/cassandra/db/marshal/DynamicCompositeType.java
index d314bd9..cade725 100644
--- a/src/java/org/apache/cassandra/db/marshal/DynamicCompositeType.java
+++ b/src/java/org/apache/cassandra/db/marshal/DynamicCompositeType.java
@@ -21,6 +21,8 @@ import java.nio.charset.CharacterCodingException;
 import java.nio.ByteBuffer;
 import java.util.HashMap;
 import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
 
 import org.apache.cassandra.cql3.Term;
 import org.slf4j.Logger;
@@ -59,21 +61,18 @@ public class DynamicCompositeType extends 
AbstractCompositeType
 private final Map aliases;
 
 // interning instances
-private static final Map, DynamicCompositeType> 
instances = new HashMap, DynamicCompositeType>();
+private static final ConcurrentMap, 
DynamicCompositeType> instances = new ConcurrentHashMap, DynamicCompositeType>();
 
 public static synchronized DynamicCompositeType getInstance(TypeParser 
parser) throws ConfigurationException, SyntaxException
 {
 return getInstance(parser.getAliasParameters());
 }
 
-public static synchronized DynamicCompositeType getInstance(Map aliases)
+public static DynamicCompositeType getInstance(Map 
aliases)
 {
 DynamicCompositeType dct = instances.get(aliases);
 if (dct == null)
-{
-dct = new DynamicCompositeType(aliases);
-instances.put(aliases, dct);
-}
+dct = instances.computeIfAbsent(aliases, k ->  new 
DynamicCompositeType(k));
 return dct;
 }
 

http://git-wip-us.apache.org/repos/asf/cassandra/blob/b9b63925/src/java/org/apache/cassandra/db/marshal/ListType.java
--
diff --git a/src/java/org/apache/cassandra/db/marshal/ListType.java 
b/src/java/org/apache/cassandra/db/marshal/ListType.java
index e4c20e2..29ccaa5 100644
--- a/src/java/org/apache/cassandra/db/marshal/ListType.java
+++ b/src/java/org/apache/cassandra/db/marshal/ListType.java
@@ -19,6 +19,8 @@ package org.apache.cassandra.db.marshal;
 
 import java.nio.ByteBuffer;
 import java.util.*;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
 
 import org.apache.cassandra.cql3.Json;
 import org.apache.cassandra.cql3.Lists;
@@ -39,8 +41,8 @@ public class ListType extends CollectionType
 

[jira] [Updated] (CASSANDRA-13271) Reduce lock contention for collection types and serializers

[Robert Stupp (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Stupp updated CASSANDRA-13271:
-
Summary: Reduce lock contention for collection types and serializers  (was: 
Reduce lock contention on instance factories of ListType and SetType)

> Reduce lock contention for collection types and serializers
> ---
>
> Key: CASSANDRA-13271
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13271
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Core
>Reporter: vincent royer
>Priority: Minor
>  Labels: performance
> Fix For: 4.x
>
> Attachments: 0001-CASSANDRA-13271-computeIfAbsent.patch, 
> 0001-CASSANDRA-13271-singleton-factory-concurrency-opimiz.patch
>
>
> By doing some performance tests, i noticed that getInstance() in 
> org.apache.cassandra.db.marshal.ListType and SetType could suffer from lock 
> contention on the singleton factory getInstance(). Here is a proposal to 
> reduce lock contention by using a ConcurrentMap and the putIfAbsent method 
> rather than a synchronized method.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (CASSANDRA-13291) Replace usages of MessageDigest with Guava's Hasher

[Michael Kjellman (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15894759#comment-15894759
 ] 

Michael Kjellman commented on CASSANDRA-13291:
--

>From Hashing.java
{code}
private static class Md5Holder {
static final HashFunction MD5 = new MessageDigestHashFunction("MD5", 
"Hashing.md5()");

private Md5Holder() {
}
}
{code}

And then from MessageDigestHashFunction.java
{code}
private static MessageDigest getMessageDigest(String algorithmName) {
try {
return MessageDigest.getInstance(algorithmName);
} catch (NoSuchAlgorithmException var2) {
throw new AssertionError(var2);
}
}
{code}

> Replace usages of MessageDigest with Guava's Hasher
> ---
>
> Key: CASSANDRA-13291
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13291
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Core
>Reporter: Michael Kjellman
>Assignee: Michael Kjellman
> Attachments: CASSANDRA-13291-trunk.diff
>
>
> During my profiling of C* I frequently see lots of aggregate time across 
> threads being spent inside the MD5 MessageDigest implementation. Given that 
> there are tons of modern alternative hashing functions better than MD5 
> available -- both in terms of providing better collision resistance and 
> actual computational speed -- I wanted to switch out our usage of MD5 for 
> alternatives (like adler128 or murmur3_128) and test for performance 
> improvements.
> Unfortunately, I found given the fact we use MessageDigest everywhere --  
> switching out the hashing function to something like adler128 or murmur3_128 
> (for example) -- which don't ship with the JDK --  wasn't straight forward.
> The goal of this ticket is to propose switching out usages of MessageDigest 
> directly in favor of Hasher from Guava. This means going forward we can 
> change a single line of code to switch the hashing algorithm being used 
> (assuming there is an implementation in Guava).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (CASSANDRA-13291) Replace usages of MessageDigest with Guava's Hasher

[Robert Stupp (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15894754#comment-15894754
 ] 

Robert Stupp commented on CASSANDRA-13291:
--

Fair enough. Thought Guava has an alternative MD5 implementation.

> Replace usages of MessageDigest with Guava's Hasher
> ---
>
> Key: CASSANDRA-13291
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13291
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Core
>Reporter: Michael Kjellman
>Assignee: Michael Kjellman
> Attachments: CASSANDRA-13291-trunk.diff
>
>
> During my profiling of C* I frequently see lots of aggregate time across 
> threads being spent inside the MD5 MessageDigest implementation. Given that 
> there are tons of modern alternative hashing functions better than MD5 
> available -- both in terms of providing better collision resistance and 
> actual computational speed -- I wanted to switch out our usage of MD5 for 
> alternatives (like adler128 or murmur3_128) and test for performance 
> improvements.
> Unfortunately, I found given the fact we use MessageDigest everywhere --  
> switching out the hashing function to something like adler128 or murmur3_128 
> (for example) -- which don't ship with the JDK --  wasn't straight forward.
> The goal of this ticket is to propose switching out usages of MessageDigest 
> directly in favor of Hasher from Guava. This means going forward we can 
> change a single line of code to switch the hashing algorithm being used 
> (assuming there is an implementation in Guava).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

cassandra-builds git commit: Fix ant target in dev branch job

[mshuler]

Repository: cassandra-builds
Updated Branches:
  refs/heads/master 481db1991 -> 0c4fc4713


Fix ant target in dev branch job


Project: http://git-wip-us.apache.org/repos/asf/cassandra-builds/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra-builds/commit/0c4fc471
Tree: http://git-wip-us.apache.org/repos/asf/cassandra-builds/tree/0c4fc471
Diff: http://git-wip-us.apache.org/repos/asf/cassandra-builds/diff/0c4fc471

Branch: refs/heads/master
Commit: 0c4fc4713582ac4d624198620373d89378fd365d
Parents: 481db19
Author: Michael Shuler 
Authored: Fri Mar 3 11:36:32 2017 -0600
Committer: Michael Shuler 
Committed: Fri Mar 3 11:36:32 2017 -0600

--
 jenkins-dsl/cassandra_job_dsl_seed.groovy | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra-builds/blob/0c4fc471/jenkins-dsl/cassandra_job_dsl_seed.groovy
--
diff --git a/jenkins-dsl/cassandra_job_dsl_seed.groovy 
b/jenkins-dsl/cassandra_job_dsl_seed.groovy
index a390ba4..857ea0c 100644
--- a/jenkins-dsl/cassandra_job_dsl_seed.groovy
+++ b/jenkins-dsl/cassandra_job_dsl_seed.groovy
@@ -260,7 +260,7 @@ cassandraBranches.each {
 
 
 /**
- * Parameterized Dev Branch `ant testall`
+ * Parameterized Dev Branch `ant test-all`
  */
 job('Cassandra-devbranch-testall') {
 description(jobDescription)
@@ -295,7 +295,7 @@ job('Cassandra-devbranch-testall') {
 steps {
 buildDescription('', buildDescStr)
 shell("git clean -xdff ; git clone ${buildsRepo}")
-shell('./cassandra-builds/build-scripts/cassandra-unittest.sh testall')
+shell('./cassandra-builds/build-scripts/cassandra-unittest.sh 
test-all')
 }
 publishers {
 junit {

[jira] [Commented] (CASSANDRA-13291) Replace usages of MessageDigest with Guava's Hasher

[Michael Kjellman (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15894744#comment-15894744
 ] 

Michael Kjellman commented on CASSANDRA-13291:
--

[~snazy] I haven't benchmarked trunk with this patch -- however Guava 
internally for the md5 implementation calls the exact same thing we were 
calling -- MessageDigest to get an instance of md5 -- so it should be identical 
baring the additional overhead for the allocation of the Guava Hasher object 
wrapper.

CASSANDRA-13292 is where we would need to do some benchmarking.

> Replace usages of MessageDigest with Guava's Hasher
> ---
>
> Key: CASSANDRA-13291
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13291
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Core
>Reporter: Michael Kjellman
>Assignee: Michael Kjellman
> Attachments: CASSANDRA-13291-trunk.diff
>
>
> During my profiling of C* I frequently see lots of aggregate time across 
> threads being spent inside the MD5 MessageDigest implementation. Given that 
> there are tons of modern alternative hashing functions better than MD5 
> available -- both in terms of providing better collision resistance and 
> actual computational speed -- I wanted to switch out our usage of MD5 for 
> alternatives (like adler128 or murmur3_128) and test for performance 
> improvements.
> Unfortunately, I found given the fact we use MessageDigest everywhere --  
> switching out the hashing function to something like adler128 or murmur3_128 
> (for example) -- which don't ship with the JDK --  wasn't straight forward.
> The goal of this ticket is to propose switching out usages of MessageDigest 
> directly in favor of Hasher from Guava. This means going forward we can 
> change a single line of code to switch the hashing algorithm being used 
> (assuming there is an implementation in Guava).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (CASSANDRA-13294) Possible data loss on upgrade 2.1 - 3.0

[Marcus Eriksson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Marcus Eriksson updated CASSANDRA-13294:

Status: Patch Available  (was: Open)

patch that appends the separator when getting sstables: 
https://github.com/krummas/cassandra/commits/marcuse/13294
http://cassci.datastax.com/view/Dev/view/krummas/job/krummas-marcuse-13294-testall/
http://cassci.datastax.com/view/Dev/view/krummas/job/krummas-marcuse-13294-dtest/

Needs upgrade dtest as well, will work on that

> Possible data loss on upgrade 2.1 - 3.0
> ---
>
> Key: CASSANDRA-13294
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13294
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Marcus Eriksson
>Assignee: Marcus Eriksson
>Priority: Blocker
> Fix For: 3.0.x
>
>
> After finishing a compaction we delete the compacted away files. This is done 
> [here|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/db/lifecycle/LogFile.java#L328-L337]
>  which uses 
> [this|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/db/lifecycle/LogRecord.java#L265-L271]
>  to get the files - we get all files starting with {{absoluteFilePath}}. 
> Absolute file path is generated 
> [here|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/io/sstable/Descriptor.java#L142-L153].
>  For 3.0 version files the filename looks like this: 
> {{/blabla/keyspace1/standard1-bdb031c0ff7b11e6940fdd0479dd8912/mc-1332-big}} 
> but for 2.1 version files, they look like this: 
> {{/blabla/keyspace1/standard1-bdb031c0ff7b11e6940fdd0479dd8912/keyspace1-standard1-ka-2}}.
> The problem is then that if we were to finish a compaction including the 
> legacy file, we would actually delete all legacy files having a generation 
> starting with '2'



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

cassandra-builds git commit: Add Parameterized Dev Branch Jobs to DSL

[mshuler]

Repository: cassandra-builds
Updated Branches:
  refs/heads/master 2fff2e6b1 -> 481db1991


Add Parameterized Dev Branch Jobs to DSL


Project: http://git-wip-us.apache.org/repos/asf/cassandra-builds/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra-builds/commit/481db199
Tree: http://git-wip-us.apache.org/repos/asf/cassandra-builds/tree/481db199
Diff: http://git-wip-us.apache.org/repos/asf/cassandra-builds/diff/481db199

Branch: refs/heads/master
Commit: 481db1991136b05b5b6d3ec90da9183a93259df8
Parents: 2fff2e6
Author: Michael Shuler 
Authored: Fri Mar 3 11:18:37 2017 -0600
Committer: Michael Shuler 
Committed: Fri Mar 3 11:18:37 2017 -0600

--
 jenkins-dsl/cassandra_job_dsl_seed.groovy | 110 -
 1 file changed, 109 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra-builds/blob/481db199/jenkins-dsl/cassandra_job_dsl_seed.groovy
--
diff --git a/jenkins-dsl/cassandra_job_dsl_seed.groovy 
b/jenkins-dsl/cassandra_job_dsl_seed.groovy
index 4b1dbdd..a390ba4 100644
--- a/jenkins-dsl/cassandra_job_dsl_seed.groovy
+++ b/jenkins-dsl/cassandra_job_dsl_seed.groovy
@@ -15,6 +15,7 @@ def buildDescStr = 'REF = ${GIT_BRANCH}  COMMIT = 
${GIT_COMMIT}'
 def cassandraBranches = ['cassandra-2.2', 'cassandra-3.0', 'cassandra-3.11', 
'trunk']
 // Ant test targets
 def testTargets = ['test', 'test-all', 'test-burn', 'test-cdc', 
'test-compression']
+// Dtest test targets
 def dtestTargets = ['dtest', 'dtest-novnode', 'dtest-offheap']  // dtest-large 
target exists, but no large servers to run on..
 
 
@@ -250,6 +251,113 @@ cassandraBranches.each {
 }
 }
 }
+}
+
+
+//
+// Parameterized Dev Branch Job Definitions
+//
+
 
-// The End.
+/**
+ * Parameterized Dev Branch `ant testall`
+ */
+job('Cassandra-devbranch-testall') {
+description(jobDescription)
+jdk(jdkLabel)
+label(slaveLabel)
+logRotator {
+numToKeep(10)
+}
+wrappers {
+timeout {
+noActivity(1200)
+}
+}
+throttleConcurrentBuilds {
+categories(['Cassandra'])
+}
+parameters {
+stringParam('REPO', 'apache', 'The github user/org to clone cassandra 
repo from')
+stringParam('BRANCH', 'trunk', 'The branch of cassandra to checkout')
+}
+scm {
+git {
+remote {
+url('https://github.com/${REPO}/cassandra.git')
+}
+branch('${BRANCH}')
+extensions {
+cleanAfterCheckout()
+}
+}
+}
+steps {
+buildDescription('', buildDescStr)
+shell("git clean -xdff ; git clone ${buildsRepo}")
+shell('./cassandra-builds/build-scripts/cassandra-unittest.sh testall')
+}
+publishers {
+junit {
+testResults('**/TEST-*.xml')
+testDataPublishers {
+stabilityTestDataPublisher()
+}
+}
+postBuildTask {
+task('.', 'echo "Finding job process orphans.."; if pgrep -af 
${JOB_BASE_NAME}; then pkill -9 -f ${JOB_BASE_NAME}; fi')
+}
+}
+}
+
+/**
+ * Parameterized Dev Branch dtest
+ */
+job('Cassandra-devbranch-dtest') {
+description(jobDescription)
+jdk(jdkLabel)
+label(slaveLabel)
+logRotator {
+numToKeep(10)
+}
+wrappers {
+timeout {
+noActivity(1200)
+}
+}
+throttleConcurrentBuilds {
+categories(['Cassandra'])
+}
+parameters {
+stringParam('REPO', 'apache', 'The github user/org to clone cassandra 
repo from')
+stringParam('BRANCH', 'trunk', 'The branch of cassandra to checkout')
+}
+scm {
+git {
+remote {
+url('https://github.com/${REPO}/cassandra.git')
+}
+branch('${BRANCH}')
+extensions {
+cleanAfterCheckout()
+}
+}
+}
+steps {
+buildDescription('', buildDescStr)
+shell("git clean -xdff ; git clone ${buildsRepo} ; git clone 
${dtestRepo}")
+shell('./cassandra-builds/build-scripts/cassandra-dtest.sh')
+}
+publishers {
+archiveArtifacts('test_stdout.txt')
+junit {
+testResults('cassandra-dtest/nosetests.xml')
+testDataPublishers {
+stabilityTestDataPublisher()
+}
+}
+postBuildTask {
+task('.', 'echo "Finding job process orphans.."; if pgrep -af 
${JOB_BASE_NAME}; then pkill -9 -f ${JOB_BASE_NAME}; fi')
+

[jira] [Updated] (CASSANDRA-13234) Add histogram for delay to deliver hints

[Jeff Jirsa (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeff Jirsa updated CASSANDRA-13234:
---
Fix Version/s: (was: 3.11.x)
   (was: 3.0.x)
   4.0

> Add histogram for delay to deliver hints
> 
>
> Key: CASSANDRA-13234
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13234
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Observability
>Reporter: Jeff Jirsa
>Assignee: Jeff Jirsa
>Priority: Minor
> Fix For: 4.0
>
>
> There is very little visibility into hint delivery in general - having 
> histograms available to understand how long it takes to deliver hints is 
> useful for operators to better identify problems. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (CASSANDRA-13271) Reduce lock contention on instance factories of ListType and SetType

[vincent royer (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15894716#comment-15894716
 ] 

vincent royer commented on CASSANDRA-13271:
---

Sorry, this fixed in https://github.com/strapdata/cassandra/tree/13271-trunk

> Reduce lock contention on instance factories of ListType and SetType
> 
>
> Key: CASSANDRA-13271
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13271
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Core
>Reporter: vincent royer
>Priority: Minor
>  Labels: performance
> Fix For: 4.x
>
> Attachments: 0001-CASSANDRA-13271-computeIfAbsent.patch, 
> 0001-CASSANDRA-13271-singleton-factory-concurrency-opimiz.patch
>
>
> By doing some performance tests, i noticed that getInstance() in 
> org.apache.cassandra.db.marshal.ListType and SetType could suffer from lock 
> contention on the singleton factory getInstance(). Here is a proposal to 
> reduce lock contention by using a ConcurrentMap and the putIfAbsent method 
> rather than a synchronized method.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (CASSANDRA-12888) Incremental repairs broken for MVs and CDC

[Benjamin Roth (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15894645#comment-15894645
 ] 

Benjamin Roth commented on CASSANDRA-12888:
---

For detailed explanation an excerpt from that discussion:
-

... there are still possible scenarios where it's possible to break consistency 
by repairing the base and the view separately even with QUORUM writes:

Initial state:

Base replica A: {k0=v0, ts=0}
Base replica B: {k0=v0, ts=0}
Base replica C: {k0=v0, ts=0}
View paired replica A: {v1=k0, ts=0}
View paired replica B: {v0=k0, ts=0}
View paired replica C: {v0=k0, ts=0}

Base replica A receives write {k1=v1, ts=1}, propagates to view paired replica 
A and dies.

Current state is:
Base replica A: {k1=v1, ts=1}
Base replica B: {k0=v0, ts=0}
Base replica C: {k0=v0, ts=0}
View paired replica A: {v1=k1, ts=1}
View paired replica B: {v0=k0, ts=0}
View paired replica C: {v0=k0, ts=0}

Base replica B and C receives write {k2=v2, ts=2}, write to their paired 
replica. Write is successful at QUORUM.

Current state is:
Base replica A: {k1=v1, ts=1}
Base replica B: {k2=v2, ts=2}
Base replica C: {k2=v2, ts=2}
View paired replica A: {v1=k1, ts=1}
View paired replica B: {v2=k2, ts=2}
View paired replica C: {v2=k2, ts=2}

A returns from the dead. Repair base table:
Base replica A: {k2=v2, ts=2}
Base replica B: {k2=v2, ts=2}
Base replica C: {k2=v2, ts=2}

Repair MV:
View paired replica A: {v1=k1, ts=1} and {v2=k2, ts=2}
View paired replica B: {v1=k1, ts=1} and {v2=k2, ts=2}
View paired replica C: {v1=k1, ts=1} and {v2=k2, ts=2}

So, this requires replica A to generate a tombstone for {v1=k1, ts=1} during 
repair of base table.

> Incremental repairs broken for MVs and CDC
> --
>
> Key: CASSANDRA-12888
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12888
> Project: Cassandra
>  Issue Type: Bug
>  Components: Streaming and Messaging
>Reporter: Stefan Podkowinski
>Assignee: Benjamin Roth
>Priority: Critical
> Fix For: 3.0.x, 3.11.x
>
>
> SSTables streamed during the repair process will first be written locally and 
> afterwards either simply added to the pool of existing sstables or, in case 
> of existing MVs or active CDC, replayed on mutation basis:
> As described in {{StreamReceiveTask.OnCompletionRunnable}}:
> {quote}
> We have a special path for views and for CDC.
> For views, since the view requires cleaning up any pre-existing state, we 
> must put all partitions through the same write path as normal mutations. This 
> also ensures any 2is are also updated.
> For CDC-enabled tables, we want to ensure that the mutations are run through 
> the CommitLog so they can be archived by the CDC process on discard.
> {quote}
> Using the regular write path turns out to be an issue for incremental 
> repairs, as we loose the {{repaired_at}} state in the process. Eventually the 
> streamed rows will end up in the unrepaired set, in contrast to the rows on 
> the sender site moved to the repaired set. The next repair run will stream 
> the same data back again, causing rows to bounce on and on between nodes on 
> each repair.
> See linked dtest on steps to reproduce. An example for reproducing this 
> manually using ccm can be found 
> [here|https://gist.github.com/spodkowinski/2d8e0408516609c7ae701f2bf1e515e8]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (CASSANDRA-12888) Incremental repairs broken for MVs and CDC

[Benjamin Roth (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15894639#comment-15894639
 ] 

Benjamin Roth commented on CASSANDRA-12888:
---

A repair must go through the write path expect for some special cases. I also 
first had the idea to avoid it completely but in discussion with 
[~pauloricardomg] it turned out that this may introduce inconsistencies that 
these could only be fixed by a view rebuild because it leaves stale rows.
I know that all this stuff is totally counter-intuitive but just streaming 
"blindly" all sstables (incl. MV tables) down is not correct. This is why I am 
trying to improve the mutation based approached.

If the Sstables for MVs get corrupted or lost, the only way to fix it is to 
rebuild that view again. There is no way (at least none I see atm) that would 
consistenly repair a view from other nodes.

The underlying principle is:
- A view must always be consistent to its base-table
- A view does not have to be consistent among nodes, thats handled by repairing 
the base table

Thats also why you don't have to run a repair before building a view. 
Nevertheless it would not help anyway because you NEVER have a 100% guaranteed 
consistent state. A repair only guarantees consistency until the point of 
repair.

The "know what you are doing" option is offered by CASSANDRA-13066 btw. 
In this ticket I also adopted the election of CFs (tables + mvs) when doing a 
keyspace repair depending if the MV is repaired by stream or by mutation.

> Incremental repairs broken for MVs and CDC
> --
>
> Key: CASSANDRA-12888
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12888
> Project: Cassandra
>  Issue Type: Bug
>  Components: Streaming and Messaging
>Reporter: Stefan Podkowinski
>Assignee: Benjamin Roth
>Priority: Critical
> Fix For: 3.0.x, 3.11.x
>
>
> SSTables streamed during the repair process will first be written locally and 
> afterwards either simply added to the pool of existing sstables or, in case 
> of existing MVs or active CDC, replayed on mutation basis:
> As described in {{StreamReceiveTask.OnCompletionRunnable}}:
> {quote}
> We have a special path for views and for CDC.
> For views, since the view requires cleaning up any pre-existing state, we 
> must put all partitions through the same write path as normal mutations. This 
> also ensures any 2is are also updated.
> For CDC-enabled tables, we want to ensure that the mutations are run through 
> the CommitLog so they can be archived by the CDC process on discard.
> {quote}
> Using the regular write path turns out to be an issue for incremental 
> repairs, as we loose the {{repaired_at}} state in the process. Eventually the 
> streamed rows will end up in the unrepaired set, in contrast to the rows on 
> the sender site moved to the repaired set. The next repair run will stream 
> the same data back again, causing rows to bounce on and on between nodes on 
> each repair.
> See linked dtest on steps to reproduce. An example for reproducing this 
> manually using ccm can be found 
> [here|https://gist.github.com/spodkowinski/2d8e0408516609c7ae701f2bf1e515e8]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (CASSANDRA-12888) Incremental repairs broken for MVs and CDC

[Stefan Podkowinski (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15894617#comment-15894617
 ] 

Stefan Podkowinski commented on CASSANDRA-12888:


If you build from an inconsistent node and some missing data never makes it 
into a view, regular repairs alone won't be able to fix that. Is that your 
point? Although I see that this could happen, I'd instead still prefer to just 
tell the user to do a full repair before creating a view, if that would be a 
big issue.

> Incremental repairs broken for MVs and CDC
> --
>
> Key: CASSANDRA-12888
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12888
> Project: Cassandra
>  Issue Type: Bug
>  Components: Streaming and Messaging
>Reporter: Stefan Podkowinski
>Assignee: Benjamin Roth
>Priority: Critical
> Fix For: 3.0.x, 3.11.x
>
>
> SSTables streamed during the repair process will first be written locally and 
> afterwards either simply added to the pool of existing sstables or, in case 
> of existing MVs or active CDC, replayed on mutation basis:
> As described in {{StreamReceiveTask.OnCompletionRunnable}}:
> {quote}
> We have a special path for views and for CDC.
> For views, since the view requires cleaning up any pre-existing state, we 
> must put all partitions through the same write path as normal mutations. This 
> also ensures any 2is are also updated.
> For CDC-enabled tables, we want to ensure that the mutations are run through 
> the CommitLog so they can be archived by the CDC process on discard.
> {quote}
> Using the regular write path turns out to be an issue for incremental 
> repairs, as we loose the {{repaired_at}} state in the process. Eventually the 
> streamed rows will end up in the unrepaired set, in contrast to the rows on 
> the sender site moved to the repaired set. The next repair run will stream 
> the same data back again, causing rows to bounce on and on between nodes on 
> each repair.
> See linked dtest on steps to reproduce. An example for reproducing this 
> manually using ccm can be found 
> [here|https://gist.github.com/spodkowinski/2d8e0408516609c7ae701f2bf1e515e8]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (CASSANDRA-12888) Incremental repairs broken for MVs and CDC

[T Jake Luciani (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15894542#comment-15894542
 ] 

T Jake Luciani commented on CASSANDRA-12888:


Just to restate the reason we need to put MV base table mutations through the 
regular write path (vs just streaming the MV and the base) from CASSANDRA-6477
bq. Let's say you put non-PK columns A and B into a Materialized view. Replica 
1 has All of Column A, Replica 2 has All of column B. The build would end up 
with no data in the MV. You would need to subsequentally repair the data to 
build the MV. 



> Incremental repairs broken for MVs and CDC
> --
>
> Key: CASSANDRA-12888
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12888
> Project: Cassandra
>  Issue Type: Bug
>  Components: Streaming and Messaging
>Reporter: Stefan Podkowinski
>Assignee: Benjamin Roth
>Priority: Critical
> Fix For: 3.0.x, 3.11.x
>
>
> SSTables streamed during the repair process will first be written locally and 
> afterwards either simply added to the pool of existing sstables or, in case 
> of existing MVs or active CDC, replayed on mutation basis:
> As described in {{StreamReceiveTask.OnCompletionRunnable}}:
> {quote}
> We have a special path for views and for CDC.
> For views, since the view requires cleaning up any pre-existing state, we 
> must put all partitions through the same write path as normal mutations. This 
> also ensures any 2is are also updated.
> For CDC-enabled tables, we want to ensure that the mutations are run through 
> the CommitLog so they can be archived by the CDC process on discard.
> {quote}
> Using the regular write path turns out to be an issue for incremental 
> repairs, as we loose the {{repaired_at}} state in the process. Eventually the 
> streamed rows will end up in the unrepaired set, in contrast to the rows on 
> the sender site moved to the repaired set. The next repair run will stream 
> the same data back again, causing rows to bounce on and on between nodes on 
> each repair.
> See linked dtest on steps to reproduce. An example for reproducing this 
> manually using ccm can be found 
> [here|https://gist.github.com/spodkowinski/2d8e0408516609c7ae701f2bf1e515e8]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (CASSANDRA-13295) test failure in upgrade_tests.upgrade_through_versions_test.TestUpgrade_current_2_1_x_To_indev_3_x.bootstrap_test

[Michael Shuler (JIRA)]

Michael Shuler created CASSANDRA-13295:
--

 Summary: test failure in 
upgrade_tests.upgrade_through_versions_test.TestUpgrade_current_2_1_x_To_indev_3_x.bootstrap_test
 Key: CASSANDRA-13295
 URL: https://issues.apache.org/jira/browse/CASSANDRA-13295
 Project: Cassandra
  Issue Type: Bug
Reporter: Michael Shuler
 Attachments: node1_debug.log, node1_gc.log, node1.log, 
node2_debug.log, node2_gc.log, node2.log, node3_debug.log, node3_gc.log, 
node3.log, node4_debug.log, node4_gc.log, node4.log

example failure:

http://cassci.datastax.com/job/cassandra-3.11_large_dtest/22/testReport/upgrade_tests.upgrade_through_versions_test/TestUpgrade_current_2_1_x_To_indev_3_x/bootstrap_test

{code}
Error Message

03 Mar 2017 12:58:17 [node2] Missing: ['127.0.0.4.* now UP']:
INFO  [HANDSHAKE-/127.0.0.4] 2017-03-03 12:54:12,1.
See system.log for remainder

Stacktrace

  File "/usr/lib/python2.7/unittest/case.py", line 329, in run
testMethod()
  File 
"/home/automaton/cassandra-dtest/upgrade_tests/upgrade_through_versions_test.py",
 line 700, in bootstrap_test
self.upgrade_scenario(after_upgrade_call=(self._bootstrap_new_node,))
  File 
"/home/automaton/cassandra-dtest/upgrade_tests/upgrade_through_versions_test.py",
 line 376, in upgrade_scenario
call()
  File 
"/home/automaton/cassandra-dtest/upgrade_tests/upgrade_through_versions_test.py",
 line 681, in _bootstrap_new_node
nnode.start(use_jna=True, wait_other_notice=240, wait_for_binary_proto=True)
  File "/home/automaton/venv/local/lib/python2.7/site-packages/ccmlib/node.py", 
line 686, in start
node.watch_log_for_alive(self, from_mark=mark, timeout=wait_other_notice)
  File "/home/automaton/venv/local/lib/python2.7/site-packages/ccmlib/node.py", 
line 503, in watch_log_for_alive
self.watch_log_for(tofind, from_mark=from_mark, timeout=timeout, 
filename=filename)
  File "/home/automaton/venv/local/lib/python2.7/site-packages/ccmlib/node.py", 
line 471, in watch_log_for
raise TimeoutError(time.strftime("%d %b %Y %H:%M:%S", time.gmtime()) + " [" 
+ self.name + "] Missing: " + str([e.pattern for e in tofind]) + ":\n" + 
reads[:50] + ".\nSee {} for remainder".format(filename))
{code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (CASSANDRA-12888) Incremental repairs broken for MVs and CDC

[Stefan Podkowinski (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15894495#comment-15894495
 ] 

Stefan Podkowinski commented on CASSANDRA-12888:


Before moving on in the discussion, I think [~pauloricardomg]'s suggestion 
still stands and deserves some more thoughts:

bq. As discussed on the mailing list and CASSANDRA-12905 we cannot ensure view 
consistency with sstable-based streaming of base/MVs in all scenarios, 
specially with repair, so I'm afraid this is not a viable solution just yet 
(needs to be further discussed). An alternative is to provide an option for 
repair to allow repairing base and MV separately when you know what you are 
doing©.

Actually having to repair both, base and view table to get back into a 
consistent state, is something I would expect as a user and something that 
would happen anyway during my regular repairs, whereas the mutation based 
approach is probably unheard of by most users and totally unexpected, given how 
we handled anti-entropy in the past. Although we talk about "views", I think 
most users will understand that these are actually separate tables and should 
be handled as that from a operational perspective. So do we really want to 
introduce additional complexity that may bite us in the back at some point, 
instead of just relying on regular repairs?



> Incremental repairs broken for MVs and CDC
> --
>
> Key: CASSANDRA-12888
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12888
> Project: Cassandra
>  Issue Type: Bug
>  Components: Streaming and Messaging
>Reporter: Stefan Podkowinski
>Assignee: Benjamin Roth
>Priority: Critical
> Fix For: 3.0.x, 3.11.x
>
>
> SSTables streamed during the repair process will first be written locally and 
> afterwards either simply added to the pool of existing sstables or, in case 
> of existing MVs or active CDC, replayed on mutation basis:
> As described in {{StreamReceiveTask.OnCompletionRunnable}}:
> {quote}
> We have a special path for views and for CDC.
> For views, since the view requires cleaning up any pre-existing state, we 
> must put all partitions through the same write path as normal mutations. This 
> also ensures any 2is are also updated.
> For CDC-enabled tables, we want to ensure that the mutations are run through 
> the CommitLog so they can be archived by the CDC process on discard.
> {quote}
> Using the regular write path turns out to be an issue for incremental 
> repairs, as we loose the {{repaired_at}} state in the process. Eventually the 
> streamed rows will end up in the unrepaired set, in contrast to the rows on 
> the sender site moved to the repaired set. The next repair run will stream 
> the same data back again, causing rows to bounce on and on between nodes on 
> each repair.
> See linked dtest on steps to reproduce. An example for reproducing this 
> manually using ccm can be found 
> [here|https://gist.github.com/spodkowinski/2d8e0408516609c7ae701f2bf1e515e8]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (CASSANDRA-13223) Unable to compute when histogram overflowed

[Vladimir Bukhtoyarov (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15894447#comment-15894447
 ] 

Vladimir Bukhtoyarov commented on CASSANDRA-13223:
--

Yes I threat overflown overflow bucket as normal value.

{quote}
This could potentially end up with a flooded log files. 
{quote}
I want to speculate that logs should not be overflown because it is not really 
to execute big amount of very slow operations at rate which enough to flood. 
But I can implement simple non-blocking throttling, in order to log no often 
then *N* times per second(or other period). I suppose an implementation of rate 
limiter already exists in Cassandra code base, if not then implementation of 
throttling based on "Leaky Bucket" is very trivial.

{quote}
An option to consider would be to count the overflow bucket as Long.MAX_VALUE 
(or some other approximation between maximum register value and 
Long.MAX_VALUE). Just a thought.
{quote}
It is good idea. The anomaly big value on the graphs will be a very good alarm.




> Unable to compute when histogram overflowed
> ---
>
> Key: CASSANDRA-13223
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13223
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Vladimir Bukhtoyarov
>Priority: Minor
>
> DecayingEstimatedHistogramReservoir throws exception when value upper max 
> recorded to reservoir. It is very undesired behavior, because functionality 
> like logging or monitoring should never fail with exception. Current behavior 
> of DecayingEstimatedHistogramReservoir violates contract for 
> [Reservoir|https://github.com/dropwizard/metrics/blob/3.2-development/metrics-core/src/main/java/com/codahale/metrics/Reservoir.java],
>  as you can see javadocs for Reservoir says nothing that implementation can 
> throw exception in getSnapshot method. As result all Dropwizzard/Metrics 
> reporters are broken, because nobody expect that metric will throw exception 
> on get, for example our monitoring pipeline is broken with exception:
> {noformat}
> com.fasterxml.jackson.databind.JsonMappingException: Unable to compute when 
> histogram overflowed (through reference chain: 
> java.util.UnmodifiableSortedMap["org.apache.cassandra.metrics.Table
> .ColUpdateTimeDeltaHistogram.all"])
> at 
> com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:339)
> at 
> com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:299)
> at 
> com.fasterxml.jackson.databind.ser.std.StdSerializer.wrapAndThrow(StdSerializer.java:342)
> at 
> com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeFields(MapSerializer.java:620)
> at 
> com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:519)
> at 
> com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:31)
> at 
> com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:130)
> at 
> com.fasterxml.jackson.databind.ObjectMapper.writeValue(ObjectMapper.java:2436)
> at 
> com.fasterxml.jackson.core.base.GeneratorBase.writeObject(GeneratorBase.java:355)
> at 
> com.fasterxml.jackson.core.JsonGenerator.writeObjectField(JsonGenerator.java:1442)
> at 
> com.codahale.metrics.json.MetricsModule$MetricRegistrySerializer.serialize(MetricsModule.java:188)
> at 
> com.codahale.metrics.json.MetricsModule$MetricRegistrySerializer.serialize(MetricsModule.java:171)
> at 
> com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:130)
> at 
> com.fasterxml.jackson.databind.ObjectWriter$Prefetch.serialize(ObjectWriter.java:1428)
> at 
> com.fasterxml.jackson.databind.ObjectWriter._configAndWriteValue(ObjectWriter.java:1129)
> at 
> com.fasterxml.jackson.databind.ObjectWriter.writeValue(ObjectWriter.java:967)
> at 
> com.codahale.metrics.servlets.MetricsServlet.doGet(MetricsServlet.java:176)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
> at 
> org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:845)
> at 
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1689)
> at 
> com.ringcentral.slf4j.CleanMDCFilter.doFilter(CleanMDCFilter.java:18)
> at 
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
> at 
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
> at 
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
>  

[jira] [Commented] (CASSANDRA-13234) Add histogram for delay to deliver hints

[Aleksey Yeschenko (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15894277#comment-15894277
 ] 

Aleksey Yeschenko commented on CASSANDRA-13234:
---

Going by the book, all of this should go to trunk only, not being a bug fix, by 
the way.

> Add histogram for delay to deliver hints
> 
>
> Key: CASSANDRA-13234
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13234
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Observability
>Reporter: Jeff Jirsa
>Assignee: Jeff Jirsa
>Priority: Minor
> Fix For: 3.0.x, 3.11.x
>
>
> There is very little visibility into hint delivery in general - having 
> histograms available to understand how long it takes to deliver hints is 
> useful for operators to better identify problems. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (CASSANDRA-13289) Make it possible to monitor an ideal consistency level separate from actual consistency level

[Romain Hardouin (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15894255#comment-15894255
 ] 

Romain Hardouin commented on CASSANDRA-13289:
-

Very interesting! 
Do you have an idea of the overhead that this monitoring layer would add? If 
this adds too much overhead we should be able to enable/disable it at runtime 
via nodetool.

> Make it possible to monitor an ideal consistency level separate from actual 
> consistency level
> -
>
> Key: CASSANDRA-13289
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13289
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
>Reporter: Ariel Weisberg
>Assignee: Ariel Weisberg
>
> As an operator there are several issues related to multi-datacenter 
> replication and consistency you may want to have more information on from 
> your production database.
> For instance. If your application writes at LOCAL_QUORUM how often are those 
> writes failing to achieve EACH_QUORUM at other data centers. If you failed 
> your application over to one of those data centers roughly how inconsistent 
> might it be given the number of writes that didn't propagate since the last 
> incremental repair?
> You might also want to know roughly what the latency of writes would be if 
> you switched to a different consistency level. For instance you are writing 
> at LOCAL_QUORUM and want to know what would happen if you switched to 
> EACH_QUORUM.
> The proposed change is to allow an ideal_consistency_level to be specified in 
> cassandra.yaml as well as get/set via JMX. If no ideal consistency level is 
> specified no additional tracking is done.
> if an ideal consistency level is specified then the 
> {{AbstractWriteResponesHandler}} will contain a delegate WriteResponseHandler 
> that tracks whether the ideal consistency level is met before a write times 
> out. It also tracks the latency for achieving the ideal CL  of successful 
> writes.
> These two metrics would be reported on a per keyspace basis.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (CASSANDRA-13223) Unable to compute when histogram overflowed

[JIRA]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15894242#comment-15894242
 ] 

Per Otterström commented on CASSANDRA-13223:


bq. I do not ignore big latency record, I write the maximum value of resolution

Right, I missed that. Basically you are merging the count of the highest 
counting bucket with the overflow bucket?

It would be nice with some logging on this. The downside I see with that is 
that this code is part of many critical code paths, including write-path and 
read-path. This could potentially end up with a flooded log files.  

An option to consider would be to count the overflow bucket as Long.MAX_VALUE 
(or some other approximation between maximum register value and 
Long.MAX_VALUE). Just a thought. Then values would really stand out in graphs, 
and you would know where to look.

The existing solution with IllegalStateException is actually derived from 
[EstimatedHistogram|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/utils/EstimatedHistogram.java#L231].
 I don't know the full background story on the original behavior. 
[~brandon.williams], do you know if there is a particular reason for throwing 
the IllegalStateException, or would you be OK with the changes proposed here?


> Unable to compute when histogram overflowed
> ---
>
> Key: CASSANDRA-13223
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13223
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Vladimir Bukhtoyarov
>Priority: Minor
>
> DecayingEstimatedHistogramReservoir throws exception when value upper max 
> recorded to reservoir. It is very undesired behavior, because functionality 
> like logging or monitoring should never fail with exception. Current behavior 
> of DecayingEstimatedHistogramReservoir violates contract for 
> [Reservoir|https://github.com/dropwizard/metrics/blob/3.2-development/metrics-core/src/main/java/com/codahale/metrics/Reservoir.java],
>  as you can see javadocs for Reservoir says nothing that implementation can 
> throw exception in getSnapshot method. As result all Dropwizzard/Metrics 
> reporters are broken, because nobody expect that metric will throw exception 
> on get, for example our monitoring pipeline is broken with exception:
> {noformat}
> com.fasterxml.jackson.databind.JsonMappingException: Unable to compute when 
> histogram overflowed (through reference chain: 
> java.util.UnmodifiableSortedMap["org.apache.cassandra.metrics.Table
> .ColUpdateTimeDeltaHistogram.all"])
> at 
> com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:339)
> at 
> com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:299)
> at 
> com.fasterxml.jackson.databind.ser.std.StdSerializer.wrapAndThrow(StdSerializer.java:342)
> at 
> com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeFields(MapSerializer.java:620)
> at 
> com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:519)
> at 
> com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:31)
> at 
> com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:130)
> at 
> com.fasterxml.jackson.databind.ObjectMapper.writeValue(ObjectMapper.java:2436)
> at 
> com.fasterxml.jackson.core.base.GeneratorBase.writeObject(GeneratorBase.java:355)
> at 
> com.fasterxml.jackson.core.JsonGenerator.writeObjectField(JsonGenerator.java:1442)
> at 
> com.codahale.metrics.json.MetricsModule$MetricRegistrySerializer.serialize(MetricsModule.java:188)
> at 
> com.codahale.metrics.json.MetricsModule$MetricRegistrySerializer.serialize(MetricsModule.java:171)
> at 
> com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:130)
> at 
> com.fasterxml.jackson.databind.ObjectWriter$Prefetch.serialize(ObjectWriter.java:1428)
> at 
> com.fasterxml.jackson.databind.ObjectWriter._configAndWriteValue(ObjectWriter.java:1129)
> at 
> com.fasterxml.jackson.databind.ObjectWriter.writeValue(ObjectWriter.java:967)
> at 
> com.codahale.metrics.servlets.MetricsServlet.doGet(MetricsServlet.java:176)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
> at 
> org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:845)
> at 
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1689)
> at 
> com.ringcentral.slf4j.CleanMDCFilter.doFilter(CleanMDCFilter.java:18)
> at 
> 

[jira] [Commented] (CASSANDRA-13271) Reduce lock contention on instance factories of ListType and SetType

[Robert Stupp (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15894204#comment-15894204
 ] 

Robert Stupp commented on CASSANDRA-13271:
--

The change of the type of the parameter {{isMultiCell}} in 
{{ListType.getInstace()}} from primitive {{boolean}} to {{Boolean}} doesn't 
make sense.

||trunk|[branch|https://github.com/apache/cassandra/compare/trunk...snazy:13271-trunk]|[testall|http://cassci.datastax.com/view/Dev/view/snazy/job/snazy-13271-trunk-testall/lastSuccessfulBuild/]|[dtest|http://cassci.datastax.com/view/Dev/view/snazy/job/snazy-13271-trunk-dtest/lastSuccessfulBuild/]

When the above issue is resolved, I think we can commit this one.

> Reduce lock contention on instance factories of ListType and SetType
> 
>
> Key: CASSANDRA-13271
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13271
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Core
>Reporter: vincent royer
>Priority: Minor
>  Labels: performance
> Fix For: 4.x
>
> Attachments: 0001-CASSANDRA-13271-computeIfAbsent.patch, 
> 0001-CASSANDRA-13271-singleton-factory-concurrency-opimiz.patch
>
>
> By doing some performance tests, i noticed that getInstance() in 
> org.apache.cassandra.db.marshal.ListType and SetType could suffer from lock 
> contention on the singleton factory getInstance(). Here is a proposal to 
> reduce lock contention by using a ConcurrentMap and the putIfAbsent method 
> rather than a synchronized method.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (CASSANDRA-11670) Rebuilding or streaming MV generates mutations larger than max_mutation_size_in_kb

[Benjamin Roth (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15894128#comment-15894128
 ] 

Benjamin Roth commented on CASSANDRA-11670:
---

Hmmm I guess this possibly breaks consistency in repair streams + MVs
In StreamReceiveTasks mutations are applied without commitlog because cf is 
flushed at the end. But MVs are not flushed.

Either:
Also flush all MVs at the end of the stream task - it is not said that this is 
actually required for all MVs as we do not know where view replica updates 
eventually go.

Or:
Enable commitlog for view replica updates even if base table does not commit 
log writes.

> Rebuilding or streaming MV generates mutations larger than 
> max_mutation_size_in_kb
> --
>
> Key: CASSANDRA-11670
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11670
> Project: Cassandra
>  Issue Type: Bug
>  Components: Configuration, Streaming and Messaging
>Reporter: Anastasia Osintseva
>Assignee: Paulo Motta
> Fix For: 3.0.10, 3.10
>
>
> I have in cluster 2 DC, in each DC - 2 Nodes. I wanted to add 1 node to each 
> DC. One node has been added successfully after I had made scrubing. 
> Now I'm trying to add node to another DC, but get error: 
> org.apache.cassandra.streaming.StreamException: Stream failed. 
> After scrubing and repair I get the same error.  
> {noformat}
> ERROR [StreamReceiveTask:5] 2016-04-27 00:33:21,082 Keyspace.java:492 - 
> Unknown exception caught while attempting to update MaterializedView! 
> messages_dump.messages
> java.lang.IllegalArgumentException: Mutation of 34974901 bytes is too large 
> for the maxiumum size of 33554432
>   at org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:264) 
> ~[apache-cassandra-3.0.5.jar:3.0.5]
>   at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:469) 
> [apache-cassandra-3.0.5.jar:3.0.5]
>   at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:384) 
> [apache-cassandra-3.0.5.jar:3.0.5]
>   at org.apache.cassandra.db.Mutation.applyFuture(Mutation.java:205) 
> [apache-cassandra-3.0.5.jar:3.0.5]
>   at org.apache.cassandra.db.Mutation.apply(Mutation.java:217) 
> [apache-cassandra-3.0.5.jar:3.0.5]
>   at 
> org.apache.cassandra.batchlog.BatchlogManager.store(BatchlogManager.java:146) 
> ~[apache-cassandra-3.0.5.jar:3.0.5]
>   at 
> org.apache.cassandra.service.StorageProxy.mutateMV(StorageProxy.java:724) 
> ~[apache-cassandra-3.0.5.jar:3.0.5]
>   at 
> org.apache.cassandra.db.view.ViewManager.pushViewReplicaUpdates(ViewManager.java:149)
>  ~[apache-cassandra-3.0.5.jar:3.0.5]
>   at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:487) 
> [apache-cassandra-3.0.5.jar:3.0.5]
>   at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:384) 
> [apache-cassandra-3.0.5.jar:3.0.5]
>   at org.apache.cassandra.db.Mutation.applyFuture(Mutation.java:205) 
> [apache-cassandra-3.0.5.jar:3.0.5]
>   at org.apache.cassandra.db.Mutation.apply(Mutation.java:217) 
> [apache-cassandra-3.0.5.jar:3.0.5]
>   at org.apache.cassandra.db.Mutation.applyUnsafe(Mutation.java:236) 
> [apache-cassandra-3.0.5.jar:3.0.5]
>   at 
> org.apache.cassandra.streaming.StreamReceiveTask$OnCompletionRunnable.run(StreamReceiveTask.java:169)
>  [apache-cassandra-3.0.5.jar:3.0.5]
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
> [na:1.8.0_11]
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
> [na:1.8.0_11]
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  [na:1.8.0_11]
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [na:1.8.0_11]
>   at java.lang.Thread.run(Thread.java:745) [na:1.8.0_11]
> ERROR [StreamReceiveTask:5] 2016-04-27 00:33:21,082 
> StreamReceiveTask.java:214 - Error applying streamed data: 
> java.lang.IllegalArgumentException: Mutation of 34974901 bytes is too large 
> for the maxiumum size of 33554432
>   at org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:264) 
> ~[apache-cassandra-3.0.5.jar:3.0.5]
>   at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:469) 
> ~[apache-cassandra-3.0.5.jar:3.0.5]
>   at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:384) 
> ~[apache-cassandra-3.0.5.jar:3.0.5]
>   at org.apache.cassandra.db.Mutation.applyFuture(Mutation.java:205) 
> ~[apache-cassandra-3.0.5.jar:3.0.5]
>   at org.apache.cassandra.db.Mutation.apply(Mutation.java:217) 
> ~[apache-cassandra-3.0.5.jar:3.0.5]
>   at 
> org.apache.cassandra.batchlog.BatchlogManager.store(BatchlogManager.java:146) 
> ~[apache-cassandra-3.0.5.jar:3.0.5]
>   at 
> 

[jira] [Updated] (CASSANDRA-13294) Possible data loss on upgrade 2.1 - 3.0

[Marcus Eriksson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Marcus Eriksson updated CASSANDRA-13294:

Reviewer: Stefania

> Possible data loss on upgrade 2.1 - 3.0
> ---
>
> Key: CASSANDRA-13294
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13294
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Marcus Eriksson
>Assignee: Marcus Eriksson
>Priority: Blocker
> Fix For: 3.0.x
>
>
> After finishing a compaction we delete the compacted away files. This is done 
> [here|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/db/lifecycle/LogFile.java#L328-L337]
>  which uses 
> [this|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/db/lifecycle/LogRecord.java#L265-L271]
>  to get the files - we get all files starting with {{absoluteFilePath}}. 
> Absolute file path is generated 
> [here|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/io/sstable/Descriptor.java#L142-L153].
>  For 3.0 version files the filename looks like this: 
> {{/blabla/keyspace1/standard1-bdb031c0ff7b11e6940fdd0479dd8912/mc-1332-big}} 
> but for 2.1 version files, they look like this: 
> {{/blabla/keyspace1/standard1-bdb031c0ff7b11e6940fdd0479dd8912/keyspace1-standard1-ka-2}}.
> The problem is then that if we were to finish a compaction including the 
> legacy file, we would actually delete all legacy files having a generation 
> starting with '2'



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (CASSANDRA-13291) Replace usages of MessageDigest with Guava's Hasher

[Robert Stupp (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15894050#comment-15894050
 ] 

Robert Stupp commented on CASSANDRA-13291:
--

[~mkjellman] can you provide some micro benchmarks for both implementations?

> Replace usages of MessageDigest with Guava's Hasher
> ---
>
> Key: CASSANDRA-13291
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13291
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Core
>Reporter: Michael Kjellman
>Assignee: Michael Kjellman
> Attachments: CASSANDRA-13291-trunk.diff
>
>
> During my profiling of C* I frequently see lots of aggregate time across 
> threads being spent inside the MD5 MessageDigest implementation. Given that 
> there are tons of modern alternative hashing functions better than MD5 
> available -- both in terms of providing better collision resistance and 
> actual computational speed -- I wanted to switch out our usage of MD5 for 
> alternatives (like adler128 or murmur3_128) and test for performance 
> improvements.
> Unfortunately, I found given the fact we use MessageDigest everywhere --  
> switching out the hashing function to something like adler128 or murmur3_128 
> (for example) -- which don't ship with the JDK --  wasn't straight forward.
> The goal of this ticket is to propose switching out usages of MessageDigest 
> directly in favor of Hasher from Guava. This means going forward we can 
> change a single line of code to switch the hashing algorithm being used 
> (assuming there is an implementation in Guava).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (CASSANDRA-13294) Possible data loss on upgrade 2.1 - 3.0

[Marcus Eriksson (JIRA)]

Marcus Eriksson created CASSANDRA-13294:
---

 Summary: Possible data loss on upgrade 2.1 - 3.0
 Key: CASSANDRA-13294
 URL: https://issues.apache.org/jira/browse/CASSANDRA-13294
 Project: Cassandra
  Issue Type: Bug
Reporter: Marcus Eriksson
Assignee: Marcus Eriksson
Priority: Blocker
 Fix For: 3.0.x


After finishing a compaction we delete the compacted away files. This is done 
[here|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/db/lifecycle/LogFile.java#L328-L337]
 which uses 
[this|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/db/lifecycle/LogRecord.java#L265-L271]
 to get the files - we get all files starting with {{absoluteFilePath}}. 
Absolute file path is generated 
[here|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/io/sstable/Descriptor.java#L142-L153].
 For 3.0 version files the filename looks like this: 
{{/blabla/keyspace1/standard1-bdb031c0ff7b11e6940fdd0479dd8912/mc-1332-big}} 
but for 2.1 version files, they look like this: 
{{/blabla/keyspace1/standard1-bdb031c0ff7b11e6940fdd0479dd8912/keyspace1-standard1-ka-2}}.

The problem is then that if we were to finish a compaction including the legacy 
file, we would actually delete all legacy files having a generation starting 
with '2'



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Comment Edited] (CASSANDRA-12888) Incremental repairs broken for MVs and CDC

[Benjamin Roth (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15818955#comment-15818955
 ] 

Benjamin Roth edited comment on CASSANDRA-12888 at 3/3/17 9:57 AM:
---

Hi Victor,

We use MVs in Production with billions of records without known data loss.
Painful + slow refers to repairs and range movements (e.g. bootstrap +
decommission). Also (as mentioned in this ticket) incremental repairs dont
work, so full repair creates some overhead. Until 3.10 there are bugs
leading to write timeouts, even to NPEs and completely blocked mutation
stages. This could even bring your cluster down. In 3.10 some issues have
been resolved - actually we use a patched trunk version which is 1-2 months
old.

Depending on your model, MVs can help a lot from a developer perspective.
Some cases are very resource intensive to manage without MVs, requiring
distributed locks and/or CAS.
For append-only workloads, it may be simpler to NOT use MVs at the moment.
They aren't very complex and MVs wont help that much compared to the
problems that may raise with them.

Painful scenarios: There is no recipe for that. You may or may not
encounter performance issues, depending on your model and your workload.
I'd recommend not to use MVs that use a different partition key on the MV
than on the base table as this requires inter-node communication for EVERY
write operation. So you can easily kill your cluster with bulk operations
(like in streaming).

At the moment our cluster runs stable but it took months to find all the
bottlenecks, race conditions, resume from failures and so on. So my
recommendation: You can get it work but you need time and you should not
start with critical data, at least if it is not backed by another stable
storage. And you should use 3.10 when it is finally released or build your
own version from trunk. I would not recommend to use < 3.10 for MVs.

Btw.: Our own patched version does some dirty tricks, that may lead to
inconsistencies in some situations but we prefer some possible
inconsistencies (we can deal with) over performance bottlenecks. I created
several tickets to improve MV performance in some streaming situations but
it will take some time to really improve that situation.

Does this answer your question?


was (Author: brstgt):
Hi Victor,

We use MVs in Production with billions of records without known data loss.
Painful + slow refers to repairs and range movements (e.g. bootstrap +
decommission). Also (as mentioned in this ticket) incremental repairs dont
work, so full repair creates some overhead. Until 3.10 there are bugs
leading to write timeouts, even to NPEs and completely blocked mutation
stages. This could even bring your cluster down. In 3.10 some issues have
been resolved - actually we use a patched trunk version which is 1-2 months
old.

Depending on your model, MVs can help a lot from a developer perspective.
Some cases are very resource intensive to manage without MVs, requiring
distributed locks and/or CAS.
For append-only workloads, it may be simpler to NOT use MVs at the moment.
They aren't very complex and MVs wont help that much compared to the
problems that may raise with them.

Painful scenarios: There is no recipe for that. You may or may not
encounter performance issues, depending on your model and your workload.
I'd recommend not to use MVs that use a different partition key on the MV
than on the base table as this requires inter-node communication for EVERY
write operation. So you can easily kill your cluster with bulk operations
(like in streaming).

At the moment our cluster runs stable but it took months to find all the
bottlenecks, race conditions, resume from failures and so on. So my
recommendation: You can get it work but you need time and you should not
start with critical data, at least if it is not backed by another stable
storage. And you should use 3.10 when it is finally released or build your
own version from trunk. I would not recommend to use < 3.10 for MVs.

Btw.: Our own patched version does some dirty tricks, that may lead to
inconsistencies in some situations but we prefer some possible
inconsistencies (we can deal with) over performance bottlenecks. I created
several tickets to improve MV performance in some streaming situations but
it will take some time to really improve that situation.

Does this answer your question?






-- 
Benjamin Roth
Prokurist

Jaumo GmbH · www.jaumo.com
Wehrstraße 46 · 73035 Göppingen · Germany
Phone +49 7161 304880-6 · Fax +49 7161 304880-1
AG Ulm · HRB 731058 · Managing Director: Jens Kammerer


> Incremental repairs broken for MVs and CDC
> --
>
> Key: CASSANDRA-12888
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12888
> Project: Cassandra
>  Issue Type: Bug
>  Components: Streaming and 

[jira] [Comment Edited] (CASSANDRA-12888) Incremental repairs broken for MVs and CDC

[Benjamin Roth (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15894006#comment-15894006
 ] 

Benjamin Roth edited comment on CASSANDRA-12888 at 3/3/17 9:55 AM:
---

I am about to hack a proof of concept for that issue.

Concept:
Each mutation and each partition update have a "repairedAt" flag. This will be 
passed along through the whole write path like MV updates and serialization for 
remote MV updates. Then repair + non-repair mutations have to be separated in 
memtables and flushed to separate SSTables. From what I can see it should be 
easier to maintain a memtable each for repaired and non-repaired data than 
tracking the repair state within a memtable.
Passing repair state to replicas isn't even necessary as replicas should not be 
repaired directly anyway, so no need for a repairedAt state.

My question is:
How important is the exact value of "repairedAt". Is it possible to merge 
updates with different repair timestamps into a single memtable and finally 
flush them to an SSTable with repairedAt set to the latest or earliest 
repairedAt timestamps of all mutations in the memtable?
Or would that produce repair-inconsistencies or sth?

Any feedback?


was (Author: brstgt):
I am about to hack a proof of concept for that issue.

Concept:
Each mutation and each partition update have a "repairedAt" flag. This will be 
passed along through the whole write path like MV updates and serialization for 
remote MV updates. Then repair + non-repair mutations have to be separated in 
memtables and flushed to separate SSTables. From what I can see it should be 
easier to maintain a memtable each for repaired and non-repaired data than 
tracking the repair state within a memtable.

My question is:
How important is the exact value of "repairedAt". Is it possible to merge 
updates with different repair timestamps into a single memtable and finally 
flush them to an SSTable with repairedAt set to the latest or earliest 
repairedAt timestamps of all mutations in the memtable?
Or would that produce repair-inconsistencies or sth?

Any feedback?

> Incremental repairs broken for MVs and CDC
> --
>
> Key: CASSANDRA-12888
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12888
> Project: Cassandra
>  Issue Type: Bug
>  Components: Streaming and Messaging
>Reporter: Stefan Podkowinski
>Assignee: Benjamin Roth
>Priority: Critical
> Fix For: 3.0.x, 3.11.x
>
>
> SSTables streamed during the repair process will first be written locally and 
> afterwards either simply added to the pool of existing sstables or, in case 
> of existing MVs or active CDC, replayed on mutation basis:
> As described in {{StreamReceiveTask.OnCompletionRunnable}}:
> {quote}
> We have a special path for views and for CDC.
> For views, since the view requires cleaning up any pre-existing state, we 
> must put all partitions through the same write path as normal mutations. This 
> also ensures any 2is are also updated.
> For CDC-enabled tables, we want to ensure that the mutations are run through 
> the CommitLog so they can be archived by the CDC process on discard.
> {quote}
> Using the regular write path turns out to be an issue for incremental 
> repairs, as we loose the {{repaired_at}} state in the process. Eventually the 
> streamed rows will end up in the unrepaired set, in contrast to the rows on 
> the sender site moved to the repaired set. The next repair run will stream 
> the same data back again, causing rows to bounce on and on between nodes on 
> each repair.
> See linked dtest on steps to reproduce. An example for reproducing this 
> manually using ccm can be found 
> [here|https://gist.github.com/spodkowinski/2d8e0408516609c7ae701f2bf1e515e8]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (CASSANDRA-12888) Incremental repairs broken for MVs and CDC

[Benjamin Roth (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15894006#comment-15894006
 ] 

Benjamin Roth commented on CASSANDRA-12888:
---

I am about to hack a proof of concept for that issue.

Concept:
Each mutation and each partition update have a "repairedAt" flag. This will be 
passed along through the whole write path like MV updates and serialization for 
remote MV updates. Then repair + non-repair mutations have to be separated in 
memtables and flushed to separate SSTables. From what I can see it should be 
easier to maintain a memtable each for repaired and non-repaired data than 
tracking the repair state within a memtable.

My question is:
How important is the exact value of "repairedAt". Is it possible to merge 
updates with different repair timestamps into a single memtable and finally 
flush them to an SSTable with repairedAt set to the latest or earliest 
repairedAt timestamps of all mutations in the memtable?
Or would that produce repair-inconsistencies or sth?

Any feedback?

> Incremental repairs broken for MVs and CDC
> --
>
> Key: CASSANDRA-12888
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12888
> Project: Cassandra
>  Issue Type: Bug
>  Components: Streaming and Messaging
>Reporter: Stefan Podkowinski
>Assignee: Benjamin Roth
>Priority: Critical
> Fix For: 3.0.x, 3.11.x
>
>
> SSTables streamed during the repair process will first be written locally and 
> afterwards either simply added to the pool of existing sstables or, in case 
> of existing MVs or active CDC, replayed on mutation basis:
> As described in {{StreamReceiveTask.OnCompletionRunnable}}:
> {quote}
> We have a special path for views and for CDC.
> For views, since the view requires cleaning up any pre-existing state, we 
> must put all partitions through the same write path as normal mutations. This 
> also ensures any 2is are also updated.
> For CDC-enabled tables, we want to ensure that the mutations are run through 
> the CommitLog so they can be archived by the CDC process on discard.
> {quote}
> Using the regular write path turns out to be an issue for incremental 
> repairs, as we loose the {{repaired_at}} state in the process. Eventually the 
> streamed rows will end up in the unrepaired set, in contrast to the rows on 
> the sender site moved to the repaired set. The next repair run will stream 
> the same data back again, causing rows to bounce on and on between nodes on 
> each repair.
> See linked dtest on steps to reproduce. An example for reproducing this 
> manually using ccm can be found 
> [here|https://gist.github.com/spodkowinski/2d8e0408516609c7ae701f2bf1e515e8]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (CASSANDRA-13066) Fast streaming with materialized views

[Benjamin Roth (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Benjamin Roth updated CASSANDRA-13066:
--
Fix Version/s: 4.0
   Status: Patch Available  (was: Open)

Depends upon 13064+13065

https://github.com/Jaumo/cassandra/commits/CASSANDRA-13066

> Fast streaming with materialized views
> --
>
> Key: CASSANDRA-13066
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13066
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Benjamin Roth
>Assignee: Benjamin Roth
> Fix For: 4.0
>
>
> I propose adding a configuration option to send streams of tables with MVs 
> not through the regular write path.
> This may be either a global option or better a CF option.
> Background:
> A repair of a CF with an MV that is much out of sync creates many streams. 
> These streams all go through the regular write path to assert local 
> consistency of the MV. This again causes a read before write for every single 
> mutation which again puts a lot of pressure on the node - much more than 
> simply streaming the SSTable down.
> In some cases this can be avoided. Instead of only repairing the base table, 
> all base + mv tables would have to be repaired. But this can break eventual 
> consistency between base table and MV. The proposed behaviour is always safe, 
> when having append-only MVs. It also works when using CL_QUORUM writes but it 
> cannot be absolutely guaranteed, that a quorum write is applied atomically, 
> so this can also lead to inconsistencies, if a quorum write is started but 
> one node dies in the middle of a request.
> So, this proposal can help a lot in some situations but also can break 
> consistency in others. That's why it should be left upon the operator if that 
> behaviour is appropriate for individual use cases.
> This issue came up here:
> https://issues.apache.org/jira/browse/CASSANDRA-12888?focusedCommentId=15736599=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15736599



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)