Merge branch cassandra-3.0 into cassandra-3.11
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/a7c45be9 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/a7c45be9 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/a7c45be9 Branch: refs/heads/trunk Commit: a7c45be936bf8aa04b657529964c41b0d82c50e2 Parents: 88a41fb dd187d1 Author: Benjamin Lerer <b.le...@gmail.com> Authored: Tue Dec 12 10:38:33 2017 +0100 Committer: Benjamin Lerer <b.le...@gmail.com> Committed: Tue Dec 12 10:42:50 2017 +0100 ---------------------------------------------------------------------- CHANGES.txt | 2 +- NEWS.txt | 8 +- bin/cassandra | 19 +- conf/cassandra-env.ps1 | 914 ++++++++++--------- conf/cassandra-env.sh | 12 + .../apache/cassandra/service/StartupChecks.java | 74 ++ .../org/apache/cassandra/utils/HeapUtils.java | 93 +- .../cassandra/utils/JVMStabilityInspector.java | 24 +- .../utils/JVMStabilityInspectorTest.java | 28 +- 9 files changed, 632 insertions(+), 542 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/cassandra/blob/a7c45be9/CHANGES.txt ---------------------------------------------------------------------- diff --cc CHANGES.txt index 6e9a0bd,20ccc4b..116c7e9 --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,15 -1,4 +1,14 @@@ -3.0.16 +3.11.2 + * Fix imbalanced disks when replacing node with same address with JBOD (CASSANDRA-14084) + * Reload compaction strategies when disk boundaries are invalidated (CASSANDRA-13948) + * Remove OpenJDK log warning (CASSANDRA-13916) + * Prevent compaction strategies from looping indefinitely (CASSANDRA-14079) + * Cache disk boundaries (CASSANDRA-13215) + * Add asm jar to build.xml for maven builds (CASSANDRA-11193) + * Round buffer size to powers of 2 for the chunk cache (CASSANDRA-13897) + * Update jackson JSON jars (CASSANDRA-13949) + * Avoid locks when checking LCS fanout and if we should defrag (CASSANDRA-13930) +Merged from 3.0: - 3.0.16 * Fix SStable ordering by max timestamp in SinglePartitionReadCommand (CASSANDRA-14010) * Accept role names containing forward-slash (CASSANDRA-14088) * Optimize CRC check chance probability calculations (CASSANDRA-14094) http://git-wip-us.apache.org/repos/asf/cassandra/blob/a7c45be9/NEWS.txt ---------------------------------------------------------------------- diff --cc NEWS.txt index 9812b22,1bbe1aa..0c32278 --- a/NEWS.txt +++ b/NEWS.txt @@@ -18,7 -18,13 +18,13 @@@ using the provided 'sstableupgrade' too Upgrading --------- - - Nothing specific to this release, but please see previous upgrading sections. - - Cassandra is now relying on the JVM options to properly shutdown on OutOfMemoryError. By default it will - rely on the OnOutOfMemoryError option as the ExitOnOutOfMemoryError and CrashOnOutOfMemoryError options - are not supported by the older 1.7 and 1.8 JVMs. A warning will be logged at startup if none of those JVM - options are used. See CASSANDRA-13006 for more details. - - Cassandra is not logging anymore by default an Heap histogram on OutOfMemoryError. To enable that behavior - set the 'cassandra.printHeapHistogramOnOutOfMemoryError' System property to 'true'. See CASSANDRA-13006 - for more details. ++ - Cassandra is now relying on the JVM options to properly shutdown on OutOfMemoryError. By default it will ++ rely on the OnOutOfMemoryError option as the ExitOnOutOfMemoryError and CrashOnOutOfMemoryError options ++ are not supported by the older 1.7 and 1.8 JVMs. A warning will be logged at startup if none of those JVM ++ options are used. See CASSANDRA-13006 for more details ++ - Cassandra is not logging anymore by default an Heap histogram on OutOfMemoryError. To enable that behavior ++ set the 'cassandra.printHeapHistogramOnOutOfMemoryError' System property to 'true'. See CASSANDRA-13006 ++ for more details. Materialized Views ------------------- http://git-wip-us.apache.org/repos/asf/cassandra/blob/a7c45be9/bin/cassandra ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/cassandra/blob/a7c45be9/conf/cassandra-env.ps1 ---------------------------------------------------------------------- diff --cc conf/cassandra-env.ps1 index 2df7054,405ed92..49d03ce --- a/conf/cassandra-env.ps1 +++ b/conf/cassandra-env.ps1 @@@ -393,70 -393,98 +393,80 @@@ Function SetCassandraEnvironmen $env:JVM_OPTS="$env:JVM_OPTS -XX:HeapDumpPath=$env:CASSANDRA_HEAPDUMP_DIR\cassandra-$unixTimestamp-pid$pid.hprof" } - if ($env:JVM_VERSION.CompareTo("1.8.0") -eq -1 -or [convert]::ToInt32($env:JVM_PATCH_VERSION) -lt 40) - { - echo "Cassandra 3.0 and later require Java 8u40 or later." - exit - } - - # Specifies the default port over which Cassandra will be available for - # JMX connections. - $JMX_PORT="7199" - - # store in env to check if it's avail in verification - $env:JMX_PORT=$JMX_PORT - - # Configure the following for JEMallocAllocator and if jemalloc is not available in the system - # library path. - # set LD_LIBRARY_PATH=<JEMALLOC_HOME>/lib/ - # $env:JVM_OPTS="$env:JVM_OPTS -Djava.library.path=<JEMALLOC_HOME>/lib/" - - # jmx: metrics and administration interface - # - # add this if you're having trouble connecting: - # $env:JVM_OPTS="$env:JVM_OPTS -Djava.rmi.server.hostname=<public name>" - # - # see - # https://blogs.oracle.com/jmxetc/entry/troubleshooting_connection_problems_in_jconsole - # for more on configuring JMX through firewalls, etc. (Short version: - # get it working with no firewall first.) - # - # Due to potential security exploits, Cassandra ships with JMX accessible - # *only* from localhost. To enable remote JMX connections, uncomment lines below - # with authentication and ssl enabled. See https://wiki.apache.org/cassandra/JmxSecurity - # - #$env:JVM_OPTS="$env:JVM_OPTS -Dcom.sun.management.jmxremote.port=$JMX_PORT" - #$env:JVM_OPTS="$env:JVM_OPTS -Dcom.sun.management.jmxremote.rmi.port=$JMX_PORT" - # - # JMX SSL options - #$env:JVM_OPTS="$env:JVM_OPTS -Dcom.sun.management.jmxremote.ssl=true" - #$env:JVM_OPTS="$env:JVM_OPTS -Dcom.sun.management.jmxremote.ssl.need.client.auth=true" - #$env:JVM_OPTS="$env:JVM_OPTS -Dcom.sun.management.jmxremote.ssl.enabled.protocols=<enabled-protocols>" - #$env:JVM_OPTS="$env:JVM_OPTS -Dcom.sun.management.jmxremote.ssl.enabled.cipher.suites=<enabled-cipher-suites>" - #$env:JVM_OPTS="$env:JVM_OPTS -Djavax.net.ssl.keyStore=C:/keystore" - #$env:JVM_OPTS="$env:JVM_OPTS -Djavax.net.ssl.keyStorePassword=<keystore-password>" - #$env:JVM_OPTS="$env:JVM_OPTS -Djavax.net.ssl.trustStore=C:/truststore" - #$env:JVM_OPTS="$env:JVM_OPTS -Djavax.net.ssl.trustStorePassword=<truststore-password>" - # - # JMX auth options - #$env:JVM_OPTS="$env:JVM_OPTS -Dcom.sun.management.jmxremote.authenticate=true" - ## Basic file based authn & authz - #$env:JVM_OPTS="$env:JVM_OPTS -Dcom.sun.management.jmxremote.password.file=C:/jmxremote.password" - #$env:JVM_OPTS="$env:JVM_OPTS -Dcom.sun.management.jmxremote.access.file=C:/jmxremote.access" - - ## Custom auth settings which can be used as alternatives to JMX's out of the box auth utilities. - ## JAAS login modules can be used for authentication by uncommenting these two properties. - ## Cassandra ships with a LoginModule implementation - org.apache.cassandra.auth.CassandraLoginModule - - ## which delegates to the IAuthenticator configured in cassandra.yaml - #$env:JVM_OPTS="$env:JVM_OPTS -Dcassandra.jmx.remote.login.config=CassandraLogin" - #$env:JVM_OPTS="$env:JVM_OPTS -Djava.security.auth.login.config=C:/cassandra-jaas.config" - - ## Cassandra also ships with a helper for delegating JMX authz calls to the configured IAuthorizer, - ## uncomment this to use it. Requires one of the two authentication options to be enabled - #$env:JVM_OPTS="$env:JVM_OPTS -Dcassandra.jmx.authorizer=org.apache.cassandra.auth.jmx.AuthorizationProxy" - - # Default JMX setup, bound to local loopback address only - $env:JVM_OPTS="$env:JVM_OPTS -Dcassandra.jmx.local.port=$JMX_PORT" - - $env:JVM_OPTS="$env:JVM_OPTS $env:JVM_EXTRA_OPTS" - } - if ($env:JVM_VERSION.CompareTo("1.8.0") -eq -1 -or [convert]::ToInt32($env:JVM_PATCH_VERSION) -lt 40) - { - echo "Cassandra 3.0 and later require Java 8u40 or later." - exit - } - - # enable assertions. disabling this in production will give a modest - # performance benefit (around 5%). - $env:JVM_OPTS = "$env:JVM_OPTS -ea" - - # Specifies the default port over which Cassandra will be available for - # JMX connections. - $JMX_PORT="7199" - - # store in env to check if it's avail in verification - $env:JMX_PORT=$JMX_PORT - - # enable thread priorities, primarily so we can give periodic tasks - # a lower priority to avoid interfering with client workload - $env:JVM_OPTS="$env:JVM_OPTS -XX:+UseThreadPriorities" - # allows lowering thread priority without being root on linux - probably - # not necessary on Windows but doesn't harm anything. - # see http://tech.stolsvik.com/2010/01/linux-java-thread-priorities-workar - $env:JVM_OPTS="$env:JVM_OPTS -XX:ThreadPriorityPolicy=42" - - $env:JVM_OPTS="$env:JVM_OPTS -XX:+HeapDumpOnOutOfMemoryError" - + # stop the jvm on OutOfMemoryError as it can result in some data corruption + # uncomment the preferred option + # ExitOnOutOfMemoryError and CrashOnOutOfMemoryError require a JRE greater or equals to 1.7 update 101 or 1.8 update 92 + # $env:JVM_OPTS="$env:JVM_OPTS -XX:+ExitOnOutOfMemoryError" + # $env:JVM_OPTS="$env:JVM_OPTS -XX:+CrashOnOutOfMemoryError" + $env:JVM_OPTS="$env:JVM_OPTS -XX:OnOutOfMemoryError=""taskkill /F /PID %p""" + + # print an heap histogram on OutOfMemoryError + # $env:JVM_OPTS="$env:JVM_OPTS -Dcassandra.printHeapHistogramOnOutOfMemoryError=true" + - # Per-thread stack size. - $env:JVM_OPTS="$env:JVM_OPTS -Xss256k" - - # Larger interned string table, for gossip's benefit (CASSANDRA-6410) - $env:JVM_OPTS="$env:JVM_OPTS -XX:StringTableSize=1000003" - - # Make sure all memory is faulted and zeroed on startup. - # This helps prevent soft faults in containers and makes - # transparent hugepage allocation more effective. - #$env:JVM_OPTS="$env:JVM_OPTS -XX:+AlwaysPreTouch" - - # Biased locking does not benefit Cassandra. - $env:JVM_OPTS="$env:JVM_OPTS -XX:-UseBiasedLocking" ++ if ($env:JVM_VERSION.CompareTo("1.8.0") -eq -1 -or [convert]::ToInt32($env:JVM_PATCH_VERSION) -lt 40) ++ { ++ echo "Cassandra 3.0 and later require Java 8u40 or later." ++ exit ++ } + - # Enable thread-local allocation blocks and allow the JVM to automatically - # resize them at runtime. - $env:JVM_OPTS="$env:JVM_OPTS -XX:+UseTLAB -XX:+ResizeTLAB" ++ # Specifies the default port over which Cassandra will be available for ++ # JMX connections. ++ $JMX_PORT="7199" + - # http://www.evanjones.ca/jvm-mmap-pause.html - $env:JVM_OPTS="$env:JVM_OPTS -XX:+PerfDisableSharedMem" ++ # store in env to check if it's avail in verification ++ $env:JMX_PORT=$JMX_PORT + + # Configure the following for JEMallocAllocator and if jemalloc is not available in the system + # library path. + # set LD_LIBRARY_PATH=<JEMALLOC_HOME>/lib/ + # $env:JVM_OPTS="$env:JVM_OPTS -Djava.library.path=<JEMALLOC_HOME>/lib/" + - # uncomment to have Cassandra JVM listen for remote debuggers/profilers on port 1414 - # $env:JVM_OPTS="$env:JVM_OPTS -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=1414" - - # Prefer binding to IPv4 network intefaces (when net.ipv6.bindv6only=1). See - # http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6342561 (short version: - # comment out this entry to enable IPv6 support). - $env:JVM_OPTS="$env:JVM_OPTS -Djava.net.preferIPv4Stack=true" - + # jmx: metrics and administration interface + # + # add this if you're having trouble connecting: + # $env:JVM_OPTS="$env:JVM_OPTS -Djava.rmi.server.hostname=<public name>" + # + # see + # https://blogs.oracle.com/jmxetc/entry/troubleshooting_connection_problems_in_jconsole + # for more on configuring JMX through firewalls, etc. (Short version: + # get it working with no firewall first.) + # + # Due to potential security exploits, Cassandra ships with JMX accessible + # *only* from localhost. To enable remote JMX connections, uncomment lines below + # with authentication and ssl enabled. See https://wiki.apache.org/cassandra/JmxSecurity + # + #$env:JVM_OPTS="$env:JVM_OPTS -Dcom.sun.management.jmxremote.port=$JMX_PORT" - #$env:JVM_OPTS="$env:JVM_OPTS -Dcom.sun.management.jmxremote.ssl=false" ++ #$env:JVM_OPTS="$env:JVM_OPTS -Dcom.sun.management.jmxremote.rmi.port=$JMX_PORT" ++ # ++ # JMX SSL options ++ #$env:JVM_OPTS="$env:JVM_OPTS -Dcom.sun.management.jmxremote.ssl=true" ++ #$env:JVM_OPTS="$env:JVM_OPTS -Dcom.sun.management.jmxremote.ssl.need.client.auth=true" ++ #$env:JVM_OPTS="$env:JVM_OPTS -Dcom.sun.management.jmxremote.ssl.enabled.protocols=<enabled-protocols>" ++ #$env:JVM_OPTS="$env:JVM_OPTS -Dcom.sun.management.jmxremote.ssl.enabled.cipher.suites=<enabled-cipher-suites>" ++ #$env:JVM_OPTS="$env:JVM_OPTS -Djavax.net.ssl.keyStore=C:/keystore" ++ #$env:JVM_OPTS="$env:JVM_OPTS -Djavax.net.ssl.keyStorePassword=<keystore-password>" ++ #$env:JVM_OPTS="$env:JVM_OPTS -Djavax.net.ssl.trustStore=C:/truststore" ++ #$env:JVM_OPTS="$env:JVM_OPTS -Djavax.net.ssl.trustStorePassword=<truststore-password>" ++ # ++ # JMX auth options + #$env:JVM_OPTS="$env:JVM_OPTS -Dcom.sun.management.jmxremote.authenticate=true" ++ ## Basic file based authn & authz + #$env:JVM_OPTS="$env:JVM_OPTS -Dcom.sun.management.jmxremote.password.file=C:/jmxremote.password" - $env:JVM_OPTS="$env:JVM_OPTS -Dcassandra.jmx.local.port=$JMX_PORT -XX:+DisableExplicitGC" ++ #$env:JVM_OPTS="$env:JVM_OPTS -Dcom.sun.management.jmxremote.access.file=C:/jmxremote.access" + - $env:JVM_OPTS="$env:JVM_OPTS $env:JVM_EXTRA_OPTS" ++ ## Custom auth settings which can be used as alternatives to JMX's out of the box auth utilities. ++ ## JAAS login modules can be used for authentication by uncommenting these two properties. ++ ## Cassandra ships with a LoginModule implementation - org.apache.cassandra.auth.CassandraLoginModule - ++ ## which delegates to the IAuthenticator configured in cassandra.yaml ++ #$env:JVM_OPTS="$env:JVM_OPTS -Dcassandra.jmx.remote.login.config=CassandraLogin" ++ #$env:JVM_OPTS="$env:JVM_OPTS -Djava.security.auth.login.config=C:/cassandra-jaas.config" ++ ++ ## Cassandra also ships with a helper for delegating JMX authz calls to the configured IAuthorizer, ++ ## uncomment this to use it. Requires one of the two authentication options to be enabled ++ #$env:JVM_OPTS="$env:JVM_OPTS -Dcassandra.jmx.authorizer=org.apache.cassandra.auth.jmx.AuthorizationProxy" + - #$env:JVM_OPTS="$env:JVM_OPTS -XX:+UnlockCommercialFeatures -XX:+FlightRecorder" ++ # Default JMX setup, bound to local loopback address only ++ $env:JVM_OPTS="$env:JVM_OPTS -Dcassandra.jmx.local.port=$JMX_PORT" ++ ++ $env:JVM_OPTS="$env:JVM_OPTS $env:JVM_EXTRA_OPTS" + } http://git-wip-us.apache.org/repos/asf/cassandra/blob/a7c45be9/conf/cassandra-env.sh ---------------------------------------------------------------------- diff --cc conf/cassandra-env.sh index 5a02f79,ffc773f..189576b --- a/conf/cassandra-env.sh +++ b/conf/cassandra-env.sh @@@ -214,6 -247,30 +214,18 @@@ if [ "x$CASSANDRA_HEAPDUMP_DIR" != "x" JVM_OPTS="$JVM_OPTS -XX:HeapDumpPath=$CASSANDRA_HEAPDUMP_DIR/cassandra-`date +%s`-pid$$.hprof" fi + # stop the jvm on OutOfMemoryError as it can result in some data corruption + # uncomment the preferred option + # ExitOnOutOfMemoryError and CrashOnOutOfMemoryError require a JRE greater or equals to 1.7 update 101 or 1.8 update 92 + # For OnOutOfMemoryError we cannot use the JVM_OPTS variables because bash commands split words + # on white spaces without taking quotes into account + # JVM_OPTS="$JVM_OPTS -XX:+ExitOnOutOfMemoryError" + # JVM_OPTS="$JVM_OPTS -XX:+CrashOnOutOfMemoryError" + JVM_ON_OUT_OF_MEMORY_ERROR_OPT="-XX:OnOutOfMemoryError=kill -9 %p" + + # print an heap histogram on OutOfMemoryError + # JVM_OPTS="$JVM_OPTS -Dcassandra.printHeapHistogramOnOutOfMemoryError=true" + -# uncomment to have Cassandra JVM listen for remote debuggers/profilers on port 1414 -# JVM_OPTS="$JVM_OPTS -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=1414" - -# uncomment to have Cassandra JVM log internal method compilation (developers only) -# JVM_OPTS="$JVM_OPTS -XX:+UnlockDiagnosticVMOptions -XX:+LogCompilation" -# JVM_OPTS="$JVM_OPTS -XX:+UnlockCommercialFeatures -XX:+FlightRecorder" - -# Prefer binding to IPv4 network intefaces (when net.ipv6.bindv6only=1). See -# http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6342561 (short version: -# comment out this entry to enable IPv6 support). -JVM_OPTS="$JVM_OPTS -Djava.net.preferIPv4Stack=true" - # jmx: metrics and administration interface # # add this if you're having trouble connecting: http://git-wip-us.apache.org/repos/asf/cassandra/blob/a7c45be9/src/java/org/apache/cassandra/service/StartupChecks.java ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/cassandra/blob/a7c45be9/src/java/org/apache/cassandra/utils/HeapUtils.java ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/cassandra/blob/a7c45be9/src/java/org/apache/cassandra/utils/JVMStabilityInspector.java ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/cassandra/blob/a7c45be9/test/unit/org/apache/cassandra/utils/JVMStabilityInspectorTest.java ---------------------------------------------------------------------- --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org