[GitHub] cassandra pull request #272: CASSANDRA-14757 Fix accessing java.nio.Bits.tot...
GitHub user snazy opened a pull request: https://github.com/apache/cassandra/pull/272 CASSANDRA-14757 Fix accessing java.nio.Bits.totalCapacity/TOTAL_CAPACITY and AtomicLong You can merge this pull request into a Git repository by running: $ git pull https://github.com/snazy/cassandra 14757-gcinspector-bits Alternatively you can review and apply these changes as the patch at: https://github.com/apache/cassandra/pull/272.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #272 commit e9ac192def04b513cfc7e9b4964f571c87416dee Author: Robert Stupp Date: 2018-09-18T17:05:17Z Fix accessing java.nio.Bits.totalCapacity/TOTAL_CAPACITY and AtomicLong --- - To unsubscribe, e-mail: pr-unsubscr...@cassandra.apache.org For additional commands, e-mail: pr-h...@cassandra.apache.org
[GitHub] cassandra-dtest pull request #31: CASSANDRA-9608 fix jmxutils.py
Github user snazy closed the pull request at: https://github.com/apache/cassandra-dtest/pull/31 --- - To unsubscribe, e-mail: pr-unsubscr...@cassandra.apache.org For additional commands, e-mail: pr-h...@cassandra.apache.org
[GitHub] cassandra-dtest issue #31: CASSANDRA-9608 fix jmxutils.py
Github user snazy commented on the issue: https://github.com/apache/cassandra-dtest/pull/31 Thanks! Committed as f45a06b2efd08e9971d29b0e15c9ba388e4ae6bd --- - To unsubscribe, e-mail: pr-unsubscr...@cassandra.apache.org For additional commands, e-mail: pr-h...@cassandra.apache.org
[GitHub] cassandra issue #236: 9608 trunk
Github user snazy commented on the issue: https://github.com/apache/cassandra/pull/236 No, it's no longer needed. I removed it. --- - To unsubscribe, e-mail: pr-unsubscr...@cassandra.apache.org For additional commands, e-mail: pr-h...@cassandra.apache.org
[GitHub] cassandra-dtest pull request #31: CASSANDRA-9608 fix jmxutils.py
GitHub user snazy opened a pull request: https://github.com/apache/cassandra-dtest/pull/31 CASSANDRA-9608 fix jmxutils.py You can merge this pull request into a Git repository by running: $ git pull https://github.com/snazy/cassandra-dtest 9608-fix-jmxutils Alternatively you can review and apply these changes as the patch at: https://github.com/apache/cassandra-dtest/pull/31.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #31 commit 9c248272b85c95b4b347e1c8277dfd0b8c94aa94 Author: Robert Stupp Date: 2018-07-24T18:51:44Z fix jmxutils.py --- - To unsubscribe, e-mail: pr-unsubscr...@cassandra.apache.org For additional commands, e-mail: pr-h...@cassandra.apache.org
[GitHub] cassandra pull request #236: 9608 trunk
Github user snazy commented on a diff in the pull request:
https://github.com/apache/cassandra/pull/236#discussion_r204382546
--- Diff: src/java/org/apache/cassandra/io/util/FileUtils.java ---
@@ -350,13 +396,38 @@ public static String getRelativePath(String basePath,
String path)
public static void clean(ByteBuffer buffer)
{
-if (buffer == null)
+if (buffer == null || !buffer.isDirect())
return;
-if (isCleanerAvailable && buffer.isDirect())
+
+// TODO Once we can get rid of Java 8, it's simpler to call
sun.misc.Unsafe.invokeCleaner(ByteBuffer),
+// but need to take care of the attachment handling (i.e. whether
'buf' is a duplicate or slice) - that
+// is different in sun.misc.Unsafe.invokeCleaner and this
implementation.
+
+try
{
-DirectBuffer db = (DirectBuffer) buffer;
-if (db.cleaner() != null)
-db.cleaner().clean();
+if (buffer.isDirect())
+{
+Object cleaner =
mhDirectBufferCleaner.bindTo(buffer).invoke();
--- End diff --
Woops. There shouldn't be any else.
---
-
To unsubscribe, e-mail: pr-unsubscr...@cassandra.apache.org
For additional commands, e-mail: pr-h...@cassandra.apache.org
[GitHub] cassandra pull request #236: 9608 trunk
Github user snazy commented on a diff in the pull request:
https://github.com/apache/cassandra/pull/236#discussion_r204381413
--- Diff: src/java/org/apache/cassandra/io/util/FileUtils.java ---
@@ -64,24 +68,31 @@
public static final long ONE_TB = 1024 * ONE_GB;
private static final DecimalFormat df = new DecimalFormat("#.##");
-public static final boolean isCleanerAvailable;
private static final AtomicReference>
fsErrorHandler = new AtomicReference<>(Optional.empty());
+private static Class clsDirectBuffer;
+private static MethodHandle mhDirectBufferCleaner;
+private static MethodHandle mhCleanerClean;
+
static
{
-boolean canClean = false;
try
{
+clsDirectBuffer = Class.forName("sun.nio.ch.DirectBuffer");
+Method mDirectBufferCleaner =
clsDirectBuffer.getMethod("cleaner");
+mhDirectBufferCleaner =
MethodHandles.lookup().unreflect(mDirectBufferCleaner);
+Method mCleanerClean =
mDirectBufferCleaner.getReturnType().getMethod("clean");
+mhCleanerClean =
MethodHandles.lookup().unreflect(mCleanerClean);
+
ByteBuffer buf = ByteBuffer.allocateDirect(1);
-((DirectBuffer) buf).cleaner().clean();
-canClean = true;
+clean(buf);
}
catch (Throwable t)
{
+logger.info("Cannot initialize optimized memory deallocator.
Some data, both in-memory and on-disk, may live longer due to garbage
collection.");
JVMStabilityInspector.inspectThrowable(t);
-logger.info("Cannot initialize un-mmaper. (Are you using a
non-Oracle JVM?) Compacted data files will not be removed promptly. Consider
using an Oracle JVM or using standard disk access mode");
+throw new RuntimeException(t);
--- End diff --
Pretty much every JVM should have a `Cleaner` nowadays. Not having would
break a lot of assumá¹tions IMO.
Changed the log message to ERROR.
---
-
To unsubscribe, e-mail: pr-unsubscr...@cassandra.apache.org
For additional commands, e-mail: pr-h...@cassandra.apache.org
[GitHub] cassandra pull request #236: 9608 trunk
Github user snazy commented on a diff in the pull request: https://github.com/apache/cassandra/pull/236#discussion_r204380974 --- Diff: build.xml --- @@ -1665,7 +1763,7 @@ - + --- End diff -- Oops, re-added --- - To unsubscribe, e-mail: pr-unsubscr...@cassandra.apache.org For additional commands, e-mail: pr-h...@cassandra.apache.org
[GitHub] cassandra pull request #236: 9608 trunk
Github user snazy commented on a diff in the pull request:
https://github.com/apache/cassandra/pull/236#discussion_r204380932
--- Diff:
src/java/org/apache/cassandra/cql3/functions/JavaBasedUDFunction.java ---
@@ -591,14 +592,17 @@ private NameEnvironmentAnswer findType(String
className)
String resourceName = className.replace('.', '/') + ".class";
-try (InputStream is =
UDFunction.udfClassLoader.getResourceAsStream(resourceName))
+try
--- End diff --
Ah - I've changed that part twice. But yea, one try-catch is absolutely
sufficient.
---
-
To unsubscribe, e-mail: pr-unsubscr...@cassandra.apache.org
For additional commands, e-mail: pr-h...@cassandra.apache.org
[GitHub] cassandra pull request #236: 9608 trunk
Github user snazy commented on a diff in the pull request: https://github.com/apache/cassandra/pull/236#discussion_r199440838 --- Diff: build.xml --- @@ -110,16 +118,35 @@ - + + + + + +
[GitHub] cassandra pull request #236: 9608 trunk
Github user snazy commented on a diff in the pull request:
https://github.com/apache/cassandra/pull/236#discussion_r199440796
--- Diff: src/java/org/apache/cassandra/io/util/FileUtils.java ---
@@ -67,23 +71,84 @@
public static final boolean isCleanerAvailable;
private static final AtomicReference>
fsErrorHandler = new AtomicReference<>(Optional.empty());
+private static Class clsDirectBuffer;
+private static MethodHandle mhDirectBufferCleaner;
+private static MethodHandle mhCleanerClean;
+private static MethodHandle mhDirectBufferAddress;
+
static
{
boolean canClean = false;
try
{
+clsDirectBuffer = Class.forName("sun.nio.ch.DirectBuffer");
+Method mDirectBufferCleaner =
clsDirectBuffer.getMethod("cleaner");
+mhDirectBufferCleaner =
MethodHandles.lookup().unreflect(mDirectBufferCleaner);
+Method mCleanerClean =
mDirectBufferCleaner.getReturnType().getMethod("clean");
+mhCleanerClean =
MethodHandles.lookup().unreflect(mCleanerClean);
+Method mDirectBufferAddress =
clsDirectBuffer.getMethod("address");
+mhDirectBufferAddress =
MethodHandles.lookup().unreflect(mDirectBufferAddress);
+
ByteBuffer buf = ByteBuffer.allocateDirect(1);
-((DirectBuffer) buf).cleaner().clean();
+cleanerClean(buf);
canClean = true;
}
catch (Throwable t)
{
+logger.info("Cannot initialize un-mmaper. Compacted data files
will not be removed promptly.", t);
JVMStabilityInspector.inspectThrowable(t);
-logger.info("Cannot initialize un-mmaper. (Are you using a
non-Oracle JVM?) Compacted data files will not be removed promptly. Consider
using an Oracle JVM or using standard disk access mode");
}
isCleanerAvailable = canClean;
}
+public static boolean isDirectBuffer(ByteBuffer buf)
+{
+return clsDirectBuffer.isInstance(buf);
--- End diff --
sure
---
-
To unsubscribe, e-mail: pr-unsubscr...@cassandra.apache.org
For additional commands, e-mail: pr-h...@cassandra.apache.org
[GitHub] cassandra pull request #236: 9608 trunk
Github user snazy commented on a diff in the pull request: https://github.com/apache/cassandra/pull/236#discussion_r199440835 --- Diff: build.xml --- @@ -462,7 +501,9 @@ - + --- End diff -- done --- - To unsubscribe, e-mail: pr-unsubscr...@cassandra.apache.org For additional commands, e-mail: pr-h...@cassandra.apache.org
[GitHub] cassandra pull request #236: 9608 trunk
Github user snazy commented on a diff in the pull request: https://github.com/apache/cassandra/pull/236#discussion_r199440825 --- Diff: build.xml --- @@ -703,6 +744,13 @@ + + --- End diff -- Otherwise an older asm (and previously an older netty version as well) landed in `build/lib/jars` via a transitive dependency. --- - To unsubscribe, e-mail: pr-unsubscr...@cassandra.apache.org For additional commands, e-mail: pr-h...@cassandra.apache.org
[GitHub] cassandra pull request #236: 9608 trunk
Github user snazy commented on a diff in the pull request:
https://github.com/apache/cassandra/pull/236#discussion_r199440779
--- Diff:
src/java/org/apache/cassandra/cql3/functions/JavaBasedUDFunction.java ---
@@ -591,14 +612,48 @@ private NameEnvironmentAnswer findType(String
className)
String resourceName = className.replace('.', '/') + ".class";
-try (InputStream is =
UDFunction.udfClassLoader.getResourceAsStream(resourceName))
+// up to Java 8:
+// returns a non-null InputStream for class files
+// since Java 11:
+// returns a non-null InputStream for class files for
application classes
+// returns null for class files for system modules (e.g.
java.base)
+try
{
-if (is != null)
+InputStream is =
UDFunction.udfClassLoader.getResourceAsStream(resourceName);
+try
+{
+if (is == null)
+{
+// For Java 11 try to see whether the class
actually exists and read the
+// class file data via the class' module. (This is
necessary at least
+// for 9-ea build 123)
--- End diff --
Hm - seems that was actually a bug in Java and it's been fixed.
---
-
To unsubscribe, e-mail: pr-unsubscr...@cassandra.apache.org
For additional commands, e-mail: pr-h...@cassandra.apache.org
[GitHub] cassandra pull request #236: 9608 trunk
Github user snazy commented on a diff in the pull request:
https://github.com/apache/cassandra/pull/236#discussion_r199440701
--- Diff: src/java/org/apache/cassandra/io/util/FileUtils.java ---
@@ -106,11 +171,57 @@ public static void createHardLink(File from, File to)
}
}
+private static final File tempDir = new
File(System.getProperty("java.io.tmpdir"));
+private static final AtomicLong tempFileNum = new AtomicLong();
+
+public static File getTempDir()
+{
+return tempDir;
+}
+
+/**
+ * Pretty much like {@link File#createTempFile(String, String, File)},
but with
+ * the guarantee that the "random" part of the generated file name
between
+ * {@code prefix} and {@code suffix} is a positive, increasing {@code
long} value.
+ */
public static File createTempFile(String prefix, String suffix, File
directory)
{
try
{
-return File.createTempFile(prefix, suffix, directory);
+// Do not use java.io.File.createTempFile(), because some
tests rely on the
--- End diff --
Right, I think that's reasonable.
---
-
To unsubscribe, e-mail: pr-unsubscr...@cassandra.apache.org
For additional commands, e-mail: pr-h...@cassandra.apache.org
[GitHub] cassandra pull request #236: 9608 trunk
Github user snazy commented on a diff in the pull request: https://github.com/apache/cassandra/pull/236#discussion_r199440821 --- Diff: conf/jvm11.options --- @@ -0,0 +1,89 @@ +### +#jvm11.options# +# # +# See jvm.options. This file is specific for Java 11 and newer. # +### + +# +# GC SETTINGS # +# + + + +### CMS Settings +#-XX:+UseParNewGC +#-XX:+UseConcMarkSweepGC +#-XX:+CMSParallelRemarkEnabled +#-XX:SurvivorRatio=8 +#-XX:MaxTenuringThreshold=1 +#-XX:CMSInitiatingOccupancyFraction=75 +#-XX:+UseCMSInitiatingOccupancyOnly +#-XX:CMSWaitDuration=1 +#-XX:+CMSParallelInitialMarkEnabled +#-XX:+CMSEdenChunksRecordAlways +## some JVMs will fill up their heap when accessed via JMX, see CASSANDRA-6541 +#-XX:+CMSClassUnloadingEnabled + + + +### G1 Settings +## Use the Hotspot garbage-first collector. +-XX:+UseG1GC +-XX:+ParallelRefProcEnabled + +# +## Have the JVM do less remembered set work during STW, instead +## preferring concurrent GC. Reduces p99.9 latency. +-XX:G1RSetUpdatingPauseTimePercent=5 +# +## Main G1GC tunable: lowering the pause target will lower throughput and vise versa. +## 200ms is the JVM default and lowest viable setting +## 1000ms increases throughput. Keep it smaller than the timeouts in cassandra.yaml. +-XX:MaxGCPauseMillis=500 + +## Optional G1 Settings +# Save CPU time on large (>= 16GB) heaps by delaying region scanning +# until the heap is 70% full. The default in Hotspot 8u40 is 40%. +#-XX:InitiatingHeapOccupancyPercent=70 + +# For systems with > 8 cores, the default ParallelGCThreads is 5/8 the number of logical cores. +# Otherwise equal to the number of cores when 8 or less. +# Machines with > 10 cores should try setting these to <= full cores. +#-XX:ParallelGCThreads=16 +# By default, ConcGCThreads is 1/4 of ParallelGCThreads. +# Setting both to the same value can reduce STW durations. +#-XX:ConcGCThreads=16 + + +### JPMS + +-Djdk.attach.allowAttachSelf=true +--add-exports java.base/jdk.internal.misc=ALL-UNNAMED +--add-opens java.base/jdk.internal.module=ALL-UNNAMED +--add-exports java.base/jdk.internal.ref=ALL-UNNAMED +--add-exports java.base/sun.nio.ch=ALL-UNNAMED +--add-exports java.management.rmi/com.sun.jmx.remote.internal.rmi=ALL-UNNAMED +--add-exports java.rmi/sun.rmi.registry=ALL-UNNAMED +--add-exports java.rmi/sun.rmi.server=ALL-UNNAMED +--add-opens jdk.management/com.sun.management.internal=ALL-UNNAMED + + +### GC logging options -- uncomment to enable + +# Java 11 (and newer) GC logging options: +# See description of https://bugs.openjdk.java.net/browse/JDK-8046148 for details about the syntax +# The following is the equivalent to -XX:+PrintGCDetails -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=10M +#-Xlog:gc=info,heap*=trace,age*=debug,safepoint=info,promotion*=trace:file=/var/log/cassandra/gc.log:time,uptime,pid,tid,level:filecount=10,filesize=10240 --- End diff -- nice catch! --- - To unsubscribe, e-mail: pr-unsubscr...@cassandra.apache.org For additional commands, e-mail: pr-h...@cassandra.apache.org
[GitHub] cassandra pull request #236: 9608 trunk
Github user snazy commented on a diff in the pull request:
https://github.com/apache/cassandra/pull/236#discussion_r199440783
--- Diff: src/java/org/apache/cassandra/utils/NativeLibrary.java ---
@@ -78,7 +77,14 @@
static
{
FILE_DESCRIPTOR_FD_FIELD =
FBUtilities.getProtectedField(FileDescriptor.class, "fd");
-FILE_CHANNEL_FD_FIELD =
FBUtilities.getProtectedField(FileChannelImpl.class, "fd");
+try
--- End diff --
Because of the `Class.forName` declaring the checked exception.
---
-
To unsubscribe, e-mail: pr-unsubscr...@cassandra.apache.org
For additional commands, e-mail: pr-h...@cassandra.apache.org
[GitHub] cassandra pull request #236: 9608 trunk
Github user snazy commented on a diff in the pull request: https://github.com/apache/cassandra/pull/236#discussion_r199440768 --- Diff: src/java/org/apache/cassandra/io/util/FileUtils.java --- @@ -67,23 +71,84 @@ public static final boolean isCleanerAvailable; private static final AtomicReference> fsErrorHandler = new AtomicReference<>(Optional.empty()); +private static Class clsDirectBuffer; +private static MethodHandle mhDirectBufferCleaner; +private static MethodHandle mhCleanerClean; +private static MethodHandle mhDirectBufferAddress; --- End diff -- done --- - To unsubscribe, e-mail: pr-unsubscr...@cassandra.apache.org For additional commands, e-mail: pr-h...@cassandra.apache.org
[GitHub] cassandra pull request #236: 9608 trunk
Github user snazy commented on a diff in the pull request:
https://github.com/apache/cassandra/pull/236#discussion_r199440790
--- Diff:
src/java/org/apache/cassandra/security/ThreadAwareSecurityManager.java ---
@@ -208,7 +225,5 @@ public void checkPackageAccess(String pkg)
RuntimePermission perm = new
RuntimePermission("accessClassInPackage." + pkg);
throw new AccessControlException("access denied: " + perm,
perm);
}
-
-super.checkPackageAccess(pkg);
--- End diff --
Otherwise JavaScript based UDFs stop working due to JPMS, that would raise
errors like
`Caused by: java.security.AccessControlException: access denied
("java.lang.RuntimePermission"
"accessClassInPackage.jdk.internal.org.objectweb.asm")`
---
-
To unsubscribe, e-mail: pr-unsubscr...@cassandra.apache.org
For additional commands, e-mail: pr-h...@cassandra.apache.org
[GitHub] cassandra pull request #236: 9608 trunk
Github user snazy commented on a diff in the pull request:
https://github.com/apache/cassandra/pull/236#discussion_r199440755
--- Diff: src/java/org/apache/cassandra/io/util/FileUtils.java ---
@@ -67,23 +71,84 @@
public static final boolean isCleanerAvailable;
private static final AtomicReference>
fsErrorHandler = new AtomicReference<>(Optional.empty());
+private static Class clsDirectBuffer;
+private static MethodHandle mhDirectBufferCleaner;
+private static MethodHandle mhCleanerClean;
+private static MethodHandle mhDirectBufferAddress;
+
static
{
boolean canClean = false;
try
{
+clsDirectBuffer = Class.forName("sun.nio.ch.DirectBuffer");
+Method mDirectBufferCleaner =
clsDirectBuffer.getMethod("cleaner");
+mhDirectBufferCleaner =
MethodHandles.lookup().unreflect(mDirectBufferCleaner);
+Method mCleanerClean =
mDirectBufferCleaner.getReturnType().getMethod("clean");
+mhCleanerClean =
MethodHandles.lookup().unreflect(mCleanerClean);
+Method mDirectBufferAddress =
clsDirectBuffer.getMethod("address");
+mhDirectBufferAddress =
MethodHandles.lookup().unreflect(mDirectBufferAddress);
+
ByteBuffer buf = ByteBuffer.allocateDirect(1);
-((DirectBuffer) buf).cleaner().clean();
+cleanerClean(buf);
canClean = true;
}
catch (Throwable t)
{
+logger.info("Cannot initialize un-mmaper. Compacted data files
will not be removed promptly.", t);
--- End diff --
Sounds good
---
-
To unsubscribe, e-mail: pr-unsubscr...@cassandra.apache.org
For additional commands, e-mail: pr-h...@cassandra.apache.org
