This is an automated email from the ASF dual-hosted git repository.
LuciferYang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new e491395f9129 [SPARK-53327][SQL] Workaround datasketches-memory Java 25
support
e491395f9129 is described below
commit e491395f91295166a7d96eec7446ccf59c0406a8
Author: Cheng Pan <[email protected]>
AuthorDate: Mon May 11 18:10:58 2026 +0800
[SPARK-53327][SQL] Workaround datasketches-memory Java 25 support
### What changes were proposed in this pull request?
This PR ported the change of
`org.apache.datasketches.memory.internal.ResourceImpl` in
https://github.com/apache/datasketches-memory/pull/272 to the Spark codebase,
and in the `dev/make-distribution.sh`, use `zip` command to delete the
`ResourceImpl.class` from `datasketches-memory-3.0.2.jar`
### Why are the changes needed?
To unblock Spark JDK 25 support before Apache DataSketches community
releases the `datasketches-memory` 3.0.3. (I have tried my best to push this
release, but unfortunately, it has not happened.)
```
Caused by: java.lang.ExceptionInInitializerError
at
org.apache.datasketches.memory.WritableMemory.writableWrap(WritableMemory.java:333)
at
org.apache.datasketches.memory.WritableMemory.writableWrap(WritableMemory.java:275)
at
org.apache.datasketches.hll.ToByteArrayImpl.toCouponByteArray(ToByteArrayImpl.java:206)
at
org.apache.datasketches.hll.AbstractCoupons.toUpdatableByteArray(AbstractCoupons.java:130)
at
org.apache.datasketches.hll.HllSketch.toUpdatableByteArray(HllSketch.java:466)
at
org.apache.spark.sql.catalyst.expressions.aggregate.HllSketchAgg.serialize(datasketchesAggregates.scala:185)
...
Caused by: java.lang.IllegalArgumentException: Unsupported JDK Major
Version. It must be one of 1.8, 8, 11, 17, 21: 25
at
org.apache.datasketches.memory.internal.ResourceImpl.checkJavaVersion(ResourceImpl.java:155)
at
org.apache.datasketches.memory.internal.ResourceImpl.parseJavaVersion(ResourceImpl.java:391)
at
org.apache.datasketches.memory.internal.ResourceImpl.<clinit>(ResourceImpl.java:87)
... 26 more (ThriftServerQueryTestSuite.scala:240)
```
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Verified on both dev mode and release mode.
- dev mode, compiled `class` files from source code take precedence over
3rd libs
```
$ export JAVA_HOME=/path/of/openjdk-25
$ build/sbt clean package -Phive,hive-thriftserver
$ SPARK_PREPEND_CLASSES=true bin/spark-shell
NOTE: SPARK_PREPEND_CLASSES is set, placing locally compiled Spark classes
ahead of assembly.
WARNING: Using incubator modules: jdk.incubator.vector
WARNING: package sun.security.action not in java.base
Using Spark's default log4j profile:
org/apache/spark/log4j2-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use
setLogLevel(newLevel).
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 5.0.0-SNAPSHOT
/_/
Using Scala version 2.13.18 (OpenJDK 64-Bit Server VM, Java 25.0.3)
Type in expressions to have them evaluated.
Type :help for more information.
26/05/09 12:39:34 WARN NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
WARNING: A terminally deprecated method in sun.misc.Unsafe has been called
WARNING: sun.misc.Unsafe::allocateMemory has been called by
io.netty.util.internal.PlatformDependent0$2
(file:/Users/chengpan/Projects/apache-spark/assembly/target/scala-2.13/jars/netty-common-4.2.12.Final.jar)
WARNING: Please consider reporting this to the maintainers of class
io.netty.util.internal.PlatformDependent0$2
WARNING: sun.misc.Unsafe::allocateMemory will be removed in a future release
Spark context Web UI available at http://10.242.159.140:4040
Spark context available as 'sc' (master = local[*], app id =
local-1778301575516).
Spark session available as 'spark'.
scala> org.apache.datasketches.memory.WritableMemory.allocate(3)
Warning: Java versions > Java 11 can only operate in restricted mode where
no off-heap operations are allowed!
val res0: org.apache.datasketches.memory.WritableMemory =
### HeapWritableMemoryImpl SUMMARY ###
Type Info : Writable + Heap +
NativeOrder + Memory +
Header Comment :
Call Parameters : .toHexString(..., 0, 3), hashCode: 1778430092
UnsafeObj, hashCode : byte[], 97094552
UnsafeObjHeader : 16
ByteBuf, hashCode : null
RegionOffset : 0
Capacity : 3
CumBaseOffset : 16
MemReqSvr, hashCode : null
is Alive : true
Read Only : false
Type Byte Order : LITTLE_ENDIAN
Native Byte Order : LITTLE_ENDIAN
JDK Runtime Version : 25.0
### END SUMMARY
scala> spark.sql("""
| SELECT hll_sketch_estimate(hll_sketch_agg(col))
| FROM VALUES (1), (1), (2), (2), (3) tab(col);
| """).show()
+--------------------------------------------+
|hll_sketch_estimate(hll_sketch_agg(col, 12))|
+--------------------------------------------+
| 3|
+--------------------------------------------+
scala>
```
- release mode, `ResourceImpl.class` is delted from
`datasketches-memory-3.0.2.jar` so it is always picked up from
`spark-catalyst_2.13-*.jar`
```
$ export JAVA_HOME=/path/of/openjdk-25
$ dev/make-distribution.sh -Phive -Phive-thriftserver
$ cd dist
$ bin/spark-shell
WARNING: Using incubator modules: jdk.incubator.vector
WARNING: package sun.security.action not in java.base
Using Spark's default log4j profile:
org/apache/spark/log4j2-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use
setLogLevel(newLevel).
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 5.0.0-SNAPSHOT
/_/
Using Scala version 2.13.18 (OpenJDK 64-Bit Server VM, Java 25.0.3)
Type in expressions to have them evaluated.
Type :help for more information.
26/05/09 12:33:21 WARN NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
WARNING: A terminally deprecated method in sun.misc.Unsafe has been called
WARNING: sun.misc.Unsafe::allocateMemory has been called by
io.netty.util.internal.PlatformDependent0$2
(file:/Users/chengpan/Projects/apache-spark/dist/jars/netty-common-4.2.12.Final.jar)
WARNING: Please consider reporting this to the maintainers of class
io.netty.util.internal.PlatformDependent0$2
WARNING: sun.misc.Unsafe::allocateMemory will be removed in a future release
Spark context Web UI available at http://10.242.159.140:4040
Spark context available as 'sc' (master = local[*], app id =
local-1778301202896).
Spark session available as 'spark'.
scala> org.apache.datasketches.memory.WritableMemory.allocate(3)
Warning: Java versions > Java 11 can only operate in restricted mode where
no off-heap operations are allowed!
val res0: org.apache.datasketches.memory.WritableMemory =
### HeapWritableMemoryImpl SUMMARY ###
Type Info : Writable + Heap +
NativeOrder + Memory +
Header Comment :
Call Parameters : .toHexString(..., 0, 3), hashCode: 1237647680
UnsafeObj, hashCode : byte[], 1777520630
UnsafeObjHeader : 16
ByteBuf, hashCode : null
RegionOffset : 0
Capacity : 3
CumBaseOffset : 16
MemReqSvr, hashCode : null
is Alive : true
Read Only : false
Type Byte Order : LITTLE_ENDIAN
Native Byte Order : LITTLE_ENDIAN
JDK Runtime Version : 25.0
### END SUMMARY
scala> spark.sql("""
| SELECT hll_sketch_estimate(hll_sketch_agg(col))
| FROM VALUES (1), (1), (2), (2), (3) tab(col);
| """).show()
+--------------------------------------------+
|hll_sketch_estimate(hll_sketch_agg(col, 12))|
+--------------------------------------------+
| 3|
+--------------------------------------------+
scala>
```
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #54477 from pan3793/SPARK-53327.
Authored-by: Cheng Pan <[email protected]>
Signed-off-by: yangjie01 <[email protected]>
---
dev/checkstyle-suppressions.xml | 2 +
dev/make-distribution.sh | 5 +
docs/_plugins/build_api_docs.rb | 2 +
pom.xml | 4 +
.../datasketches/memory/internal/ResourceImpl.java | 561 +++++++++++++++++++++
5 files changed, 574 insertions(+)
diff --git a/dev/checkstyle-suppressions.xml b/dev/checkstyle-suppressions.xml
index 9925ae406dbd..55b1ed40cb9e 100644
--- a/dev/checkstyle-suppressions.xml
+++ b/dev/checkstyle-suppressions.xml
@@ -70,4 +70,6 @@
files="src/test/java/org/apache/spark/util/collection/TestTimSort.java"
/>
<suppress checks=".*"
files="src/main/java/org/apache/spark/sql/execution/streaming/state/StateMessage.java"/>
+ <suppress checks=".*"
+
files="src/main/java/org/apache/datasketches/memory/internal/ResourceImpl.java"/>
</suppressions>
diff --git a/dev/make-distribution.sh b/dev/make-distribution.sh
index 428a3ed3f112..e43e4afbd0e2 100755
--- a/dev/make-distribution.sh
+++ b/dev/make-distribution.sh
@@ -203,6 +203,11 @@ echo "Build flags: $@" >> "$DISTDIR/RELEASE"
# Copy jars
cp -r "$SPARK_HOME"/assembly/target/scala*/jars/* "$DISTDIR/jars/"
+# SPARK-53327: Use the modified ResourceImpl.class in spark-catalyst which is
compatible with Java 25
+if [ -f "$DISTDIR"/jars/datasketches-memory-3.0.2.jar ]; then
+ zip -d "$DISTDIR"/jars/datasketches-memory-3.0.2.jar
org/apache/datasketches/memory/internal/ResourceImpl.class
+fi
+
# Only create the yarn directory if the yarn artifacts were built.
if [ -f
"$SPARK_HOME"/common/network-yarn/target/scala*/spark-*-yarn-shuffle.jar ]; then
mkdir "$DISTDIR/yarn"
diff --git a/docs/_plugins/build_api_docs.rb b/docs/_plugins/build_api_docs.rb
index cf58465ab5b6..1ef80bfaf09a 100644
--- a/docs/_plugins/build_api_docs.rb
+++ b/docs/_plugins/build_api_docs.rb
@@ -48,6 +48,8 @@ def build_spark_if_necessary
command = "NO_PROVIDED_SPARK_JARS=0 build/sbt -Phive -Pkinesis-asl clean
package"
puts "Running '#{command}'; this may take a few minutes..."
system(command) || raise("Failed to build Spark")
+ # SPARK-53327: Use the modified ResourceImpl.class in spark-catalyst which
is compatible with Java 25
+ system("zip -d assembly/target/scala-2.13/jars/datasketches-memory-3.0.2.jar
org/apache/datasketches/memory/internal/ResourceImpl.class")
$spark_package_is_built = true
end
diff --git a/pom.xml b/pom.xml
index 5ab2b89543ef..750b44203944 100644
--- a/pom.xml
+++ b/pom.xml
@@ -215,6 +215,10 @@
<commons-cli.version>1.11.0</commons-cli.version>
<bouncycastle.version>1.84</bouncycastle.version>
<tink.version>1.20.0</tink.version>
+ <!--
+ TODO: Once upgrade datasketches to a version that supports Java 25,
+ SPARK-53327 workaround should be reverted.
+ -->
<datasketches.version>6.2.0</datasketches.version>
<netty.version>4.2.12.Final</netty.version>
<netty-tcnative.version>2.0.77.Final</netty-tcnative.version>
diff --git
a/sql/catalyst/src/main/java/org/apache/datasketches/memory/internal/ResourceImpl.java
b/sql/catalyst/src/main/java/org/apache/datasketches/memory/internal/ResourceImpl.java
new file mode 100644
index 000000000000..0ec73caca72a
--- /dev/null
+++
b/sql/catalyst/src/main/java/org/apache/datasketches/memory/internal/ResourceImpl.java
@@ -0,0 +1,561 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.datasketches.memory.internal;
+
+import static org.apache.datasketches.memory.internal.UnsafeUtil.unsafe;
+import static org.apache.datasketches.memory.internal.Util.characterPad;
+
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import org.apache.datasketches.memory.MemoryBoundsException;
+import org.apache.datasketches.memory.MemoryRequestServer;
+import org.apache.datasketches.memory.ReadOnlyException;
+import org.apache.datasketches.memory.Resource;
+
+// Ported from https://github.com/apache/datasketches-memory/pull/272
+// which relaxes the `checkJavaVersion` to allow Java 25
+
+/**
+ * Implements the root Resource methods plus some common static variables and
check methods.
+ *
+ * @author Lee Rhodes
+ */
+@SuppressWarnings("restriction")
+public abstract class ResourceImpl implements Resource {
+ static final String JDK;
+ static final int JDK_MAJOR; //8, 11, 17, etc
+
+ //Used to convert "type" to bytes: bytes = longs << LONG_SHIFT
+ static final int BOOLEAN_SHIFT = 0;
+ static final int BYTE_SHIFT = 0;
+ static final long SHORT_SHIFT = 1;
+ static final long CHAR_SHIFT = 1;
+ static final long INT_SHIFT = 2;
+ static final long LONG_SHIFT = 3;
+ static final long FLOAT_SHIFT = 2;
+ static final long DOUBLE_SHIFT = 3;
+
+ //class type IDs. Do not change the bit orders
+ //The lowest 3 bits are set dynamically
+ // 0000 0XXX Group 1
+ static final int WRITABLE = 0; //bit 0 = 0
+ static final int READONLY = 1; //bit 0
+ static final int REGION = 2; //bit 1
+ static final int DUPLICATE = 4; //bit 2, for Buffer only
+
+ // 000X X000 Group 2
+ static final int HEAP = 0; //bits 3,4 = 0
+ static final int DIRECT = 8; //bit 3
+ static final int MAP = 16; //bit 4, Map is effectively Direct
+
+ // 00X0 0000 Group 3 ByteOrder
+ static final int NATIVE_BO = 0; //bit 5 = 0
+ static final int NONNATIVE_BO = 32;//bit 5
+
+ // 0X00 0000 Group 4
+ static final int MEMORY = 0; //bit 6 = 0
+ static final int BUFFER = 64; //bit 6
+
+ // X000 0000 Group 5
+ static final int BYTEBUF = 128; //bit 7
+
+ /**
+ * The java line separator character as a String.
+ */
+ public static final String LS = System.getProperty("line.separator");
+
+ static final String NOT_MAPPED_FILE_RESOURCE = "This is not a memory-mapped
file resource";
+ static final String THREAD_EXCEPTION_TEXT = "Attempted access outside owning
thread";
+
+ private static AtomicBoolean JAVA_VERSION_WARNING_PRINTED = new
AtomicBoolean(false);
+
+ static {
+ final String jdkVer = System.getProperty("java.version");
+ final int[] p = parseJavaVersion(jdkVer);
+ JDK = p[0] + "." + p[1];
+ JDK_MAJOR = (p[0] == 1) ? p[1] : p[0];
+ }
+
+ //set by the leaf nodes
+ long capacityBytes;
+ long cumOffsetBytes;
+ long offsetBytes;
+ int typeId;
+ Thread owner = null;
+
+ /**
+ * The root of the Memory inheritance hierarchy
+ */
+ ResourceImpl() { }
+
+ //MemoryRequestServer logic
+
+ /**
+ * User specified MemoryRequestServer. Set here and by leaf nodes.
+ */
+ MemoryRequestServer memReqSvr = null;
+
+ @Override
+ public MemoryRequestServer getMemoryRequestServer() {
+ return memReqSvr;
+ }
+
+ @Override
+ public boolean hasMemoryRequestServer() {
+ return memReqSvr != null;
+ }
+
+ @Override
+ public void setMemoryRequestServer(final MemoryRequestServer memReqSvr) {
this.memReqSvr = memReqSvr; }
+
+ //***
+
+ /**
+ * Check the requested offset and length against the allocated size.
+ * The invariants equation is: {@code 0 <= reqOff <= reqLen <= reqOff +
reqLen <= allocSize}.
+ * If this equation is violated an {@link MemoryBoundsException} will be
thrown.
+ * @param reqOff the requested offset
+ * @param reqLen the requested length
+ * @param allocSize the allocated size.
+ * @throws MemoryBoundsException if the given arguments constitute a
violation
+ * of the invariants equation expressed above.
+ */
+ public static void checkBounds(final long reqOff, final long reqLen, final
long allocSize) {
+ if ((reqOff | reqLen | (reqOff + reqLen) | (allocSize - (reqOff +
reqLen))) < 0) {
+ throw new MemoryBoundsException(
+ "reqOffset: " + reqOff + ", reqLength: " + reqLen
+ + ", (reqOff + reqLen): " + (reqOff + reqLen) + ", allocSize: "
+ allocSize);
+ }
+ }
+
+ /**
+ * Checks the runtime Java Version string. Note that Java 17 and 21 is
allowed only because some clients do not use the
+ * WritableMemory.allocateDirect(..) and related functions, which will not
work with Java versions >= 14.
+ * The on-heap functions may work with 17 and 21, nonetheless, versions >
Java 11 are not officially supported.
+ * Caveat emptor.
+ * @param jdkVer the <i>System.getProperty("java.version")</i> string of the
form "p0.p1.X"
+ * @param p0 The first number group
+ * @param p1 The second number group
+ */
+ static void checkJavaVersion(final String jdkVer, final int p0, final int p1
) {
+ final boolean ok = ((p0 == 1) && (p1 == 8)) || (p0 == 8) || (p0 == 11) ||
(p0 == 17 || (p0 == 21) || (p0 == 25));
+ if (!ok) { throw new IllegalArgumentException(
+ "Unsupported JDK Major Version. It must be one of 1.8, 8, 11, 17, 21,
25: " + jdkVer);
+ }
+ if (p0 > 11 && JAVA_VERSION_WARNING_PRINTED.compareAndSet(false, true)) {
+ System.err.println(
+ "Warning: Java versions > Java 11 can only operate in restricted
mode where no off-heap operations are allowed!");
+ }
+ }
+
+ void checkNotReadOnly() {
+ if (isReadOnly()) {
+ throw new ReadOnlyException("Cannot write to a read-only Resource.");
+ }
+ }
+
+ /**
+ * This checks that the current thread is the same as the given owner thread.
+ * @Throws IllegalStateException if it is not.
+ * @param owner the given owner thread.
+ */
+ static final void checkThread(final Thread owner) {
+ if (owner != Thread.currentThread()) {
+ throw new IllegalStateException(THREAD_EXCEPTION_TEXT);
+ }
+ }
+
+ /**
+ * @throws IllegalStateException if this Resource is AutoCloseable, and
already closed, i.e., not <em>alive</em>.
+ */
+ void checkValid() {
+ if (!isAlive()) {
+ throw new IllegalStateException("this Resource is AutoCloseable, and
already closed, i.e., not <em>alive</em>.");
+ }
+ }
+
+ /**
+ * Checks that this resource is still valid and throws a
MemoryInvalidException if it is not.
+ * Checks that the specified range of bytes is within bounds of this
resource, throws
+ * {@link MemoryBoundsException} if it's not: i. e. if offsetBytes < 0,
or length < 0,
+ * or offsetBytes + length > {@link #getCapacity()}.
+ * @param offsetBytes the given offset in bytes of this object
+ * @param lengthBytes the given length in bytes of this object
+ * @throws IllegalStateException if this resource is AutoCloseable and is no
longer valid, i.e.,
+ * it has already been closed.
+ * @throws MemoryBoundsException if this resource violates the memory bounds
of this resource.
+ */
+ public final void checkValidAndBounds(final long offsetBytes, final long
lengthBytes) {
+ checkValid();
+ checkBounds(offsetBytes, lengthBytes, getCapacity());
+ }
+
+ /**
+ * Checks that this resource is still valid and throws a
MemoryInvalidException if it is not.
+ * Checks that the specified range of bytes is within bounds of this
resource, throws
+ * {@link MemoryBoundsException} if it's not: i. e. if offsetBytes < 0,
or length < 0,
+ * or offsetBytes + length > {@link #getCapacity()}.
+ * Checks that this operation is a read-only operation and throws a
ReadOnlyException if not.
+ * @param offsetBytes the given offset in bytes of this object
+ * @param lengthBytes the given length in bytes of this object
+ * @Throws MemoryInvalidException if this resource is AutoCloseable and is
no longer valid, i.e.,
+ * it has already been closed.
+ * @Throws MemoryBoundsException if this resource violates the memory bounds
of this resource.
+ * @Throws ReadOnlyException if the associated operation is not a Read-only
operation.
+ */
+ final void checkValidAndBoundsForWrite(final long offsetBytes, final long
lengthBytes) {
+ checkValid();
+ checkBounds(offsetBytes, lengthBytes, getCapacity());
+ if (isReadOnly()) {
+ throw new ReadOnlyException("Memory is read-only.");
+ }
+ }
+
+ @Override
+ public void close() {
+ /* Overridden by the leaf sub-classes that need AutoCloseable. */
+ }
+
+ @Override
+ public final boolean equalTo(final long thisOffsetBytes, final Resource that,
+ final long thatOffsetBytes, final long lengthBytes) {
+ if (that == null) { return false; }
+ return CompareAndCopy.equals(this, thisOffsetBytes, (ResourceImpl) that,
thatOffsetBytes, lengthBytes);
+ }
+
+ @Override
+ public void force() { //overridden by Map Leaves
+ throw new UnsupportedOperationException(NOT_MAPPED_FILE_RESOURCE);
+ }
+
+ //Overridden by ByteBuffer Leaves. Used internally and for tests.
+ ByteBuffer getByteBuffer() {
+ return null;
+ }
+
+ @Override
+ public final ByteOrder getTypeByteOrder() {
+ return isNativeOrder(getTypeId()) ? Util.NATIVE_BYTE_ORDER :
Util.NON_NATIVE_BYTE_ORDER;
+ }
+
+ @Override
+ public long getCapacity() {
+ checkValid();
+ return capacityBytes;
+ }
+
+ @Override
+ public long getCumulativeOffset(final long addOffsetBytes) {
+ return cumOffsetBytes + addOffsetBytes;
+ }
+
+ @Override
+ public long getRelativeOffset() {
+ return offsetBytes;
+ }
+
+ //Overridden by all leaves
+ int getTypeId() {
+ return typeId;
+ }
+
+ //Overridden by Heap and ByteBuffer leaves. Made public as getArray() in
BaseWritableMemoryImpl and BaseWritableBufferImpl
+ Object getUnsafeObject() {
+ return null;
+ }
+
+ @Override
+ public boolean hasByteBuffer() {
+ return (getTypeId() & BYTEBUF) > 0;
+ }
+
+ @Override
+ public final boolean isByteOrderCompatible(final ByteOrder byteOrder) {
+ final ByteOrder typeBO = getTypeByteOrder();
+ return typeBO == ByteOrder.nativeOrder() && typeBO == byteOrder;
+ }
+
+ static final boolean isBuffer(final int typeId) {
+ return (typeId & BUFFER) > 0;
+ }
+
+ @Override
+ public boolean isCloseable() {
+ return (getTypeId() & (MAP | DIRECT)) > 0 && isAlive();
+ }
+
+ @Override
+ public final boolean isDirect() {
+ return getUnsafeObject() == null;
+ }
+
+ @Override
+ public boolean isDuplicate() {
+ return (getTypeId() & DUPLICATE) > 0;
+ }
+
+ @Override
+ public final boolean isHeap() {
+ checkValid();
+ return getUnsafeObject() != null;
+ }
+
+ @Override
+ public boolean isLoaded() { //overridden by Map Leaves
+ throw new IllegalStateException(NOT_MAPPED_FILE_RESOURCE);
+ }
+
+ @Override
+ public boolean isMapped() {
+ return (getTypeId() & MAP) > 0;
+ }
+
+ @Override
+ public boolean isMemory() {
+ return (getTypeId() & BUFFER) == 0;
+ }
+
+ static final boolean isNativeOrder(final int typeId) { //not used
+ return (typeId & NONNATIVE_BO) == 0;
+ }
+
+ @Override
+ public boolean isNonNativeOrder() {
+ return (getTypeId() & NONNATIVE_BO) > 0;
+ }
+
+ @Override
+ public final boolean isReadOnly() {
+ checkValid();
+ return (getTypeId() & READONLY) > 0;
+ }
+
+ @Override
+ public boolean isRegionView() {
+ return (getTypeId() & REGION) > 0;
+ }
+
+ @Override
+ public boolean isSameResource(final Resource that) {
+ checkValid();
+ if (that == null) { return false; }
+ final ResourceImpl that1 = (ResourceImpl) that;
+ that1.checkValid();
+ if (this == that1) { return true; }
+ return getCumulativeOffset(0) == that1.getCumulativeOffset(0)
+ && getCapacity() == that1.getCapacity()
+ && getUnsafeObject() == that1.getUnsafeObject()
+ && getByteBuffer() == that1.getByteBuffer();
+ }
+
+ //Overridden by Direct and Map leaves
+ @Override
+ public boolean isAlive() {
+ return true;
+ }
+
+ @Override
+ public void load() { //overridden by Map leaves
+ throw new IllegalStateException(NOT_MAPPED_FILE_RESOURCE);
+ }
+
+ private static String pad(final String s, final int fieldLen) {
+ return characterPad(s, fieldLen, ' ' , true);
+ }
+
+ /**
+ * Returns first two number groups of the java version string.
+ * @param jdkVer the java version string from
System.getProperty("java.version").
+ * @return first two number groups of the java version string.
+ * @throws IllegalArgumentException for an improper Java version string.
+ */
+ static int[] parseJavaVersion(final String jdkVer) {
+ final int p0, p1;
+ try {
+ String[] parts = jdkVer.trim().split("^0-9\\.");//grab only number
groups and "."
+ parts = parts[0].split("\\."); //split out the number groups
+ p0 = Integer.parseInt(parts[0]); //the first number group
+ p1 = (parts.length > 1) ? Integer.parseInt(parts[1]) : 0; //2nd number
group, or 0
+ } catch (final NumberFormatException | ArrayIndexOutOfBoundsException e) {
+ throw new IllegalArgumentException("Improper Java -version string: " +
jdkVer + LS + e);
+ }
+ checkJavaVersion(jdkVer, p0, p1);
+ return new int[] {p0, p1};
+ }
+
+ //REACHABILITY FENCE
+ static void reachabilityFence(final Object obj) { }
+
+ final static int removeNnBuf(final int typeId) { return typeId &
~NONNATIVE_BO & ~BUFFER; }
+
+ final static int setReadOnlyBit(final int typeId, final boolean readOnly) {
+ return readOnly ? typeId | READONLY : typeId & ~READONLY;
+ }
+
+ /**
+ * Returns a formatted hex string of an area of this object.
+ * Used primarily for testing.
+ * @param state the ResourceImpl
+ * @param preamble a descriptive header
+ * @param offsetBytes offset bytes relative to the MemoryImpl start
+ * @param lengthBytes number of bytes to convert to a hex string
+ * @return a formatted hex string in a human readable array
+ */
+ static final String toHex(final ResourceImpl state, final String preamble,
final long offsetBytes, final int lengthBytes,
+ final boolean withData) {
+ final long capacity = state.getCapacity();
+ ResourceImpl.checkBounds(offsetBytes, lengthBytes, capacity);
+ final StringBuilder sb = new StringBuilder();
+ final Object uObj = state.getUnsafeObject();
+ final String uObjStr;
+ final long uObjHeader;
+ if (uObj == null) {
+ uObjStr = "null";
+ uObjHeader = 0;
+ } else {
+ uObjStr = uObj.getClass().getSimpleName() + ", " + (uObj.hashCode() &
0XFFFFFFFFL);
+ uObjHeader = UnsafeUtil.getArrayBaseOffset(uObj.getClass());
+ }
+ final ByteBuffer bb = state.getByteBuffer();
+ final String bbStr = bb == null ? "null"
+ : bb.getClass().getSimpleName() + ", " + (bb.hashCode() &
0XFFFFFFFFL);
+ final MemoryRequestServer memReqSvr = state.getMemoryRequestServer();
+ final String memReqStr = memReqSvr != null
+ ? memReqSvr.getClass().getSimpleName() + ", " + (memReqSvr.hashCode()
& 0XFFFFFFFFL)
+ : "null";
+ final long cumBaseOffset = state.getCumulativeOffset(0);
+ sb.append(preamble).append(LS);
+ sb.append("UnsafeObj, hashCode : ").append(uObjStr).append(LS);
+ sb.append("UnsafeObjHeader : ").append(uObjHeader).append(LS);
+ sb.append("ByteBuf, hashCode : ").append(bbStr).append(LS);
+ sb.append("RegionOffset :
").append(state.getRelativeOffset()).append(LS);
+ if (ResourceImpl.isBuffer(state.typeId)) {
+ sb.append("Start :
").append(((PositionalImpl)state).getStart()).append(LS);
+ sb.append("Position :
").append(((PositionalImpl)state).getPosition()).append(LS);
+ sb.append("End :
").append(((PositionalImpl)state).getEnd()).append(LS);
+ }
+ sb.append("Capacity : ").append(capacity).append(LS);
+ sb.append("CumBaseOffset : ").append(cumBaseOffset).append(LS);
+ sb.append("MemReqSvr, hashCode : ").append(memReqStr).append(LS);
+ sb.append("is Alive : ").append(state.isAlive()).append(LS);
+ sb.append("Read Only : ").append(state.isReadOnly()).append(LS);
+ sb.append("Type Byte Order :
").append(state.getTypeByteOrder().toString()).append(LS);
+ sb.append("Native Byte Order :
").append(ByteOrder.nativeOrder().toString()).append(LS);
+ sb.append("JDK Runtime Version : ").append(JDK).append(LS);
+ //Data detail
+ if (withData) {
+ sb.append("Data, bytes : 0 1 2 3 4 5 6 7");
+
+ for (long i = 0; i < lengthBytes; i++) {
+ final int b = unsafe.getByte(uObj, cumBaseOffset + offsetBytes + i) &
0XFF;
+ if (i % 8 == 0) { //row header
+ sb.append(String.format("%n%20s: ", offsetBytes + i));
+ }
+ sb.append(String.format("%02x ", b));
+ }
+ sb.append(LS);
+ }
+ sb.append("### END SUMMARY");
+ return sb.toString();
+ }
+
+ @Override
+ public final String toString(final String header, final long offsetBytes,
final int lengthBytes,
+ final boolean withData) {
+ checkValid();
+ final String klass = this.getClass().getSimpleName();
+ final String s1 = String.format("(..., %d, %d)", offsetBytes, lengthBytes);
+ final long hcode = hashCode() & 0XFFFFFFFFL;
+ final String call = ".toHexString" + s1 + ", hashCode: " + hcode;
+ final StringBuilder sb = new StringBuilder();
+ sb.append("### ").append(klass).append(" SUMMARY ###").append(LS);
+ sb.append("Type Info : ").append(typeDecode(typeId)).append(LS +
LS);
+ sb.append("Header Comment : ").append(header).append(LS);
+ sb.append("Call Parameters : ").append(call);
+ return toHex(this, sb.toString(), offsetBytes, lengthBytes, withData);
+ }
+
+ @Override
+ public final String toString() {
+ return toString("", 0, (int)this.getCapacity(), false);
+ }
+
+ /**
+ * Decodes the resource type. This is primarily for debugging.
+ * @param typeId the given typeId
+ * @return a human readable string.
+ */
+ static final String typeDecode(final int typeId) {
+ final StringBuilder sb = new StringBuilder();
+ final int group1 = typeId & 0x7;
+ switch (group1) { // 0000 0XXX
+ case 0 : sb.append(pad("Writable + ",32)); break;
+ case 1 : sb.append(pad("ReadOnly + ",32)); break;
+ case 2 : sb.append(pad("Writable + Region + ",32)); break;
+ case 3 : sb.append(pad("ReadOnly + Region + ",32)); break;
+ case 4 : sb.append(pad("Writable + Duplicate + ",32)); break;
+ case 5 : sb.append(pad("ReadOnly + Duplicate + ",32)); break;
+ case 6 : sb.append(pad("Writable + Region + Duplicate + ",32)); break;
+ case 7 : sb.append(pad("ReadOnly + Region + Duplicate + ",32)); break;
+ default: break;
+ }
+ final int group2 = (typeId >>> 3) & 0x3;
+ switch (group2) { // 000X X000
+ case 0 : sb.append(pad("Heap + ",15)); break;
+ case 1 : sb.append(pad("Direct + ",15)); break;
+ case 2 : sb.append(pad("Map + Direct + ",15)); break;
+ case 3 : sb.append(pad("Map + Direct + ",15)); break;
+ default: break;
+ }
+ final int group3 = (typeId >>> 5) & 0x1;
+ switch (group3) { // 00X0 0000
+ case 0 : sb.append(pad("NativeOrder + ",17)); break;
+ case 1 : sb.append(pad("NonNativeOrder + ",17)); break;
+ default: break;
+ }
+ final int group4 = (typeId >>> 6) & 0x1;
+ switch (group4) { // 0X00 0000
+ case 0 : sb.append(pad("Memory + ",9)); break;
+ case 1 : sb.append(pad("Buffer + ",9)); break;
+ default: break;
+ }
+ final int group5 = (typeId >>> 7) & 0x1;
+ switch (group5) { // X000 0000
+ case 0 : sb.append(pad("",10)); break;
+ case 1 : sb.append(pad("ByteBuffer",10)); break;
+ default: break;
+ }
+ return sb.toString();
+ }
+
+ @Override
+ public final long xxHash64(final long offsetBytes, final long lengthBytes,
final long seed) {
+ checkValid();
+ return XxHash64.hash(getUnsafeObject(), getCumulativeOffset(0) +
offsetBytes, lengthBytes, seed);
+ }
+
+ @Override
+ public final long xxHash64(final long in, final long seed) {
+ return XxHash64.hash(in, seed);
+ }
+
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
