This is an automated email from the ASF dual-hosted git repository.
leerho pushed a commit to branch update_7.0.X
in repository https://gitbox.apache.org/repos/asf/datasketches-memory.git
The following commit(s) were added to refs/heads/update_7.0.X by this push:
new b604d73b update update_7.0.X
b604d73b is described below
commit b604d73b4b1557ae3cec24baba2cfa1b78d76393
Author: Lee Rhodes <[email protected]>
AuthorDate: Sun May 31 17:52:06 2026 -0700
update update_7.0.X
---
.github/workflows/auto-jdk-os-matrix.yml | 2 +-
README.md | 278 ++++++++++++-------------------
docs/maven.md | 50 ------
3 files changed, 111 insertions(+), 219 deletions(-)
diff --git a/.github/workflows/auto-jdk-os-matrix.yml
b/.github/workflows/auto-jdk-os-matrix.yml
index 20e80ce4..ffa710fd 100644
--- a/.github/workflows/auto-jdk-os-matrix.yml
+++ b/.github/workflows/auto-jdk-os-matrix.yml
@@ -1,4 +1,4 @@
-name: Auto JDK & OS Matrix Test & Install
+name: JDK & OS Matrix Test
on:
# push:
diff --git a/README.md b/README.md
index d103a97f..ce8c68c6 100644
--- a/README.md
+++ b/README.md
@@ -23,214 +23,156 @@
=================
-# DataSketches Java Memory Component
-This Memory component is general purpose, has no external runtime dependencies
and can be used in any
-application that needs to manage data structures inside or outside the Java
heap.
+# DataSketches Memory Project
+This Memory Project was initially developed to enable the DataSketches Java
Project to manage off-heap data structures efficiently. The first version was
developed as an internal project at Yahoo in 2014 using Java 8 and released to
open source in 2015. This Memory Project is general purpose, has no external
runtime dependencies and can be used in any application that needs to manage
data structures on or off the Java heap.
-Please visit the main [DataSketches website](https://datasketches.apache.org)
for more information.
+## Basic Capabilities
+* **Speed**: Fast Read & Write using Unsafe
+* **Types**: All Java primitives plus Java Strings
+* **Target**: A contiguous *region* of on-heap or off-heap memory. (Note: Java
FFM later named this *region* a *MemorySegment*)
+* **Addressing**: Arbitrary Byte Offset (long) as well as relative positional
offset.
+* **Size**: A region of off-heap memory may exceed the standard Java limit of
2^31-1 bytes.
-If you are interested in making contributions to this Memory component please
see our
-[Community](https://datasketches.apache.org/docs/Community/) page.
+To make this useful, this project also provided:
-The goal of this component of the DataSketches library is to provide a high
performance access API for accessing six different types of memory resources.
Each of the six resource types is accessed using different API methods.
+## Capability Set A:
+* Wrapping of Java primitive arrays as a region.
+* Wrapping of ByteBuffers (both on and off-heap) as a region.
+* Creation and immediate closing of off-heap regions of memory.
+* Creation and immediate closing of off-heap memory-mapped regions of a file.
-Note: *primitive* = *{byte, short, int, long, float, double}*
+Providing these capabilities without sacrificing performance required
leveraging internal JVM classes such as *Unsafe*, *FileChannelImpl*,
*DirectBuffer*, *MappedByteBuffer internals* and *sun.nio.ch*.
-### Contiguous bytes on the Java Heap constructed by:
+However, starting with Java 11, Oracle started restricting access to a many
JVM internal classes. With Java 17 the internals of *FileChannelImpl* and
*MappedByteBuffer* were hidden or relocated. This blocked the ability to
provide direct access to file-mapped memory.
-* **Heap:**
- * *Memory.wrap(primitive[])*
- * *WritableMemory.allocate(int)*
- * *WritableMemory.writableWrap(primitive[])*
+Meanwhile, the Java Panama project, which had the potential to provide all the
above capabilities as part of the Java language, became available as the
Foreign Function and Memory (FFM) API in the form of "Incubation" code in Java
17 and as "Preview" code in Java 21.
-* **Heap via ByteBuffer**
- * *Memory.wrap(ByteBuffer.allocate(int))*
- * *WritableMemory.writableWrap(ByteBuffer.allocate(int))*
+Unfortunately, the DataSketches Memory Project releases 4.X, 5.X and 6.X for
Java 17 and 21 were developed using the Incubation and Preview versions of FFM
not realizing that Incubation and Preview codes are only available until the
next java version is released, after which they are no longer available.
-### Contiguous bytes off the Java Heap constructed by:
+The actual LTS release of FFM did not occur until Java 25 was released on
September 16, 2025. Since Java 17 was released on September 14, 2021, there is
a gap of 4 years where Oracle had blocked access to internals of the JVM
without providing a LTS release API with comparable capabilities. This means
that for the LTS Java versions 17 and 21, the DataSketches Memory Project could
not provide access to file-mapped memory. Nonetheless, it is still possible to
provide these capabilities:
-* **Direct:**
- * *WritableMemory.allocateDirect(long)* method.
+## Capability Set B:
+* Wrapping of Java primitive arrays as a region.
+* Wrapping of ByteBuffers (both on and off-heap) as a region.
+* Creation and immediate closing of off-heap regions of memory.
-* **Direct via ByteBuffer**
- * *WritableMemory.writableWrap(ByteBuffer.allocateDirect(int))*
+The objective of this 7.X release of the Memory Project is to provide
**Capability Set A** with Java 11 and **Capability Set B** for Java versions 17
and 21 for users that are still dependent on the these older LTS releases.
-* **Memory-Mapped Files**
- * *WritableMemory.writableMap(File)* method.
+In addition, with this release the build code has been completely refactored:
-# Releases 3.0.0 (inclusive) to 4.0.0 (exclusive)
-These are transitional releases that also supports Java 8 and 11.
-However, the goal of this set of releases is to migrate the API to what it
will be in release 4.0.0.
-The 4.0.0 release will require Java 17 and will utilize the Project Panama
(FFM) capabilites introduced in Java 17.
+* A clean all-Maven build, install and deploy process ( no shell scripts
required ).
+* New GitHub Actions Workflows that test this project in matrix of 3 OSs
(Windows, MacOS, Ubuntu) and 3 Java versions (11, 17, 21).
+* All code jar files, including the test jars, can be successfully run as jars.
+* The Apache release source is a multi-module Maven project
+* The the release to Maven Central is two jar sets as two new artifacts:
+ * datasketches-memory-java11
+ * datasketches-memory-java17_21
+* All documentation has been brought up-to-date.
-Some of the capabilites of releases [2.0.0, 3.0.0) have been removed because
they cannot be supported in release 4.0.0 with Panama-17.
-For example:
+Because the FFM capabilities built into Java 25 (released September, 2025) are
a major superset of this Memory Project's capabilities (open sourced in 2015),
all of these capabilities are now available directly from the Java Language.
There is little reason to continue to provide new releases of this project.
Alas, this project must come to an end. Any further releases will only be for
bug fixes. Nonetheless, we were able to provide these fast, low-level, off-heap
memory capabilities for [...]
-* The ability to directly import and export UTF-8 encoded strings has been
removed.
- * The UTF-8 code in Memory 1.0.0 through 2.2.1 is out of date and needed
to be removed anyway.
- * Java already has built-in UTF-8 encoding.
-* The ability to directly import and export boolean arrays has been removed.
- * The Java Language Specification does not define the byte storage format
for boolean arrays, thus Panama-17 doesn't support boolean arrays.
-* Other minor changes to the API is best understood by examining the Javadocs
directly.
+## Comments about ByteBuffer
+Although the ByteBuffer provides similar capabilities, it has some severe
limitations.
-Release 3.0.0 includes two bug fixes (Issues #194, #195).
+* The deallocation is governed by the Garbage Collector. In large systems this
delay can lead to out-of-memory failures.
+* The addressing is limited to 2^31 -1 elements.
+* Reading and writing multi-byte primitive arrays require recasting the
ByteBuffer to a <*type*>Buffer where the alignment becomes modulo the *type*.
This severely complicates the creation of packed structs.
+* The API has some serious footguns, which can catch even experienced
programmers and can be challenging to debug:
+ * Every duplicate() or slice() resets the endianness to Big Endian.
+ * Numerous common operations silently discard the *mark*.
-It is our expectation that this set of releases will be the last that support
Java 8 and 11.
+----
-The comments in the following section for releases [2.0.0, 3.0.0) still apply.
+## Release Comments
-# Releases 2.0.0 (inclusive) to 3.0.0 (exclusive)
-Starting with release *datasketches-memory-2.0.0*, this Memory component
supports Java 8 and 11. Providing access to the off-heap resources in Java 8
only requires reflection. However, **Java 9 introduced the Java Platform Module
System (JPMS) where access to these internal classes requires starting up the
JVM with special JPMS arguments.** The actual JVM arguments required will
depend on how the user intends to use the Memory API, the Java version used to
run the user's application and [...]
+### For Use With Java 8
+Please download datasketches-memory 3.X from the website
[downloads](https://datasketches.apache.org/docs/Community/Downloads.html) page
and refer to the README.md there.
+The 3.X set of releases will be the last that support Java 8.
-Also see the [usage examples](docs/usage-examples.md) for more information.
+### Releases 4.X, 5.X and 6.X
+Deprecated with the 7.X releases.
-## USE AS A LIBRARY (using jars from Maven Central)
-In this environment, the user is using the Jars from Maven Central as a
library dependency and not attempting to build the Memory component from the
source code or run the Memory component tests.
+### Releases 7.X (THIS RELEASE)
+* This release no longer uses JPMS but the jar manifests do provide an
Automatic Module Name.
+* In addition to the source code obtained from Apache, this release provides
two sets of jars in Maven Central based on two artifacts:
-* If you are running Java 8, no extra JVM arguments are required.
-* If you are running Java 11-13 and only using the **Heap** related API, no
extra JVM arguments are required.
+#### Artifact: datasketches-memory-java11 (Capability Set A)
+This artifact only supports Java 11
-Otherwise, if you are running Java 11-13 and ...
+#### Artifact: datasketches-memory-java17_21 (Capability Set B)
+This artifact only supports Java 17 and Java 21
-* If your application **is not a JPMS module** use the following table. Choose
the columns that describe your use of the Memory API. If any of the columns
contain a *Yes*, then the JVM argument in the first column of the row
containing a *Yes* will be required. If you are not sure the extent of the
Memory API being used, there is no harm in specifying all 4 JVM arguments.
Note: do not embed any spaces in the full argument.
+----
-| JVM Arguments for non-JPMS Applications | Direct ByteBuffer |
Direct | MemoryMapped Files |
-| :----------------------------------------------------: | :---------------: |
:----: | :----------------: |
-| --add-exports java.base/jdk.internal.misc= ALL-UNNAMED | |
Yes | |
-| --add-exports java.base/jdk.internal.ref= ALL-UNNAMED | |
Yes | Yes |
-| --add-opens java.base/java.nio= ALL-UNNAMED | |
Yes | Yes |
-| --add-opens java.base/sun.nio.ch= ALL-UNNAMED | |
| Yes |
+## LIBRARY USAGE (using jars from Maven Central)
+In this environment, the user is using the Jars from Maven Central as a
library dependency and not attempting to build this Memory Project from the
source code.
-* If your application **is a JPMS module** use the following table. Choose the
columns that describe your use of the Memory API. If any of the columns
contain a *Yes*, then the JVM argument in the first column of the row
containing a *Yes* will be required. If you are not sure the extent of the
Memory API being used, there is no harm in specifying all 4 JVM arguments.
Note: do not embed any spaces in the full argument.
-| JVM Arguments for JPMS Applications |
Direct ByteBuffer | Direct | MemoryMapped Files |
-| :-----------------------------------------------------------------------: |
:---------------: | :----: | :----------------: |
-| --add-exports java.base/jdk.internal.misc= org.apache.datasketches.memory |
| Yes | |
-| --add-exports java.base/jdk.internal.ref= org.apache.datasketches.memory |
| Yes | Yes |
-| --add-opens java.base/java.nio= org.apache.datasketches.memory |
| Yes | Yes |
-| --add-opens java.base/sun.nio.ch= org.apache.datasketches.memory |
Yes | | Yes |
+### If you are running Java 11 you must use artifact
*datasketches-memory-java11*
+You will need to add these java modular arguments to your JVM:
+* --add-exports=java.base/jdk.internal.misc=ALL-UNNAMED
+* --add-exports=java.base/sun.nio.ch=ALL-UNNAMED
+* --add-opens=java.base/java.nio=ALL-UNNAMED
+* --add-opens=java.base/sun.nio.ch=ALL-UNNAMED
-## DEVELOPER USAGE
-In this environment the developer needs to build the Memory component from
source and run the Memory Component tests. There are two use cases. The first
is for a *System Developer* that needs to build and test their own Jar from
source for a specific Java version. The second use case is for a *Memory
Component Developer and Contributor*.
+### If you are running Java 17 or Java 21 you must use artifact
*datasketches-memory-java17_21*
+You will need to add these java modular arguments to your JVM:
-* System Developer
- * Compile, test and create a Jar for a specific Java version
- * use the provided script for this purpose
+* --add-exports=java.base/jdk.internal.misc=ALL-UNNAMED
+* --add-exports=java.base/sun.nio.ch=ALL-UNNAMED
-* Memory Component Developer / Contributor
- * Compile & test the library from source code using:
- * Eclipse (version)
- * IntelliJ (version)
- * Maven (version)
- * Command-line or scripts
- * The developer must have installed in their development system at least
JDK versions 8 and 11.
- * Unless building with the provided script, the developer must have a
valid [Maven toolchain configuration](docs/maven-toolchains.md).
+## DEVELOPER USAGE
+In this environment the developer needs to build the Memory component from
source and run the Memory Component tests.
+* Fork this project at https://github.com/apache/datasketches-memory into your
own GitHub development site. The 7.0.X releases will be located in the branch
7.0.X.
+* Provide a
[toolchains.xml](https://maven.apache.org/guides/mini/guide-using-toolchains.html)
in your local system at `~/.m2/toolchains.xml` and configured with at least
JDK versions 11, 17 and 21.
+* You must have gpg-agent running in your terminal.
+* Your local machine JAVA_HOME should be set to at least Java 8, but Java 17
or higher is recommended.
-### Build Instructions
+### Maven Build Instructions
__NOTES:__
-1) This component accesses resource files for testing. As a result, the
directory elements
- of the full absolute path of the target installation directory must qualify
as Java identifiers.
- In other words, the directory elements must not have any space characters
(or non-Java identifier
- characters) in any of the path elements. This is required by the Oracle
Java Specification in
- order to ensure location-independent access to resources:
- [See Oracle Location-Independent Access to
Resources](https://docs.oracle.com/javase/8/docs/technotes/guides/lang/resources.html)
-
-#### Dependencies
-There are no run-time dependencies. See the pom.xml file for test dependencies.
-
-#### Maven build instructions
-The Maven build requires the following JDKs to compile:
+* This component accesses resource files for testing. As a result, the
directory elements of the full absolute path of the target installation
directory must qualify as Java identifiers. In other words, the directory
elements must not have any space characters (or non-Java identifier characters)
in any of the path elements. This is required by the Oracle Java Specification
in order to ensure location-independent access to resources: [See Oracle
Location-Independent Access to Resources](h [...]
-- JDK8/Hotspot
-- JDK11/Hotspot
+__TESTS:__
-Before building, first ensure that your local environment has been configured
according to the [Maven Toolchains Configuration](docs/maven-toolchains.md).
+* There are two types of tests: unit tests and continuous integration (CI)
tests.
+The CI tests target the released jars and run all the unit tests against Java
versions 11, 17 and 21 and three different OSs, Windows, MacOS and Ubuntu.
-There are two types of tests: normal unit tests and continuous integration(CI)
tests.
-The CI tests target the Multi-Release (MR) JAR and run the entire test suite
using a specific version of Java. Running the CI test command also runs the
default unit tests.
-
-To run normal unit tests:
-
- mvn clean test
+To run normal unit tests on the forked site:
+* mvn clean test
+* The test results can be found in `/target/test-output`
To run javadoc on this multi-module project, use:
-
- mvn clean javadoc:javadoc -DskipTests=true
-
-To build the multi-release JAR, use:
-
- mvn clean package
-
-To run the eclipse plugin on this multi-module project, use:
-
- mvn clean eclipse:eclipse -DskipTests=true
-
-To install jars built from the downloaded source:
-
- mvn clean install -DskipTests=true
-
-This will create the following Jars:
-
-* datasketches-memory-X.Y.Z.jar The compiled main class files.
-* datasketches-memory-X.Y.Z-tests.jar The compiled test class files.
-* datasketches-memory-X.Y.Z-sources.jar The main source files.
-* datasketches-memory-X.Y.Z-test-sources.jar The test source files
-* datasketches-memory-X.Y.Z-javadoc.jar The compressed Javadocs.
-
-#### Building for a specific java version
-
-A build script named **package-single-release-jar.sh** has been provided to
package a JAR for a specific java version. This is necessary in cases where a
developer is unable to install all the required versions of the JDK that are
required as part of the Maven build.
-
-The build script performs the following steps:
-
-1. Sets up staging directories under **target/** for the package files
-2. Uses git commands to gather information about the current Git commit and
branch
-3. Compiles java source tree
-4. Packages a JAR containing compiled sources together with the Manifest,
License and Notice files
-5. Checks and tests the assembled JAR by using the API to access four
different resource types
-
-The build script is located in the **tools/scripts/** directory and requires
the following arguments:
-
-* JDK Home Directory - The first argument is the absolute path of JDK home
directory e.g. $JAVA_HOME
-* Git Version Tag - The second argument is the Git Version Tag for this
deployment e.g. 1.0.0-SNAPSHOT, 1.0.0-RC1, 1.0.0 etc.
-* Project Directory - The third argument is the absolute path of
project.basedir e.g. /src/apache-datasketches-memory
-
-For example, if the project base directory is `/src/datasketches-memory`;
-
-To run the script for a release version:
-
- ./tools/scripts/package-single-release-jar.sh $JAVA_HOME 2.1.0
/src/datasketches-memory
-
-To run the script for a snapshot version:
-
- ./tools/scripts/package-single-release-jar.sh $JAVA_HOME 2.2.0-SNAPSHOT
/src/datasketches-memory
-
-To run the script for an RC version:
-
- ./tools/scripts/package-single-release-jar.sh $JAVA_HOME 2.1.0-RC1
/src/datasketches-memory
-
-Note that the script does **not** use the _Git Version Tag_ to adjust the
working copy to a remote tag - it is expected that the user has a pristine copy
of the desired branch/tag available **before** using the script.
+* mvn clean javadoc:javadoc -DskipTests=true
+* The javadocs will be located in `/target/site/apidocs/`
+
+To install the jars in ~/.m2
+* mvn clean install -DskipTests=true
+
+This will create the following sets of Jars and POM, with associated GPG .asc
signatures in the folder `~/.m2/repository/org/apache/datasketches/` and under
+
+`datasketches-memory-java11/X.Y.Z/`:
+* datasketches-memory-java11-X.Y.Z.jar
+* datasketches-memory-java11-X.Y.Z-tests.jar
+* datasketches-memory-java11-X.Y.Z-sources.jar
+* datasketches-memory-java11-X.Y.Z-test-sources.jar
+* datasketches-memory-java11-X.Y.Z-javadoc.jar
+* datasketches-memory-java11-X.Y.Z-pom
+
+And under:
+`datasketches-memory-java17_21/X.Y.Z/`:
+* datasketches-memory-java17_21-X.Y.Z.jar
+* datasketches-memory-java17_21-X.Y.Z-tests.jar
+* datasketches-memory-java17_21-X.Y.Z-sources.jar
+* datasketches-memory-java17_21-X.Y.Z-test-sources.jar
+* datasketches-memory-java17_21-X.Y.Z-javadoc.jar
+* datasketches-memory-java17_21-X.Y.Z-pom
---
-### Further documentation for contributors
-
-For more information on the project configuration, the following topics are
discussed in more
-detail:
+Please visit the main [DataSketches website](https://datasketches.apache.org)
for more information about the DataSketches Library and all the different
languages it is available in.
-* [Maven Configuration](docs/maven.md)
-* [Maven Toolchains Configuration](docs/maven-toolchains.md)
-* [Multi-Release Jar](docs/multi-release-jar.md)
-* [Java Platform Module System](docs/module-system.md)
-* [Usage examples](docs/usage-examples.md)
-
-In order to build and contribute to this project, please read the relevant IDE
documentation:
-
-- [Eclipse IDE Setup](docs/eclipse.md)
-- [IntelliJ IDE Setup](docs/intellij.md)
-
-For releasing to AppNexus, please use the `sign-deploy-jar.sh` script in the
scripts directory.
-See the documentation within the script for usage instructions.
+If you are interested in making contributions to this Memory Project or to any
of the other DataSketches Projects please visit our
+[Community](https://datasketches.apache.org/docs/Community/) page.
diff --git a/docs/maven.md b/docs/maven.md
deleted file mode 100644
index 5fd9868f..00000000
--- a/docs/maven.md
+++ /dev/null
@@ -1,50 +0,0 @@
-<!--
- Licensed to the Apache Software Foundation (ASF) under one
- or more contributor license agreements. See the NOTICE file
- distributed with this work for additional information
- regarding copyright ownership. The ASF licenses this file
- to you under the Apache License, Version 2.0 (the
- "License"); you may not use this file except in compliance
- with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing,
- software distributed under the License is distributed on an
- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- KIND, either express or implied. See the License for the
- specific language governing permissions and limitations
- under the License.
--->
-
-# Maven Configuration: Multi-Module Project
-
-This project is a multi-module Maven project. A multi-module Maven project
consists of an
-aggregator project (the `datasketches-memory` project), together with a set of
submodules.
-The aggregator's configuration is inherited by each submodule, thus reducing
duplication.
-
-Datasketches Memory makes use of some features of the Java platform, for
example, `Unsafe`,
-which have evolved in Java versions 9 and above. Therefore, a multi-module
project allows us to
-add support for later versions of Java by using independent Maven modules to
target
-platform-specific APIs. For example, to deallocate references a
`sun.misc.Cleaner` will be used
-in Java8, but the `jdk.internal.ref.Cleaner` is used in Java 11.
-
-This project has been divided into the following submodules:
-
-* datasketches-memory-java8 (base version of the JVM that is currently
supported)
-* datasketches-memory-java11 (Java11 equivalent of some platform specific
classes in
-
-### Artifact assembly
-
-During the Maven package phase, the [Maven exec
plugin](https://www.mojohaus.org/exec-maven-plugin/index.html)
-invokes a custom bash script `package-mr-jar.sh` located in the
`tools/scripts` directory.
-This script assembles sources and compiled classes from the individual maven
modules into
-a coherent set of JARs that appear to the end user to originate from a single
module.
-
-The following jars are assembled by the bash package script:
-
-* datasketches-memory-X.Y.Z.jar The compiled main class files.
-* datasketches-memory-X.Y.Z-tests.jar The compiled test class files.
-* datasketches-memory-X.Y.Z-sources.jar The main source files.
-* datasketches-memory-X.Y.Z-test-sources.jar The test source files
-* datasketches-memory-X.Y.Z-javadoc.jar The compressed Javadocs.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
