jvdvegt opened a new issue, #469:
URL: https://github.com/apache/maven-build-cache-extension/issues/469
### New feature, improvement proposal
In our CI/CD pipeline (Linux, Java 21), we encountered this error:
```
Caused by: java.lang.OutOfMemoryError: Cannot reserve 1490778611 bytes of
direct buffer memory (allocated: 1768309554, limit: 3221225472)
at java.nio.Bits.reserveMemory (Bits.java:178)
at java.nio.DirectByteBuffer.<init> (DirectByteBuffer.java:127)
at java.nio.ByteBuffer.allocateDirect (ByteBuffer.java:360)
at sun.nio.ch.Util.getTemporaryDirectBuffer (Util.java:242)
at sun.nio.ch.IOUtil.read (IOUtil.java:303)
at sun.nio.ch.IOUtil.read (IOUtil.java:283)
at sun.nio.ch.FileChannelImpl.read (FileChannelImpl.java:234)
at sun.nio.ch.ChannelInputStream.read (ChannelInputStream.java:74)
at sun.nio.ch.ChannelInputStream.read (ChannelInputStream.java:103)
at java.nio.file.Files.read (Files.java:3224)
at java.nio.file.Files.readAllBytes (Files.java:3275)
at org.apache.maven.buildcache.hash.SHA$Algorithm.hash (SHA.java:70)
at org.apache.maven.buildcache.hash.HashAlgorithm.hash
(HashAlgorithm.java:36)
at org.apache.maven.buildcache.CacheControllerImpl.artifactDto
(CacheControllerImpl.java:640)
at org.apache.maven.buildcache.CacheControllerImpl.artifactDtos
(CacheControllerImpl.java:628)
at org.apache.maven.buildcache.CacheControllerImpl.save
(CacheControllerImpl.java:508)
at org.apache.maven.buildcache.BuildCacheMojosExecutionStrategy.execute
(BuildCacheMojosExecutionStrategy.java:160)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute
(MojoExecutor.java:159)
...```
We're using the build cache for a 1.7GB artifact. The build cache code
currently reads all bytes of an artifact into a byte array, and computes the
SHA-hash in one go:
https://github.com/apache/maven-build-cache-extension/blob/b827d52410d96375e2e5ebafe3b95cf559649809/src/main/java/org/apache/maven/buildcache/hash/SHA.java#L70
.
For some hashes, there is a 'zero allocation hash' implementation:
https://github.com/apache/maven-build-cache-extension/blob/b827d52410d96375e2e5ebafe3b95cf559649809/src/main/java/org/apache/maven/buildcache/hash/Zah.java#L101
. Unfortunately, there is no such version for SHA (all available hashes are
defined
[here](https://github.com/apache/maven-build-cache-extension/blob/b827d52410d96375e2e5ebafe3b95cf559649809/src/main/java/org/apache/maven/buildcache/hash/HashFactory.java#L40)).
Changing the SHA-implementation to use memory-mapping seems pretty straight
forward, but changing the current behaviour might not be desirable. If I were
to add a memory-mapped SHA implementation, would it be okay to add 4
SHA-entries to the HashFactory with `_MM` added to their name?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
