franz1981 edited a comment on pull request #3711: URL: https://github.com/apache/activemq-artemis/pull/3711#issuecomment-908895949
I see I didn't state clearly on https://issues.apache.org/jira/browse/ARTEMIS-3449 that the major benefit of this change is not just speeding up file reading, but it reducing the chances of OOM due to lack of native resources too (see [io_util::readBytes](https://github.com/frohoff/jdk8u-jdk/blob/da0da73ab82ed714dc5be94acd2f0d00fbdfe2e9/src/share/native/java/io/io_util.c#L95-L99) for reference). In the original code, byte[] was used to perform read vs `SequentialFile`, using a `RandomAccessFile` under the hood as explained on https://github.com/apache/activemq-artemis/pull/2844#issuecomment-533606931. Sadly, `RandomAccessFile::read` is using platform `malloc/free` (for writing/reading > [8192](https://github.com/frohoff/jdk8u-jdk/blob/da0da73ab82ed714dc5be94acd2f0d00fbdfe2e9/src/share/native/java/io/io_util.c#L58), that's > of the AMQP default max frame size on the broker) under the hood that can massively degrade over time due to fragmentation (if using the default system memory allocator) until been unable to allocate anything, eventually causing broker to hit OOM. With the current approach of this PR we both save this to happen while benefitting from the speedup mentioned in https://github.com/apache/activemq-artemis/pull/2844#issuecomment-533906809 and reported here: ``` Benchmark (chunkSize) (fileName) (fileSize) (ioSize) Mode Cnt Score Error Units ChunkedFileBenchmark.readFullyFile 32768 /tmp/raf.tmp 10485760 32768 avgt 4 2153.527 ± 715.641 us/op ChunkedFileBenchmark.readFullyFileWithNio 32768 /tmp/raf.tmp 10485760 32768 avgt 4 1775.999 ± 831.715 us/op ``` `readFullyFile` is using `byte[]` , while `readFullyFileWithNio` is using direct `ByteBuffer` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
