DadanielZ commented on a change in pull request #1890: HADOOP-16854 Fix to
prevent OutOfMemoryException and Make the threadpool and bytebuffer pool common
across all AbfsOutputStream instances
URL: https://github.com/apache/hadoop/pull/1890#discussion_r393387356
##########
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsOutputStream.java
##########
@@ -285,14 +311,22 @@ private synchronized void writeCurrentBufferToService()
throws IOException {
final byte[] bytes = buffer;
final int bytesLength = bufferIndex;
- buffer = byteBufferPool.getBuffer(false, bufferSize).array();
bufferIndex = 0;
final long offset = position;
position += bytesLength;
- if (threadExecutor.getQueue().size() >= maxConcurrentRequestCount * 2) {
+ if (Runtime.getRuntime().freeMemory() < MIN_MEMORY_THRESHOLD
+ && buffersToBeReturned.get() > 0) {
+ LOG.debug("Low memory");
Review comment:
what if` Runtime.getRuntime().freeMemory() < MIN_MEMORY_THRESHOLD` but
`buffersToBeReturned.get()` is `0`?
Can we make the LOG more meaningful like "Low memory, less than 100 MB is
available, the existing read tasks consumes 2048 MB, waiting for completion."
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
