odraese commented on a change in pull request #626: [HIVE-21686] ensure that
memory allocator does not evict using brute …
URL: https://github.com/apache/hive/pull/626#discussion_r283519647
##########
File path:
llap-server/src/java/org/apache/hadoop/hive/llap/cache/BuddyAllocator.java
##########
@@ -277,7 +301,74 @@ public void allocateMultiple(MemoryBuffer[] dest, int
size, BufferObjectFactory
long threadId = arenaCount > 1 ? Thread.currentThread().getId() : 0;
int destAllocIx = allocateFast(dest, null, 0, dest.length,
freeListIx, allocationSize, (int)(threadId % arenaCount), arenaCount);
- if (destAllocIx == dest.length) return;
+ if (destAllocIx == dest.length) {
+ return;
+ }
+ // Try to allocate memory if we haven't allocated all the way to maxSize
yet; very rare.
+ destAllocIx = allocateWithExpand(dest, destAllocIx, freeListIx,
allocationSize, arenaCount);
+ if (destAllocIx == dest.length) {
+ return;
+ }
+
+ int forceEvictAttempt = 0;
+ long totalForceEvictedBytes = 0;
+ long currentEvicted;
+ int startArenaIx;
+ int emptyAttempt = 0;
+
+ // We called reserveMemory so we know that there's memory waiting for us
somewhere.
+ // But that can mean that the reserved memory is fragmented thus unusable
+ // or we have a common class of rare race conditions related to the order
of locking/checking of
+ // different allocation areas this is due to the fact that Allocations
arrive in a burst fashion.
+ // To mitigate that:
+ // - First we try again (fast path):
+ // - Second try to push on the memoryManger/evictor to evict more.
+ // - Third by pass the memory manager and use discards method to brute
force evict if needed.
+ while (totalForceEvictedBytes < maxForcedEvictionSize && emptyAttempt <
MAX_DISCARD_ATTEMPTS) {
+ // make sure we hit a different start on each new attempt
+ startArenaIx = (int) ((threadId + forceEvictAttempt) % arenaCount);
+ destAllocIx =
+ allocateFast(dest, null, destAllocIx, dest.length, freeListIx,
allocationSize, startArenaIx, arenaCount);
+ if (destAllocIx == dest.length) {
+ return;
+ }
+ destAllocIx =
+ allocateWithSplit(dest,
+ null,
+ destAllocIx,
+ dest.length,
+ freeListIx,
+ allocationSize,
+ startArenaIx,
+ arenaCount,
+ -1);
+ if (destAllocIx == dest.length) {
+ return;
+ }
+
+ if (forceEvictAttempt >= MAX_FAST_ATTEMPT) {
+ // Try to evict more starting from how much memory we are missing
allocLog2 << dest.length - destAllocIx
+ // Exponentially increase the eviction request based on
forceEvictAttempt counter.
+ // Why Exponentially increase, usually query fragments triggers a
burst of allocation at the same time
+ // Thus IMO to avoid lock contentions it is worth to over evict thus
next allocation will get free lunch.
+ currentEvicted =
+ memoryManager.evictMemory(allocLog2 << dest.length - destAllocIx +
forceEvictAttempt - MAX_FAST_ATTEMPT);
+ // Note that memoryManager.evictMemory is blocking but eventually will
return 0 if all the memory is locked
+ // thus mark empty attempts and bail out after MAX_DISCARD_ATTEMPTS
+ totalForceEvictedBytes += currentEvicted;
Review comment:
This while loop might also exit with a single (large) forced eviction?
Specifically if the number of unallocated buffers cause a large shift of
allocLog2, the totalForceEvictedBytes might get > maxForcedEvictionSize and
therefore leave the loop without the chance of using any of the evicted memory.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
