keith-turner commented on code in PR #5185:
URL: https://github.com/apache/accumulo/pull/5185#discussion_r1885809482
##########
server/manager/src/main/java/org/apache/accumulo/manager/compaction/coordinator/CompactionCoordinator.java:
##########
@@ -528,82 +551,138 @@ protected CompactionMetadata
createExternalCompactionMetadata(CompactionJob job,
}
- protected CompactionMetadata reserveCompaction(CompactionJobQueues.MetaJob
metaJob,
- String compactorAddress, ExternalCompactionId externalCompactionId) {
+ private class ReserveCompactionTask implements Supplier<CompactionMetadata> {
+
+ // Use a soft reference for this in case free memory gets low while this
is sitting in the queue
+ // waiting to process. This object can contain the tablets list of files
and if there are lots
+ // of tablet with lots of files then that could start to cause memory
problems.
+ private final SoftReference<CompactionJobQueues.MetaJob> metaJobRef;
Review Comment:
This was trying to deal with a problem I have been seeing while testing
using the
[FlakyBulkBatchWriter](https://github.com/apache/accumulo-testing/blob/807b62c01c4aae422a155be549fbf19664157b9c/src/main/java/org/apache/accumulo/testing/continuous/FlakyBulkBatchWriter.java#L55)
in accumulo-testing that will sometimes submit a large numbers of files for a
tablet. This can cause lots of memory pressure on the manager when lots of
tablet have lots of files and the manager is trying to keep a lot of that in
memory. For this case its really hard to reason about what will be on the
thread pool queue and how much memory it is using. So decided to use a soft
refernce for now, but I am uncertain about it. Opened #5188 and added comment
pointing to that issue in this PR. I think if #5188 were implemented that this
soft reference could be removed. Would still want to restrict compactors from
having multiple request queued/running. I think if compactors can only have
one thing queued
for reservation and the queued data is only compaction jobs and no tablet
metadata, then the memory usage would be unlikely to cause a problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
