pheecian commented on a change in pull request #6717: fix filesystem offload
oom based on https://github.com/apache/pulsar/…
URL: https://github.com/apache/pulsar/pull/6717#discussion_r407047545
##########
File path:
tiered-storage/file-system/src/main/java/org/apache/bookkeeper/mledger/offload/filesystem/impl/FileSystemManagedLedgerOffloader.java
##########
@@ -188,12 +191,15 @@ public void run() {
AtomicLong haveOffloadEntryNumber = new AtomicLong(0);
long needToOffloadFirstEntryNumber = 0;
CountDownLatch countDownLatch;
+ //avoid prefetch too much data into memory
+ ArrayBlockingQueue<Boolean> tasks = new
ArrayBlockingQueue<>(PREFETCH_ROUNDS);
do {
long end = Math.min(needToOffloadFirstEntryNumber +
ENTRIES_PER_READ - 1, readHandle.getLastAddConfirmed());
log.debug("read ledger entries. start: {}, end: {}",
needToOffloadFirstEntryNumber, end);
LedgerEntries ledgerEntriesOnce =
readHandle.readAsync(needToOffloadFirstEntryNumber, end).get();
Review comment:
the blockingQueue is to block the whole do while loop(to be precisely, the
read and submit) Here, not try to limit fileSystemWriter. I understand that the
FileSystemWriter will be executed orderly. My intention is to limit the whole
do while loop because the do while loop((readAsync().get()) read data from
ledger in advance too much too quickly(and lead to consume much memory), more
than FileSystemWriter Threads can handle timely. So read in advance, or put in
other words, prefetch, should be controlled to some degree. So I introduce a
blocking queue and hence a producer-consumer mock-up. the do while loop(read
and submit) acts like a producer, the FileSystemWriter acts as a consumer.
@congbobo184
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
