dxichen commented on code in PR #1687:
URL: https://github.com/apache/samza/pull/1687#discussion_r1336400804
##########
samza-core/src/main/java/org/apache/samza/storage/blobstore/util/BlobStoreUtil.java:
##########
@@ -670,12 +676,18 @@ private CompletableFuture<Void> removeTTL(DirIndex
dirIndex, Metadata metadata)
metadata.getJobName(), metadata.getJobId(),
metadata.getTaskName(), metadata.getStoreName());
List<FileBlob> fileBlobs = file.getBlobs();
for (FileBlob fileBlob : fileBlobs) {
- String opname = "removeTTL for fileBlob: " + file.getFileName() + "
with blobId: {}" + fileBlob.getBlobId();
- Supplier<CompletionStage<Void>> ttlRemovalAction = () ->
- blobStoreManager.removeTTL(fileBlob.getBlobId(),
requestMetadata).toCompletableFuture();
- CompletableFuture<Void> ttlRemovalFuture =
- FutureUtil.executeAsyncWithRetries(opname, ttlRemovalAction,
isCauseNonRetriable(), executor, retryPolicyConfig);
- updateTTLsFuture.add(ttlRemovalFuture);
+ if (ttlRemovedBlobIdsCache.getIfPresent(fileBlob.getBlobId()) == null)
{
+ String opname = "removeTTL for fileBlob: " + file.getFileName() + "
with blobId: {}" + fileBlob.getBlobId();
+ Supplier<CompletionStage<Void>> ttlRemovalAction = () ->
+ blobStoreManager.removeTTL(fileBlob.getBlobId(),
requestMetadata).toCompletableFuture()
+ .thenRun(() ->
ttlRemovedBlobIdsCache.put(fileBlob.getBlobId(), new Object()));
Review Comment:
Lets also store the time this was removed for debugging purposes and add it
in the log below
##########
samza-core/src/main/java/org/apache/samza/storage/blobstore/util/BlobStoreUtil.java:
##########
@@ -90,6 +92,9 @@ public class BlobStoreUtil {
private final SnapshotIndexSerde snapshotIndexSerde;
private final RetryPolicyConfig retryPolicyConfig;
+ // LRU cache for blob ids that were TTL removed
+ private final Cache<String, Object> ttlRemovedBlobIdsCache =
CacheBuilder.newBuilder().maximumSize(1000).build();
Review Comment:
Could you verify how many blobs are usually stored per store? we could use
that to inform on this constant
##########
samza-core/src/main/java/org/apache/samza/storage/blobstore/util/BlobStoreUtil.java:
##########
@@ -21,6 +21,8 @@
import com.google.common.annotations.VisibleForTesting;
import com.google.common.base.Preconditions;
+import com.google.common.cache.Cache;
Review Comment:
Any particular reason we are using an LRU cache instead of an in mem set
here? we are not using anything cache specific ie invalidation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
