Csaba Ringhofer has uploaded this change for review. ( http://gerrit.cloudera.org:8080/24037
Change subject: IMPALA-14739: Harden CatalogdMetaProvider.Weigher for edge cases ...................................................................... IMPALA-14739: Harden CatalogdMetaProvider.Weigher for edge cases Handle two cases safer: 1. > 2GB cache entries These were truncated before the patch to 2GB to fit to int32, and this underreporting meant that the cache could grow beyond its supposed limit. This is improved by using byte_size / 16 as weight, allowing sizes up to 32GB, which seems unrealistically large to me. 2. eviction of currently loaded entries The "piggy-backing" mechanism uses CompletableFuture as values while loading objects. Evicting this before loading finishes leads to not writing back the loaded object to cache as it is assumed that it was invalidated. The patch protects against weight based eviction by weighing these entries as 0, which leads the weight based eviction to ignore them. This is the recommended way to "lock" entries in weight bounded guava / caffeine caches. Time based evection can still remove entries while loading (1 hour by default). Both time and weight based eviction should be rare - one needs >1hour loading time, the other needs many new entries added while loading to push out the entry from LRU cache. Change-Id: Id525b9b0578fb7f9cb3e0f8f4fa32f6fdae313b9 --- M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java 1 file changed, 50 insertions(+), 11 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/37/24037/1 -- To view, visit http://gerrit.cloudera.org:8080/24037 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Id525b9b0578fb7f9cb3e0f8f4fa32f6fdae313b9 Gerrit-Change-Number: 24037 Gerrit-PatchSet: 1 Gerrit-Owner: Csaba Ringhofer <[email protected]>
