Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/21557 )
Change subject: IMPALA-13088, IMPALA-13109: Use RoaringBitmap instead of sorted vector of int64s ...................................................................... Patch Set 9: Code-Review+1 (3 comments) http://gerrit.cloudera.org:8080/#/c/21557/9/be/src/exec/iceberg-delete-builder.h File be/src/exec/iceberg-delete-builder.h: http://gerrit.cloudera.org:8080/#/c/21557/9/be/src/exec/iceberg-delete-builder.h@119 PS9, Line 119: RoaringBitmap64 As we discussed offline, memory used by roaring bitmap could be tracked the following way: an allocator could be created that saves mem tracker to a thread local variable - as a thread only works for a specific fragment instance, this would be a clear mapping roaring bitmap can use a global allocator that gets the thread local mem tracker and allocates memory through that we could check before each batch whether the memory limit was exceeded - if yes, then it could try to increase reservation or fail the query if it can't get more memory This mainly just a reminder, I don't mean to implement this in the current patch. http://gerrit.cloudera.org:8080/#/c/21557/9/fe/src/main/java/org/apache/impala/planner/IcebergDeleteNode.java File fe/src/main/java/org/apache/impala/planner/IcebergDeleteNode.java: http://gerrit.cloudera.org:8080/#/c/21557/9/fe/src/main/java/org/apache/impala/planner/IcebergDeleteNode.java@177 PS9, Line 177: getChild(1).getCardinality() == -1 Is it possible for these to be unknown? If not, then this if could be replaced with a precondition. http://gerrit.cloudera.org:8080/#/c/21557/9/fe/src/main/java/org/apache/impala/planner/IcebergDeleteNode.java@201 PS9, Line 201: long roaringBitmapSize = (long) Math.ceil(1.5 * rhsCard); Can you move the 1.5 to a constant? Also, I think that this can underestimate memory needs: - if there are relatively few rows per file, then the per 64K block overhead can be significant - the sorted vectors are likely to allocate more memory than needed to store elements (*1.5). -- To view, visit http://gerrit.cloudera.org:8080/21557 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib769965d094149e99c43e0044914d9ecccc76107 Gerrit-Change-Number: 21557 Gerrit-PatchSet: 9 Gerrit-Owner: Zoltan Borok-Nagy <[email protected]> Gerrit-Reviewer: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: Daniel Becker <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Kurt Deschler <[email protected]> Gerrit-Reviewer: Michael Smith <[email protected]> Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]> Gerrit-Comment-Date: Wed, 10 Jul 2024 15:50:43 +0000 Gerrit-HasComments: Yes
