Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/24512 )
Change subject: POC/slop: decrease per file memory usage in catalog ...................................................................... Patch Set 4: (4 comments) > (4 comments) > > Direction seems fine. I think there is some inaccuracy in comments > that could be cleaned up. Yes, there is a lot to clean up, I don't even understand some parts written by AI (try to express this with POC/slop at the beginning) My plan is to merge https://gerrit.cloudera.org/#/c/24515/ and then re-evaluate this patch. Impala used much more memory than I expected, so I was digging into how these flatbuffer arrays look like. Then I realized that most of unexplained usage comes from partitionList_, the mem usage of other parts is "not that bad" http://gerrit.cloudera.org:8080/#/c/24512/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/24512/1//COMMIT_MSG@22 PS1, Line 22: based on measurements: > Was there actually any rearranging? I will investigate this, Claude gave me conflicting answers (order in definition matters vs order of calling setters in builder api matters) http://gerrit.cloudera.org:8080/#/c/24512/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/24512/2//COMMIT_MSG@26 PS2, Line 26: - added optimized block info for the common case of > I don't see any changes like this. See my pref comment, I am still trying to understand how the layout in memory is set exactly. http://gerrit.cloudera.org:8080/#/c/24512/1/be/src/scheduling/scheduler-test-util.cc File be/src/scheduling/scheduler-test-util.cc: http://gerrit.cloudera.org:8080/#/c/24512/1/be/src/scheduling/scheduler-test-util.cc@420 PS1, Line 420: FbFileDescBuilder fb_file_desc_builder(fb_builder); > Why'd this change? The order of members changed, path and length are no longer the first 2 arguments. http://gerrit.cloudera.org:8080/#/c/24512/2/fe/src/main/java/org/apache/impala/util/IcebergUtil.java File fe/src/main/java/org/apache/impala/util/IcebergUtil.java: http://gerrit.cloudera.org:8080/#/c/24512/2/fe/src/main/java/org/apache/impala/util/IcebergUtil.java@1300 PS2, Line 1300: // This matches the order in the generated createFbIcebergMetadata() and saves > I'm not sure what createFbIcebergMetadata is referring to. Will clean this up later. it looks a file that flat buffer would generate. -- To view, visit http://gerrit.cloudera.org:8080/24512 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id48c7785ec88e16a9b79ac910add2c0a240869e3 Gerrit-Change-Number: 24512 Gerrit-PatchSet: 4 Gerrit-Owner: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Michael Smith <[email protected]> Gerrit-Comment-Date: Fri, 26 Jun 2026 11:50:14 +0000 Gerrit-HasComments: Yes
