Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/21729 )
Change subject: IMPALA-13185: Include runtime filter source in key ...................................................................... Patch Set 5: (2 comments) http://gerrit.cloudera.org:8080/#/c/21729/5/fe/src/main/java/org/apache/impala/planner/PlanNode.java File fe/src/main/java/org/apache/impala/planner/PlanNode.java: http://gerrit.cloudera.org:8080/#/c/21729/5/fe/src/main/java/org/apache/impala/planner/PlanNode.java@1364 PS5, Line 1364: if (!tupleCacheInfo_.mergeChild(build.getTupleCacheInfo())) { > I'm working through what needs to happen here. split_specs seems like it co TFileSplitGeneratorSpec has enough information, but that information is encoded as flatbuffers (FbFileDesc / FbFileBlock / FbFileMetadata). For this case, we're on a filesystem without block locations and there are no FbFileBlock objects (so no variability from block locations / disk indexes / etc). We assert that in the scheduler here: https://github.com/apache/impala/blob/master/be/src/scheduling/scheduler.cc#L135 I think that means that FbFileDesc looks fine to hash. FbFileMetadata has Iceberg information (FbIcebergMetadata). I don't know all of the fields involved there, but my first thought is that is looks hashable. http://gerrit.cloudera.org:8080/#/c/21729/5/fe/src/main/java/org/apache/impala/planner/RuntimeFilterGenerator.java File fe/src/main/java/org/apache/impala/planner/RuntimeFilterGenerator.java: http://gerrit.cloudera.org:8080/#/c/21729/5/fe/src/main/java/org/apache/impala/planner/RuntimeFilterGenerator.java@403 PS5, Line 403: if (serialCtx.includeRFTarget(target.node)) { : // Only serialize the relevant target PlanNode when using tuple caching. > I'm not clear what context is missing. This section doesn't skip runtime fi I would appreciate a few words saying that the list of targets includes other PlanNodes (which is why we need to limit ourselves to the target PlanNode). That's something I didn't realize when I was first looking in this area. -- To view, visit http://gerrit.cloudera.org:8080/21729 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0077964be5acdb588d76251a6a39e57a0f42bb5a Gerrit-Change-Number: 21729 Gerrit-PatchSet: 5 Gerrit-Owner: Michael Smith <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Joe McDonnell <[email protected]> Gerrit-Reviewer: Kurt Deschler <[email protected]> Gerrit-Reviewer: Michael Smith <[email protected]> Gerrit-Reviewer: Steve Carlin <[email protected]> Gerrit-Reviewer: Yida Wu <[email protected]> Gerrit-Comment-Date: Wed, 25 Sep 2024 17:15:20 +0000 Gerrit-HasComments: Yes
