Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/20612 )
Change subject: IMPALA-3825: Delegate runtime filter aggregation to some executors ...................................................................... Patch Set 10: (16 comments) looks great, mostly nits http://gerrit.cloudera.org:8080/#/c/20612/10/be/src/runtime/coordinator.cc File be/src/runtime/coordinator.cc: http://gerrit.cloudera.org:8080/#/c/20612/10/be/src/runtime/coordinator.cc@478 PS10, Line 478: f->set_pending_count(num_agg); It looks a bit strange that pending count is set at line 432, then we set it here again. Can you move that logic here, e.g. in an else block? http://gerrit.cloudera.org:8080/#/c/20612/10/be/src/runtime/query-state.cc File be/src/runtime/query-state.cc: http://gerrit.cloudera.org:8080/#/c/20612/10/be/src/runtime/query-state.cc@438 PS10, Line 438: this_krpc_address The new address compare function could be used here. http://gerrit.cloudera.org:8080/#/c/20612/10/be/src/runtime/runtime-filter-bank.h File be/src/runtime/runtime-filter-bank.h: http://gerrit.cloudera.org:8080/#/c/20612/10/be/src/runtime/runtime-filter-bank.h@100 PS10, Line 100: /// Filters are aggregated, first locally in this RuntimeFilterBank, if there are multiple Can you mention the new aggregation mechanism in the class comment? http://gerrit.cloudera.org:8080/#/c/20612/10/be/src/runtime/runtime-filter-bank.h@196 PS10, Line 196: has complete nit: has completed? http://gerrit.cloudera.org:8080/#/c/20612/10/be/src/runtime/runtime-filter-bank.h@197 PS10, Line 197: has not complete nit: that has not completed? http://gerrit.cloudera.org:8080/#/c/20612/10/be/src/runtime/runtime-filter-bank.h@198 PS10, Line 198: RUNTIME_FILTER_WAIT_TIME_MS Can you mention RUNTIME_FILTER_WAIT_TIME_MS handling in the commit message? http://gerrit.cloudera.org:8080/#/c/20612/10/be/src/runtime/runtime-filter-bank.h@199 PS10, Line 199: not nit: doesn't happen? http://gerrit.cloudera.org:8080/#/c/20612/10/be/src/runtime/runtime-filter-bank.h@227 PS10, Line 227: hold nit: holds http://gerrit.cloudera.org:8080/#/c/20612/10/be/src/runtime/runtime-filter-bank.h@246 PS10, Line 246: number nit: number of? http://gerrit.cloudera.org:8080/#/c/20612/10/be/src/runtime/runtime-filter-bank.cc File be/src/runtime/runtime-filter-bank.cc: http://gerrit.cloudera.org:8080/#/c/20612/10/be/src/runtime/runtime-filter-bank.cc@139 PS10, Line 139: AllocateScratchBloomFilter Would it be hard to allocate the filter only when we first need it? It look strange that the filters are both initialized and allocated in ClaimBufferReservation. http://gerrit.cloudera.org:8080/#/c/20612/10/be/src/runtime/runtime-filter-bank.cc@722 PS10, Line 722: CombinePeerAndLocalUpdates Shouldn't we mark the filter as always true? http://gerrit.cloudera.org:8080/#/c/20612/10/be/src/runtime/runtime-filter-bank.cc@756 PS10, Line 756: canceled nit: cancelled http://gerrit.cloudera.org:8080/#/c/20612/7/be/src/scheduling/scheduler.cc File be/src/scheduling/scheduler.cc: http://gerrit.cloudera.org:8080/#/c/20612/7/be/src/scheduling/scheduler.cc@317 PS7, Line 317: > I think it still make sense. I see, I agree that this makes sense if there are a lot of runtime filters. Sending the runtime filters to each executor will be still a pretty large work for the coordinator though, so in the long run it would be better to move that work to the aggregating executors too. http://gerrit.cloudera.org:8080/#/c/20612/10/be/src/util/network-util.h File be/src/util/network-util.h: http://gerrit.cloudera.org:8080/#/c/20612/10/be/src/util/network-util.h@107 PS10, Line 107: KrpcAddressMatch naming: I think that KrpcAddressEquals would be clearer http://gerrit.cloudera.org:8080/#/c/20612/10/common/thrift/ImpalaInternalService.thrift File common/thrift/ImpalaInternalService.thrift: http://gerrit.cloudera.org:8080/#/c/20612/10/common/thrift/ImpalaInternalService.thrift@54 PS10, Line 54: aggregation nit: aggregator? http://gerrit.cloudera.org:8080/#/c/20612/10/common/thrift/ImpalaInternalService.thrift@54 PS10, Line 54: that nit: "that" not needed -- To view, visit http://gerrit.cloudera.org:8080/20612 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I11d38ed0f223d6e5b32a19ebe725af7738ee4ab0 Gerrit-Change-Number: 20612 Gerrit-PatchSet: 10 Gerrit-Owner: Riza Suminto <[email protected]> Gerrit-Reviewer: Abhishek Rawat <[email protected]> Gerrit-Reviewer: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Kurt Deschler <[email protected]> Gerrit-Reviewer: Michael Smith <[email protected]> Gerrit-Reviewer: Riza Suminto <[email protected]> Gerrit-Comment-Date: Mon, 11 Dec 2023 16:47:13 +0000 Gerrit-HasComments: Yes
