Sailesh Mukil has posted comments on this change. Change subject: IMPALA-3610: Account for memory used by filters in the coordinator ......................................................................
Patch Set 2: (3 comments) http://gerrit.cloudera.org:8080/#/c/3618/2/be/src/runtime/coordinator.cc File be/src/runtime/coordinator.cc: PS2, Line 2077: state->DiscardFilter(filter_mem_tracker_.get()); Isn't this essentially a no-op, because an always_true_filter will always just consist of a NULL bloom_filter? Or can a partitioned join filter have just one ALWAYS_TRUE_FILTER (in which case this is fine, because state->bloom_filter would have been set by another update)? PS2, Line 2090: state->pending_count == 0 Wouldn't this always be 0 for a broadcast join filter? Given that we always set state->pending_count to 1 for a broadcast join filter with a remote target (in UpdateFilterRoutingTable())? Which makes this check redundant. One way to handle this is to modify the code in UpdateFilterRoutingTable() to set the pending_count to 3 (or MAX_BROADCAST_FILTER_PRODUCERS) for broadcast join filters; and change the conditional in L2065 to be: if (state->pending_count == 0 || (state->desc.is_broadcast_join && state->pending_count < MAX_BROADCAST_FILTER_PRODUCERS)) {...} But this seems a little messy and if you think of a cleaner way, that would be much better. PS2, Line 2110: for (const auto& target: state->targets) { : // Don't publish the filter to targets that are in the same fragment as the join : // that produced it. : if (target.is_local) continue; : target_fragment_instance_idxs.insert(target.fragment_instance_idxs.begin(), : target.fragment_instance_idxs.end()); : } Seems like state->targets is read only at this point, because it's set only in UpdateFilterRoutingTable() which runs before sending out the remote fragments. You could move the BloomFilter::ToThrift() before this (i.e. after L2109) and also release the filter_lock_ before running this for loop. -- To view, visit http://gerrit.cloudera.org:8080/3618 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: I3c52c8a1c2e79ef370c77bf264885fc859678d1b Gerrit-PatchSet: 2 Gerrit-Project: Impala Gerrit-Branch: cdh5-trunk Gerrit-Owner: Henry Robinson <[email protected]> Gerrit-Reviewer: Dan Hecht <[email protected]> Gerrit-Reviewer: Henry Robinson <[email protected]> Gerrit-Reviewer: Mostafa Mokhtar <[email protected]> Gerrit-Reviewer: Sailesh Mukil <[email protected]> Gerrit-HasComments: Yes
