ganeshashree commented on PR #57813: URL: https://github.com/apache/spark/pull/57813#issuecomment-5243652215
> I was trying to comment each lines but feel like talking about general direction is much better than that. > > I see this to be implemented as a variance of LeftOuter, but the spec of LeftAnti is actually a mirror of LeftSemi, meaning so many optimizations we have made for LeftSemi can be applied to LeftAnti. For example, the row in the left side doesn't need to be stored as long as it finds "any" matched row in the right side. This is also applied when the new row in the right side is matched with the left side's row in the state store; we can just remove it from state store rather than updating the matched flag. We don't need to check with matched flag since we wouldn't leave the row which was matched in the state store. > > It would give a lot of benefit if we start from LeftSemi, not LeftOuter. Still, LeftAnti should be a hybrid of LeftSemi and LeftOuter (since we need to produce the outer result as the main output), but for the matched rows, the optimization for LeftSemi can be applied to LeftAnti. Thanks, agreed. LeftSemi is the right framing. Reworked in `772c64e` as a hybrid of left semi and left outer: - left row that matches on arrival isn't stored; - already-stored left row matched by a later right row is removed from state (`getJoinedRowsAndRemoveMatched`), not flag-flipped; - right side processed first, as in LeftSemi. Only the eviction-time emission of surviving never-matched left rows stays left-outer-shaped. Since every survivor is unmatched by construction, the `matched` flag is no longer read at eviction, so I dropped the v4 `skipUpdatingMatchedFlag` special-casing too. Outputs unchanged; state size for matched left rows now matches LeftSemi. Updated `assertNumStateRows` accordingly. PTAL. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
