Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/19452#discussion_r144372504
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamingSymmetricHashJoinExec.scala
---
@@ -387,9 +400,10 @@ case class StreamingSymmetricHashJoinExec(
* input row from this side and the matched
row from the other side
*/
def storeAndJoinWithOtherSide(
- otherSideJoiner: OneSideHashJoiner)(
- generateJoinedRow: (UnsafeRow, UnsafeRow) => JoinedRow):
Iterator[InternalRow] = {
-
+ otherSideJoiner: OneSideHashJoiner,
+ joinedFilter: InternalRow => Boolean)(
--- End diff --
maybe add this to the constructor like the other filter.
you can rename it to something more consistent like "preJoinFilter" and
"postJoinFilter"
also add docs.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
