XdithyX commented on code in PR #56486:
URL: https://github.com/apache/spark/pull/56486#discussion_r3500260293
##########
sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/SupportsPushDownJoin.java:
##########
@@ -63,6 +69,157 @@ boolean pushDownJoin(
Predicate condition
);
+ /**
+ * Returns pushed operators this data source can preserve when pushing down
a join.
+ * <p>
+ * Spark uses this as a safety check before attempting join push down. If
the join input
+ * already contains pushed operators not listed here, Spark will not push
down the join.
+ * This prevents a data source from silently dropping pushed state it does
not understand.
+ *
+ * @since 4.3.0
+ */
+ default Set<PushedOperator> supportedPushedOperatorsForJoin() {
+ return Set.of();
+ }
+
+ /**
+ * Pushes down the join of the current {@code SupportsPushDownJoin} and the
other side of join
+ * {@code SupportsPushDownJoin}, with information about pushed operators
from each side.
+ *
+ * @param other {@code SupportsPushDownJoin} that this {@code
SupportsPushDownJoin}
+ * gets joined with.
+ * @param joinType the type of join.
+ * @param leftSideRequiredColumnsWithAliases required output of the
+ * left side {@code
SupportsPushDownJoin}
+ * @param rightSideRequiredColumnsWithAliases required output of the
+ * right side {@code
SupportsPushDownJoin}
+ * @param condition join condition. Columns are named after the specified
aliases in
+ * {@code leftSideRequiredColumnsWithAliases} and {@code
rightSideRequiredColumnsWithAliases}
+ * @param joinPushDownInfo information about pushed operators from each side
of the join.
+ * @return True if join has been successfully pushed down.
+ *
+ * @since 4.3.0
+ */
+ default boolean pushDownJoin(
+ SupportsPushDownJoin other,
+ JoinType joinType,
+ ColumnWithAlias[] leftSideRequiredColumnsWithAliases,
+ ColumnWithAlias[] rightSideRequiredColumnsWithAliases,
+ Predicate condition,
+ JoinPushDownInfo joinPushDownInfo) {
Review Comment:
Hi @cloud-fan ,
That makes sense. The main tradeoff I see is that we would no longer have
Spark-side capability negotiation for existing third-party connectors. Instead,
the existing `pushDownJoin` contract would need to require connectors to return
false if they cannot preserve previously pushed state such as table samples.
I’m not aware of any third-party connectors that implement
`SupportsPushDownJoin` and `SupportsPushDownTableSample` together, and JDBC
appears to be the only known OSS connector in Spark covering both APIs. Given
that `SupportsPushDownJoin` is `@Evolving`, I agree that avoiding the new
public API and documenting the contract on pushDownJoin seems acceptable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
