cloud-fan commented on code in PR #56486:
URL: https://github.com/apache/spark/pull/56486#discussion_r3502700805
##########
sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/SupportsPushDownJoin.java:
##########
@@ -63,6 +69,157 @@ boolean pushDownJoin(
Predicate condition
);
+ /**
+ * Returns pushed operators this data source can preserve when pushing down
a join.
+ * <p>
+ * Spark uses this as a safety check before attempting join push down. If
the join input
+ * already contains pushed operators not listed here, Spark will not push
down the join.
+ * This prevents a data source from silently dropping pushed state it does
not understand.
+ *
+ * @since 4.3.0
+ */
+ default Set<PushedOperator> supportedPushedOperatorsForJoin() {
+ return Set.of();
+ }
+
+ /**
+ * Pushes down the join of the current {@code SupportsPushDownJoin} and the
other side of join
+ * {@code SupportsPushDownJoin}, with information about pushed operators
from each side.
+ *
+ * @param other {@code SupportsPushDownJoin} that this {@code
SupportsPushDownJoin}
+ * gets joined with.
+ * @param joinType the type of join.
+ * @param leftSideRequiredColumnsWithAliases required output of the
+ * left side {@code
SupportsPushDownJoin}
+ * @param rightSideRequiredColumnsWithAliases required output of the
+ * right side {@code
SupportsPushDownJoin}
+ * @param condition join condition. Columns are named after the specified
aliases in
+ * {@code leftSideRequiredColumnsWithAliases} and {@code
rightSideRequiredColumnsWithAliases}
+ * @param joinPushDownInfo information about pushed operators from each side
of the join.
+ * @return True if join has been successfully pushed down.
+ *
+ * @since 4.3.0
+ */
+ default boolean pushDownJoin(
+ SupportsPushDownJoin other,
+ JoinType joinType,
+ ColumnWithAlias[] leftSideRequiredColumnsWithAliases,
+ ColumnWithAlias[] rightSideRequiredColumnsWithAliases,
+ Predicate condition,
+ JoinPushDownInfo joinPushDownInfo) {
Review Comment:
yea it's still `@Evolving` so we'd better keep it clean, a doc change is fine
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
