cloud-fan commented on code in PR #36918:
URL: https://github.com/apache/spark/pull/36918#discussion_r930791167
##########
sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/SupportsRuntimeFiltering.java:
##########
@@ -57,4 +59,30 @@ public interface SupportsRuntimeFiltering extends Scan {
* @param filters data source filters used to filter the scan at runtime
*/
void filter(Filter[] filters);
+
+ /**
+ * Filters this scan using runtime predicates.
+ * <p>
+ * The provided expressions must be interpreted as a set of predicates that
are ANDed together.
+ * Implementations may use the predicates to prune initially planned {@link
InputPartition}s.
+ * <p>
+ * If the scan also implements {@link SupportsReportPartitioning}, it must
preserve
+ * the originally reported partitioning during runtime filtering. While
applying runtime
+ * predicates, the scan may detect that some {@link InputPartition}s have no
matching data. It
+ * can omit such partitions entirely only if it does not report a specific
partitioning.
+ * Otherwise, the scan can replace the initially planned {@link
InputPartition}s that have no
+ * matching data with empty {@link InputPartition}s but must preserve the
overall number of
+ * partitions.
+ * <p>
+ * Note that Spark will call {@link Scan#toBatch()} again after filtering
the scan at runtime.
+ *
+ * @param predicates data source V2 predicates used to filter the scan at
runtime
+ */
+ default void filter(Predicate[] predicates) {
+ Filter[] filters = new Filter[predicates.length];
+ for (int i = 0; i < predicates.length; i++) {
+ filters[i] = PredicateUtils.toV1(predicates[i]).get();
Review Comment:
We should drop the predicates that can't be translated to v1.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
