cloud-fan commented on code in PR #58370:
URL: https://github.com/apache/spark/pull/58370#discussion_r3892275058
##########
sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/SupportsRuntimeFiltering.java:
##########
@@ -51,6 +50,13 @@ public interface SupportsRuntimeFiltering extends
SupportsRuntimeV2Filtering {
* The provided expressions must be interpreted as a set of filters that are
ANDed together.
* Implementations may use the filters to prune initially planned {@link
InputPartition}s.
* <p>
+ * Spark tracks runtime-filter eligibility by root attribute. If {@link
#filterAttributes()}
+ * returns a nested reference, this method may receive a filter on another
nested field under
+ * the same root. Implementations must inspect each filter and use only
filters they can apply.
+ * Nested paths are encoded in a V1 {@link Filter} as unquoted dot-separated
names such as
Review Comment:
**Non-blocking (P2):** [P2] Describe per-part quoting in V1 filter names
`PredicateUtils.toV1` uses `NamedReference.toString`, and
`FieldReference.toString` applies `quoteIfNeeded` to each path part. Thus
`Seq("parent", "child.with.dot")` reaches this callback as
parent.`child.with.dot`, not as an entirely unquoted name. A connector
following this text can split or bind a legal nested name incorrectly. Please
say that parts are dot-separated and individually quoted as needed, with
parent.`child.with.dot` as an example.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Relation.scala:
##########
@@ -201,16 +202,23 @@ case class DataSourceV2ScanRelation(
* Resolved attributes that the scan declares for runtime filtering via
* [[SupportsRuntimeV2Filtering.filterAttributes]] or
* [[SupportsRuntimeCatalystFiltering.filterAttributes]]. Empty when the scan
- * implements neither interface or exposes no attributes.
+ * implements neither interface or exposes no attributes. Accessing this
value also validates
+ * attributes returned by
[[SupportsRuntimeCatalystFiltering.fullyPushedFilterAttributes]].
*/
lazy val runtimeFilterAttrs: AttributeSet = {
checkRuntimeFilteringInterfaces()
+ checkFullyPushedFilterAttrs()
Review Comment:
**Non-blocking (P2):** [P2] Resolve fully-pushed attributes on this entry
path
This helper only rejects nested references. A one-part value returned by
`fullyPushedFilterAttributes()` that is absent from `output` therefore survives
`runtimeFilterAttrs`: `filterAttributes()` is resolved, but the fully-pushed
array is not resolved until `fullyPushedRuntimeFilterAttrs` is forced.
`DataSourceV2Strategy` does not force that lazy value when
`scalarSubqueryFilters` is empty, so the documented build-time validation
becomes query-shape dependent. Please resolve and cache the fully-pushed
declarations here, reuse them in the secondary accessor, and assert this entry
point with `MissingFullyPushedFilterAttributeScan`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
