AbhinavBattu opened a new pull request, #57936:
URL: https://github.com/apache/spark/pull/57936
### What changes were proposed in this pull request?
`OrderedFilters` groups pushed down filters by the position of the last
schema
field they reference, so that a filter can be applied to a partially parsed
row
as soon as its referenced fields are set. The group arrays are sized by
`requiredSchema.fields.length`, but filters without references (`AlwaysTrue`,
`AlwaysFalse`) are assigned to index 0. When the required schema has no
fields
the arrays are empty and the assignment throws.
This PR skips the grouping when the required schema has no fields. In that
case
`skipRow()` can never be invoked, because it asserts that the index points
to a
field of the required schema, so there is no position at which such a filter
could be applied.
Note that `StructFilters.pushedFilters()` is correct in keeping these
filters:
it checks that a filter does not reference a field missing from the schema,
and
a filter without references does not. Reference-free filters are also worth
keeping in general, since a pushed `AlwaysFalse` lets the reader skip rows.
The
faulty assumption was in `OrderedFilters`, which required not just that every
reference is present, but that at least one field exists.
`JsonFilters` already handles the same input, because it spreads
reference-free
filters over the set of schema field names rather than pinning them to index
0.
With an empty schema that set is empty and the filter is dropped, which is
why
JSON does not crash on an equivalent query.
The reordering step immediately below the loop already guarded `len > 0`; the
assignment loop did not.
### Why are the changes needed?
The following query fails at execution time:
```sql
CREATE TABLE m(c0 INT) USING CSV;
INSERT INTO m VALUES (1);
SELECT COUNT(*) FROM m WHERE (SELECT BOOL_OR(true) FROM m);
```
```
java.lang.ArrayIndexOutOfBoundsException: Index 0 out of bounds for length 0
at
org.apache.spark.sql.catalyst.OrderedFilters.$anonfun$predicates$3(OrderedFilters.scala:63)
at
org.apache.spark.sql.catalyst.OrderedFilters.<init>(OrderedFilters.scala:50)
at
org.apache.spark.sql.catalyst.csv.UnivocityParser.<init>(UnivocityParser.scala:119)
```
Through the SQL path the exception surfaces wrapped as
`[FAILED_READ_FILE.NO_HINT]` with the `ArrayIndexOutOfBoundsException` as its
cause.
The `WHERE` clause is an uncorrelated boolean scalar subquery referencing no
columns. Since SPARK-43402 such predicates are retained as data filters; at
execution the finished subquery is substituted as `Literal(true)` and
translated
to `sources.AlwaysTrue`, a filter with no references. With `COUNT(*)` the
required schema is empty, so both conditions hold at once.
`OrderedFilters` is also used by the Avro reader, which is affected in the
same
way and is fixed by the same change.
### Does this PR introduce _any_ user-facing change?
Yes. A query that pushes down a filter without column references while
requiring
no columns previously failed with `ArrayIndexOutOfBoundsException` and now
returns the correct result.
### How was this patch tested?
Two new tests, both confirmed to fail before this change and pass after it.
`StructFiltersSuite` is shared by `OrderedFiltersSuite` and
`JsonFiltersSuite`,
so the unit test covers both implementations and shows that JSON was already
correct. The test asserts only that building the filters succeeds, because
with
an empty required schema there is nothing observable to assert: the
predicates
are private and `skipRow()` cannot be called at all.
```
build/sbt 'catalyst/testOnly *OrderedFiltersSuite *JsonFiltersSuite'
```
An end to end test was added to `CSVSuite`, which is inherited by
`CSVv1Suite`
and `CSVv2Suite`, so the query from the JIRA is covered on both the DSv1 and
DSv2 reader paths and its result is checked.
```
build/sbt 'sql/testOnly *CSVv1Suite *CSVv2Suite -- -z "SPARK-58484"'
```
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
