pingzh opened a new issue, #7616:
URL: https://github.com/apache/datafusion-comet/issues/7616
### Describe the bug
Partial consumers of raw `QueryExecution.toRdd` or PySpark `DataFrame.rdd`
can encounter a `make_date` error from a later row that Spark would never
evaluate. A native projection can evaluate that row as part of a batch before
the consumer stops.
The existing Scala `Dataset.rdd` safeguards do not cover all of these entry
points. The gap already exposes invalid-calendar errors with ANSI enabled. The
Date32 fix in #7343 also exposes it with ANSI disabled: an overflowing valid
date now raises `ARITHMETIC_OVERFLOW` instead of returning NULL.
### Steps to reproduce
For the ANSI-off Date32 case, use #7343 at `8905cda0a` or a revision
containing its kernel fix. Use the regression shape from the earlier PySpark
bridge test:
1. With Comet disabled, write a single-partition Parquet input with 4,096
rows. Set integer `y=2024` on the first 4,095 rows and `y=6000000` on the final
row; retain `id`.
2. Set ANSI and AQE off and `spark.comet.batchSize=8192`.
3. Read the Parquet input and project:
```sql
unix_date(make_date(y, 1, 1)) AS d,
repeat(sha2(CAST(id AS STRING), 512), 512) AS padding
```
4. Consume only the first serialized batch through the JVM
`javaToPython().take(1)` bridge used by PySpark. Compare Comet enabled and
disabled. The large padding makes Python serialization emit a batch before
reaching the overflowing final row; a two-row example can instead cause Spark
to consume both rows.
5. Cover both the uncapped DataFrame and `limit(4096)`, including when the
physical plan is prepared before the RDD action. Add corresponding raw
`QueryExecution.toRdd` and public PySpark partial-consumer coverage.
The integer `unix_date` output avoids unrelated Python date-range limits.
The [earlier JVM bridge
regression](https://github.com/apache/datafusion-comet/blob/d1ffa2e83ab06717df05f3d60a2c71964df05139/spark/src/test/scala/org/apache/comet/CometTemporalExpressionSuite.scala)
provides the complete test setup.
### Expected behavior
- Match Spark when a raw or PySpark consumer stops before a throwing row,
including reused/preplanned DataFrames and AQE on/off.
- Cover ANSI-on invalid-calendar errors and Date32 overflow in both ANSI
modes. Full consumption must still raise the required error.
- Preserve existing Scala Dataset safeguards and native collect-limit caps.
- Benchmark ordinary root projections, limits above one (including
`show()`), and materializing/write consumers before selecting a broader
fallback strategy.
### Additional context
Requested in [this review of
#7343](https://github.com/apache/datafusion-comet/pull/7343#pullrequestreview-5894195234).
The broader external-iterator routing was deferred from #7343 to keep the
overflow fix focused; this issue tracks that remaining correctness work. It
follows the consumer guards introduced in #7283.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
