sunchao commented on code in PR #5177:
URL: https://github.com/apache/datafusion-comet/pull/5177#discussion_r3935073414
##########
spark/src/main/scala/org/apache/comet/serde/operator/CometNativeScan.scala:
##########
@@ -142,8 +142,10 @@ object CometNativeScan extends
CometOperatorSerde[CometScanExec] with CometTypeS
builder.clearChildren()
if (scan.conf.getConf(SQLConf.PARQUET_FILTER_PUSHDOWN_ENABLED)) {
+ val supportedDataFilters = scan.supportedDataFilters
+ commonBuilder.setHasDataFilters(supportedDataFilters.nonEmpty)
Review Comment:
**[P2] Carry runtime scalar-subquery filters into the overflow fallback**
`scan.supportedDataFilters` excludes filters containing a `PlanExpression`,
so this bit remains false when the only filter is `id <=> (SELECT max(k) FROM
lookup)`. At execution, `CometNativeScanExec` resolves that scalar subquery and
tries to append the predicate. With
`spark.comet.expression.EqualNullSafe.enabled=false`, `exprToProto` drops the
resolved predicate without setting `has_data_filters`. Both the bit and the
native filter list stay empty, so the native planner passes `None` and enables
checked timestamp conversion before Spark's remaining filter runs.
I reproduced the Spark 4.1.3 side with dictionary-only `id` values 1 and 3,
a subquery result of 2, and an overflowing `TIMESTAMP_MILLIS` `ts`. Spark's
scan shows `PushedFilters: [EqualNullSafe(id,ScalarSubquery#...)]` and returns
no rows; an unfiltered read of `ts` raises `long overflow`. The current Comet
outcome follows from the planning and serialization paths but was not run end
to end locally. Could we carry the presence of these late-bound filters through
runtime serialization, even if they cannot be serialized, and add this case to
the regression tests?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
