sunchao commented on PR #24668:
URL: https://github.com/apache/datafusion/pull/24668#issuecomment-5455764743
### [P2] Preserve ordering when retained negation wraps temporal UDFs
@Amogh-2404, the original grouping, `LEAD`, and composed-projection
reproductions now pass on `2bad6666a`. There is a remaining regression
involving ordered temporal functions:
```sql
SELECT i, j FROM s
ORDER BY -(-from_unixtime(i)), j
LIMIT 1;
```
I compared head `2bad6666a` against base `a38bb10c3` using a
single-partition unbounded `StreamingTable`, ordered by `(i ASC NULLS LAST, j
ASC NULLS LAST)`, with `target_partitions=1` and `batch_size=1`. Both columns
are `Int64`; the source emits `(1,0)` and `(1,1)` in separate batches, then
remains pending.
Base returns `(1,0)` immediately. Head introduces a global `SortExec` and
fails `SanityCheckPlan` with `Cannot execute pipeline breaking queries`. The
control using `ORDER BY from_unixtime(i), j` works on both revisions. I also
reproduced the difference with nullable input and this projected form:
```sql
SELECT k, j
FROM (SELECT -(-from_unixtime(i)) AS k, j FROM s)
ORDER BY k, j LIMIT 1;
```
On an ordered timestamp source, even the one-key `ORDER BY
-(-date_trunc('day', i)) LIMIT 1` query changes from immediate output on base
to the same planning error on head.
The [retained-expression
branch](https://github.com/apache/datafusion/blob/2bad6666a5ee333a90aaa0a7d55f94ea78e00afe/datafusion/optimizer/src/simplify_expressions/expr_simplifier.rs#L1352-L1360)
exposes this because these functions advertise output ordering but inherit an
unknown `Interval<Null>` bound. `NegativeExpr::get_properties` attempts to
negate that bound, errors, and loses the known ordering. The supplied values do
not overflow.
Could ordering inference handle unknown UDF bounds while retaining checked
evaluation, with streaming regressions for both direct ordering and projected
keys? Fixing only projection substitution would leave the direct `ORDER BY`
case broken.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
