cool9850311 opened a new issue, #27127: URL: https://github.com/apache/datafusion/issues/27127
### Describe the bug `file_row_index()` works when called bare, but fails at **execution** time as soon as the call is wrapped in any enclosing expression — on the same table, same data, default configuration: ``` Execution error: file_row_index() is source dependent and cannot be evaluated directly ``` ```sql SELECT file_row_index(), column1 FROM big ORDER BY column1 LIMIT 3; -- ok SELECT file_row_index() + 1, column1 FROM big ORDER BY column1 LIMIT 3; -- Execution error ``` The marker UDF has to be rewritten into the file scan's projection. That rewrite does not happen when a `RepartitionExec` sits between the `DataSourceExec` and the `ProjectionExec`, and whether it does is decided by the round-robin heuristic in `enforce_distribution`, which compares the estimated row count against `batch_size`: ```rust // datafusion/physical-optimizer/src/ensure_requirements/enforce_distribution.rs:1167 Precision::Exact(n_rows) => n_rows > batch_size, ``` Three independent observations line up with that heuristic being the trigger: | change | result | |---|---| | 20 000-row table (default `batch_size` = 8192) | **fails** | | same query on a 3-row table | ok | | same 20 000-row table, `SET datafusion.execution.batch_size = 30000` | ok | | same 20 000-row table, `SET datafusion.execution.target_partitions = 1` | ok | So a query that works during development against a small table starts failing once the table grows past one batch. #### On the documented behaviour [`scalar_functions.md`](https://github.com/apache/datafusion/blob/main/docs/source/user-guide/sql/scalar_functions.md) says: > This function is intended to be rewritten at file-scan time. If the input file is > not known (for example, if this function is evaluated outside a file scan, or was > not pushed down into one), direct evaluation returns an error. That documents the error *mechanism* for a call with no file-scan context. In this report the call is written directly against a file scan, and the **bare form works on the same table** — so the deciding factor is the expression shape plus a physical-planning decision, not where the user put the call. The docs' own example passes only because it has 3 rows. For precedent: #23531 was the same `"... is source dependent and cannot be evaluated directly"` error surfacing from a different pushdown gap (`input_file_name()` in a Parquet `RowFilter`), and it was treated and fixed as a bug. ### To Reproduce In `datafusion-cli`, stock configuration — no `SET` required: ```sql COPY (SELECT value AS column1 FROM range(0, 20000)) TO '/tmp/df_fri_repro/big.parquet' STORED AS PARQUET; CREATE EXTERNAL TABLE big STORED AS PARQUET LOCATION '/tmp/df_fri_repro/big.parquet'; -- works: 0, 1, 2 SELECT file_row_index(), column1 FROM big ORDER BY column1 LIMIT 3; -- BUG: Execution error SELECT file_row_index() + 1, column1 FROM big ORDER BY column1 LIMIT 3; ``` Which expression shapes break: | shape | result | |---|---| | `file_row_index()` | ok | | `file_row_index() AS r` | ok | | `file_row_index() + 1` | **fails** | | `arrow_typeof(file_row_index())` | **fails** | | `SELECT r + 1 FROM (SELECT file_row_index() AS r FROM big)` | **fails** — the subquery is flattened, so this is not a workaround | It is not arithmetic specifically; any wrapping breaks it. The two escapes are keeping the call bare or setting `target_partitions = 1`, which gives up all parallelism. `EXPLAIN` on the failing query **succeeds**, and shows the repartition between the scan and the projection: ``` ProjectionExec RepartitionExec DataSourceExec ``` ### Expected behavior Either: 1. the rewrite survives a `RepartitionExec` between scan and projection — e.g. push the projection below it, or perform the rewrite before distribution enforcement — so that `file_row_index() + 1` behaves like `file_row_index()`; or 2. if successful pushdown is a hard precondition, reject the query at **plan** time with an actionable message, instead of accepting the plan, letting `EXPLAIN` succeed, and failing once execution has started. Silently depending on row count vs `batch_size` is the part that seems worth fixing either way. ### Additional context - Reproduced on `c120619de` (`48.0.0-rc2-4901-gc120619de`), `datafusion-cli` reporting `57.0.0-dev`, macOS arm64, default `target_partitions` (= core count). - `file_row_index()` was added in #22604, with examples from #24634. - Found while sweeping `datafusion.execution.batch_size` over the sqllogictest corpus for #26700. It is not batch-size-specific though: it reproduces at the default `batch_size` on any table larger than one batch. - Possibly related: #25459 (leaf expression pushdown epic), #20135 (metadata columns in `ListingTable` epic). ### Willingness to contribute I plan to contribute a fix for this bug independently. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
