thisisnic commented on issue #33464: URL: https://github.com/apache/arrow/issues/33464#issuecomment-5646220502
Had another look at this. The root cause of issue 1 is in libarrow, not the R package, which is why #39542 didn't work out. The scan node binds the pushdown filter against the bare dataset schema: https://github.com/apache/arrow/blob/0241012442c00da87b777e24a37e1d155e80e806/cpp/src/arrow/dataset/scanner.cc#L179-L182 but binds the projection against the dataset schema plus the augmented fields: https://github.com/apache/arrow/blob/0241012442c00da87b777e24a37e1d155e80e806/cpp/src/arrow/dataset/scanner.cc#L241-L249 So `mutate(f = add_filename())` works, but any filter referencing `__filename` fails at bind time before the plan runs. The R package passes the unbound filter straight into the scan node here: https://github.com/apache/arrow/blob/0241012442c00da87b777e24a37e1d155e80e806/r/src/compute-exec.cpp#L290-L292 The R package also re-applies the same filter as a FilterNode after the scan: https://github.com/apache/arrow/blob/0241012442c00da87b777e24a37e1d155e80e806/r/R/query-engine.R#L94-L98 and the scan node's output schema does include the augmented fields: https://github.com/apache/arrow/blob/0241012442c00da87b777e24a37e1d155e80e806/cpp/src/arrow/dataset/scanner.cc#L1083-L1088 so once the bind in `NormalizeScanOptions` matches the projection, the filter evaluates correctly with no further R changes. I checked the downstream consumers of the bound filter and they already tolerate unknown refs: Parquet row group pruning uses `FindOneOrNone` and skips them (https://github.com/apache/arrow/blob/0241012442c00da87b777e24a37e1d155e80e806/cpp/src/arrow/dataset/file_parquet.cc#L948-L951), and the per-format column selection all goes through `MaterializedFields()`, which already includes projection refs like `__filename` in the working `mutate()` case. R reprex (dev build at 0241012): ```r library(arrow, warn.conflicts = FALSE) library(dplyr, warn.conflicts = FALSE) td <- tempfile() write_dataset(group_by(mtcars, cyl), td) ds <- open_dataset(td) ds |> mutate(f = add_filename()) |> filter(grepl("cyl=4", f)) |> collect() #> Error: Invalid: No match for FieldRef.Name(__filename) in mpg: double ... ds |> filter(grepl("cyl=4", add_filename())) |> collect() #> Error: Invalid: No match for FieldRef.Name(__filename) in mpg: double ... # works, because the filter becomes a FilterNode after the scan ds |> mutate(f = add_filename()) |> collapse() |> filter(grepl("cyl=4", f)) |> collect() ``` Python is affected too, but by two additional paths. pyarrow pre-binds the filter against the dataset schema in Python (https://github.com/apache/arrow/blob/0241012442c00da87b777e24a37e1d155e80e806/python/pyarrow/_compute.pyx#L2983-L2990) and `ScannerBuilder::Filter` re-checks against the dataset schema (https://github.com/apache/arrow/blob/0241012442c00da87b777e24a37e1d155e80e806/cpp/src/arrow/dataset/scanner.cc#L937-L944). Expression-based projection has the same problem because `ScannerBuilder::Project(exprs, names)` doesn't add the augmented fields, unlike the name-based overload (https://github.com/apache/arrow/blob/0241012442c00da87b777e24a37e1d155e80e806/cpp/src/arrow/dataset/scanner.cc#L928-L935). Only name-based projection works. Python reprex (pyarrow 25.0.1): ```python import tempfile, pyarrow as pa, pyarrow.dataset as ds, pyarrow.compute as pc d = tempfile.mkdtemp() ds.write_dataset(pa.table({"x": [1, 2, 3], "part": ["a", "a", "b"]}), d, format="parquet", partitioning=["part"]) dataset = ds.dataset(d, format="parquet", partitioning=["part"]) # works dataset.to_table(columns=["x", "__filename"]) # ArrowInvalid: No match for FieldRef.Name(__filename) in x: int64 ... dataset.to_table(columns={"f": pc.field("__filename")}) # ArrowInvalid: No match for FieldRef.Name(__filename) in x: int64 ... dataset.to_table(filter=pc.field("__filename") != "") ``` Proposed fix for the R case: bind the filter in `NormalizeScanOptions` against the dataset schema plus `kAugmentedFields`, mirroring the projection. The Python paths would need `ScannerBuilder::Filter`, `ScannerBuilder::Project(exprs, names)` and the Python-side bind updated as well, which could be a follow-up. Issue 2 (referencing `add_filename()` after a `collapse()`) isn't fixed by this. The ProjectNode after the scan has already dropped the augmented columns by then, and the error hint added in #32630 already covers that case. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
