taiyang-li opened a new pull request, #12272:
URL: https://github.com/apache/gluten/pull/12272
## Background
When running queries like below on Gluten (e.g. with the Velox backend),
`input_file_name()` returns an empty string:
```sql
SELECT a.event, input_file_name() AS fname
FROM parquet_table a
JOIN (SELECT X AS k1, 123L AS k2) b
ON a.event = b.k1 AND a.device_id = b.k2;
```
The physical plan contains (key fragment):
```
ProjectExecTransformer [..., input_file_name#1816 AS fname]
+- BroadcastHashJoinExecTransformer ..., BuildLeft
:- InputIteratorTransformer[..., input_file_name#1816]
: +- RowToColumnar
: +- *(1) Project [..., input_file_name() AS input_file_name#1816]
← BUG
: +- ColumnarToRow
: +- BroadcastQueryStage 0
: +- ColumnarBroadcastExchange
: +- LocalTableScan [...]
← no file context
+- ProjectExecTransformer [..., input_file_name#1816]
+- FileScanTransformer parquet ...[..., input_file_name#1816]
← same ExprId
```
## Root cause
`PushDownInputFileExpression.PreOffload` originally injects `Project [...,
input_file_name() AS attr#N]` above **every** `LeafExecNode`.
1. When BHJ's build side is a `LocalTableScanExec` / `RangeExec` /
`RDDScanExec` etc. (no real file context), `input_file_name()` has no
`InputFileBlockHolder` thread-local and always returns `""`.
2. Both leaves end up reusing the same `ExprId`. When BHJ resolves `left ++
right`, the outer `Project` is rebound to the build-side empty attribute, so
the final query returns an empty file name.
## Fix
Only inject `input_file_name()` on leaves that can really populate
`InputFileBlockHolder`:
- `FileSourceScanExec`
- v2 `BatchScanExec`
- Hive table scan (`HiveTableScanExecTransformer.isHiveTableScan`)
- `BatchScanExecTransformerBase` (already special-cased)
In addition, the `ProjectExec` match in `PreOffload` now requires that the
subtree actually contains at least one such file-aware source via the new
`hasInputFileRelatedSource` helper. This avoids producing a fake
`input_file_name` attribute on non-file leaves and avoids polluting the common
ExprId with the empty string from the BHJ build side.
## Test
Added `input_file_name() with BHJ build-side LocalRelation must return real
path` in `ScalarFunctionsValidateSuite` (backends-velox) covering:
- `fname` column is non-empty for every joined row;
- `fname` contains the real parquet path;
- compared against vanilla Spark.
The existing `test("input_file_name")` (file/Hive scan paths) is unchanged
because those scans are still in the whitelist.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
