r7raul1984 opened a new issue, #13629:
URL: https://github.com/apache/gluten/issues/13629
### Backend
VL (Velox)
### Bug description
Recursive Hive scans can include data files stored under hidden directories.
When `mapreduce.input.fileinputformat.input.dir.recursive=true`,
`HivePartitionConverter` recursively enters subdirectories before filtering
hidden paths. The final filtering checks only each file's name, so normally
named files inside `_temporary`, `.backup`, or nested hidden directories
can be included in the scan.
### Example directory layout
```text
table/
root-data
visible/
nested-data
_temporary/
nested-uncommitted-data
.backup/
nested-backup-data
_temporary/
attempt/
uncommitted-data
.backup/
backup-data
```
All files in this example contain a non-empty line of text, such as `1\n`.
### Expected behavior
With recursive scanning enabled, only these files should be included:
- `root-data`
- `visible/nested-data`
Hidden directories and their descendants should be skipped, matching
Hadoop `TextInputFormat`.
### Actual behavior
`HivePartitionConverter` includes all six files, including four files under
hidden directories. This can cause scans to read temporary or backup data
and return unexpected rows.
With recursive scanning disabled, both `HivePartitionConverter` and
Hadoop `TextInputFormat` include only `root-data`.
### Reproduction and validation
The file-listing discrepancy was reproduced by compiling the current
`HivePartitionConverter` source and invoking its file-listing method,
then comparing the results with Hadoop `TextInputFormat` on the same
directory.
This reproduction checks file selection directly; it does not execute a
native reader or a SQL query.
Before the fix:
| Recursive scanning | Hadoop file count | Gluten file count |
| --- | --- | --- |
| Disabled | 1 | 1 |
| Enabled | 2 | 6 |
A regression test was added under:
```text
gluten-ut/spark35/src/test/scala/org/apache/spark/sql/hive/execution/GlutenHiveTableScanSuite.scala
```
Test name:
```text
Gluten - Hive recursive file listing excludes hidden directories
```
The test compares file enumeration with Hadoop `TextInputFormat`, including
visible subdirectories, hidden directories at multiple levels, and hidden
files.
After filtering hidden paths before descending into directories, the
regression test passed with Spark 3.5 / Scala 2.12 and the Velox profile:
one test succeeded, zero failed.
### Affected code
```text
gluten-substrait/src/main/scala/org/apache/spark/sql/hive/HivePartitionConverter.scala
```
Both the initial directory listing and recursive directory traversal need
to filter hidden paths before descending.
### Related issues
- https://github.com/apache/gluten/issues/3535
- https://github.com/apache/gluten/pull/3536
These introduced recursive Hive directory scanning. This issue concerns
hidden-directory filtering in that scanning path.
This issue description was prepared with assistance from
OpenAI Codex (gpt-6.1-sol).
### Gluten version
main branch
### Spark version
_No response_
### Spark configurations
```text
spark.hadoop.mapreduce.input.fileinputformat.input.dir.recursive=true
```
Spark forwards this setting to Hadoop using the `spark.hadoop.` prefix.
The reproduction sets Hadoop configuration
`mapreduce.input.fileinputformat.input.dir.recursive=true` directly when
constructing `HivePartitionConverter`. The regression test covers both
`true` and `false`.
### System information
_No response_
### Relevant logs
```bash
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]