aglinxinyuan opened a new pull request, #6488:
URL: https://github.com/apache/texera/pull/6488
### What changes were proposed in this PR?
On a Windows dev machine without a native Hadoop installation, every
workflow execution fails at execution start. Hadoop's local file system applies
POSIX permissions on **every** file/directory creation by shelling out to
`%HADOOP_HOME%\bin\winutils.exe`, with no configuration flag to disable it —
and Iceberg's `HadoopFileIO` (local warehouse, used by both the default
`postgres` catalog and the `hadoop` catalog) writes through exactly that path:
```
ExecutionStatsService (create runtime-stats Iceberg table)
└─ IcebergUtil.createTable
└─ JdbcTableOperations.doCommit → HadoopFileIO
└─ RawLocalFileSystem.mkdirs / create
└─ RawLocalFileSystem.setPermission ← on every file/dir
creation
└─ Shell.getSetPermissionCommand ← requires winutils.exe
chmod
└─ FileNotFoundException: Hadoop bin directory does not
exist
```
POSIX permission bits carry no meaning on NTFS, so this PR skips them there
instead of failing:
| File | Change |
|---|---|
| `WinutilsFreeLocalFileSystem.scala` (new) | A `LocalFileSystem` whose
`setPermission`/`setOwner` skip the winutils shell-out. Self-gating: when
winutils **is** installed, or on non-Windows hosts, it delegates to Hadoop's
default behavior. |
| `IcebergUtil.scala` | New private `newLocalHadoopConf()` used by
`createHadoopCatalog`/`createPostgresCatalog`. Only when `Shell.WINDOWS &&
!Shell.hasWinutilsPath()` does it select the new file system via `fs.file.impl`
(and disable the scheme-keyed `FileSystem` cache so an earlier cached instance
cannot bypass the override). On all other hosts it returns the default
`Configuration` unchanged. |
Before → after:
| Environment | Before | After |
|---|---|---|
| Windows, no winutils (stock checkout) | any workflow run fails immediately
| works |
| Windows with winutils installed | works | unchanged (gate off, shim
delegates) |
| Linux / macOS / CI | works | unchanged (gate off, shim delegates) |
### Any related issues, documentation, discussions?
Closes #6487
### How was this PR tested?
- New regression test in `IcebergUtilSpec` (create + load a table via
`createHadoopCatalog` in a temp local warehouse). Written first against stock
`main` on a Windows machine without winutils: it fails with the exact
production error (`RawLocalFileSystem.setPermission →
Shell.getSetPermissionCommand → FileNotFoundException: Hadoop bin directory
does not exist`), and passes with this fix. On Linux CI it exercises the
unchanged default path.
- New `WinutilsFreeLocalFileSystemSpec`: selection through `fs.file.impl`
(same mechanism production uses), `mkdirs`/`create`/read/delete round-trip, and
`setPermission` no-throw.
- Manually verified on Windows: workflow execution against the default
`postgres` catalog previously failed at `ExecutionStatsService` table creation
and now succeeds end-to-end (`JdbcCatalog` + `HadoopFileIO` → local warehouse
write).
- `WorkflowCore/scalafmtCheckAll` and `WorkflowCore/scalafixAll --check`
pass.
### Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Fable 5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
