aglinxinyuan opened a new pull request, #6528:
URL: https://github.com/apache/texera/pull/6528
### What changes were proposed in this PR?
Backport of apache/texera#6488 to `release/v1.2`.
On a Windows dev machine without a native Hadoop installation, every
workflow execution fails at execution start when the `postgres` or `hadoop`
Iceberg catalog type is configured. Hadoop's local file system applies POSIX
permissions on **every** file/directory creation by shelling out to
`%HADOOP_HOME%\bin\winutils.exe`, with no configuration flag to disable it —
and Iceberg's `HadoopFileIO` (the local warehouse used by the `postgres` and
`hadoop` catalog types) writes through exactly that path.
`release/v1.2` carries the same regression: it is on hadoop-common
**3.4.3**, and 3.4+ applies POSIX permissions on every local file/directory
creation, which pulls winutils in:
```
ExecutionStatsService (create runtime-stats Iceberg table)
└─ IcebergUtil.createTable
└─ JdbcTableOperations.doCommit → HadoopFileIO
└─ RawLocalFileSystem.mkdirs / create
└─ RawLocalFileSystem.setPermission ← on every file/dir
creation
└─ Shell.getSetPermissionCommand ← requires winutils.exe
chmod
└─ FileNotFoundException: Hadoop bin directory does not
exist
```
POSIX permission bits carry no meaning on NTFS, so this skips them there
instead of failing:
| File | Change |
|---|---|
| `WinutilsFreeLocalFileSystem.scala` (new) | A `LocalFileSystem` whose
`setPermission`/`setOwner` skip the winutils shell-out. Self-gating: when
winutils **is** installed, or on non-Windows hosts, it delegates to Hadoop's
default behavior. |
| `IcebergUtil.scala` | New `newLocalHadoopConf()` used by
`createHadoopCatalog`/`createPostgresCatalog`. Only when `Shell.WINDOWS &&
!Shell.hasWinutilsPath()` does it select the new file system via `fs.file.impl`
(and disable the scheme-keyed `FileSystem` cache so an earlier cached instance
cannot bypass the override). On all other hosts it returns the default
`Configuration` unchanged. |
Clean cherry-pick of the squash commit (`aefae7b`) — no conflicts. The
change is confined to `common/workflow-core` Scala and adds no new
dependencies, so no `LICENSE-binary`/`NOTICE-binary` regeneration is needed.
### Any related issues, documentation, discussions?
Backport of #6488 (Closes #6487).
### How was this PR tested?
- Both specs run green on `release/v1.2` with Java 17: `IcebergUtilSpec`
(local-warehouse create/load regression test + both branches of the injectable
winutils gate) and `WinutilsFreeLocalFileSystemSpec` (selection via
`fs.file.impl`, file round-trip, `setPermission`/`setOwner` no-throw).
- `WorkflowCore/scalafmtCheckAll` and `WorkflowCore/scalafixAll --check`
pass.
No `release/*` label is added — this PR *is* the backport, so the automated
backport check correctly skips.
### Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.8)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
