aglinxinyuan opened a new issue, #6487: URL: https://github.com/apache/texera/issues/6487
## Describe the bug On a Windows development machine, every workflow execution fails immediately at execution start, before any operator runs: ``` java.lang.RuntimeException: java.io.FileNotFoundException: Hadoop bin directory does not exist: C:\...\Coursier\cache\...\org\apache\hadoop\bin -see https://wiki.apache.org/hadoop/WindowsProblems ``` ## Root cause Iceberg's `HadoopFileIO` (used for the local warehouse by both the default `postgres` catalog and the `hadoop` catalog) writes through Hadoop's `LocalFileSystem`. On Windows, hadoop-common applies POSIX permissions on **every** file/directory creation by shelling out to `%HADOOP_HOME%\bin\winutils.exe`, and there is no configuration flag to disable this: ``` ExecutionStatsService (create runtime-stats Iceberg table) └─ IcebergUtil.createTable └─ JdbcTableOperations.doCommit → HadoopFileIO └─ RawLocalFileSystem.mkdirs / create └─ RawLocalFileSystem.setPermission ← on every file/dir creation └─ Shell.getSetPermissionCommand ← requires winutils.exe chmod └─ FileNotFoundException: Hadoop bin directory does not exist ``` Hadoop only skips the winutils path when its native library (`hadoop.dll`) is loadable, which also requires a native Hadoop installation. Neither ships with Texera, so a stock checkout cannot run workflows on Windows. | | | |---|---| | Affected storage paths | workflow results, runtime statistics, console messages (all Iceberg local-warehouse writes) | | Catalog types affected | `postgres` (default), `hadoop` | | Not affected | `rest` catalog (S3FileIO), Linux/macOS hosts | | Hadoop version | hadoop-common 3.5.0 (`RawLocalFileSystem.setPermission`) | ## To Reproduce 1. On Windows, without `HADOOP_HOME` pointing to a native Hadoop installation (the default for a fresh checkout), start Texera and run any workflow. 2. Execution fails immediately with the stack trace above. ## Expected behavior Workflows run on a stock Windows checkout. POSIX permission bits carry no meaning on NTFS, so Hadoop's chmod-on-create can be safely skipped there instead of failing the execution. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
