mengw15 commented on code in PR #6488:
URL: https://github.com/apache/texera/pull/6488#discussion_r3605838073
##########
common/workflow-core/src/main/scala/org/apache/texera/amber/util/IcebergUtil.scala:
##########
@@ -58,6 +59,27 @@ object IcebergUtil {
// Unique suffix for LARGE_BINARY field encoding
private val LARGE_BINARY_FIELD_SUFFIX = "__texera_large_binary_ptr"
+ /**
+ * Creates the Hadoop `Configuration` used by catalogs that access the
local file
+ * system through `HadoopFileIO`.
+ *
+ * On Windows hosts without a native Hadoop installation (winutils.exe),
Hadoop's
+ * default local file system fails on every write because it shells out to
winutils
+ * for chmod. In that case, swap in [[WinutilsFreeLocalFileSystem]], which
skips
+ * permission operations, so local development works without installing
winutils.
+ * On all other hosts the default configuration is returned unchanged.
+ */
+ private def newLocalHadoopConf(): Configuration = {
Review Comment:
Dug into this — the fix is needed independently of #6150. The crash is not
in the `hadoop` catalog type but in `HadoopFileIO`'s local-warehouse writes
(`RawLocalFileSystem.setPermission` → winutils), which the `postgres` catalog —
kept by #6150 — goes through as well (`JdbcTableOperations.doCommit →
HadoopFileIO`); `rest` writes through `S3FileIO` and is unaffected.
On "some recent changes": the timeline pins it to the early-July dependency
bumps. Windows dev worked on hadoop-common 3.3.1; after #6201 (3.4.3) and #6227
(3.5.0) every execution crashes at its first local Iceberg write — 3.4+/3.5
applies POSIX permissions on every file/directory creation (as the issue
notes), which is what pulls winutils in. So this is a regression from the
CVE-driven bumps, not a never-worked path.
Two small asks: (1) the description's "default `postgres` catalog" framing
and the before/after table predate #6049 — the stock default is `rest` now,
worth refreshing since it lands in the squash commit message; (2) for #6150:
the regression test drives `HadoopFileIO` through `createHadoopCatalog` as a
lightweight vehicle — when the `hadoop` type is removed, please re-anchor it
(e.g. against the local file system directly) rather than dropping it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
