mengw15 commented on code in PR #6488:
URL: https://github.com/apache/texera/pull/6488#discussion_r3605498707
##########
common/workflow-core/src/main/scala/org/apache/texera/amber/util/IcebergUtil.scala:
##########
@@ -58,6 +59,27 @@ object IcebergUtil {
// Unique suffix for LARGE_BINARY field encoding
private val LARGE_BINARY_FIELD_SUFFIX = "__texera_large_binary_ptr"
+ /**
+ * Creates the Hadoop `Configuration` used by catalogs that access the
local file
+ * system through `HadoopFileIO`.
+ *
+ * On Windows hosts without a native Hadoop installation (winutils.exe),
Hadoop's
+ * default local file system fails on every write because it shells out to
winutils
+ * for chmod. In that case, swap in [[WinutilsFreeLocalFileSystem]], which
skips
+ * permission operations, so local development works without installing
winutils.
+ * On all other hosts the default configuration is returned unchanged.
+ */
+ private def newLocalHadoopConf(): Configuration = {
Review Comment:
we discussed dropping the `hadoop` catalog type in #6049: no deployment or
CI selects it, Iceberg documents `HadoopCatalog` as only safe on file systems
with atomic rename (HDFS, which we don't use), and when I tried it for the unit
job it flooded `CommitFailedException` under amber's concurrent result commits.
#6150 tracks the removal.
@aglinxinyuan given the `hadoop` catalog type may be removed, is this PR
still needed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
