johntomcat7408-cmyk opened a new pull request, #4333: URL: https://github.com/apache/amoro/pull/4333
## Why are the changes needed? AMS currently constructs optimizing commits with tables whose Hadoop FileIO runs as the catalog service identity. In secured, table-owner-controlled warehouses, optimizer tasks can succeed while the AMS commit fails to write metadata or clean up files. This is the first scoped part of #4268. It covers AMS-side optimizing commits only; optimizer task writes and other maintenance operations are unchanged. ## Brief change log - Add opt-in `hdfs.impersonation.enabled` configuration, disabled by default. An explicit table value overrides the direct catalog value and the catalog `table.` default. - Scope owner impersonation to table loading for AMS optimizing commits so the resulting FileIO retains the owner identity for commit-time reads, writes, deletes, and executor-thread cleanup. - Resolve the current owner from Hive Metastore for Hive-backed Iceberg, Mixed-Iceberg, and Mixed-Hive tables, and from refreshed Iceberg metadata for other Iceberg, Mixed-Iceberg, and internal tables. - Require SIMPLE or KERBEROS authentication, HadoopFileIO, and a non-empty owner. Proxy and configuration failures are propagated without falling back to the catalog service user. - Keep ordinary table loads unchanged and document the Hadoop proxy-user and authorization requirements. ## How was this patch tested? - [x] Added positive and negative tests for configuration precedence, owner resolution, caller and executor identities, concrete FileIO entry points, unsupported configurations, missing owners, and no-fallback behavior. - [ ] Screenshots are not applicable because this change has no UI surface. - [x] Ran focused tests and validation locally. Commands and results: - `.\mvnw.cmd test -pl amoro-format-iceberg -am -Dtest=TestAuthenticatedHadoopFileIO,TestTableOwnerResolver -Dsurefire.failIfNoSpecifiedTests=false` — 17 tests passed. - `.\mvnw.cmd test -pl amoro-format-mixed/amoro-mixed-hive -am -Dtest=TestMixedHiveTablesImpersonation -Dsurefire.failIfNoSpecifiedTests=false` — 2 tests passed. - `TestInternalIcebergHandlerImpersonation` — 1 test passed in the AMS-focused run. - `.\mvnw.cmd validate -pl amoro-format-iceberg,amoro-format-mixed/amoro-mixed-hive,amoro-ams -am` — all 15 reactor modules passed Spotless and Checkstyle validation. - `git diff HEAD^ HEAD --check` — passed. The AMS-focused run could not initialize `TestOptimizingQueue`'s Hive metastore on Windows because `HADOOP_HOME`/`winutils.exe` is unavailable. A real Kerberos/MiniDFS/HMS environment and the full test suite were not run locally. ## Compatibility and risks - The feature is opt-in and disabled by default. - There are no Thrift, REST, database schema, or dependency changes. - When enabled, an invalid owner, unsupported authentication/FileIO, or Hadoop proxy authorization failure intentionally fails the optimizing commit instead of retrying as the service user. - The conventional `hdfs` owner is not special-cased; cluster proxy-user policy remains authoritative. ## Provenance and contribution boundaries The implementation is original to this PR. It builds on the existing `TableMetaStore#doAsImpersonating` and authenticated FileIO patterns already present in upstream commit `96a4d781aac60c4080b3f4a41322848e2308649a`. No code was copied from open or unmerged pull requests or third-party sources. ## Documentation - Does this pull request introduce a new feature? yes - If yes, how is the feature documented? `docs/admin-guides/managing-catalogs.md` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
