difin commented on code in PR #6474:
URL: https://github.com/apache/hive/pull/6474#discussion_r3495783220
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveTableUtil.java:
##########
@@ -244,6 +244,7 @@ public static Table deserializeTable(Configuration config,
String name) {
table = readTableObjectFromFile(location, config);
}
checkAndSetIoConfig(config, table);
+ IcebergVendedCredentialUtil.applyFromJobConf(table, config);
Review Comment:
I tested in the Gravitino q-test: you are correct - vended credentials in
`table.io().properties()` survive ser/de to executors. But I believe we still
need parsing these credentials and moving them to Hive configuration for the
following 2 reasons:
We still need to put credentials in JobConf because Iceberg FileIO and
Hadoop S3A do not share state, S3A only reads Configuration, and Tez/LLAP use
S3A for many s3:// paths.
`applyFromJobConf` in `HiveTableUtil.deserializeTable` is still needed
because there can be setups where REST catalogs vend S3 connection settings for
their own environment while Hive is running in a different network. For
example, In the Gravitino q-test, Gravitino (in Docker) vends
`s3.endpoint=http://minio:9000`, which Hive tasks on the host cannot reach.
Hive session config sets a reachable endpoint (`http://<host>:9000`).
`applyFromJobConf` takes the vended credentials, replaces only the endpoint and
path-style fields with the session values, leaves access keys unchanged, and
installs the result on the FileIO via `setCredentials()`. In my debug,
deserialized `table.io().properties()` still contained `http://minio:9000`, and
need to be overridden, so this step is required even when vended keys survive
ser/de.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
