szehon-ho edited a comment on issue #3087: URL: https://github.com/apache/iceberg/issues/3087#issuecomment-917078226
Thanks for the replies guys. I started working on a POC for Peter's option: https://github.com/apache/iceberg/pull/3099 I also see many advantages of using Hive's native [RetryingMetaStoreClient](https://github.com/apache/hive/blob/master/standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/RetryingMetaStoreClient.java). To repeat some of his and also add some of my own: * RetryingMetaStoreClient is used today in Hive and Spark, and is more mature with all known Hive failure types. HiveClientPool will have to catch up to all the Hive exception types to retry: https://github.com/apache/iceberg/pull/2844 for some missing exceptions, for example. * RetryingMetaStoreClient handles UGI impersonation logic for reconnect (needed in Kerberized environments) * ClientPoolImpl does not support any configuration of retry and retry-backoff. It hard-codes to 1 retry and no backoff, which may not be ideal in all cases (the default in RetryingMetaStoreClient is 3 retries and 1s backoff for instance). * For user experience, using RetryingMetaStoreClient would unify all Hive configs. For instance in Spark, setting 'spark.hadoop.hive.metastore.client.connect.retry.delay' will set it for all Hive connections (for Iceberg and non-Iceberg tables). Whereas if we add configs to ClientPool, it will be per catalog level, which may not make sense to user, and the config flags will be different than what they are used to. @rdblue can you see if these justifications make sense, or if you guys still feel strongly about adding more logic in ClientPool instead? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
