umehrot2 commented on issue #954: org.apache.hudi.org.apache.hadoop_hive.metastore.api.NoSuchObjectException: <hivedb.tableName> table not found URL: https://github.com/apache/incubator-hudi/issues/954#issuecomment-542446001 @gfn9cho @vinothchandar we are aware of this issue in Hudi. It does not currently work with glue-catalog. We have solved it on our side, and will be pushing a PR soon. For your information, here is explanation: - Currently Hudi is interacting with Hive through two different ways: - Creation of table statement is submitted directly to Hive via JDBC https://github.com/apache/incubator-hudi/blob/master/hudi-hive/src/main/java/org/apache/hudi/hive/HoodieHiveClient.java#L472 . Thus, Hive will internally create the right metastore client (i.e. Glue if `hive.metastore.client.factory.class` is set to `com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory` in hive-site) - Whereas partition listing among other things are being done by directly calling hive metastore APIs using hive metastore client: https://github.com/apache/incubator-hudi/blob/master/hudi-hive/src/main/java/org/apache/hudi/hive/HoodieHiveClient.java#L240 - Now in Hudi code, standard specific implementation of the metastore client (not glue metastore client) is being instantiated: https://github.com/apache/incubator-hudi/blob/master/hudi-hive/src/main/java/org/apache/hudi/hive/HoodieHiveClient.java#L109 . - Ideally this instantiation of metastore client should be left to Hive through https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L5045 for it to consider other implementations of metastore client that might be configured. That is why your table gets created in Glue metastore, but while reading or scanning partitions it is talking to the local hive metastore where it does not find the table created.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
