[GitHub] [iceberg] bkahloon opened a new issue #2202: Glue catalog table location not updated if creating Iceberg catalogs, databases or tables

[GitBox]

bkahloon opened a new issue #2202:
URL: https://github.com/apache/iceberg/issues/2202


   I seem to be having some difficulties with integrating AWS Glue catalog and 
Flink. If I try to register an Iceberg catalog, databases or tables from the 
Flink SQL client the appropriate names show up in Glue catalog but the location 
field is not updated (preventing me from querying it using some other query 
engine). 
   
   To provide some more details this was my workflow:
   
   1. create an iceberg catalog, database and table from the sql client.
   2.  Submit a flink datastream api job to write to this iceberg table, I just 
provide the table reference (also load the catalog) with the database.tablename 
reference (couldn't use SQL because CDC wasn't supported, see issue #2172)
   3. Flink is able to write to the appropriate s3 bucket and the metadata 
files are getting updated as well.
   4. When I look in the AWS Glue catalog UI, even though the table name shows 
up the table location, schema is not picked up in Glue. 
   
   The create catalog, database and table DDL queries I issued via the SQL 
client
   
   ```
   CREATE CATALOG iceberg_catalog WITH (
     'type'='iceberg',
     'catalog-type'='iceberg',
     'catalog-impl'='org.apache.iceberg.aws.glue.GlueCatalog',
     'lock-impl'='org.apache.iceberg.aws.glue.DynamoLockManager',
     'lock.table'='icebergGlueLockTable',
     'warehouse'='s3://iceberg-flink'
   );
   
   create database iceberg_catalog.iceberg_temp;
   
   CREATE TABLE iceberg_catalog.iceberg_temp.shipments_temp(
     shipment_id INT,
     origin STRING,
     is_arrived BOOLEAN,
     order_date DATE
   )PARTITIONED BY  (order_date);
   ```
   
   <img width="984" alt="Screen Shot 2021-02-02 at 10 32 45 PM 1" 
src="https://user-images.githubusercontent.com/42993708/106703571-b70da280-65a7-11eb-956f-ffd5f4351ca7.png";>
    
   <img width="1037" alt="Screen Shot 2021-02-02 at 11 00 03 PM" 
src="https://user-images.githubusercontent.com/42993708/106705167-af033200-65aa-11eb-86af-d013cd7820a5.png";>
   
   <img width="1049" alt="Screen Shot 2021-02-02 at 11 00 14 PM" 
src="https://user-images.githubusercontent.com/42993708/106705175-b296b900-65aa-11eb-8a36-a39089120e0b.png";>
   
   <img width="577" alt="Screen Shot 2021-02-02 at 11 01 01 PM" 
src="https://user-images.githubusercontent.com/42993708/106705183-ba565d80-65aa-11eb-9530-9d4f4cbc693a.png";>
   
   
   <img width="877" alt="Screen Shot 2021-02-02 at 11 01 17 PM" 
src="https://user-images.githubusercontent.com/42993708/106705192-bf1b1180-65aa-11eb-87aa-a751c27fb613.png";>
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For additional commands, e-mail: issues-h...@iceberg.apache.org