kumarpritam863 opened a new pull request, #18582: URL: https://github.com/apache/iceberg/pull/18582
The Flink dynamic sink's `TableUpdater.findOrCreateTable` creates a missing namespace only when the table lookup throws `NoSuchNamespaceException`. The Glue catalog never does: `GlueTableOperations.getGlueTable` returns `null` on `EntityNotFoundException`, so a table in a database that does not exist looks like a table that does not exist yet and `loadTable` throws `NoSuchTableException`. The namespace is never created, and `createTable` then fails with ``` Cannot commit <catalog>.<db>.<table> because Glue cannot find the requested entity ``` (`GlueTableOperations`, `EntityNotFoundException` -> `NotFoundException`). A job that writes to a database that does not exist yet fails on every record and restarts. Seen on a cluster with a Flink dynamic sink writing to a new Glue database; the task manager log carried the message above. ### Change `findOrCreateTable` creates the namespace when the lookup reported `NoSuchNamespaceException` (as before) or when `namespaceIsMissing(identifier)` is true: the catalog is a `SupportsNamespaces`, the identifier has a namespace, and `namespaceExists` is false. The extra call happens only on the path that is about to create a table. A namespace created concurrently is still tolerated. The change and its test are identical in the three Flink version directories: `v1.20`, `v2.2`, `v2.3`. ### Testing - New `TestTableUpdater#testTableCreationWhenCatalogReportsMissingNamespaceAsNoSuchTable`: an `InMemoryCatalog` whose `loadTable` throws `NoSuchTableException` for a missing namespace, as Glue does. It fails without the change (`NoSuchNamespaceException: Cannot create table glue_like_namespace.myTable. Namespace does not exist`) and passes with it. - `TestTableUpdater` passes in all three versions (15 tests each). - `spotlessApply` is clean. Not run against a real Glue catalog; the behaviour of `getGlueTable` and `GlueTableOperations` is taken from the source. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
