nimesh1601 opened a new pull request, #17847:
URL: https://github.com/apache/iceberg/pull/17847
## What & Why
Metadata tables were detected from the **last part of an identifier alone**
(`MetadataTableType.from(identifier.name())`). This caused three related
problems:
1. **A regular table that shares a metadata table name was unusable
(#10550).** A table like `db.files` was interpreted as the `files` metadata
table of namespace `db`, so it could not be created, loaded, or written — in
Spark it surfaced as `Cannot write into v1 table` / appeared as a non-Iceberg
table.
2. **Wrong identifier in the "not found" error (#13115).** Loading
`ns1.ns2.snapshots` when the base table `ns1.ns2` doesn't exist reported `Table
does not exist: ns1.ns2` instead of the identifier the caller actually
requested.
3. **REST catalog failure on a shallow metadata identifier (#13115).**
`loadTable("ns1", "snapshots")` fell back to loading a base table with an
**empty namespace**, producing `Malformed request: Ambiguous URI empty segment`
instead of a normal `NoSuchTableException`.
## Approach
Introduce `MetadataTableType.from(TableIdentifier)` that treats an
identifier as a metadata table **only when its namespace has at least two
levels** — i.e. the base table (the identifier without its last part) still has
a namespace. Route the catalog code through it:
- `BaseMetastoreCatalog.loadMetadataTable` / `isValidMetadataIdentifier`
- `MetadataTableUtils.hasMetadataTableName`
- `RESTSessionCatalog.loadTable` (metadata-table fallback)
and report the **requested identifier** when the base table is missing.
Because all metastore-backed catalogs (Hive, JDBC, Hadoop, Glue, DynamoDB,
Nessie, BigQuery, Snowflake, ECS, InMemory) extend `BaseMetastoreCatalog`, they
inherit the fix; `RESTSessionCatalog` is the only catalog that reimplements
`loadTable`, so it gets the equivalent change. Spark/Flink use explicit `$`/`#`
delimiters and are unaffected.
| Identifier | Before | After |
|---|---|---|
| `db.files` (real table, 1-level ns) | hijacked as metadata of `db` |
regular table `db.files` |
| `ns1.ns2.snapshots`, base missing | `Table does not exist: ns1.ns2` |
`Table does not exist: ns1.ns2.snapshots` |
| `ns1.snapshots` via REST | `Ambiguous URI empty segment` | clean
`NoSuchTableException` |
| `db.tbl.snapshots` (normal metadata table) | works | works (unchanged) |
## Behavior change / limitation
Metadata tables of a **root-namespace (empty-namespace) base table**
accessed via dotted syntax (e.g. `tbl.snapshots` where `tbl` lives at the
catalog root) are no longer recognized — a table sharing a metadata name in a
single-level namespace now wins. This ambiguity is inherent (`X.snapshots`
can't mean both) and matches the direction agreed in the prior PRs; it is not
reachable on REST, which rejects empty-namespace identifiers.
## Tests
- New `TestMetadataTableType` unit tests for `from(TableIdentifier)`.
- New parameterized `CatalogTests.tableSharingMetadataTableName` (all 16
metadata-table types × `{ns, ns1.ns2}`) covering create/load of a table that
shares a metadata name, resolution of its own metadata table, and correct "not
found" messaging. Inherited by every catalog suite; verified on InMemory, JDBC,
Hadoop, and REST. `spotlessCheck` and `:iceberg-core:revapi` pass.
## Credits
Builds on prior work by @nastra in #11963 and @adutra in #13223, which
stalled via the stale bot.
Closes #10550
Closes #13115
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
