kotwal-itpro opened a new pull request, #8795:
URL: https://github.com/apache/hop/pull/8795
Fixes #8794.
On the native Spark engine, Iceberg tables in PATH mode were never written
at their path. They ended up under
`${java.io.tmpdir}/hop-iceberg-path-catalog-warehouse/`, in a sub-folder named
after the whole table URI. Reads looked in the same place, so a write followed
by a read worked, which is why the existing tests didn't notice.
## Cause
A PATH table was addressed as `` hop_iceberg.`file:///data/orders` `` in one
Hadoop catalog whose warehouse is in the temp folder. Iceberg's `SparkCatalog`
only treats an identifier as a location when it is a `PathIdentifier`, which
Spark creates for `spark.read.format("iceberg").load(path)`, not for a quoted
name in SQL or `writeTo()`. So the Hadoop catalog saw a table called
`file:///data/orders` in the default namespace and stored it at
`<warehouse>/file:/data/orders`.
## Fix
Each folder that holds PATH tables gets its own Hadoop catalog,
`hop_iceberg_<hash of the folder>`, with that folder as its warehouse, and the
table is addressed by its folder name: `` hop_iceberg_1a2b3c4d5e6f.`orders` ``.
A Hadoop catalog stores a table of the default namespace at
`<warehouse>/<name>`, which is exactly the table path. The catalog is
registered on the session the first time a table in that folder is used.
Because every PATH caller goes through
`SparkLakeTableSupport.icebergPathSqlIdentifier()` and
`resolveLakeTargetSqlId()`, this covers reads (including time travel), writes
in every save mode, merges, and the maintenance procedures (`CALL
hop_iceberg_<hash>.system.expire_snapshots(table => '`orders`')` etc.).
I looked at reading with `load(path)` instead, but writes have no path
equivalent that can also create a table, and merge and maintenance need a SQL
identifier anyway, so a catalog per folder keeps one code path for everything.
## Upgrading
Tables written with 2.19 are still in the temp folder. The lakehouse guide
now has a note on copying such a table to its path. They aren't moved
automatically: the temp folder may already have been cleaned, and silently
moving data around felt wrong. Happy to add an automatic fallback (read from
the old location when the table isn't at the path) if you think it's worth it.
## Testing
- `SparkLakeTableIcebergPathTest` now also checks that the table's metadata
is at the table path and that nothing is written to the old warehouse. That
assertion fails on main and passes with this change.
- New unit tests for the folder/table mapping: the parent folder is the
warehouse, tables in one folder share a catalog, other folders get another, and
a path without a parent folder is rejected.
- All 195 tests of `hop-engines-spark` pass.
- Built the full client (`-Pfull`) and ran `integration-tests/iceberg`
(round trip, overwrite and time travel, merge) with `hop-run`: all pass, the
tables are at `/tmp/hop-it-iceberg/...`, and
`hop-iceberg-path-catalog-warehouse` is no longer created.
The docs that described `hop_iceberg.uri` (lakehouse guide, output, merge,
maintenance, catalog pages) are updated.
- [x] I hereby declare this contribution to be licensed under the [Apache
License Version 2.0, January 2004](http://www.apache.org/licenses/LICENSE-2.0)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
