dennismdejong opened a new pull request, #13711: URL: https://github.com/apache/gravitino/pull/13711
### What changes were proposed in this pull request? Docs only: explain why writes to a Paimon primary-key table can be missing from the Iceberg view served by Gravitino's Iceberg REST service, as reported in #13560. For a primary-key table, Paimon's Iceberg metadata publication selects only the data files that have been merged to the highest LSM level (`shouldAddFileToIceberg` in Paimon's `IcebergCommitCallback`, threshold `numLevels - 1`, level 5 by default). A plain `INSERT` writes files below that level, so it is committed to the Paimon table and is readable by Paimon readers, but it never reaches the Iceberg metadata until a full compaction moves the merged result to the top level. Only the first commit of a table is published from a full snapshot, which is why the first write is visible and the later ones look lost. Paimon documents this in [Iceberg Compatibility / Primary Key Tables](https://paimon.apache.org/docs/master/iceberg/primary-key-table/). Gravitino's Iceberg REST service applies the client's `add-snapshot` / `set-snapshot-ref` updates verbatim through Iceberg's `CatalogHandlers.updateTable`, so it serves exactly the metadata Paimon published. - `docs/lakehouse-paimon-catalog.md`: new "Iceberg Compatibility" section describing `metadata.iceberg.storage` = `rest-catalog` usage with the Iceberg REST service, the primary-key publication rule, and the ways to establish a visibility boundary (full compaction, `compaction.optimization-interval` / `full-compaction.delta-commits`, deletion-vector mode). - `docs/iceberg-rest-service.md`: note next to the managed-table-type table that metadata is served as the producing engine published it, linking to the new section. ### Why are the changes needed? Users following the Paimon and Iceberg REST setup see later inserts missing from the Iceberg view and report it as a Gravitino bug (#13560). Documenting the publication rule and the compaction remedies answers the report without a code change. Fix: #13560 ### Does this PR introduce _any_ user-facing change? Docs only, no behaviour change. ### How was this patch tested? Not a code change. Verified that the two new relative links and their anchors resolve to existing headings, and that both external links (`https://paimon.apache.org/docs/master/iceberg/`, `https://paimon.apache.org/docs/master/iceberg/primary-key-table/`) return HTTP 200. The described behaviour was reproduced outside Gravitino with Apache Paimon 1.4.1 and a primary-key table writing Iceberg metadata to a local metadata directory: after two batch inserts the table holds four data files at level 0 while the published Iceberg metadata still references only the first two, and with a configuration where compaction reaches the highest level (`num-levels` = `2`, `num-sorted-run.compaction-trigger` = `1`) the published metadata contains all rows. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
