Hi all, Thanks a lot for the detailed review comments. I updated the Iceberg create_from_lake draft based on the first round of feedback:
https://docs.google.com/document/d/1REmpQ2r5W12OvW2CU9tNjDWjpkFJegiTZnE1IZbBH_U/edit?tab=t.0 The main updates are: 1. Partition mapping Iceberg transform partition support is treated as a follow-up dependency on the proposed Fluss implicit partition design: https://docs.google.com/document/d/1wDOkB-MuLrn3QYqL0fMlr1rj-lq891UbBcqXJSzCNcU/edit?tab=t.0 If that proposal is accepted, supported transforms such as day(ts) can be mapped to Fluss virtual partition keys. Until then, transform partitions are rejected by create_from_lake with a clear validation error. For partition spec evolution, Fluss only maps the latest/current Iceberg partition spec into Fluss metadata, while historical data reading still goes through the Iceberg reader for the captured snapshot. 2. Bucket configuration For Iceberg lake primary-key tables, Fluss uses the Iceberg bucketing strategy. A compatible bucket(N, ...) transform provides the Fluss bucket count. If the Iceberg table has no bucket transform, users must specify bucket.num for the Fluss primary-key table. 3. Initial readable LakeSnapshot Both log-table and primary-key paths now register an initial readable LakeSnapshot using the captured Iceberg snapshot ID plus Fluss bucket offsets. This makes the union-read boundary explicit. 4. Delete handling and deletion vectors The boundary remains the Iceberg reader. Fluss does not implement custom equality-delete, position-delete, or deletion-vector merging. 5. Write freeze The source Iceberg table is expected to be externally frozen during create_from_lake. Fluss captures the current Iceberg snapshot ID and re-checks it before exposing the created Fluss table/partition as ready. 6. Identifier-field duplicates Iceberg identifier fields are advisory. In v1, Fluss does not add a full uniqueness validation during bootstrap. KV rebuild follows Fluss upsert semantics, and duplicate identifier values are not guaranteed to produce a deterministic retained row. 7. Unsupported types and empty tables Unsupported Iceberg column types are rejected during metadata/schema mapping before bootstrap starts. Tables without a current Iceberg snapshot are also rejected in v1. Please take another look when you have time. Best, Liting ________________________________ 发件人: Mehul Batra <[email protected]> 发送时间: 2026年6月26日 19:23 收件人: [email protected] <[email protected]> 主题: Re: [DISCUSS] Draft design: Support upgrading existing Iceberg tables to Fluss tables Hi Liting, Thanks for the FIP, and agreed that building Iceberg support on top of FIP-25 is a great starting point. In addition to the partition and bucket questions above, a few more from my side: 1: Delete handling vs deletion vectors. The draft routes equality/position deletes through the Iceberg reader and rejects tables the reader can't resolve. On our side we're planning to move Iceberg PK toward deletion vectors and v3 rather than delete files. Would the bootstrap reader treat a v3 table with DVs the same way, i.e. just read snapshot-visible rows through the reader, or does DV support shift the reject boundary at all? 2: Write freeze enforcement. The flow assumes writes to the source Iceberg table are stopped and bootstrap reads a fixed snapshot. Is that freeze enforced or advisory? For example, if a commit lands on the source table after we capture the snapshot but before bootstrap finishes, do we detect it, or are those rows silently missed? 3: Identifier-field uniqueness. Iceberg identifier fields are advisory and don't enforce uniqueness. If two rows share the same identifier values in the source, what does the KV rebuild do, last-writer-wins overwrite or detect and reject? It would be good to state the expected behavior explicitly. 4: v3 type mapping. As we move toward v3, types like Variant and nanosecond timestamps come into scope. Are there source column types we can't map to Fluss yet, and if so do we reject at create_from_lake time or fail mid-bootstrap? Looking forward to your reply. Best Regards, Mehul Batra On Fri, Jun 19, 2026 at 9:04 PM Zhe Wang <[email protected]> wrote: > Thanks for drafting this. The overall direction of keeping this as a delta > design over FIP-25 looks reasonable to me. > > A few questions came to mind: > > 1. For the log-table path, should we still capture and persist the source > Iceberg snapshot ID even though no KV bootstrap is required? It could serve > as the historical read boundary and make the no-catch-up behavior explicit. > > 2. For mapping Iceberg identifier fields to Fluss primary keys, can we make > the validation and failure semantics more explicit? Iceberg identifier > fields are not enforced uniqueness constraints, so it would be good to > define what happens if duplicate identifier values exist in the source > snapshot. > > 3. Should the design explicitly reject unsupported Iceberg format versions > or delete encodings, for example v3 deletion vectors, unless the selected > Iceberg reader can guarantee correct snapshot-visible rows? > > I agree that delete-file handling should stay inside the Iceberg reader > boundary. Fluss should not implement custom equality/position delete > merging unless the Iceberg reader cannot provide correct snapshot-visible > rows, in which case rejecting the table sounds safer for the first version. > > > Best regards, > Zhe Wang > > > Yuxia Luo <[email protected]> 于2026年6月17日周三 10:44写道： > > > Hi, Liting > > Thanks a lot for this FIP, and apologies for the late reply. > > A note first: the content, interfaces, and implementation of FIP-25 may > > still change, but the overall direction should stay roughly the same. > > Building Iceberg support on top of FIP-25 is a great starting point. > > > > I have a couple of small questions: > > 1: Partition mapping. Iceberg supports transform-based partitioning, e.g. > > day(ts). How do we plan to map this onto Fluss partitions? It'd be good > to > > clarify how transform partitions (and possibly partition spec evolution) > > are handled. > > > > 2: Bucket configuration. For an Iceberg primary-key table, there isn't an > > explicit bucket concept like in Paimon. But a Fluss primary-key table > needs > > a bucket count. Wha would the bucket count be for the resulting Fluss > > primary-key table? > > > > Looking forward to your reply. > > > > On 2026/05/21 06:18:47 "Liting Liu (litiliu) via dev" wrote: > > > Hi community, > > > > > > I drafted a design for supporting create_from_lake on existing Iceberg > > tables: > > > > > > > > > https://docs.google.com/document/d/1REmpQ2r5W12OvW2CU9tNjDWjpkFJegiTZnE1IZbBH_U/edit?tab=t.0 > > > > > > The proposal follows the existing Paimon create_from_lake design as > much > > as possible, and only introduces Iceberg-specific deltas where needed. > > > > > > The main difference is how the target Fluss table type is derived from > > Iceberg metadata: > > > > > > * Iceberg format v1 tables are mapped to Fluss log tables. > > > * Iceberg format v2 tables without identifier fields are mapped to > > Fluss log tables. > > > * Iceberg format v2 tables with non-empty identifier fields are > > mapped to Fluss primary-key tables and reuse the existing bootstrap flow. > > > > > > For primary-key tables, the design reuses the same coordinator > > lifecycle, single holdPartition model, Tiering Service bootstrap > pipeline, > > snapshot commit flow, partition activation, and recovery behavior from > the > > Paimon design. The Iceberg-specific part is mainly metadata extraction > and > > using the Iceberg reader to read snapshot-visible rows during bootstrap. > > > > > > Could you please help review the draft and share feedback, especially > on: > > > > > > * Whether the table type mapping based on Iceberg format version > and > > identifier fields is reasonable. > > > * Whether relying on the Iceberg reader for equality/position > delete > > handling during bootstrap is the right boundary. > > > * Whether there are any Iceberg-specific cases missing from the > > current design. > > > > > > Thanks > > > > > >
