Hi, all. If no any further comments, I'd like to start vote in next monday.
On 2026/08/03 02:28:22 Yuxia Luo wrote: > Hi Lorenzo and Mehul, > > Thank you for the detailed feedback. My sincere apologies for the delayed > response. I’d like to revive this FIP and move it forward again. > > I have narrowed the FIP to focus only on removing mandatory system columns. > PARTITION_TIMESTAMP has been removed and can be discussed separately if > needed. > > Replies to Lorenzo > ------------------ > > 1. Scope > > I agree that the previous scope was misleading. > > Removing the system columns does not remove lake access. FULL mode will > continue reading a lake snapshot before switching to Fluss, and $lake will > continue using the native Paimon or Iceberg source. > > The lake reader and tiering writer will support both legacy tables with > system columns and new tables with only user-defined columns. > > We may still provide an offset system column for offset-ordered lake reads > in the future. Since adding such a column changes the user-visible physical > schema and sort order, it should require an explicit table option. > > 2. Union reads > > This FIP does not change the existing FULL union-read behavior. > > The lake-to-Fluss transition uses snapshot-level per-bucket offsets stored > with the lake snapshot metadata. It does not depend on the __offset value > of each lake row. > > 3. Auto-partitioned table restriction > > This restriction belonged to PARTITION_TIMESTAMP and is no longer part of > this FIP. > > Replies to Mehul > ---------------- > > 1. Schema and legacy detection > > The schema inspection will remain inside each lake-format plugin. When an > existing lake table is opened, the Paimon or Iceberg implementation will > classify it as: > > - legacy, when all three expected system columns exist; or > - clean, when none of them exists. > > The mode will be derived from the physical lake schema rather than stored > separately in Fluss metadata. A partial set of system columns or unexpected > types will cause schema validation to fail. Manual changes to a > Fluss-managed lake schema remain unsupported. > > 2. PARTITION_TIMESTAMP and partition boundaries > > Questions about timestamp-to-partition mapping, validation of manually > partitioned tables, and CDC boundary handling are no longer part of this > FIP. > > A separate proposal for partition-based historical reads should define > those semantics explicitly, including duplicate handling. > > 3. Existing union-read boundary > > FULL mode will continue using the snapshot-level per-bucket offsets reported > through the existing tiering metadata path. Removing per-row system columns > does not change this transition point. > > 4. Backward compatibility > > Existing lake tables retain the legacy schema. A table dropped and recreated > after the upgrade will use the clean schema, so users depending directly on > __bucket, __offset, or __timestamp must migrate first. > > The safe upgrade order is: > > 1. Flink connector and lake plugin > 2. Tiering service > 3. Fluss cluster > > The cluster is upgraded last so it cannot create clean tables while old > writers or lake readers are still running. > > 5. Paimon and Iceberg > > Yes, the change applies to both Paimon and Iceberg. Their catalog, writer, > and reader implementations must both support the legacy and clean layouts. > > Regarding the FIP name, the broader name made sense when > PARTITION_TIMESTAMP was included. Since that feature has been removed, I > propose keeping the current title. > > Thanks again for the feedback. > > > On 2026/03/03 19:35:23 Mehul Batra wrote: > > Also I think the current name of the FIP doesn't do justice to the scope, > > can we rename it something like: > > *Optional System Columns and Partition-Based Historical Reads* > > > > On Wed, Mar 4, 2026 at 12:41 AM Mehul Batra <[email protected]> > > wrote: > > > > > Hi Yuxia, > > > > > > First of all thank you for leading this, It's an important aspect as this > > > is non-trivial storage cost in Parquet/ORC files for columns that most > > > consumers never read and schema introspection gets polluted too. > > > I've been going through FIP-27 in detail and have a few questions I'd like > > > to clarify before implementation begins. Grouping them by area: > > > > > > *1. Schema & Legacy Detection* > > > > > > 1a. When datalake is re-enabled on an existing table and the lake table > > > already exists with system columns, where does the schema inspection > > > happen do we add a new method to the LakeCatalog interface (e.g., > > > getTableSchema(TablePath)), or is this handled at the Fluss server > > > metadata level outside the plugin boundary? > > > > > > 1b. Is the legacy/clean mode decision persisted in Fluss table metadata > > > (e.g., as a property like fluss.lake.schema.mode = legacy | clean), or is > > > it re-derived by inspecting the lake table schema each time? If > > > re-derived, > > > what happens if someone manually alters the lake table schema externally? > > > > > > *2. PARTITION_TIMESTAMP Mode* > > > > > > 2a. The FIP shows a day-granularity example for timestamp-to-partition > > > mapping. Can we document the exact mapping for all supported time-unit > > > values (hour, day, month, quarter, year)? I assume it follows the same > > > DateTimeFormatter patterns in PartitionUtils, but it would be good to > > > make this explicit. > > > > > > 2b. Should the Flink connector fail fast at job submission time (via > > > ValidationException) if PARTITION_TIMESTAMP is used on a > > > non-auto-partitioned table? Or do we allow it for manually partitioned > > > tables as well? > > > > > > 2c. For PK tables with CDC, how are duplicates at the partition boundary > > > resolved during the union read? Is it the same snapshot-then-changelog > > > pattern that FULL mode uses today? The FIP mentions "downstream > > > idempotency" but CDC duplicate handling is non-trivial it would help to be > > > more specific here. > > > > > > *3. Union Read Boundary* > > > > > > 3a. How is the exact transition point from lake historical reads to Fluss > > > log reads determined per-partition is it the per-partition tiering > > > watermark stored in Fluss server metadata? > > > > > > *4. Backward Compatibility* > > > > > > 4a. If a user drops and recreates a table with the same name post-upgrade, > > > the new lake table will not have system columns. Should we warn users > > > about > > > this schema change, especially if they have downstream jobs that depend on > > > __offset or __bucket? > > > > > > *5. Scope* > > > > > > 5a. The changes apply to both Paimon and Iceberg lake catalogs, correct? > > > Both PaimonLakeCatalog and IcebergLakeCatalog currently append system > > > columns independently. > > > > > > Thanks for the FIP, happy to help with the implementation once these are > > > clarified. > > > > > > > > > Best Regards, > > > Mehul Batra > > > > > > On Mon, Mar 2, 2026 at 7:26 PM Lorenzo Affetti via dev < > > > [email protected]> wrote: > > > > > >> Hello! I went through the FIP another time as I did not remember doing it > > >> already :) > > >> > > >> I have additional questions beyond the first 2. > > >> > > >> Let me paste those here and add: > > >> > > >> 1. Isn't the scope of the FIP misleading? > > >> This FIP seems to be about removing system columns, but it primarily > > >> proposes a new read mode named PARTITION_TIMESTAMP. > > >> Is this because removing those columns prevents users from accessing data > > >> on the lake? > > >> If so: > > >> - how do user are supposed to do that now > > >> - What would change > > >> > > >> 2. How does this relate to union reads? > > >> I am quite new to the community and Fluss. Could you explain how the new > > >> PARTITION_TIMESTAMP mode relates to union reads? > > >> If the answer is not obvious, perhaps this warrants a section in the FIP. > > >> > > >> 3. Why *"*Only auto partitioned table is supported in this mode"? > > >> Why only for partitions generated by Fluss, and not for any partition > > >> that > > >> represents a timestamp? > > >> > > >> On Wed, Feb 4, 2026 at 4:50 PM Lorenzo Affetti < > > >> [email protected]> wrote: > > >> > > >> > Hello Yuxia! > > >> > Thanks for the great FIP! > > >> > I have some questions: > > >> > > > >> > 1. Isn't the scope of the FIP misleading? > > >> > It seems this FIP is about removing system columns, but it primarily > > >> > proposes a new read mode named PARTITION_TIMESTAMP. > > >> > > > >> > 2. How does this relate to union reads? > > >> > I am quite new to the community and Fluss. Could you explain how the > > >> > new > > >> > PARTITION_TIMESTAMP mode relates to union reads? > > >> > If the answer is not obvious, perhaps this warrants a section in the > > >> FIP. > > >> > > > >> > Thank you! > > >> > > > >> > On Tue, Jan 20, 2026 at 8:20 AM yuxia <[email protected]> > > >> wrote: > > >> > > > >> >> Hi, all. > > >> >> > > >> >> Currently, every Fluss lake table is automatically provisioned with > > >> three > > >> >> mandatory system columns, __bucket , __offset , __timstamp (intended > > >> for > > >> >> bucket and offset-based subscription as well as addition informartion > > >> >> check). > > >> >> While originally designed to allow clients to pinpoint specific data > > >> >> offsets of specific buckets, the practical evolution of the ecosystem > > >> has > > >> >> rendered this default behavior suboptimal for the dowstream since the > > >> >> dowstream warehouse or BI tools do not expect these internal metadata > > >> >> fields. > > >> >> > > >> >> > > >> >> So, I'd like to propose FIP-27: Remove Mandatory System Columns From > > >> >> Fluss Lake Tables [1] to remove the three mandatory system columns > > >> while > > >> >> still keep compability. > > >> >> > > >> >> Welcome your feedback and suggestions on this proposal. Looking > > >> >> forward > > >> >> to a productive discussion! > > >> >> > > >> >> [1]: > > >> >> > > >> https://cwiki.apache.org/confluence/display/FLUSS/FIP-27%3A+Remove+Mandatory+System+Columns+From+Fluss+Lake+Tables > > >> >> > > >> >> Best regards, > > >> >> Yuxia > > >> >> > > >> > > > >> > > > >> > -- > > >> > Lorenzo Affetti > > >> > Senior Software Engineer @ Flink Team > > >> > Ververica <http://www.ververica.com> > > >> > > > >> > > >> > > >> -- > > >> Lorenzo Affetti > > >> Senior Software Engineer @ Flink Team > > >> Ververica <http://www.ververica.com> > > >> > > > > > >
