Hi Lorenzo and Mehul, Thank you for the detailed feedback. My sincere apologies for the delayed response. I’d like to revive this FIP and move it forward again.
I have narrowed the FIP to focus only on removing mandatory system columns. PARTITION_TIMESTAMP has been removed and can be discussed separately if needed. Replies to Lorenzo ------------------ 1. Scope I agree that the previous scope was misleading. Removing the system columns does not remove lake access. FULL mode will continue reading a lake snapshot before switching to Fluss, and $lake will continue using the native Paimon or Iceberg source. The lake reader and tiering writer will support both legacy tables with system columns and new tables with only user-defined columns. We may still provide an offset system column for offset-ordered lake reads in the future. Since adding such a column changes the user-visible physical schema and sort order, it should require an explicit table option. 2. Union reads This FIP does not change the existing FULL union-read behavior. The lake-to-Fluss transition uses snapshot-level per-bucket offsets stored with the lake snapshot metadata. It does not depend on the __offset value of each lake row. 3. Auto-partitioned table restriction This restriction belonged to PARTITION_TIMESTAMP and is no longer part of this FIP. Replies to Mehul ---------------- 1. Schema and legacy detection The schema inspection will remain inside each lake-format plugin. When an existing lake table is opened, the Paimon or Iceberg implementation will classify it as: - legacy, when all three expected system columns exist; or - clean, when none of them exists. The mode will be derived from the physical lake schema rather than stored separately in Fluss metadata. A partial set of system columns or unexpected types will cause schema validation to fail. Manual changes to a Fluss-managed lake schema remain unsupported. 2. PARTITION_TIMESTAMP and partition boundaries Questions about timestamp-to-partition mapping, validation of manually partitioned tables, and CDC boundary handling are no longer part of this FIP. A separate proposal for partition-based historical reads should define those semantics explicitly, including duplicate handling. 3. Existing union-read boundary FULL mode will continue using the snapshot-level per-bucket offsets reported through the existing tiering metadata path. Removing per-row system columns does not change this transition point. 4. Backward compatibility Existing lake tables retain the legacy schema. A table dropped and recreated after the upgrade will use the clean schema, so users depending directly on __bucket, __offset, or __timestamp must migrate first. The safe upgrade order is: 1. Flink connector and lake plugin 2. Tiering service 3. Fluss cluster The cluster is upgraded last so it cannot create clean tables while old writers or lake readers are still running. 5. Paimon and Iceberg Yes, the change applies to both Paimon and Iceberg. Their catalog, writer, and reader implementations must both support the legacy and clean layouts. Regarding the FIP name, the broader name made sense when PARTITION_TIMESTAMP was included. Since that feature has been removed, I propose keeping the current title. Thanks again for the feedback. On 2026/03/03 19:35:23 Mehul Batra wrote: > Also I think the current name of the FIP doesn't do justice to the scope, > can we rename it something like: > *Optional System Columns and Partition-Based Historical Reads* > > On Wed, Mar 4, 2026 at 12:41 AM Mehul Batra <[email protected]> > wrote: > > > Hi Yuxia, > > > > First of all thank you for leading this, It's an important aspect as this > > is non-trivial storage cost in Parquet/ORC files for columns that most > > consumers never read and schema introspection gets polluted too. > > I've been going through FIP-27 in detail and have a few questions I'd like > > to clarify before implementation begins. Grouping them by area: > > > > *1. Schema & Legacy Detection* > > > > 1a. When datalake is re-enabled on an existing table and the lake table > > already exists with system columns, where does the schema inspection > > happen do we add a new method to the LakeCatalog interface (e.g., > > getTableSchema(TablePath)), or is this handled at the Fluss server > > metadata level outside the plugin boundary? > > > > 1b. Is the legacy/clean mode decision persisted in Fluss table metadata > > (e.g., as a property like fluss.lake.schema.mode = legacy | clean), or is > > it re-derived by inspecting the lake table schema each time? If re-derived, > > what happens if someone manually alters the lake table schema externally? > > > > *2. PARTITION_TIMESTAMP Mode* > > > > 2a. The FIP shows a day-granularity example for timestamp-to-partition > > mapping. Can we document the exact mapping for all supported time-unit > > values (hour, day, month, quarter, year)? I assume it follows the same > > DateTimeFormatter patterns in PartitionUtils, but it would be good to > > make this explicit. > > > > 2b. Should the Flink connector fail fast at job submission time (via > > ValidationException) if PARTITION_TIMESTAMP is used on a > > non-auto-partitioned table? Or do we allow it for manually partitioned > > tables as well? > > > > 2c. For PK tables with CDC, how are duplicates at the partition boundary > > resolved during the union read? Is it the same snapshot-then-changelog > > pattern that FULL mode uses today? The FIP mentions "downstream > > idempotency" but CDC duplicate handling is non-trivial it would help to be > > more specific here. > > > > *3. Union Read Boundary* > > > > 3a. How is the exact transition point from lake historical reads to Fluss > > log reads determined per-partition is it the per-partition tiering > > watermark stored in Fluss server metadata? > > > > *4. Backward Compatibility* > > > > 4a. If a user drops and recreates a table with the same name post-upgrade, > > the new lake table will not have system columns. Should we warn users about > > this schema change, especially if they have downstream jobs that depend on > > __offset or __bucket? > > > > *5. Scope* > > > > 5a. The changes apply to both Paimon and Iceberg lake catalogs, correct? > > Both PaimonLakeCatalog and IcebergLakeCatalog currently append system > > columns independently. > > > > Thanks for the FIP, happy to help with the implementation once these are > > clarified. > > > > > > Best Regards, > > Mehul Batra > > > > On Mon, Mar 2, 2026 at 7:26 PM Lorenzo Affetti via dev < > > [email protected]> wrote: > > > >> Hello! I went through the FIP another time as I did not remember doing it > >> already :) > >> > >> I have additional questions beyond the first 2. > >> > >> Let me paste those here and add: > >> > >> 1. Isn't the scope of the FIP misleading? > >> This FIP seems to be about removing system columns, but it primarily > >> proposes a new read mode named PARTITION_TIMESTAMP. > >> Is this because removing those columns prevents users from accessing data > >> on the lake? > >> If so: > >> - how do user are supposed to do that now > >> - What would change > >> > >> 2. How does this relate to union reads? > >> I am quite new to the community and Fluss. Could you explain how the new > >> PARTITION_TIMESTAMP mode relates to union reads? > >> If the answer is not obvious, perhaps this warrants a section in the FIP. > >> > >> 3. Why *"*Only auto partitioned table is supported in this mode"? > >> Why only for partitions generated by Fluss, and not for any partition that > >> represents a timestamp? > >> > >> On Wed, Feb 4, 2026 at 4:50 PM Lorenzo Affetti < > >> [email protected]> wrote: > >> > >> > Hello Yuxia! > >> > Thanks for the great FIP! > >> > I have some questions: > >> > > >> > 1. Isn't the scope of the FIP misleading? > >> > It seems this FIP is about removing system columns, but it primarily > >> > proposes a new read mode named PARTITION_TIMESTAMP. > >> > > >> > 2. How does this relate to union reads? > >> > I am quite new to the community and Fluss. Could you explain how the new > >> > PARTITION_TIMESTAMP mode relates to union reads? > >> > If the answer is not obvious, perhaps this warrants a section in the > >> FIP. > >> > > >> > Thank you! > >> > > >> > On Tue, Jan 20, 2026 at 8:20 AM yuxia <[email protected]> > >> wrote: > >> > > >> >> Hi, all. > >> >> > >> >> Currently, every Fluss lake table is automatically provisioned with > >> three > >> >> mandatory system columns, __bucket , __offset , __timstamp (intended > >> for > >> >> bucket and offset-based subscription as well as addition informartion > >> >> check). > >> >> While originally designed to allow clients to pinpoint specific data > >> >> offsets of specific buckets, the practical evolution of the ecosystem > >> has > >> >> rendered this default behavior suboptimal for the dowstream since the > >> >> dowstream warehouse or BI tools do not expect these internal metadata > >> >> fields. > >> >> > >> >> > >> >> So, I'd like to propose FIP-27: Remove Mandatory System Columns From > >> >> Fluss Lake Tables [1] to remove the three mandatory system columns > >> while > >> >> still keep compability. > >> >> > >> >> Welcome your feedback and suggestions on this proposal. Looking forward > >> >> to a productive discussion! > >> >> > >> >> [1]: > >> >> > >> https://cwiki.apache.org/confluence/display/FLUSS/FIP-27%3A+Remove+Mandatory+System+Columns+From+Fluss+Lake+Tables > >> >> > >> >> Best regards, > >> >> Yuxia > >> >> > >> > > >> > > >> > -- > >> > Lorenzo Affetti > >> > Senior Software Engineer @ Flink Team > >> > Ververica <http://www.ververica.com> > >> > > >> > >> > >> -- > >> Lorenzo Affetti > >> Senior Software Engineer @ Flink Team > >> Ververica <http://www.ververica.com> > >> > > >
