Hello devs, I have raised a proposal on Log Enrichment via Append Columns [1].
A common Fluss enrichment pipeline today looks like the following: edge device --> log table A --> Flink job (enrichment) --> log table B --> consumer If A has 50 columns and the enrichment adds 3, the 50 base columns are stored twice - once in A, once in B. At scale this is real operational and infrastructure cost - 2x storage, 2x write I/O, 2x fetch I/O for any downstream that consumes B plus the additional architectural and operational complexity of running and maintaining a two pipelines. We propose a first-class primitive that lets the same row gain columns over time, with the system enforcing "completeness" at the read boundary. With this proposal, enrichment job(s) writes enrichment columns back to the same log table A via appendColumns; there is no log table B. Different enrichment tasks can also happen at different cadence e.g. streaming for GeoIP lookup, batch for vector embedding. Base columns are stored once and enrichment columns are stored once (per column group). Consumers project whatever they need against the single table: projections that touch only base columns advance up to high watermark (HWM) as today; projections that touch enrichment column group g are clamped by enrichment water mark (EWM) at min(HWM, EWM_g), so partial rows are never observed and the system not the consumer owns the "is enrichment caught up?" decision. There are more details on the FIP and POC implementation [2]. Looking forward to your comments Best Keith References [1] https://cwiki.apache.org/confluence/display/FLUSS/FIP-45%3A+Log+Enrichment+via+Append+Columns [2] https://github.com/leekeiabstraction/fluss/tree/option02-lateMaterialized
