Hi, For auto-partitioned tables with Paimon lake tiering enabled, Fluss currently expires old partitions from metadata after the TTL window. This leads to two gaps:
1. Write gap: Late-arriving updates/inserts for expired partitions are rejected with PartitionNotExistException, even though the data could still be tiered to lake storage. 2. Lookup gap: Point lookups on expired partitions return null, even though the data still exists in Paimon. This is confusing for users because batch/lake reads can still observe historical data while Fluss online paths cannot. To address this, I'd like to propose FIP-28: Support Write and Lookup for Expired Partitions in Paimon Lake Tables[1] , which introduces: - A dedicated __historical__ partition that captures late-arriving writes for expired partitions, with correct changelog generation (UPDATE_BEFORE/UPDATE_AFTER) via old-value fallback from Paimon lake storage. - Server-side lake lookup fallback for point queries on expired partitions, so lookups return correct results as long as the data exists in Paimon. - Performance isolation between real-time and historical paths to ensure late-data handling does not impact real-time write/lookup performance. A POC implementation is available at my fork[2] for reference. Welcome your feedback and suggestions on this proposal. Looking forward to a productive discussion! [1] https://cwiki.apache.org/confluence/display/FLUSS/FIP-28%3A+Improve+historical+Partition+Support+for+Fluss+Datalake-Enabled+Tables [2] https://github.com/luoyuxia/fluss/commits/start-support-write-lookup-historial-partition/ Best regards, Yuxia
