litiliu opened a new issue, #3435: URL: https://github.com/apache/fluss/issues/3435
### Search before asking - [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar. ### Motivation ### Background Currently Fluss partitioning requires partition keys to be declared as physical table columns. For primary-key tables, the partition key also has to be a subset of the primary key. On the write path, the partition value is extracted directly from each row. This means users need to materialize partition columns such as `dt`, `hh`, or `event_day` in the table schema and in every written row, even when the partition value can be derived from an existing field such as `event_time` or from ingestion time. ### Problem For ingestion and lakehouse-style use cases, this leaks storage layout into the logical table schema and makes pipelines more cumbersome: - Upstream sources may not contain the desired partition column. - Users have to add derived columns only for partition routing. - The derived partition column can become inconsistent with the source timestamp column. - Primary-key table definitions are harder because the partition column must be included in the primary key. - It is different from lake formats such as Iceberg, where hidden partitions and partition transforms are common. Fluss already supports auto partition creation, but auto partitioning still depends on an explicit partition column in the table schema. It does not provide implicit or hidden partitioning. ### Solution Add support for implicit / hidden partitions, or partition transforms, so the partition value can be derived without exposing an extra physical column in the table schema. A minimal first step could support time-based transforms on one existing timestamp/date column, then extend to multi-level transforms later. ### Anything else? This probably needs a design/FIP because it touches table metadata, validation, client write routing, catalog integration, and partition pruning semantics. Related current behavior in the codebase: - `TableDescriptor` validates partition keys against schema/primary-key columns. - `PartitionGetter` extracts partition values from row fields. - Auto partitioning only creates partitions automatically; it does not make partition columns implicit. ### Willingness to contribute - [x] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
