zhuxiangyi commented on PR #8334: URL: https://github.com/apache/paimon/pull/8334#issuecomment-4857283032
@JingsongLi Thanks for the review. This is indeed a significant change, so let me describe our real use case in detail — and I'd love to hear your suggestions. **Background.** We have a wide feature/data-source cache table for our risk engine. The modeling groups fields of the same kind (one data-source response / one feature family) into a single `struct`. Roughly: ~276 top-level columns ≈ 267 structs + 9 scalars; the number of sub-fields per struct ranges from a few up to ~2599; flattening everything into top-level columns would be ~22k columns. **Why we keep it nested instead of flattening.** At this scale, ~22k top-level columns become hard to work with for us — the schema is serialized into every snapshot/manifest, columnar footer & per-column stats metadata grow (especially painful for the small incremental files data evolution produces), and engine planning/codegen cost rises noticeably; day-to-day schema evolution also gets unwieldy. Modeling "one data source = one struct" lets us manage a source as a unit and prune by group on read, which fits us better. If there's a better modeling approach here, I'm very open to it. **Read pattern.** This table is only read by primary key (row id), pulling one or more whole structs to feed the risk engine — no aggregation, no filtering, no sub-field predicate pushdown. So nesting has essentially no downside for our reads, and top-level column pruning already reads only the structs actually requested. **Why we need sub-field-level updates.** We backfill specific sub-fields inside a group over historical data (when a feature definition changes / data is fixed — e.g. recomputing 8 of the ~2599 features in one group), across large historical row ranges. With the existing top-level (whole-column) evolution, changing those few sub-fields forces rewriting the entire struct (up to ~2599 fields) across history — large write amplification; and when a group is maintained by multiple pipelines, whole-column rewrites also clobber each other. Sub-field-level writes aligned by row id let us write only the backfilled leaves and reassemble the rest from the original files by row id, which is exactly the pain point this PR targets. **Known trade-offs.** The feature currently supports one level of nesting, and partially-written struct files don't contribute that column's stats to pushdown — which doesn't affect our "point-read only, no pushdown" usage, but it is a limitation and I've noted it in the description. If you think there's a more suitable direction (either in modeling or in the implementation), I'm happy to discuss and adjust, and to add more docs/tests. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
