praveenc7 commented on issue #17821:
URL: https://github.com/apache/pinot/issues/17821#issuecomment-4020140663
> Can you describe your use case? `enableDynamicStarTreeCreation` was
introduced to reduce server overhead of re-creating the star-tree index, but
when a new segment is created, it should take the latest star-tree config. Why
do you want to keep old star-tree config?
@Jackie-Jiang We want to safely roll out star-tree indexes for a large
REALTIME table. The plan is:
1. Add starTreeIndexConfigs to the table config with
enableDynamicStarTreeCreation=false
2. New segments from real-time ingestion naturally get star-tree at
creation time
3. Old segments remain unchanged — no server-side reload overhead or risk
of OOM/disk spikes
4. Once we've validated storage/performance impact on new segments, enable
enableDynamicStarTreeCreation=true and trigger a reload to backfill old segments
The issue is that PurgeTask rebuilds segments from scratch — it reads all
records and creates a brand new segment via `SegmentIndexCreationDriverImpl`.
Without this fix, SegmentGeneratorConfig(tableConfig, schema)
unconditionally picks up starTreeIndexConfigs, adding star-tree to old
segments that never had it during purge rebuilds.
My understanding was that `enableDynamicStarTreeCreation`=false guards
against star-tree modifications to existing segments regardless of the code
path — consistent with how SegmentPreProcessor skips all star-tree
processing (no add, no delete, no modify) when this flag is false. Is that
not the intended semantics for this flag?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
