deepthi912 opened a new pull request, #19415:
URL: https://github.com/apache/pinot/pull/19415
## Summary
Draft: extends the existing `time` tier selector with an optional
`segmentAgeField` that picks which `SegmentZKMetadata` timestamp is compared
against the age threshold.
Today `TimeBasedTierSegmentSelector` computes `now -
segmentZKMetadata.getEndTimeMs()`. `endTime` is the max data timestamp in the
segment — for batch/historical ingest (Iceberg/Parquet), this can lie years in
the past regardless of when the segment was actually built. Such segments match
any age threshold immediately at ingest and skip past every lifecycle tier
meant to catch them later.
Same selector type (`"time"`), same class, one optional parameter.
## Config
```json
"tierConfigs": [
{
"name": "cold_tier",
"segmentSelectorType": "time",
"segmentAge": "30m",
"segmentAgeField": "creationTime",
"storageType": "pinot_server",
...
}
]
```
Accepted values:
- `endTime` (default, backward-compatible) — uses
`SegmentZKMetadata#getEndTimeMs()`, unchanged behaviour.
- `creationTime` — uses `SegmentZKMetadata#getCreationTime()`, the moment
the segment file was built.
Missing / null / empty → `endTime`.
## Why creationTime matters
For Iceberg / batch ingest of historical Parquet:
- `endTime` reflects the data's timestamp (e.g. Oct 2024).
- `creationTime` reflects when the minion built the tar (e.g. Aug 2026).
With `endTime`-based aging, every freshly-ingested historical segment
matches any lifecycle tier immediately — the fresh tier is skipped entirely.
Switching to `creationTime` makes segment lifecycle track ingestion recency
rather than data recency, so `"segmentAge": "30m"` reliably means "30 minutes
after this segment first entered the cluster."
## Backward compatibility
- **Existing table configs (no `segmentAgeField`)**: Jackson deserializes to
null → factory maps to `END_TIME` → selector runs identical code path.
- **Existing Java code** calling `new TierConfig(name, selectorType, age,
list, ...)` (8-arg): backward-compat overload preserved.
- **Existing Java code** calling `new TimeBasedTierSegmentSelector(age)`:
1-arg constructor preserved as delegator to `(segmentAge, END_TIME)`.
- **Sort comparator** (`TierConfigUtils.getTierComparator`) unchanged —
still uses `segmentAgeMillis`, which the class still exposes.
The `END_TIME` code path in `selectSegment` is byte-for-byte the same as
before.
## Changes
- `TierConfig` — new optional `segmentAgeField` field + getter +
backward-compat constructor overload.
- `TimeBasedTierSegmentSelector` — new `AgeField` enum (`END_TIME`,
`CREATION_TIME`), new 2-arg constructor, `selectSegment` switches on it. Legacy
segments without `creationTime` are skipped (returns false) instead of
throwing, so a partially-populated table doesn't fail every tier evaluation.
- `TierFactory` — reads `tierConfig.getSegmentAgeField()`, parses via
`AgeField.fromConfig(...)`, threads through.
- `TierSegmentSelectorTest` — two new tests:
`testTimeBasedSegmentSelectorWithCreationTimeAgeField` (historical data +
missing creationTime + consuming guard) and `testAgeFieldParsing`
(null/empty/case/snake_case/unknown values).
## Test plan
- [x] `./mvnw -pl pinot-common,pinot-spi test
-Dtest=TierSegmentSelectorTest,TierConfigUtilsTest` — passes.
- [x] `./mvnw -pl pinot-common,pinot-spi checkstyle:check spotless:check` —
clean.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
