tarun11Mavani commented on issue #17337:
URL: https://github.com/apache/pinot/issues/17337#issuecomment-3653224951
> For this,
>
> > We should set the creationTime = max (creationTime of all merging
segments) + 1 to avoid this and fully replace all records from merging segments
with new merged segment.
>
> Can you verify this following edge case :
>
> * I have segments S1 and S2, and a merged segment MS1 that needs to be
uploaded. Before it's uploaded, some PKs in the segments S1 and S2 are
invalidated by a new segment S3.
> * Now if we upload the new merged segment MS1, does it invalidate PKs of
S3 in some rare case ? eg. if the time clashes.
If the new records in S3 have a higher comparison values than S1 or S2, it
will continue to stay even after merged segment is uploaded as the comparison
column value takes precedence.
If the record in S3 has the same comparison value as older record, record in
S1 and S2 will still get replaced with record in S3. Later when merged segment
is uploaded with creation time = (creation time of S2 + 1), it will still be
lower than the creation time of S3 so the record in S3 will remain valid. You
can read about the preference logic
[here](https://github.com/apache/pinot/blame/52db36c816f91ef8887fddd0beade5d169824296/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/upsert/BasePartitionUpsertMetadataManager.java#L518)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
