deemoliu commented on PR #17254:
URL: https://github.com/apache/pinot/pull/17254#issuecomment-3577189528
Upsert metadata TTL is enforced during segment commit. This approach has the
following pros and cons:
- Pros: Ensures keys are removed synchronously during commit, avoiding the
inconsistency risks associated with asynchronous deletion.
- Cons: Because key removal is tied to segment commit, it becomes less
flexible to control or tune the cleanup frequency. Moreover, Server replacement
or new server bootstrap may encounter issues due to the loss of watermarks
during segment commit–based cleanup.
For async deletion of upsert key, it might cause some inconsistency result,
for an example
```
{key=key1, ts = t1, col1= v1, ...}
...
{key=key1, ts = t2, col1= v2, ...}
```
if t2 - t1 > metadataTTL and v1 != v2,
- if the async jobs is completed between [t1, t1 + ttl], the key1 NOT is
removed from metadata and we got the result if we query the key1 {key=key1, ts
= t1, **col1= v1,** ...}
- if the async jobs is completed between [t1 + ttl, t2], the key1 is
**removed** from metadata, and we got the result if we query the key1
{key=key1, ts = t1, col1= v1, ...} {key=key1, ts = t2, **col1= v2**, ...}.
Will this issue happen to Dedup as well?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
