ankitsultana commented on issue #12396: URL: https://github.com/apache/pinot/issues/12396#issuecomment-1949474503
> I'd assume the issue is about during table rebalance, the consuming segment might get started even before the other immutable segments in the same table partition get fully loaded on the server (?) The case of "consuming segment starting on a server before all all immutable segments are fully loaded" is not something I am concerned about, because I think the allSegmentsLoaded boolean should be able to handle that. (correct me if I am wrong) The case I tried to highlight in #12400 is: 1. Table rebalance is started. We have two replicas of a consuming segment S0 and S1. 2. As part of the rebalance, one of the immutable segment is removed from the server hosting S0. 3. Soon after, say there's a event whose primary key was in S0. S0 will think there was no record for this key, but S1 will merge it with the correct key. 4. If a segment commit follows soon after, then S0 and S1 will diverge permanently, leading to data inconsistency/loss. Of course it can happen that the immutable segment can get removed in both replica servers. In that case replicas won't diverge, but still the data computed as part of Partial Upsert merger would be wrong in both replicas. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
