chenboat edited a comment on issue #7569: URL: https://github.com/apache/pinot/issues/7569#issuecomment-941708883
As can be seen from the event sequence below, the follow call `Removing upsert metadata for segment:` took 20s to finish and is the main culprit of the slow reload. In segment reload, an existing segment A is replicated to another copy A' to continue serving. When the segment finishes reloading, A' is destroyed. A upsert table metadata manager needs to delete all the references to the old segment in the upsert metadata table `_primaryKeyToRecordLocationMap`. This step right now takes too long for finish and should be optimized. One interesting note is that during segment reload, in fact none of the segment location is changed logically and they can be left untouched. Event |timestamp -- | -- Closed segment: rta_ads_metrics__3__72__20210611T1841Z of table: rta_ads_me... | 2021-10-12T22:01:04.719+00:00 Reloaded segment: rta_ads_metrics__3__72__20210611T1841Z in table: rta_ads_... | 2021-10-12T22:01:04.719+00:00 Removing upsert metadata for segment: rta_ads_metrics__3__72__20210611T1841... | 2021-10-12T22:00:42.629+00:00 Closing segment: rta_ads_metrics__3__72__20210611T1841Z of table: rta_ads_m... | 2021-10-12T22:00:42.628+00:00 Adding immutable segment: rta_ads_metrics__3__72__20210611T1841Z to table: ... | 2021-10-12T22:00:42.628+00:00 Replaced immutable segment: rta_ads_metrics__3__72__20210611T1841Z of table... | 2021-10-12T22:00:42.628+00:00 Trying to destroy segment : rta_ads_metrics__3__72__20210611T1841Z | 2021-10-12T22:00:42.628+00:00 Adding upsert metadata for segment: rta_ads_metrics__3__72__20210611T1841Z | 2021-10-12T22:00:41.431+00:00 Successfully loaded segment rta_ads_metrics__3__72__20210611T1841Z with con... | 2021-10-12T22:00:41.431+00:00 Found inverted index for segment: rta_ads_metrics__3__72__20210611T1841Z, c... | 2021-10-12T22:00:41.426+00:00 Found inverted index for segment: rta_ads_metrics__3__72__20210611T1841Z, c... | 2021-10-12T22:00:41.426+00:00 Found inverted index for segment: rta_ads_metrics__3__72__20210611T1841Z, c... | 2021-10-12T22:00:41.426+00:00 Reloading segment: rta_ads_metrics__3__72__20210611T1841Z in table: rta_ads... | 2021-10-12T22:00:41.378+00:00 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
