Jackie-Jiang commented on PR #9306: URL: https://github.com/apache/pinot/pull/9306#issuecomment-1241099058
> But as to the ideas `persist segment tier over server restarts` or `check all previous tiers`, they are subject to another race condition: the dataDir for tier is updated in TierConfig in TableConfig when server restarts. So instead, we may simply persist the segmentDir path over restarts, and check it out when calling addOrReplaceSegment(). This check is supposed to be done in TierBasedSegmentDirectoryLoader as those race conditions are specific to it. Will think a bit more on this. I thought about the problem of changing data dir, which is a destructive operation anyway (think of the case of server changing its root data dir). That leads me start thinking whether we should store the tier data dir in the table config. IMO that should be a server config because each server might have disk mounted under different path (not likely happen but just an example why it should be part of the server config). Basically the mapping from tier name to data dir should be within the server config, and the table config should only specify the tier name to use. Persisting local tier requires more management overhead such as where to store this meta, when the update it, clean it up when segment is removed etc. That's why checking all previous tiers might be easier. Segment deletion is another problem we need to think of. When segment is deleted, the ZK metadata is already gone. How do we find the local segment to remove? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
