JulianJaffePinterest commented on pull request #12137: URL: https://github.com/apache/druid/pull/12137#issuecomment-1017164092
@loquisgon but how does this solution avoid the same issue? Tombstones will be announced immediately, before the replacement segments are loaded, and so you still have the scenario where old data is unavailable but new data has not yet been loaded. To my understanding, you are attempting to retrofit atomic table replacement onto atomic partition replacement either way and so the same data availability issues exist here. To help me understand, what happens in your code if the historicals are under load and so take minutes to load the new physical segments? How are the tombstone segments held back in this case? More broadly, it would be a useful feature for Druid to have the ability to atomically swap a table, not just a partition (which is just a restatement of what you're trying to accomplish here I think). A common way this is done in other systems is to fully load the new version of a table and only once that's done update metadata pointers from the old to the new. Was a similar design considered here? Why or why not? If you've already considered and rejected this approach, can we add a paragraph to the design alternatives discussing why? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
