gianm edited a comment on issue #10042: URL: https://github.com/apache/druid/issues/10042#issuecomment-675732225
> We only guarantee that sketches generated from an older code version can be read by newer code versions; not the other way around. Hmm, this guarantee unfortunately isn't strong enough for how Druid is designed, for two reasons that apply during software updates: 1. When in the middle of a rolling update, older brokers routinely talk to newer historicals. We do this due to potential protocol changes across versions. For example: if we're changing the query syntax, we make sure that new Historicals can read both the old and new syntaxes; but typically new Brokers will only send the new format. Due to this design, it means the intermediate data generated by the newer historicals would be written by the newer version of datasketches, but the broker needs to read it with the older version of datasketches. 2. If you have some data servers on a newer version than others, then the new data servers might use a newer version of datasketches to write segments containing new sketch images. But then those segments might get loaded by older historicals as part of replication or balancing operations. And so they need to be readable by the older version of datasketches. I'm not familiar enough with datasketches to know if there's a good way around this. With some of our internal sketches, what've we done in the past is make the new version able to write out both the old and new formats, and to use the old format by default, unless specifically configured to use the new format. Once all servers are upgraded to the new version then they can start using the new format. Would something like that be possible with datasketches? ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
