mcvsubbu commented on PR #10451: URL: https://github.com/apache/pinot/pull/10451#issuecomment-1490781015
> > Back when we were designing pause/resume, we specifically did not want proliferation of znodes (as was proposed in the first design). We changed to fold it all into idealstate znode. cc: @kishoreg , @npawar , @navina , @sajjad-moradi > > Thanks Subbu. Could you point me to the design/reference doc, just for my understanding ? Actually in this case, we did not want to send all updates (progress status updates corresponding to reload, force_commit, rebalance) to a single Znode as that would become a bottleneck. Instead have a Znode per update type which is fixed/constant (3). This Znode is not per table. Its per type (3 types) and takes updates for all tables. Here is the design doc in which we proposed znode hierarchy specifically for pause/resume. https://docs.google.com/document/d/19uKzPRowJ8WLE0A4g6i8XLOBwqpswH2_OvFvbxxZ_X4/edit#heading=h.tvfsvrm5pwew And here is the later design doc in which we changed it to be in IDEALSTATE, thereby losing some flxibility (e.g. per partition pause). I am not saying that one is better than the other. There are pros and cons. Asking if we are changing direction here: cc: @kishoreg -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
