Jackie-Jiang commented on PR #10451: URL: https://github.com/apache/pinot/pull/10451#issuecomment-1490988991
> > @mcvsubbu There are pros and cons of storing the status within the IS: For pause/consume, it makes more sense to store the status within the IS because: > > > > * There is only one state associated with the table (a boolean flag pause) > > * It is accessed quite frequently (not ad-hoc) > > * We never need to access this flag for all tables > > > > For the controller task, IMO it makes more sense to keep the status into a separate ZNode because: > > > > * It is per task type node, instead of per table node > > * All the access is ad-hoc (when manually submitting a task, or checking status) > > * We want to know all the tasks (cross multiple tables) for a given task type > > Not sure what you mean by "frequent access". > > Also, reload table is on a per-table basis and this PR suggests we move that to a separate znode as well Say we want to take the same approach as pause consumption by storing the job status in the IS, there will be the following problems: 1. We can only use the simple field because map field is reserved for segment assignment. For job status, we want to keep different job status apart by using the map field 2. It will be hard to use the simple field to track multiple tasks of the same task type on the same table (e.g. I reloaded multiple segments within a table) 3. To gather the job status for a given job type, we need to loop over all the IS, which is very inefficient -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
