thinkharderdev commented on PR #59: URL: https://github.com/apache/arrow-ballista/pull/59#issuecomment-1147256298
> @thinkharderdev wow! a lot change 👍, I have some questions: Does the executor and scheduler have a one-to-one relationship? if one-to-multi, how to keep executor available slot in each scheduler, if each sql-request need get state update from all-executor, is there are better way ? I think this is one problem we should consider 😊 No, it is one-to-multi (or really multi-to-multi in principle). The executor available slots are stored in the state backend (either etcd or Sled). But when an executor publishes a task status update to the scheduler, then the particular scheduler which receives that request can re-asssign the executor slots without reading the backend state because the slot hasn't been returned to the pool yet. The session issue is interesting. For right now, we just save the session properties in the backend state and whenever we need to get a `SessionContext` for a session ID, we read the properties and create a new `SessionContext`. This can probably be optimized to use a `Watch` and an in-memory session registry but I kept it simple for right now. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@arrow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
