yjshen commented on pull request #1924: URL: https://github.com/apache/arrow-datafusion/pull/1924#issuecomment-1060433340
Sorry to join the party late! Thanks for @ me. Initially, I made `RuntimeEnv` a global one for minor interface plumbing while prototyping `MemoryManager`. And I even have had the formal proposal before introducing the `ObjectStore` interface to make `ObjectStore` itself global. In my humble opinion, whether the `Env` or `Store` static or not is more like a programming flavor. When implementing a lib, explicitly plumbing is favored. While implementing a framework or a standalone program, you are free to use `global`. However, when it comes to the multi-tenancy world, the issue changed significantly in my mind. We should prompt **Runtime Isolation** to among our first-class considerations. Therefore, I'm **against** making `RuntimeEnv`, especially `MemoryManager` be global among multiple tenants in DataFusion or Ballista. We should never have tasks from different clients with different priorities or shares treated equally in one manager. We may even need to consider **physical isolation** for the most crucial jobs or queries and assign exclusive executors to guarantee that no bad things happen. I'm OK with whatever configurations are passed down because of the simple usage pattern. Therefore, please provide another abstraction layer for **resource-related** or **performance-critical** components. We can afford to use resources less optimally, but never in a way that interferes with or even hinders each other. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
