yunyad opened a new pull request, #3674: URL: https://github.com/apache/texera/pull/3674
This PR implements basic operator-level parallelism optimization by modifying the GUI interface for UDFs (User Defined Functions). It corresponds to [PR 2] in the Basic Ramen plan. The details and context are discussed in [issue #3605](https://github.com/apache/texera/issues/3605). The Basic Ramen strategy assumes that between two executions of the same workflow, the workflow structure remains unchanged. This allows us to reuse past runtime statistics for optimizing operator-level resource allocation (e.g., worker count). The full implementation will be split into two PRs: - PR 1 (this PR): Add UI and backend support for operator-level parallelism in UDFs - PR 2: Extend support to all other parallelizable operators Key Changes in This PR **Updated UDF UI:** - Added number-of-workers input field to the UDF operator panel - Ensures users can configure parallelism directly through the UI **Backend Modifications:** - Refactored ResourceAllocator to support configurable parallelism - Implemented GreedyResourceAllocator to select parallelism level based on historical runtime - Integrated GreedyResourceAllocator with the UDF operator execution logic **Configuration Support:** - Added workflow-level flags to enable/disable GreedyResourceAllocator - Allows flexible toggling of Basic Ramen mode -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
