jscheffl opened a new pull request, #56457: URL: https://github.com/apache/airflow/pull/56457
So far the Edge worker running remote is using a simgle threaded loop to fetch, process, monitor tasks and upload results logs as well as heartbeat. This limits the parallelism achieveable on a Edge Worker - how many tasks can be handled in parallel. With this PR a major refactor is made to use Python AsyncIO for handling of tasks in order to improve concurrency with many tasks. Note: Before merging this needs to have proper review and testing as a lot of logic and libraries change with the risk of degraded quality/stability as of implementation glitched. Therefore this PR is WIP. FYI @dheerajturaga -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
