garvit-arora commented on PR #74577: URL: https://github.com/apache/airflow/pull/74577#issuecomment-6124008745
Thanks for putting up the API-server-specific fix. Keeping `ProcessPoolExecutor` for scheduler events preserves the reason for #39235. One concurrency concern: the API-server `ThreadPoolExecutor` uses `dag_state_change_process_pool_size`, which can be configured above 1. Both `_run_adapter_method()` and `_emit_manual_state_change_event()` call the process-wide cached `_get_process_adapter()`. That adapter lazily creates and stores its `OpenLineageClient` without synchronization. Multiple API threads could initialize separate clients/transports concurrently and then emit through the same adapter; the old process pool gave each worker its own adapter. Could the API-server pool use one thread, or provide a separate adapter per thread? A regression case with the configured pool size above 1 would help pin whichever ownership model you choose. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
