kacpermuda commented on code in PR #74577:
URL: https://github.com/apache/airflow/pull/74577#discussion_r4616951040
##########
providers/openlineage/src/airflow/providers/openlineage/plugins/listener.py:
##########
@@ -1117,41 +1129,84 @@ def _fork_execute(self, callable, callable_name: str):
os._exit(0)
@property
- def executor(self) -> ProcessPoolExecutor:
+ def executor(self) -> Executor:
+ if self._is_api_server:
+ if not self._thread_executor:
+ with self._lock:
+ if not self._thread_executor:
+ # The API server uses ThreadPoolExecutor with
max_workers=1 to serialize
+ # access to the module-level cached OpenLineageAdapter
(_get_process_adapter)
+ # across manual DAG run and task instance state change
emissions.
+ # Multiple threads must not access the adapter
concurrently because
+ # OpenLineageAdapter.get_or_create_openlineage_client
is not thread-safe.
+ # Scheduler uses ProcessPoolExecutor where each child
process has its
+ # own memory space and isolated adapter.
+ self._thread_executor = ThreadPoolExecutor(
+ max_workers=1,
+ thread_name_prefix="openlineage-api-executor",
+ )
+ return self._thread_executor
+
if not self._executor:
- self._executor = ProcessPoolExecutor(
- max_workers=conf.dag_state_change_process_pool_size(),
- initializer=_executor_initializer,
- )
+ with self._lock:
+ if not self._executor:
+ self._executor = ProcessPoolExecutor(
+ max_workers=conf.dag_state_change_process_pool_size(),
+ initializer=_executor_initializer,
+ )
return self._executor
@hookimpl
def on_starting(self, component) -> None:
self.log.debug("on_starting: %s", component.__class__.__name__)
+ from fastapi import FastAPI
+
+ if isinstance(component, FastAPI):
+ self._is_api_server = True
@hookimpl
def before_stopping(self, component) -> None:
self.log.debug("before_stopping: %s", component.__class__.__name__)
- if self._executor is None:
+ with self._lock:
+ thread_executor, self._thread_executor = self._thread_executor,
None
+ process_executor, self._executor = self._executor, None
+
+ if thread_executor is not None:
+ try:
+ thread_executor.shutdown(wait=False, cancel_futures=False)
+ timeout_val = conf.execution_timeout()
+ for thread in list(thread_executor._threads):
+ thread.join(timeout=timeout_val)
+ if thread.is_alive():
+ self.log.warning(
+ "OpenLineage API server thread did not finish
within execution_timeout=%ss "
+ "and will be abandoned.",
Review Comment:
Two problems in this block:
1. "will be abandoned" isn't accurate. `ThreadPoolExecutor` worker threads
are joined at interpreter exit (`concurrent.futures.thread._python_exit`) with
no timeout, so a hung call still delays process exit, and under gunicorn that
can run until `graceful_timeout`. The log line should say the thread is still
running rather than promise it's gone.
2. It still doesn't say how many queued events `cancel_futures=True` dropped
(my earlier question). Counting the pending futures before the cancel and
logging that would give on-call something to see.
Also, `thread.join(timeout=execution_timeout)` runs synchronously inside the
lifespan, so it blocks the event loop for up to that long, and `_threads` is a
private attribute.
##########
providers/openlineage/src/airflow/providers/openlineage/plugins/listener.py:
##########
@@ -1117,41 +1129,84 @@ def _fork_execute(self, callable, callable_name: str):
os._exit(0)
@property
- def executor(self) -> ProcessPoolExecutor:
+ def executor(self) -> Executor:
+ if self._is_api_server:
+ if not self._thread_executor:
+ with self._lock:
+ if not self._thread_executor:
+ # The API server uses ThreadPoolExecutor with
max_workers=1 to serialize
+ # access to the module-level cached OpenLineageAdapter
(_get_process_adapter)
+ # across manual DAG run and task instance state change
emissions.
+ # Multiple threads must not access the adapter
concurrently because
+ # OpenLineageAdapter.get_or_create_openlineage_client
is not thread-safe.
+ # Scheduler uses ProcessPoolExecutor where each child
process has its
+ # own memory space and isolated adapter.
+ self._thread_executor = ThreadPoolExecutor(
+ max_workers=1,
Review Comment:
`max_workers=1` doesn't fully serialize adapter access across a shutdown.
Thread A reads `self.executor` (E1), `before_stopping` detaches and shuts E1
down, A's `submit` raises `RuntimeError`, and `_reset_failed_executor` is a
no-op because `_thread_executor` is already `None`. The retry then builds E2
through `self.executor`, which nothing shuts down, and E1's in-flight call and
E2 can use `_get_process_adapter()` at the same time.
The window is small, but the comment here says the single worker guarantees
serialization. A lock around client creation in
`OpenLineageAdapter.get_or_create_openlineage_client` would fix it at the
source. Or have `before_stopping` mark the listener as stopped so the retry
doesn't build a new pool.
##########
providers/openlineage/src/airflow/providers/openlineage/plugins/listener.py:
##########
@@ -1117,41 +1129,84 @@ def _fork_execute(self, callable, callable_name: str):
os._exit(0)
@property
- def executor(self) -> ProcessPoolExecutor:
+ def executor(self) -> Executor:
+ if self._is_api_server:
+ if not self._thread_executor:
+ with self._lock:
+ if not self._thread_executor:
+ # The API server uses ThreadPoolExecutor with
max_workers=1 to serialize
+ # access to the module-level cached OpenLineageAdapter
(_get_process_adapter)
+ # across manual DAG run and task instance state change
emissions.
+ # Multiple threads must not access the adapter
concurrently because
+ # OpenLineageAdapter.get_or_create_openlineage_client
is not thread-safe.
+ # Scheduler uses ProcessPoolExecutor where each child
process has its
+ # own memory space and isolated adapter.
+ self._thread_executor = ThreadPoolExecutor(
+ max_workers=1,
+ thread_name_prefix="openlineage-api-executor",
+ )
+ return self._thread_executor
+
if not self._executor:
- self._executor = ProcessPoolExecutor(
- max_workers=conf.dag_state_change_process_pool_size(),
- initializer=_executor_initializer,
- )
+ with self._lock:
+ if not self._executor:
+ self._executor = ProcessPoolExecutor(
+ max_workers=conf.dag_state_change_process_pool_size(),
+ initializer=_executor_initializer,
+ )
return self._executor
@hookimpl
def on_starting(self, component) -> None:
self.log.debug("on_starting: %s", component.__class__.__name__)
+ from fastapi import FastAPI
Review Comment:
This hook runs in the task runner for every task and job start, and task-sdk
doesn't depend on fastapi. The provider still declares
`apache-airflow>=2.11.0`, and 2.11 doesn't install fastapi, so on those
installs this raises `ModuleNotFoundError` on every start and listeners after
it don't run. On 3.x it adds an import of fastapi to every task.
Suggest not importing it:
```python
fastapi = sys.modules.get("fastapi")
if fastapi is not None and isinstance(component, fastapi.FastAPI):
self._is_api_server = True
```
If the API server is the one calling this, fastapi is already loaded, so
this loses nothing.
##########
providers/openlineage/src/airflow/providers/openlineage/plugins/listener.py:
##########
@@ -166,10 +167,22 @@ class OpenLineageListener:
"""OpenLineage listener sends events on task instance and dag run starts,
completes and failures."""
def __init__(self):
- self._executor = None
+ self._executor: ProcessPoolExecutor | None = None
+ self._thread_executor: ThreadPoolExecutor | None = None
+ self._is_api_server: bool = False
+ self._lock = threading.Lock()
self.log = logging.getLogger(__name__)
self.extractor_manager = ExtractorManager()
self.adapter = OpenLineageAdapter()
+ if hasattr(os, "register_at_fork"):
+ os.register_at_fork(after_in_child=self._after_fork_child)
Review Comment:
Nit: every instance registers an `after_in_child` hook that is never
removed, so tests and any re-created listener accumulate them. Registering once
at module level against `get_openlineage_listener()` would avoid it.
##########
providers/openlineage/src/airflow/providers/openlineage/plugins/listener.py:
##########
@@ -1117,41 +1146,76 @@ def _fork_execute(self, callable, callable_name: str):
os._exit(0)
@property
- def executor(self) -> ProcessPoolExecutor:
+ def executor(self) -> Executor:
+ self._check_pid()
+ if self._is_api_server:
+ if not self._thread_executor:
+ with self._lock:
+ if not self._thread_executor:
+ # The API server uses ThreadPoolExecutor with
max_workers=1 to serialize
+ # access to the module-level cached OpenLineageAdapter
(_get_process_adapter)
+ # across manual DAG run and task instance state change
emissions.
+ # Multiple threads must not access the adapter
concurrently because
+ # OpenLineageAdapter.get_or_create_openlineage_client
is not thread-safe.
+ # Scheduler uses ProcessPoolExecutor where each child
process has its
+ # own memory space and isolated adapter.
+ self._thread_executor = ThreadPoolExecutor(
+ max_workers=1,
+ thread_name_prefix="openlineage-api-executor",
+ )
+ return self._thread_executor
+
if not self._executor:
- self._executor = ProcessPoolExecutor(
- max_workers=conf.dag_state_change_process_pool_size(),
- initializer=_executor_initializer,
- )
+ with self._lock:
+ if not self._executor:
+ self._executor = ProcessPoolExecutor(
+ max_workers=conf.dag_state_change_process_pool_size(),
+ initializer=_executor_initializer,
+ )
return self._executor
@hookimpl
def on_starting(self, component) -> None:
self.log.debug("on_starting: %s", component.__class__.__name__)
+ if component.__class__.__name__ == "FastAPI" or getattr(component,
"__module__", "").startswith(
+ "fastapi"
+ ):
+ self._is_api_server = True
@hookimpl
def before_stopping(self, component) -> None:
self.log.debug("before_stopping: %s", component.__class__.__name__)
- if self._executor is None:
+ self._check_pid()
+ with self._lock:
+ thread_executor, self._thread_executor = self._thread_executor,
None
+ self._semaphore = None
+ process_executor, self._executor = self._executor, None
+
+ if thread_executor is not None:
+ try:
+ thread_executor.shutdown(wait=False, cancel_futures=True)
Review Comment:
Still not addressed: the number of cancelled futures isn't logged, and the
"abandoned" warning added here is inaccurate. Details in the new inline comment
on the join loop.
##########
airflow-core/src/airflow/api_fastapi/app.py:
##########
@@ -100,6 +101,13 @@ def _initialize_api_server_stats() -> None:
@asynccontextmanager
async def lifespan(app: FastAPI):
_initialize_api_server_stats()
+ os.environ["_AIRFLOW_COMPONENT"] = "api_server"
+ from airflow.listeners.listener import get_listener_manager
+
+ try:
+ get_listener_manager().hook.on_starting(component=app)
Review Comment:
The inline import is now at module level, thanks. Still open: the PR
description mentions third-party listeners, but there's no newsfragment for
`airflow-core` now firing `on_starting` / `before_stopping` with a `FastAPI`
component. Leaving it open for a maintainer to decide.
##########
providers/openlineage/tests/unit/openlineage/plugins/test_listener.py:
##########
@@ -3267,3 +3269,393 @@ def _failing_emit():
listener.log.warning.assert_called_once()
assert "on_running" in listener.log.warning.call_args.args
+
+
@mock.patch("airflow.providers.openlineage.plugins.listener.ThreadPoolExecutor")
Review Comment:
The real-pool transport test is there now, thanks. The repeated `MagicMock`
dag-run setup is still copied in several tests (the `mock_dag_run =
MagicMock()` blocks), so a fixture would still help.
##########
providers/openlineage/src/airflow/providers/openlineage/plugins/listener.py:
##########
@@ -1117,41 +1129,84 @@ def _fork_execute(self, callable, callable_name: str):
os._exit(0)
@property
- def executor(self) -> ProcessPoolExecutor:
+ def executor(self) -> Executor:
+ if self._is_api_server:
+ if not self._thread_executor:
+ with self._lock:
+ if not self._thread_executor:
+ # The API server uses ThreadPoolExecutor with
max_workers=1 to serialize
+ # access to the module-level cached OpenLineageAdapter
(_get_process_adapter)
+ # across manual DAG run and task instance state change
emissions.
+ # Multiple threads must not access the adapter
concurrently because
+ # OpenLineageAdapter.get_or_create_openlineage_client
is not thread-safe.
+ # Scheduler uses ProcessPoolExecutor where each child
process has its
+ # own memory space and isolated adapter.
+ self._thread_executor = ThreadPoolExecutor(
+ max_workers=1,
+ thread_name_prefix="openlineage-api-executor",
+ )
+ return self._thread_executor
+
if not self._executor:
- self._executor = ProcessPoolExecutor(
- max_workers=conf.dag_state_change_process_pool_size(),
- initializer=_executor_initializer,
- )
+ with self._lock:
+ if not self._executor:
+ self._executor = ProcessPoolExecutor(
+ max_workers=conf.dag_state_change_process_pool_size(),
+ initializer=_executor_initializer,
+ )
return self._executor
@hookimpl
def on_starting(self, component) -> None:
Review Comment:
The API server check still fails open.
`get_listener_manager().hook.on_starting(component=app)` stops at the first
impl that raises, so a third-party `on_starting` that assumes a `Job` (e.g.
reads `component.job_type`, which I flagged earlier) and runs before this one
leaves `_is_api_server = False`. `app.py` logs and continues, and the next
manual state change forks again, which is #74544.
Two cheap fixes, ideally both:
- `@hookimpl(tryfirst=True)` on this hook, so a misbehaving plugin can't
starve it.
- Fail closed: use the process pool only when `on_starting` actually saw a
scheduler `Job`, and use the in-process thread pool otherwise.
A test with a raising `on_starting` registered ahead of this listener would
pin it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]