ChickenBenny commented on code in PR #76537: URL: https://github.com/apache/airflow/pull/76537#discussion_r4506567149
########## airflow-core/tests/unit/api_fastapi/test_lazy_response_fields.py: ########## @@ -0,0 +1,299 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +"""Guard against response fields that pydantic validates lazily. + +Pydantic validates an ``Iterable[...]`` field lazily, while FastAPI serializes the +response on the event loop: each item's attributes are then resolved there, on the sync +engine, so a lazy relationship or a ``selectin`` post-load becomes a query that blocks +the whole API server once the connection pool is busy. ``list[...]`` validates on +construction, inside the route, and renders to the same ``type: array`` in the OpenAPI +schema, so there is no reason to declare one of these fields lazily. + +Every pydantic model under ``airflow.api_fastapi`` is checked, not only the collection +responses: the hazard is the annotation, not the name of the class holding it. A model +defined inside a function is out of reach, and so is a response model that lives outside +the package, such as one belonging to a provider's auth manager app. + +The fields that predate this test are listed in +``generated/known_lazy_response_fields.txt`` as strict xfails. The collection ones are +tracked for migration in #76425. +""" + +from __future__ import annotations + +import functools +import importlib +import pkgutil +from collections.abc import ( + AsyncIterable, + AsyncIterator, + Generator, + Iterable, + Iterator, + Mapping, + Sequence, +) +from pathlib import Path +from types import ModuleType +from typing import Annotated, Any, get_args, get_origin + +import pydantic +import pytest +from pydantic import BaseModel, Field +from typing_extensions import TypeAliasType + +import airflow.api_fastapi + +from tests_common.test_utils.paths import AIRFLOW_ROOT_PATH + +KNOWN_LAZY_FIELDS_FILE = AIRFLOW_ROOT_PATH / "generated" / "known_lazy_response_fields.txt" + +# Only Iterable and Generator are validated lazily; the other three raise +# PydanticSchemaGenerationError at class definition instead, so they cannot reach the +# event loop. All five are listed to keep the rule one thing a reviewer has to remember. +LAZY_ORIGINS = frozenset({Iterable, Iterator, Generator, AsyncIterable, AsyncIterator}) Review Comment: **nit**: I tried this locally (pydantic 2.14.1). Without `arbitrary_types_allowed` all three raise as the comment says, but with it they're accepted with an isinstance check. `Iterator` is then iterated during serialization: ```python class M(BaseModel): model_config = ConfigDict(arbitrary_types_allowed=True) items: Iterator[int] M(items=iter(rows)).model_dump_json() # rows is only consumed here ``` `AsyncIterable` and `AsyncIterator` fail at serialization instead (`Unable to serialize unknown type: <class 'async_generator'>`), so `Iterator` is the one that could reach the event loop. Keeping it in `LAZY_ORIGINS` looks right to me, just for a different reason than the comment gives. Maybe something like: ```python # Iterable and Generator are validated lazily. Iterator is rejected unless # arbitrary_types_allowed is set, in which case it is passed through and iterated # during serialization. The async ones cannot be serialized either way. ``` No model under `airflow.api_fastapi` sets `arbitrary_types_allowed` today, so this is only about the comment. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
