dabla opened a new pull request, #75815: URL: https://github.com/apache/airflow/pull/75815
`AzureDataLakeStorageV2Hook` uploads only local files (`upload_file`, `upload_file_to_directory` open a `file_path` themselves), so a task that already holds the data in memory, such as an API response or an extraction page, has to write it to disk first. The hook also has no async method, so an async task or trigger has to hand every call to a thread. This adds: - `load_bytes(file_system_name, file_name, data, overwrite=False, **kwargs)`: `upload_file()` without the local file, through `get_file_client(...).upload_data(...)` on the hook's own client. - `get_async_conn()`: the async `DataLakeServiceClient` (`azure.storage.filedatalake.aio`), built from the connection the way `get_conn()` builds the sync one (connection string, Active Directory service principal with its proxies, account key, default credential). The connection is read with `get_async_connection` / `get_async_extra_dejson`, so the event loop is never blocked. It is not cached: an async client belongs to the loop it was created on. - `aload_bytes(...)`: the async counterpart of `load_bytes()` on that client. It closes the client and, when it has one, the token credential's own session. `load_bytes` writes through `get_file_client` rather than `create_file` as `upload_file` does, because `create_file` replaces an existing file with an empty one before the upload, even with `overwrite=False`. We have run the same code as a monkey patch in production Dags, writing JSON Lines pages to a Fabric OneLake lakehouse from async tasks. Tests: six new unit tests in `test_data_lake.py`, for `load_bytes`, `aload_bytes` with a service principal, a connection string and an account key (the sync `get_conn` must not be called), and `get_async_conn` building a real async client with a service principal. `test_data_lake.py` passes locally (26 tests). --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes — Claude Code (Opus 5.5) Generated-by: Claude Code (Opus 5.5) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
