jayamanikharyono opened a new pull request, #72768: URL: https://github.com/apache/airflow/pull/72768
`ClickHouseHook.get_conn()` and `get_client()` each called `clickhouse_connect.get_client()` directly, so every query, `run()` call, and `bulk_insert_rows()` invocation opened a brand new HTTP connection pool and incurred the client-initialization overhead (including a server metadata round trip). `bulk_insert_rows()` then closed that client itself after every call. This reworks the hook to create the client lazily once and reuse it for the hook's lifetime, matching clickhouse-connect's own client lifecycle guidance (reuse a single client, avoid creating one per query, close it explicitly when done): - `get_client()` now caches the client on the hook and only calls `clickhouse_connect.get_client()` on first access. `get_conn()` and `bulk_insert_rows()` both reuse the cached client instead of opening a new one. - Added `ClickHouseHook.close()` to close and clear the cached client, plus `__enter__`/`__exit__` so the hook can be used as a context manager. - `ClickHouseConnection.close()` is now a no-op. `DbApiHook.run()`/`get_records()`/etc. wrap `get_conn()` in `contextlib.closing(...)`, so this previously ran after *every* query — closing the real client there would have prevented it from being reused. - Since the client is now shared across calls instead of one client per call, `autogenerate_session_id` defaults to `False` unless overridden via `client_kwargs`, to avoid `ProgrammingError` from concurrent queries sharing one auto-generated session ID. No public API change — `get_conn()` and `get_client()` keep their existing signatures and return types. Unit tests updated/added to cover lazy creation, client reuse across `get_conn()` / `get_client()` / `bulk_insert_rows()`, `close()` semantics, and the new `autogenerate_session_id` default and overrides. References: - https://github.com/ClickHouse/clickhouse-connect/blob/main/docs/driver-api.mdx - https://github.com/ClickHouse/clickhouse-connect/blob/main/docs/advanced-usage.mdx --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes — Claude (Sonnet 5) Generated-by: Claude (Sonnet 5) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
