dabla opened a new pull request, #75656: URL: https://github.com/apache/airflow/pull/75656
`PostgresHook.dialect` and `MsSqlHook.dialect` are plain properties that build a new dialect on every access. The dialects cache their metadata lookups per instance (`get_column_names`, `get_target_fields`, `get_primary_keys` carry `methodtools.lru_cache`), so a cache on an object that lives for one attribute access never hits. `insert_rows` calls `_generate_insert_sql` once per chunk, and with `[core] dbapihook_resolve_target_fields` on, every chunk ran the `information_schema.columns` query again: about 10 ms and a round trip per `commit_every` rows. A `GenericTransfer` of 17 million rows in chunks of 1,000 spends a few minutes on that query alone. This makes both properties `cached_property`, as `DbApiHook.dialect` and `Db2Hook.dialect` already are: one dialect per hook, one lookup per table. Tests: for each hook, one test that the dialect is the same object on two accesses, and one that two `_generate_insert_sql` calls on the same hook look the columns up once. All four fail on `main` (the lookup runs twice). --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes — Claude Code (Fable 5.1) Generated-by: Claude Code (Fable 5.1) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
