SameerMesiah97 opened a new pull request, #68411: URL: https://github.com/apache/airflow/pull/68411
**Description** This PR is a follow-up to #66220, which added support for returning Elasticsearch SQL query results as a Polars DataFrame via a custom cursor-based reader. This change extends that implementation to support `get_df_by_chunks(df_type="polars")` by introducing a chunked reader that uses Elasticsearch SQL cursor-based pagination to yield Polars DataFrames incrementally. **Rationale** The Common SQL implementation of `get_df_by_chunks(df_type="polars")` relies on `polars.read_database`, which is not compatible with Elasticsearch's SQL API and custom cursor implementation. To provide feature parity with the existing Polars DataFrame support added in #66220, this change introduces an Elasticsearch-specific chunked reader that interacts directly with the Elasticsearch SQL API and yields Polars DataFrames of the requested chunk size. **Tests** Added unit tests verifying that: * Elasticsearch SQL responses are correctly yielded as Polars DataFrames when the result set fits within a single chunk. * Result sets are split correctly across multiple chunks. * Chunk boundaries are handled correctly when they span multiple Elasticsearch cursor pages. * Cursor cleanup is performed using the last non-null cursor and is skipped when no cursor is returned by Elasticsearch. * The hook correctly delegates to the custom chunked reader when `get_df_by_chunks(df_type="polars")` is requested. **Documentation** Added docstrings for the chunked reader describing the use of Elasticsearch SQL cursor-based pagination and chunked Polars DataFrame generation. **Backwards Compatibility** This change adds support for `get_df_by_chunks(df_type="polars")` in ElasticsearchSQLHook and does not modify existing behavior. No breaking changes are introduced. Closes: ##66290 ##### Was generative AI tooling used to co-author this PR? - [X] Yes (please specify the tool below) Generated-by: [GPT 5.5] following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
