SameerMesiah97 opened a new pull request, #66658: URL: https://github.com/apache/airflow/pull/66658
**Description** This PR implements `fetchmany` support for `ElasticsearchSQLCursor` by introducing an internal mutable row buffer (`_rows`) used to progressively consume paginated Elasticsearch SQL results. `fetchmany` now incrementally drains rows from the current response page and transparently requests additional Elasticsearch SQL cursor pages as needed, enabling compatibility with chunked dataframe retrieval methods such as `get_pandas_df_by_chunks`. **Rationale** `ElasticsearchSQLCursor` previously implemented `fetchall` using Elasticsearch SQL cursor pagination, but did not implement `fetchmany`, which prevented compatibility with APIs that rely on incremental DB-API cursor consumption semantics. This aligns the cursor behavior more closely with standard DB-API expectations and enables chunked dataframe retrieval without materializing the full result set in memory. **Tests** Added unit tests verifying that: * `fetchmany` correctly returns batches of rows across Elasticsearch cursor pages. * Sequential `fetchmany` calls progressively consume rows without duplication. * Exhausting the cursor returns all remaining rows and subsequent calls return an empty list. * Default `fetch_size` behavior is respected when no explicit size is provided. * `fetchall` correctly respects partially consumed cursor state after prior `fetchmany` calls. Updated existing cursor tests and pagination fixtures to reflect incremental row-consumption semantics and distinct paginated Elasticsearch SQL responses. **Notes** * Updated `execute` to refresh the internal row buffer whenever a new cursor page is retrieved. * Updated `rows` to return the internal buffer instead of directly proxying `response["rows"]`. * Updated cursor row handling to use explicit page-buffer semantics for incremental row consumption. **Backwards Compatibility** No breaking changes; existing `fetchall`, `get_records`, and dataframe retrieval behavior remain compatible. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
