dabla opened a new pull request, #75730:
URL: https://github.com/apache/airflow/pull/75730
`SqlToS3Operator` reads the whole query into one DataFrame;
`max_rows_per_file` only splits that frame afterwards. A result that does not
fit in memory therefore cannot be transferred with the operator, while
`GenericTransfer` already pages a source with `page_size` and
`paginated_sql_statement_clause`.
This gives `SqlToS3Operator` the same two parameters, with the same names
and the same default clause (`"{} LIMIT {} OFFSET {}"`), so the two operators
are configured alike: one query per page through the clause, one file per page
(`<key>_0`, `<key>_1`, ... — the naming `max_rows_per_file` already produces),
one page in memory at a time, and the transfer stops at the first page that
comes back short. The first page is always written, so the key exists for an
empty result. `page_size` cannot be combined with `max_rows_per_file` or
`groupby_kwargs`, which already split the output.
The per-DataFrame write and upload, and the read with its pandas dtype fix,
move out of `execute()` into `_upload_dataframe()` and `_read_dataframe()`,
shared by the paged and the unpaged path; `_partition_dataframe()` is untouched.
A hook that serves `LIMIT n OFFSET m` pages from one streamed result (a hook
with no server-side cursor can do that) then gives a memory-bounded transfer of
any size at the cost of a single query; a database source pages one `SELECT`
per page as with `GenericTransfer`, with the usual `ORDER BY` for stable pages.
Docs: a "Paging" section in the SQL to Amazon S3 transfer page with an
example in the system test DAG.
Tests: one file per page with the page SQL and the keys asserted; an exact
multiple that ends with an empty page; an empty result that still writes the
first page; a custom clause with a templated (string) `page_size`; the
exclusions. The system example is not run here.
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes — Claude Code (Fable 5.1)
Generated-by: Claude Code (Fable 5.1) following [the
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
