[
https://issues.apache.org/jira/browse/AIRFLOW-3185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16979489#comment-16979489
]
jack commented on AIRFLOW-3185:
-------------------------------
[~tomanizer] do you have a final version to PR?
> Add chunking to DBAPI_hook by implementing fetchmany and pandas chunksize
> -------------------------------------------------------------------------
>
> Key: AIRFLOW-3185
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3185
> Project: Apache Airflow
> Issue Type: Improvement
> Components: hooks
> Affects Versions: 1.10.0
> Reporter: Thomas Haederle
> Assignee: Thomas Haederle
> Priority: Minor
> Labels: easyfix
> Original Estimate: 72h
> Remaining Estimate: 72h
>
> DbApiHook currently implements get_records and get_pandas_df, where both
> methods fetch all records into memory.
> We should implement two new methods which return a generator with a
> configurable chunksize:
> - def get_many_records(self, sql, parameters=None, chunksize=20,
> iterate_singles=False):
> - def get_pandas_df_chunks(self, sql, parameters=None, chunksize=20)
> this should work for all DB hooks which inherit from this class.
> We could also adapt existing methods, but that could be problematic because
> these methods will return a generator whereas the others return either
> records or dataframes.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
