[
https://issues.apache.org/jira/browse/AIRFLOW-3185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16645608#comment-16645608
]
Thomas Haederle commented on AIRFLOW-3185:
------------------------------------------
draft code Pandas method:
{code:java}
def get_pandas_chunks(self, sql, parameters=None, chunksize=100):
"""
Executes the sql and returns a pandas dataframe
:param sql: the sql statement to be executed (str) or a list of
sql statements to execute
:type sql: str or list
:param parameters: The parameters to render the SQL query with.
:type parameters: mapping or iterable
:param chunksize: Chunksize for the dataframe generator.
:type chunksize: int
"""
import sys
from contextlib import closing
if sys.version_info[0] < 3:
sql = sql.encode('utf-8')
import pandas.io.sql as psql
with closing(self.get_sqlalchemy_engine()) as conn:
return psql.read_sql(sql, con=conn, params=parameters, chunksize=chunksize)
{code}
> Add chunking to DBAPI_hook by implementing fetchmany and pandas chunksize
> -------------------------------------------------------------------------
>
> Key: AIRFLOW-3185
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3185
> Project: Apache Airflow
> Issue Type: Improvement
> Components: hooks
> Affects Versions: 1.10.0
> Reporter: Thomas Haederle
> Assignee: Thomas Haederle
> Priority: Minor
> Labels: easyfix
> Original Estimate: 72h
> Remaining Estimate: 72h
>
> DbApiHook currently implements get_records and get_pandas_df, where both
> methods fetch all records into memory.
> We should implement two new methods which return a generator with a
> configurable chunksize:
> - def get_many_records(self, sql, parameters=None, chunksize=20,
> iterate_singles=False):
> - def get_pandas_df_chunks(self, sql, parameters=None, chunksize=20)
> this should work for all DB hooks which inherit from this class.
> We could also adapt existing methods, but that could be problematic because
> these methods will return a generator whereas the others return either
> records or dataframes.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
