Duan Shiqiang created AIRFLOW-3487:
--------------------------------------
Summary: PostgresToGoogleCloudStorageOperator can't handle big sql
result
Key: AIRFLOW-3487
URL: https://issues.apache.org/jira/browse/AIRFLOW-3487
Project: Apache Airflow
Issue Type: Bug
Components: contrib, operators
Affects Versions: 1.10.1
Reporter: Duan Shiqiang
PostgresToGoogleCloudStorageOperator is using psycopg2's default cursor to
query postgres which will execute and fetch all results in memory. For bigger
sql result it won't work if the airflow worker doesn't have that much memory.
The operator should consider using server side cursor to handle big volume of
data.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
