sgoel2be24-cyber commented on PR #73324: URL: https://github.com/apache/airflow/pull/73324#issuecomment-5947696590
Hi @molcay, thanks for taking a look. I can't run `example_postgres_to_gcs` myself: it provisions a Compute Engine VM and a GCS bucket, and I don't have a GCP environment set up for it. Instead, I ran the operator end-to-end against a real PostgreSQL server with only the GCS upload stubbed, and counted the `FETCH` statements Postgres received (`log_statement = 'all'`). Setup: PostgreSQL 16.2, Airflow 3.3.2, `apache-airflow-providers-postgres` 7.1.0 (psycopg3 path), and `apache-airflow-providers-google` 22.6.0, whose `postgres_to_gcs.py` is identical to current `main`. I compared that file with the same file plus this PR's patch, using `use_server_side_cursor=True` and JSON export. | psycopg | rows | `cursor_itersize` | version | `FETCH` statements received by Postgres | rows exported | |---|---|---|---|---|---| | 3.2.9 | 10,000 | 1,000 | `main` | 10,001 × `FETCH FORWARD 1` | 10,000 | | 3.2.9 | 10,000 | 1,000 | this PR | 1 × `FETCH FORWARD 1` + 10 × `FETCH FORWARD 1000` | 10,000 | | 3.3.6 | 100,000 | 2,000 | `main` | 100,001 × `FETCH FORWARD 1` | 100,000 | | 3.3.6 | 100,000 | 2,000 | this PR | 1 × `FETCH FORWARD 1` + 50 × `FETCH FORWARD 2000` | 100,000 | In every run the exported rows were complete and in order. The remaining `FETCH FORWARD 1` comes from the `description` property, which reads the first row to populate `cursor.description`. On localhost, 100k rows went from 6.0s to 3.3s. With real latency between the worker and the database, as in #72075, the gap is much larger because `main` makes one round trip per row. I'm happy to share the script. If someone with GCP access could run the system test, I'd appreciate it. Otherwise, let me know if you'd like this verified another way. --- Drafted-by: Claude Code (Opus 5.5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
