Marius Van Niekerk created SPARK-11881:
------------------------------------------
Summary: [SQL] JDBC Postgres fetchsize parameter ignored
Key: SPARK-11881
URL: https://issues.apache.org/jira/browse/SPARK-11881
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 1.6.0
Reporter: Marius Van Niekerk
Reference:
https://jdbc.postgresql.org/documentation/head/query.html#query-with-cursor
In order for PostgreSQL to honor the fetchSize non-zero setting, its
Connection.autoCommit needs to be set to false. Otherwise, it will just quietly
ignore the fetchSize setting.
Since the JDBC driver doesn’t serialize the result data in its memory, we don’t
want to have 2-3 times of the memory created in the driver first before Spark
can fetch anything into its serialized data storage.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
