GitHub user JoshRosen opened a pull request:
https://github.com/apache/spark/pull/14796
[SPARK-17229][SQL] PostgresDialect shouldn't widen float and short types
during reads
## What changes were proposed in this pull request?
When reading float4 and smallint columns from PostgreSQL, Spark's
`PostgresDialect` widens these types to Decimal and Integer rather than using
the narrower Float and Short types. According to
https://www.postgresql.org/docs/7.1/static/datatype.html#DATATYPE-TABLE,
Postgres maps the `smallint` type to a signed two-byte integer and the `real` /
`float4` types to single precision floating point numbers.
This patch fixes this by adding more special-cases to `getCatalystType`,
similar to what was done for the Derby JDBC dialect.
## How was this patch tested?
New test cases in `PostgresIntegrationSuite` (which I ran manually because
Jenkins can't run it right now).
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/JoshRosen/spark postgres-jdbc-type-fixes
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/14796.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #14796
----
commit c232666a806bb11985232854a4516b37d08a8cc9
Author: Josh Rosen <[email protected]>
Date: 2016-08-24T19:52:18Z
Fix handling of Float type in Postgres JDBC dialect.
commit 5c2b84ece88a106c3850ad8ba3cb64a539f1e6c4
Author: Josh Rosen <[email protected]>
Date: 2016-08-24T20:23:38Z
Similar fix for ShortType
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
