GitHub user gengliangwang opened a pull request:
https://github.com/apache/spark/pull/22151
[SPARK-25160][SQL]Avro: remove sql configuration
spark.sql.avro.outputTimestampType
## What changes were proposed in this pull request?
In the PR for supporting logical timestamp types
https://github.com/apache/spark/pull/21935, a SQL configuration
spark.sql.avro.outputTimestampType is added, so that user can specify the
output timestamp precision they want.
With PR https://github.com/apache/spark/pull/21847, the output file can be
written with user specified types.
So there is no need to have such trivial configuration. Otherwise to make
it consistent we need to add configuration for all the Catalyst types that can
be converted into different Avro types.
This PR also add a test case for user specified output schema with
different timestamp types.
## How was this patch tested?
Unit test
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/gengliangwang/spark removeOutputTimestampType
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/22151.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #22151
----
commit 088bc83a7d5f767bcdfa98d72fd5bf6f1293f8f8
Author: Gengliang Wang <gengliang.wang@...>
Date: 2018-08-20T08:02:46Z
remove sql configuration spark.sql.avro.outputTimestampType
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
