GitHub user MaxGekk opened a pull request:
https://github.com/apache/spark/pull/23132
[SPARK-26163][SQL] Parsing decimals from JSON using locale
## What changes were proposed in this pull request?
In the PR, I propose using of the locale option to parse (and infer)
decimals from JSON input. After the changes, `JacksonParser` converts input
string to `BigDecimal` and to Spark's Decimal by using
`java.text.DecimalFormat`. New behaviour can be switched off via SQL config
`spark.sql.legacy.decimalParsing.enabled`.
## How was this patch tested?
Added 2 tests to `JsonExpressionsSuite` for the `en-US`, `ko-KR`, `ru-RU`,
`de-DE` locales:
- Inferring decimal type using locale from JSON field values
- Converting JSON field values to specified decimal type using the locales.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/MaxGekk/spark-1 json-decimal-parsing-locale
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/23132.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #23132
----
commit 506417ed4b643298560a66c043f7b31beb489da3
Author: Maxim Gekk <max.gekk@...>
Date: 2018-11-10T19:49:12Z
Test for parsing decimals using locale
commit ac25fb6ed1d3d6689ad8841476c025848c87f2a3
Author: Maxim Gekk <max.gekk@...>
Date: 2018-11-10T19:51:48Z
Parsing decimals using locale
commit b784003078270a46aaf8aceb2d86dd9f13f3500c
Author: Maxim Gekk <max.gekk@...>
Date: 2018-11-10T19:54:00Z
Updating the migration guide
commit d0522093dfe1823f91100cbec3ef5d6c8a372f27
Author: Maxim Gekk <max.gekk@...>
Date: 2018-11-24T19:32:45Z
Merge remote-tracking branch 'origin/master' into
json-decimal-parsing-locale
# Conflicts:
#
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/JsonExpressionsSuite.scala
commit 722e135cf3b94e22079fd9a0fe1d309a04e76a64
Author: Maxim Gekk <max.gekk@...>
Date: 2018-11-24T19:58:39Z
Add SQL config spark.sql.legacy.decimalParsing.enabled
commit dc6c0ac0a97b31441d99ff1dd71608ae5e2eca73
Author: Maxim Gekk <max.gekk@...>
Date: 2018-11-24T20:00:31Z
Updating the migration guide
commit f15b1817fb51d453487665122473855712214692
Author: Maxim Gekk <max.gekk@...>
Date: 2018-11-24T20:38:24Z
Added a test for parsing
commit ab781d54e4f7604d64c72c3c383c549abab0a9a9
Author: Maxim Gekk <max.gekk@...>
Date: 2018-11-24T20:52:03Z
Fix test
commit 163a8b9d7d017409ae4dfa40e492680bf0e4f935
Author: Maxim Gekk <max.gekk@...>
Date: 2018-11-24T20:52:26Z
Create getDecimalParser
commit 8fb65c0db85f4bd2f76d473c5e31e772ff0d4c1d
Author: Maxim Gekk <max.gekk@...>
Date: 2018-11-24T22:11:42Z
Add a test for inferring decimals
commit 7e3a2906a96894cadc58771131d07d06ba265382
Author: Maxim Gekk <max.gekk@...>
Date: 2018-11-24T22:12:35Z
Change JsonSuite to adopt it for JsonInferSchema class
commit 83920b25f586dc242841ff0a73105ae9e43295ed
Author: Maxim Gekk <max.gekk@...>
Date: 2018-11-24T22:13:01Z
Inferring decimals from JSON
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
