Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/15610
@srowen, It seems we should fix `FastDateFormat.getInstance` too. It seems
that method also takes locale as the second argument.
```
.../sql/catalyst/json/JSONOptions.scala:
FastDateFormat.getInstance(parameters.getOrElse("dateFormat", "yyyy-MM-dd"))
.../sql/catalyst/json/JSONOptions.scala: FastDateFormat.getInstance(
.../sql/execution/datasources/csv/CSVOptions.scala:
FastDateFormat.getInstance(parameters.getOrElse("dateFormat", "yyyy-MM-dd"))
.../sql/execution/datasources/csv/CSVOptions.scala:
FastDateFormat.getInstance(
.../sql/execution/datasources/csv/CSVSuite.scala: val iso8501 =
FastDateFormat.getInstance("yyyy-MM-dd'T'HH:mm:ss.SSSZZ")
.../sql/execution/datasources/csv/CSVSuite.scala: val iso8501 =
FastDateFormat.getInstance("yyyy-MM-dd")
```
Also, IMHO, it seems okay to change the failed test
```scala
test("Float and Double Types are cast correctly with Locale") {
val originalLocale = Locale.getDefault
try {
val locale : Locale = new Locale("fr", "FR")
Locale.setDefault(locale)
assert(CSVTypeCast.castTo("1,00", FloatType) == 1.0)
assert(CSVTypeCast.castTo("1,00", DoubleType) == 1.0)
} finally {
Locale.setDefault(originalLocale)
}
}
```
to
```scala
test("Float and Double Types are cast correctly with US Locale") {
val originalLocale = Locale.getDefault
try {
val locale : Locale = new Locale("fr", "FR")
Locale.setDefault(locale)
assert(CSVTypeCast.castTo("1,00", FloatType) == 100.0)
assert(CSVTypeCast.castTo("1,00", DoubleType) == 100.0)
} finally {
Locale.setDefault(originalLocale)
}
}
```
in order to test if setting other locales does not affect the converted
results as this PR proposes the fixed locale.
One concern about this is to not consider the default locale in JVM for
parsing CSV and JSON but I think we might be able to introduce another option
to set the locale if this is problematic (with default value `Locale.US`) in
the future.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
