Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/14279
I think it is okay. Just to make sure,
- **This writes date and timestamps fine**:
```scala
val path = "/tmp"
val date = java.sql.Date.valueOf("1970-01-01")
val timestamp = java.sql.Timestamp.valueOf("1970-01-01 11:46:40.0")
val data = Seq((date, timestamp)).toDF("date", "timestamp")
// Write CSV data
data.write.format("csv").save(s"$path/csv")
spark.read.format("text").load(s"$path/csv").collect().foreach(println(_))
// Write JSON data
data.write.format("json").save(s"$path/json")
spark.read.format("text").load(s"$path/json").collect().foreach(println(_))
```
prints below:
```
[1970-01-01,1970-01-01T11:46:40.000-08:00]
[{"date":"1970-01-01","timestamp":"1970-01-01T11:46:40.000-08:00"}]
```
- **Able to be read back fine.**
```scala
val path = "/tmp"
val date = java.sql.Date.valueOf("1970-01-01")
val timestamp = java.sql.Timestamp.valueOf("1970-01-01 11:46:40.0")
val data = Seq((date, timestamp)).toDF("date", "timestamp")
// Roundtrip in reading and writing CSV data
data.write.format("csv").save(s"$path/csv")
val roundtripCsvDF =
spark.read.schema(data.schema).format("csv").load(s"$path/csv")
roundtripCsvDF.collect().foreach(println(_))
roundtripCsvDF.printSchema()
// Roundtrip in reading and writing JSON data
data.write.format("json").save(s"$path/json")
val roundtripJsonDF =
spark.read.schema(data.schema).format("json").load(s"$path/json")
roundtripJsonDF.collect().foreach(println(_))
roundtripJsonDF.printSchema()
```
prints below:
```
[1970-01-01,1970-01-01 11:46:40.0]
root
|-- date: date (nullable = true)
|-- timestamp: timestamp (nullable = true)
[1970-01-01,1970-01-01 11:46:40.0]
root
|-- date: date (nullable = true)
|-- timestamp: timestamp (nullable = true)
```
- **`FastDateFormat` is thread-safe and compatible with
`SimpleDateFormat`** (See
https://github.com/apache/commons-lang/blob/LANG_3_2/src/main/java/org/apache/commons/lang3/time/FastDateFormat.java#L30-L37).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
