Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/13912#discussion_r71124186
--- Diff: python/pyspark/sql/readwriter.py ---
@@ -328,6 +328,10 @@ def csv(self, path, schema=None, sep=None,
encoding=None, quote=None, escape=Non
applies to both date type and timestamp type.
By default, it is None
which means trying to parse times and date by
``java.sql.Timestamp.valueOf()`` and
``java.sql.Date.valueOf()``.
+ :param timezone: defines the timezone to be used for both date
type and timestamp type.
+ If a timezone is specified in the data, this will
load them after
--- End diff --
Yes, it will use the timezone specified in the input. Since `Date` and
`Timestamp` do not keep timezone information, it calculates the differences
between specified timezone in the input and in the option but the standard is
the one specified in the option.
I meant to say, for example, if `timezone` is set to `GMT`, all the read
`Date` and `Timestamp` are in `GMT` timezone after calculating the differences.
So..
If the CSV data is as below:
```
27/08/2015 00:00 PDT
27/08/2015 01:00 PDT
27/08/2015 02:00 PDT
```
If this is read as below:
```scala
spark.read
.format("csv")
.option("timezone", "GTM")
.option("dateFormat", "dd/MM/yyyy HH:mm z")
.load("path")
```
it will become as below in dataframe (difference between `GMT` and `PDT` is
7 hours).
```
27/08/2015 07:00
27/08/2015 08:00
27/08/2015 09:00
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
