This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 1008ab0 [SPARK-26178][SPARK-26243][SQL][FOLLOWUP] Replacing
SimpleDateFormat by DateTimeFormatter in comments
1008ab0 is described below
commit 1008ab0801c192e8f261001eaaf58a6c9f6e747a
Author: Maxim Gekk <[email protected]>
AuthorDate: Mon Dec 24 10:47:47 2018 +0800
[SPARK-26178][SPARK-26243][SQL][FOLLOWUP] Replacing SimpleDateFormat by
DateTimeFormatter in comments
## What changes were proposed in this pull request?
The PRs #23150 and #23196 switched JSON and CSV datasources on new
formatter for dates/timestamps which is based on `DateTimeFormatter`. In this
PR, I replaced `SimpleDateFormat` by `DateTimeFormatter` to reflect the changes.
Closes #23374 from MaxGekk/java-time-docs.
Authored-by: Maxim Gekk <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
python/pyspark/sql/readwriter.py | 28 ++++++++++++----------
python/pyspark/sql/streaming.py | 14 ++++++-----
.../org/apache/spark/sql/DataFrameReader.scala | 12 +++++-----
.../org/apache/spark/sql/DataFrameWriter.scala | 12 +++++-----
.../spark/sql/streaming/DataStreamReader.scala | 12 +++++-----
5 files changed, 42 insertions(+), 36 deletions(-)
diff --git a/python/pyspark/sql/readwriter.py b/python/pyspark/sql/readwriter.py
index 7b10512..3da0523 100644
--- a/python/pyspark/sql/readwriter.py
+++ b/python/pyspark/sql/readwriter.py
@@ -226,11 +226,12 @@ class DataFrameReader(OptionUtils):
it uses the value specified in
``spark.sql.columnNameOfCorruptRecord``.
:param dateFormat: sets the string that indicates a date format.
Custom date formats
- follow the formats at
``java.text.SimpleDateFormat``. This
+ follow the formats at
``java.time.format.DateTimeFormatter``. This
applies to date type. If None is set, it uses the
default value, ``yyyy-MM-dd``.
- :param timestampFormat: sets the string that indicates a timestamp
format. Custom date
- formats follow the formats at
``java.text.SimpleDateFormat``.
+ :param timestampFormat: sets the string that indicates a timestamp
format.
+ Custom date formats follow the formats at
+ ``java.time.format.DateTimeFormatter``.
This applies to timestamp type. If None is
set, it uses the
default value,
``yyyy-MM-dd'T'HH:mm:ss.SSSXXX``.
:param multiLine: parse one record, which may span multiple lines, per
file. If None is
@@ -406,11 +407,12 @@ class DataFrameReader(OptionUtils):
:param negativeInf: sets the string representation of a negative
infinity value. If None
is set, it uses the default value, ``Inf``.
:param dateFormat: sets the string that indicates a date format.
Custom date formats
- follow the formats at
``java.text.SimpleDateFormat``. This
+ follow the formats at
``java.time.format.DateTimeFormatter``. This
applies to date type. If None is set, it uses the
default value, ``yyyy-MM-dd``.
- :param timestampFormat: sets the string that indicates a timestamp
format. Custom date
- formats follow the formats at
``java.text.SimpleDateFormat``.
+ :param timestampFormat: sets the string that indicates a timestamp
format.
+ Custom date formats follow the formats at
+ ``java.time.format.DateTimeFormatter``.
This applies to timestamp type. If None is
set, it uses the
default value,
``yyyy-MM-dd'T'HH:mm:ss.SSSXXX``.
:param maxColumns: defines a hard limit of how many columns a record
can have. If None is
@@ -803,11 +805,12 @@ class DataFrameWriter(OptionUtils):
known case-insensitive shorten names (none, bzip2,
gzip, lz4,
snappy and deflate).
:param dateFormat: sets the string that indicates a date format.
Custom date formats
- follow the formats at
``java.text.SimpleDateFormat``. This
+ follow the formats at
``java.time.format.DateTimeFormatter``. This
applies to date type. If None is set, it uses the
default value, ``yyyy-MM-dd``.
- :param timestampFormat: sets the string that indicates a timestamp
format. Custom date
- formats follow the formats at
``java.text.SimpleDateFormat``.
+ :param timestampFormat: sets the string that indicates a timestamp
format.
+ Custom date formats follow the formats at
+ ``java.time.format.DateTimeFormatter``.
This applies to timestamp type. If None is
set, it uses the
default value,
``yyyy-MM-dd'T'HH:mm:ss.SSSXXX``.
:param encoding: specifies encoding (charset) of saved json files. If
None is set,
@@ -904,11 +907,12 @@ class DataFrameWriter(OptionUtils):
:param nullValue: sets the string representation of a null value. If
None is set, it uses
the default value, empty string.
:param dateFormat: sets the string that indicates a date format.
Custom date formats
- follow the formats at
``java.text.SimpleDateFormat``. This
+ follow the formats at
``java.time.format.DateTimeFormatter``. This
applies to date type. If None is set, it uses the
default value, ``yyyy-MM-dd``.
- :param timestampFormat: sets the string that indicates a timestamp
format. Custom date
- formats follow the formats at
``java.text.SimpleDateFormat``.
+ :param timestampFormat: sets the string that indicates a timestamp
format.
+ Custom date formats follow the formats at
+ ``java.time.format.DateTimeFormatter``.
This applies to timestamp type. If None is
set, it uses the
default value,
``yyyy-MM-dd'T'HH:mm:ss.SSSXXX``.
:param ignoreLeadingWhiteSpace: a flag indicating whether or not
leading whitespaces from
diff --git a/python/pyspark/sql/streaming.py b/python/pyspark/sql/streaming.py
index fc23b9d..b981fdc 100644
--- a/python/pyspark/sql/streaming.py
+++ b/python/pyspark/sql/streaming.py
@@ -456,11 +456,12 @@ class DataStreamReader(OptionUtils):
it uses the value specified in
``spark.sql.columnNameOfCorruptRecord``.
:param dateFormat: sets the string that indicates a date format.
Custom date formats
- follow the formats at
``java.text.SimpleDateFormat``. This
+ follow the formats at
``java.time.format.DateTimeFormatter``. This
applies to date type. If None is set, it uses the
default value, ``yyyy-MM-dd``.
- :param timestampFormat: sets the string that indicates a timestamp
format. Custom date
- formats follow the formats at
``java.text.SimpleDateFormat``.
+ :param timestampFormat: sets the string that indicates a timestamp
format.
+ Custom date formats follow the formats at
+ ``java.time.format.DateTimeFormatter``.
This applies to timestamp type. If None is
set, it uses the
default value,
``yyyy-MM-dd'T'HH:mm:ss.SSSXXX``.
:param multiLine: parse one record, which may span multiple lines, per
file. If None is
@@ -630,11 +631,12 @@ class DataStreamReader(OptionUtils):
:param negativeInf: sets the string representation of a negative
infinity value. If None
is set, it uses the default value, ``Inf``.
:param dateFormat: sets the string that indicates a date format.
Custom date formats
- follow the formats at
``java.text.SimpleDateFormat``. This
+ follow the formats at
``java.time.format.DateTimeFormatter``. This
applies to date type. If None is set, it uses the
default value, ``yyyy-MM-dd``.
- :param timestampFormat: sets the string that indicates a timestamp
format. Custom date
- formats follow the formats at
``java.text.SimpleDateFormat``.
+ :param timestampFormat: sets the string that indicates a timestamp
format.
+ Custom date formats follow the formats at
+ ``java.time.format.DateTimeFormatter``.
This applies to timestamp type. If None is
set, it uses the
default value,
``yyyy-MM-dd'T'HH:mm:ss.SSSXXX``.
:param maxColumns: defines a hard limit of how many columns a record
can have. If None is
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala
b/sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala
index 9751528..ce8e4c8 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala
@@ -375,11 +375,11 @@ class DataFrameReader private[sql](sparkSession:
SparkSession) extends Logging {
* `spark.sql.columnNameOfCorruptRecord`): allows renaming the new field
having malformed string
* created by `PERMISSIVE` mode. This overrides
`spark.sql.columnNameOfCorruptRecord`.</li>
* <li>`dateFormat` (default `yyyy-MM-dd`): sets the string that indicates a
date format.
- * Custom date formats follow the formats at `java.text.SimpleDateFormat`.
This applies to
- * date type.</li>
+ * Custom date formats follow the formats at
`java.time.format.DateTimeFormatter`.
+ * This applies to date type.</li>
* <li>`timestampFormat` (default `yyyy-MM-dd'T'HH:mm:ss.SSSXXX`): sets the
string that
* indicates a timestamp format. Custom date formats follow the formats at
- * `java.text.SimpleDateFormat`. This applies to timestamp type.</li>
+ * `java.time.format.DateTimeFormatter`. This applies to timestamp type.</li>
* <li>`multiLine` (default `false`): parse one record, which may span
multiple lines,
* per file</li>
* <li>`encoding` (by default it is not set): allows to forcibly set one of
standard basic
@@ -585,11 +585,11 @@ class DataFrameReader private[sql](sparkSession:
SparkSession) extends Logging {
* <li>`negativeInf` (default `-Inf`): sets the string representation of a
negative infinity
* value.</li>
* <li>`dateFormat` (default `yyyy-MM-dd`): sets the string that indicates a
date format.
- * Custom date formats follow the formats at `java.text.SimpleDateFormat`.
This applies to
- * date type.</li>
+ * Custom date formats follow the formats at
`java.time.format.DateTimeFormatter`.
+ * This applies to date type.</li>
* <li>`timestampFormat` (default `yyyy-MM-dd'T'HH:mm:ss.SSSXXX`): sets the
string that
* indicates a timestamp format. Custom date formats follow the formats at
- * `java.text.SimpleDateFormat`. This applies to timestamp type.</li>
+ * `java.time.format.DateTimeFormatter`. This applies to timestamp type.</li>
* <li>`maxColumns` (default `20480`): defines a hard limit of how many
columns
* a record can have.</li>
* <li>`maxCharsPerColumn` (default `-1`): defines the maximum number of
characters allowed
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala
b/sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala
index b9c4076..981b3a8 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala
@@ -530,11 +530,11 @@ final class DataFrameWriter[T] private[sql](ds:
Dataset[T]) {
* one of the known case-insensitive shorten names (`none`, `bzip2`, `gzip`,
`lz4`,
* `snappy` and `deflate`). </li>
* <li>`dateFormat` (default `yyyy-MM-dd`): sets the string that indicates a
date format.
- * Custom date formats follow the formats at `java.text.SimpleDateFormat`.
This applies to
- * date type.</li>
+ * Custom date formats follow the formats at
`java.time.format.DateTimeFormatter`.
+ * This applies to date type.</li>
* <li>`timestampFormat` (default `yyyy-MM-dd'T'HH:mm:ss.SSSXXX`): sets the
string that
* indicates a timestamp format. Custom date formats follow the formats at
- * `java.text.SimpleDateFormat`. This applies to timestamp type.</li>
+ * `java.time.format.DateTimeFormatter`. This applies to timestamp type.</li>
* <li>`encoding` (by default it is not set): specifies encoding (charset)
of saved json
* files. If it is not set, the UTF-8 charset will be used. </li>
* <li>`lineSep` (default `\n`): defines the line separator that should be
used for writing.</li>
@@ -649,11 +649,11 @@ final class DataFrameWriter[T] private[sql](ds:
Dataset[T]) {
* one of the known case-insensitive shorten names (`none`, `bzip2`, `gzip`,
`lz4`,
* `snappy` and `deflate`). </li>
* <li>`dateFormat` (default `yyyy-MM-dd`): sets the string that indicates a
date format.
- * Custom date formats follow the formats at `java.text.SimpleDateFormat`.
This applies to
- * date type.</li>
+ * Custom date formats follow the formats at
`java.time.format.DateTimeFormatter`.
+ * This applies to date type.</li>
* <li>`timestampFormat` (default `yyyy-MM-dd'T'HH:mm:ss.SSSXXX`): sets the
string that
* indicates a timestamp format. Custom date formats follow the formats at
- * `java.text.SimpleDateFormat`. This applies to timestamp type.</li>
+ * `java.time.format.DateTimeFormatter`. This applies to timestamp type.</li>
* <li>`ignoreLeadingWhiteSpace` (default `true`): a flag indicating whether
or not leading
* whitespaces from values being written should be skipped.</li>
* <li>`ignoreTrailingWhiteSpace` (default `true`): a flag indicating
defines whether or not
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/streaming/DataStreamReader.scala
b/sql/core/src/main/scala/org/apache/spark/sql/streaming/DataStreamReader.scala
index 914fa90..98589da 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/streaming/DataStreamReader.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/streaming/DataStreamReader.scala
@@ -286,11 +286,11 @@ final class DataStreamReader private[sql](sparkSession:
SparkSession) extends Lo
* `spark.sql.columnNameOfCorruptRecord`): allows renaming the new field
having malformed string
* created by `PERMISSIVE` mode. This overrides
`spark.sql.columnNameOfCorruptRecord`.</li>
* <li>`dateFormat` (default `yyyy-MM-dd`): sets the string that indicates a
date format.
- * Custom date formats follow the formats at `java.text.SimpleDateFormat`.
This applies to
- * date type.</li>
+ * Custom date formats follow the formats at
`java.time.format.DateTimeFormatter`.
+ * This applies to date type.</li>
* <li>`timestampFormat` (default `yyyy-MM-dd'T'HH:mm:ss.SSSXXX`): sets the
string that
* indicates a timestamp format. Custom date formats follow the formats at
- * `java.text.SimpleDateFormat`. This applies to timestamp type.</li>
+ * `java.time.format.DateTimeFormatter`. This applies to timestamp type.</li>
* <li>`multiLine` (default `false`): parse one record, which may span
multiple lines,
* per file</li>
* <li>`lineSep` (default covers all `\r`, `\r\n` and `\n`): defines the
line separator
@@ -347,11 +347,11 @@ final class DataStreamReader private[sql](sparkSession:
SparkSession) extends Lo
* <li>`negativeInf` (default `-Inf`): sets the string representation of a
negative infinity
* value.</li>
* <li>`dateFormat` (default `yyyy-MM-dd`): sets the string that indicates a
date format.
- * Custom date formats follow the formats at `java.text.SimpleDateFormat`.
This applies to
- * date type.</li>
+ * Custom date formats follow the formats at
`java.time.format.DateTimeFormatter`.
+ * This applies to date type.</li>
* <li>`timestampFormat` (default `yyyy-MM-dd'T'HH:mm:ss.SSSXXX`): sets the
string that
* indicates a timestamp format. Custom date formats follow the formats at
- * `java.text.SimpleDateFormat`. This applies to timestamp type.</li>
+ * `java.time.format.DateTimeFormatter`. This applies to timestamp type.</li>
* <li>`maxColumns` (default `20480`): defines a hard limit of how many
columns
* a record can have.</li>
* <li>`maxCharsPerColumn` (default `-1`): defines the maximum number of
characters allowed
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
