asfgit closed pull request #23374: [SPARK-26178][SPARK-26243][SQL][FOLLOWUP]
Replacing SimpleDateFormat by DateTimeFormatter in comments
URL: https://github.com/apache/spark/pull/23374
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/python/pyspark/sql/readwriter.py b/python/pyspark/sql/readwriter.py
index 7b10512a43294..3da052391a95b 100644
--- a/python/pyspark/sql/readwriter.py
+++ b/python/pyspark/sql/readwriter.py
@@ -226,11 +226,12 @@ def json(self, path, schema=None,
primitivesAsString=None, prefersDecimal=None,
it uses the value specified in
``spark.sql.columnNameOfCorruptRecord``.
:param dateFormat: sets the string that indicates a date format.
Custom date formats
- follow the formats at
``java.text.SimpleDateFormat``. This
+ follow the formats at
``java.time.format.DateTimeFormatter``. This
applies to date type. If None is set, it uses the
default value, ``yyyy-MM-dd``.
- :param timestampFormat: sets the string that indicates a timestamp
format. Custom date
- formats follow the formats at
``java.text.SimpleDateFormat``.
+ :param timestampFormat: sets the string that indicates a timestamp
format.
+ Custom date formats follow the formats at
+ ``java.time.format.DateTimeFormatter``.
This applies to timestamp type. If None is
set, it uses the
default value,
``yyyy-MM-dd'T'HH:mm:ss.SSSXXX``.
:param multiLine: parse one record, which may span multiple lines, per
file. If None is
@@ -406,11 +407,12 @@ def csv(self, path, schema=None, sep=None, encoding=None,
quote=None, escape=Non
:param negativeInf: sets the string representation of a negative
infinity value. If None
is set, it uses the default value, ``Inf``.
:param dateFormat: sets the string that indicates a date format.
Custom date formats
- follow the formats at
``java.text.SimpleDateFormat``. This
+ follow the formats at
``java.time.format.DateTimeFormatter``. This
applies to date type. If None is set, it uses the
default value, ``yyyy-MM-dd``.
- :param timestampFormat: sets the string that indicates a timestamp
format. Custom date
- formats follow the formats at
``java.text.SimpleDateFormat``.
+ :param timestampFormat: sets the string that indicates a timestamp
format.
+ Custom date formats follow the formats at
+ ``java.time.format.DateTimeFormatter``.
This applies to timestamp type. If None is
set, it uses the
default value,
``yyyy-MM-dd'T'HH:mm:ss.SSSXXX``.
:param maxColumns: defines a hard limit of how many columns a record
can have. If None is
@@ -803,11 +805,12 @@ def json(self, path, mode=None, compression=None,
dateFormat=None, timestampForm
known case-insensitive shorten names (none, bzip2,
gzip, lz4,
snappy and deflate).
:param dateFormat: sets the string that indicates a date format.
Custom date formats
- follow the formats at
``java.text.SimpleDateFormat``. This
+ follow the formats at
``java.time.format.DateTimeFormatter``. This
applies to date type. If None is set, it uses the
default value, ``yyyy-MM-dd``.
- :param timestampFormat: sets the string that indicates a timestamp
format. Custom date
- formats follow the formats at
``java.text.SimpleDateFormat``.
+ :param timestampFormat: sets the string that indicates a timestamp
format.
+ Custom date formats follow the formats at
+ ``java.time.format.DateTimeFormatter``.
This applies to timestamp type. If None is
set, it uses the
default value,
``yyyy-MM-dd'T'HH:mm:ss.SSSXXX``.
:param encoding: specifies encoding (charset) of saved json files. If
None is set,
@@ -904,11 +907,12 @@ def csv(self, path, mode=None, compression=None,
sep=None, quote=None, escape=No
:param nullValue: sets the string representation of a null value. If
None is set, it uses
the default value, empty string.
:param dateFormat: sets the string that indicates a date format.
Custom date formats
- follow the formats at
``java.text.SimpleDateFormat``. This
+ follow the formats at
``java.time.format.DateTimeFormatter``. This
applies to date type. If None is set, it uses the
default value, ``yyyy-MM-dd``.
- :param timestampFormat: sets the string that indicates a timestamp
format. Custom date
- formats follow the formats at
``java.text.SimpleDateFormat``.
+ :param timestampFormat: sets the string that indicates a timestamp
format.
+ Custom date formats follow the formats at
+ ``java.time.format.DateTimeFormatter``.
This applies to timestamp type. If None is
set, it uses the
default value,
``yyyy-MM-dd'T'HH:mm:ss.SSSXXX``.
:param ignoreLeadingWhiteSpace: a flag indicating whether or not
leading whitespaces from
diff --git a/python/pyspark/sql/streaming.py b/python/pyspark/sql/streaming.py
index fc23b9d99c34a..b981fdc4edc77 100644
--- a/python/pyspark/sql/streaming.py
+++ b/python/pyspark/sql/streaming.py
@@ -456,11 +456,12 @@ def json(self, path, schema=None,
primitivesAsString=None, prefersDecimal=None,
it uses the value specified in
``spark.sql.columnNameOfCorruptRecord``.
:param dateFormat: sets the string that indicates a date format.
Custom date formats
- follow the formats at
``java.text.SimpleDateFormat``. This
+ follow the formats at
``java.time.format.DateTimeFormatter``. This
applies to date type. If None is set, it uses the
default value, ``yyyy-MM-dd``.
- :param timestampFormat: sets the string that indicates a timestamp
format. Custom date
- formats follow the formats at
``java.text.SimpleDateFormat``.
+ :param timestampFormat: sets the string that indicates a timestamp
format.
+ Custom date formats follow the formats at
+ ``java.time.format.DateTimeFormatter``.
This applies to timestamp type. If None is
set, it uses the
default value,
``yyyy-MM-dd'T'HH:mm:ss.SSSXXX``.
:param multiLine: parse one record, which may span multiple lines, per
file. If None is
@@ -630,11 +631,12 @@ def csv(self, path, schema=None, sep=None, encoding=None,
quote=None, escape=Non
:param negativeInf: sets the string representation of a negative
infinity value. If None
is set, it uses the default value, ``Inf``.
:param dateFormat: sets the string that indicates a date format.
Custom date formats
- follow the formats at
``java.text.SimpleDateFormat``. This
+ follow the formats at
``java.time.format.DateTimeFormatter``. This
applies to date type. If None is set, it uses the
default value, ``yyyy-MM-dd``.
- :param timestampFormat: sets the string that indicates a timestamp
format. Custom date
- formats follow the formats at
``java.text.SimpleDateFormat``.
+ :param timestampFormat: sets the string that indicates a timestamp
format.
+ Custom date formats follow the formats at
+ ``java.time.format.DateTimeFormatter``.
This applies to timestamp type. If None is
set, it uses the
default value,
``yyyy-MM-dd'T'HH:mm:ss.SSSXXX``.
:param maxColumns: defines a hard limit of how many columns a record
can have. If None is
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala
b/sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala
index 9751528654ffb..ce8e4c8f5b82b 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala
@@ -375,11 +375,11 @@ class DataFrameReader private[sql](sparkSession:
SparkSession) extends Logging {
* `spark.sql.columnNameOfCorruptRecord`): allows renaming the new field
having malformed string
* created by `PERMISSIVE` mode. This overrides
`spark.sql.columnNameOfCorruptRecord`.</li>
* <li>`dateFormat` (default `yyyy-MM-dd`): sets the string that indicates a
date format.
- * Custom date formats follow the formats at `java.text.SimpleDateFormat`.
This applies to
- * date type.</li>
+ * Custom date formats follow the formats at
`java.time.format.DateTimeFormatter`.
+ * This applies to date type.</li>
* <li>`timestampFormat` (default `yyyy-MM-dd'T'HH:mm:ss.SSSXXX`): sets the
string that
* indicates a timestamp format. Custom date formats follow the formats at
- * `java.text.SimpleDateFormat`. This applies to timestamp type.</li>
+ * `java.time.format.DateTimeFormatter`. This applies to timestamp type.</li>
* <li>`multiLine` (default `false`): parse one record, which may span
multiple lines,
* per file</li>
* <li>`encoding` (by default it is not set): allows to forcibly set one of
standard basic
@@ -585,11 +585,11 @@ class DataFrameReader private[sql](sparkSession:
SparkSession) extends Logging {
* <li>`negativeInf` (default `-Inf`): sets the string representation of a
negative infinity
* value.</li>
* <li>`dateFormat` (default `yyyy-MM-dd`): sets the string that indicates a
date format.
- * Custom date formats follow the formats at `java.text.SimpleDateFormat`.
This applies to
- * date type.</li>
+ * Custom date formats follow the formats at
`java.time.format.DateTimeFormatter`.
+ * This applies to date type.</li>
* <li>`timestampFormat` (default `yyyy-MM-dd'T'HH:mm:ss.SSSXXX`): sets the
string that
* indicates a timestamp format. Custom date formats follow the formats at
- * `java.text.SimpleDateFormat`. This applies to timestamp type.</li>
+ * `java.time.format.DateTimeFormatter`. This applies to timestamp type.</li>
* <li>`maxColumns` (default `20480`): defines a hard limit of how many
columns
* a record can have.</li>
* <li>`maxCharsPerColumn` (default `-1`): defines the maximum number of
characters allowed
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala
b/sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala
index b9c4076994e96..981b3a8fd4ac1 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala
@@ -530,11 +530,11 @@ final class DataFrameWriter[T] private[sql](ds:
Dataset[T]) {
* one of the known case-insensitive shorten names (`none`, `bzip2`, `gzip`,
`lz4`,
* `snappy` and `deflate`). </li>
* <li>`dateFormat` (default `yyyy-MM-dd`): sets the string that indicates a
date format.
- * Custom date formats follow the formats at `java.text.SimpleDateFormat`.
This applies to
- * date type.</li>
+ * Custom date formats follow the formats at
`java.time.format.DateTimeFormatter`.
+ * This applies to date type.</li>
* <li>`timestampFormat` (default `yyyy-MM-dd'T'HH:mm:ss.SSSXXX`): sets the
string that
* indicates a timestamp format. Custom date formats follow the formats at
- * `java.text.SimpleDateFormat`. This applies to timestamp type.</li>
+ * `java.time.format.DateTimeFormatter`. This applies to timestamp type.</li>
* <li>`encoding` (by default it is not set): specifies encoding (charset)
of saved json
* files. If it is not set, the UTF-8 charset will be used. </li>
* <li>`lineSep` (default `\n`): defines the line separator that should be
used for writing.</li>
@@ -649,11 +649,11 @@ final class DataFrameWriter[T] private[sql](ds:
Dataset[T]) {
* one of the known case-insensitive shorten names (`none`, `bzip2`, `gzip`,
`lz4`,
* `snappy` and `deflate`). </li>
* <li>`dateFormat` (default `yyyy-MM-dd`): sets the string that indicates a
date format.
- * Custom date formats follow the formats at `java.text.SimpleDateFormat`.
This applies to
- * date type.</li>
+ * Custom date formats follow the formats at
`java.time.format.DateTimeFormatter`.
+ * This applies to date type.</li>
* <li>`timestampFormat` (default `yyyy-MM-dd'T'HH:mm:ss.SSSXXX`): sets the
string that
* indicates a timestamp format. Custom date formats follow the formats at
- * `java.text.SimpleDateFormat`. This applies to timestamp type.</li>
+ * `java.time.format.DateTimeFormatter`. This applies to timestamp type.</li>
* <li>`ignoreLeadingWhiteSpace` (default `true`): a flag indicating whether
or not leading
* whitespaces from values being written should be skipped.</li>
* <li>`ignoreTrailingWhiteSpace` (default `true`): a flag indicating
defines whether or not
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/streaming/DataStreamReader.scala
b/sql/core/src/main/scala/org/apache/spark/sql/streaming/DataStreamReader.scala
index 914fa90ae7e14..98589da9552cb 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/streaming/DataStreamReader.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/streaming/DataStreamReader.scala
@@ -286,11 +286,11 @@ final class DataStreamReader private[sql](sparkSession:
SparkSession) extends Lo
* `spark.sql.columnNameOfCorruptRecord`): allows renaming the new field
having malformed string
* created by `PERMISSIVE` mode. This overrides
`spark.sql.columnNameOfCorruptRecord`.</li>
* <li>`dateFormat` (default `yyyy-MM-dd`): sets the string that indicates a
date format.
- * Custom date formats follow the formats at `java.text.SimpleDateFormat`.
This applies to
- * date type.</li>
+ * Custom date formats follow the formats at
`java.time.format.DateTimeFormatter`.
+ * This applies to date type.</li>
* <li>`timestampFormat` (default `yyyy-MM-dd'T'HH:mm:ss.SSSXXX`): sets the
string that
* indicates a timestamp format. Custom date formats follow the formats at
- * `java.text.SimpleDateFormat`. This applies to timestamp type.</li>
+ * `java.time.format.DateTimeFormatter`. This applies to timestamp type.</li>
* <li>`multiLine` (default `false`): parse one record, which may span
multiple lines,
* per file</li>
* <li>`lineSep` (default covers all `\r`, `\r\n` and `\n`): defines the
line separator
@@ -347,11 +347,11 @@ final class DataStreamReader private[sql](sparkSession:
SparkSession) extends Lo
* <li>`negativeInf` (default `-Inf`): sets the string representation of a
negative infinity
* value.</li>
* <li>`dateFormat` (default `yyyy-MM-dd`): sets the string that indicates a
date format.
- * Custom date formats follow the formats at `java.text.SimpleDateFormat`.
This applies to
- * date type.</li>
+ * Custom date formats follow the formats at
`java.time.format.DateTimeFormatter`.
+ * This applies to date type.</li>
* <li>`timestampFormat` (default `yyyy-MM-dd'T'HH:mm:ss.SSSXXX`): sets the
string that
* indicates a timestamp format. Custom date formats follow the formats at
- * `java.text.SimpleDateFormat`. This applies to timestamp type.</li>
+ * `java.time.format.DateTimeFormatter`. This applies to timestamp type.</li>
* <li>`maxColumns` (default `20480`): defines a hard limit of how many
columns
* a record can have.</li>
* <li>`maxCharsPerColumn` (default `-1`): defines the maximum number of
characters allowed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
