This is an automated email from the ASF dual-hosted git repository.
srowen pushed a commit to branch branch-2.4
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-2.4 by this push:
new 4d476ed [SPARK-29494][SQL] Fix for ArrayOutofBoundsException while
converting string to timestamp
4d476ed is described below
commit 4d476ed44a36eadb0b21b88d0f6420d52a80cc9d
Author: Rahul Mahadev <[email protected]>
AuthorDate: Fri Oct 18 16:45:25 2019 -0500
[SPARK-29494][SQL] Fix for ArrayOutofBoundsException while converting
string to timestamp
### What changes were proposed in this pull request?
* Adding an additional check in `stringToTimestamp` to handle cases where
the input has trailing ':'
* Added a test to make sure this works.
### Why are the changes needed?
In a couple of scenarios while converting from String to Timestamp
`DateTimeUtils.stringToTimestamp` throws an array out of bounds exception if
there is trailing ':'. The behavior of this method requires it to return
`None` in case the format of the string is incorrect.
### Does this PR introduce any user-facing change?
No
### How was this patch tested?
Added a test in the `DateTimeTestUtils` suite to test if my fix works.
Closes #26143 from rahulsmahadev/SPARK-29494.
Lead-authored-by: Rahul Mahadev <[email protected]>
Co-authored-by: Rahul Shivu Mahadev
<[email protected]>
Signed-off-by: Sean Owen <[email protected]>
(cherry picked from commit 4cfce3e5d03b0badb4e9685499be2ab0fca5747a)
Signed-off-by: Sean Owen <[email protected]>
---
.../scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala | 2 +-
.../org/apache/spark/sql/catalyst/util/DateTimeUtilsSuite.scala | 6 ++++++
2 files changed, 7 insertions(+), 1 deletion(-)
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala
index cc3fcb2..5deb83e 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala
@@ -378,7 +378,7 @@ object DateTimeUtils {
i += 1
}
} else {
- if (b == ':' || b == ' ') {
+ if (i < segments.length && (b == ':' || b == ' ')) {
segments(i) = currentSegmentValue
currentSegmentValue = 0
i += 1
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/DateTimeUtilsSuite.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/DateTimeUtilsSuite.scala
index abdb916..4496ec8 100644
---
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/DateTimeUtilsSuite.scala
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/DateTimeUtilsSuite.scala
@@ -581,6 +581,12 @@ class DateTimeUtilsSuite extends SparkFunSuite {
}
}
+ test("trailing characters while converting string to timestamp") {
+ val s = UTF8String.fromString("2019-10-31T10:59:23Z:::")
+ val time = DateTimeUtils.stringToTimestamp(s, defaultZoneId)
+ assert(time == None)
+ }
+
test("truncTimestamp") {
def testTrunc(
level: Int,
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
