Hello Grant Henke,
I'd like you to do a code review. Please visit
http://gerrit.cloudera.org:8080/17213
to review the following change.
Change subject: [spark] revert bump to 3.1.1
......................................................................
[spark] revert bump to 3.1.1
TestKuduBackup.testRandomBackupAndRestore started failing with errors
like the following:
02:04:37.919 [ERROR - Executor task launch worker for task 0.0 in stage 0.0
(TID 0)] (Logging.scala:94) Aborting task
org.apache.spark.SparkUpgradeException: You may get a different result due to
the upgrading of Spark 3.0: writing dates before 1582-10-15 or timestamps
before 1900-01-01T00:00:00Z into Parquet INT96 files can be dangerous, as the
files may be read by Spark 2.x or legacy versions of Hive later, which uses a
legacy hybrid calendar that is different from Spark 3.0+'s Proleptic Gregorian
calendar. See more details in SPARK-31404. You can set
spark.sql.legacy.parquet.int96RebaseModeInWrite to 'LEGACY' to rebase the
datetime values w.r.t. the calendar difference during writing, to get maximum
interoperability. Or set spark.sql.legacy.parquet.int96RebaseModeInWrite to
'CORRECTED' to write the datetime values as it is, if you are 100% sure that
the written files will only be read by Spark 3.0+ or other systems that use
Proleptic Gregorian calendar.
at
org.apache.spark.sql.execution.datasources.DataSourceUtils$.newRebaseExceptionInWrite(DataSourceUtils.scala:165)
~[spark-sql_2.12-3.1.1.jar:3.1.1]
...
The error suggests some workarounds that are worth exploring further. In
the meantime, let's just revert the version bump, since it seems like
the bump will necessitate further discussion on how best to handle the
difference in behavior in the common case.
Change-Id: Ib9ca4d9e69785dd9d056fa8e62c944d56cf219ed
---
M java/gradle/dependencies.gradle
1 file changed, 1 insertion(+), 1 deletion(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/13/17213/1
--
To view, visit http://gerrit.cloudera.org:8080/17213
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ib9ca4d9e69785dd9d056fa8e62c944d56cf219ed
Gerrit-Change-Number: 17213
Gerrit-PatchSet: 1
Gerrit-Owner: Andrew Wong <[email protected]>
Gerrit-Reviewer: Grant Henke <[email protected]>
