[
https://issues.apache.org/jira/browse/SPARK-34772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ulysses you updated SPARK-34772:
--------------------------------
Description:
We will use date formatter in `HiveShim` that convert `date` to `string`, if we
set `spark.sql.legacy.timeParserPolicy=LEGACY` the `RebaseDateTime` code will
be invoked. At that moment, if `RebaseDateTime` is initialzed the first time
then context class loader is `IsolatedClientLoader`. Such error msg is thrown:
{code:java}
java.lang.IllegalArgumentException: argument "src" is null
at
com.fasterxml.jackson.databind.ObjectMapper._assertNotNull(ObjectMapper.java:4413)
at
com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3157)
at
com.fasterxml.jackson.module.scala.ScalaObjectMapper.readValue(ScalaObjectMapper.scala:187)
at
com.fasterxml.jackson.module.scala.ScalaObjectMapper.readValue$(ScalaObjectMapper.scala:186)
at
org.apache.spark.sql.catalyst.util.RebaseDateTime$$anon$1.readValue(RebaseDateTime.scala:267)
at
org.apache.spark.sql.catalyst.util.RebaseDateTime$.loadRebaseRecords(RebaseDateTime.scala:269)
at
org.apache.spark.sql.catalyst.util.RebaseDateTime$.<init>(RebaseDateTime.scala:291)
at
org.apache.spark.sql.catalyst.util.RebaseDateTime$.<clinit>(RebaseDateTime.scala)
at
org.apache.spark.sql.catalyst.util.DateTimeUtils$.toJavaDate(DateTimeUtils.scala:109)
at
org.apache.spark.sql.catalyst.util.LegacyDateFormatter.format(DateFormatter.scala:95)
at
org.apache.spark.sql.catalyst.util.LegacyDateFormatter.format$(DateFormatter.scala:94)
at
org.apache.spark.sql.catalyst.util.LegacySimpleDateFormatter.format(DateFormatter.scala:138)
at
org.apache.spark.sql.hive.client.Shim_v0_13$ExtractableLiteral$1$.unapply(HiveShim.scala:661)
at org.apache.spark.sql.hive.client.Shim_v0_13.convert$1(HiveShim.scala:785)
at
org.apache.spark.sql.hive.client.Shim_v0_13.$anonfun$convertFilters$4(HiveShim.scala:826)
{code}
was:
We will use date formatter in `HiveShim` that convert `date` to `string`, if we
set `spark.sql.legacy.timeParserPolicy=LEGACY` the `RebaseDateTime` code will
be invoked. At that moment, in custom hive metastore version case, the context
class loader is `IsolatedClientLoader` then such error msg is thrown:
{code:java}
java.lang.IllegalArgumentException: argument "src" is null
at
com.fasterxml.jackson.databind.ObjectMapper._assertNotNull(ObjectMapper.java:4413)
at
com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3157)
at
com.fasterxml.jackson.module.scala.ScalaObjectMapper.readValue(ScalaObjectMapper.scala:187)
at
com.fasterxml.jackson.module.scala.ScalaObjectMapper.readValue$(ScalaObjectMapper.scala:186)
at
org.apache.spark.sql.catalyst.util.RebaseDateTime$$anon$1.readValue(RebaseDateTime.scala:267)
at
org.apache.spark.sql.catalyst.util.RebaseDateTime$.loadRebaseRecords(RebaseDateTime.scala:269)
at
org.apache.spark.sql.catalyst.util.RebaseDateTime$.<init>(RebaseDateTime.scala:291)
at
org.apache.spark.sql.catalyst.util.RebaseDateTime$.<clinit>(RebaseDateTime.scala)
at
org.apache.spark.sql.catalyst.util.DateTimeUtils$.toJavaDate(DateTimeUtils.scala:109)
at
org.apache.spark.sql.catalyst.util.LegacyDateFormatter.format(DateFormatter.scala:95)
at
org.apache.spark.sql.catalyst.util.LegacyDateFormatter.format$(DateFormatter.scala:94)
at
org.apache.spark.sql.catalyst.util.LegacySimpleDateFormatter.format(DateFormatter.scala:138)
at
org.apache.spark.sql.hive.client.Shim_v0_13$ExtractableLiteral$1$.unapply(HiveShim.scala:661)
at org.apache.spark.sql.hive.client.Shim_v0_13.convert$1(HiveShim.scala:785)
at
org.apache.spark.sql.hive.client.Shim_v0_13.$anonfun$convertFilters$4(HiveShim.scala:826)
{code}
> RebaseDateTime loadRebaseRecords should use Spark classloader instead of
> context
> --------------------------------------------------------------------------------
>
> Key: SPARK-34772
> URL: https://issues.apache.org/jira/browse/SPARK-34772
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.2.0, 3.1.1
> Reporter: ulysses you
> Priority: Minor
>
> We will use date formatter in `HiveShim` that convert `date` to `string`, if
> we set `spark.sql.legacy.timeParserPolicy=LEGACY` the `RebaseDateTime` code
> will be invoked. At that moment, if `RebaseDateTime` is initialzed the first
> time then context class loader is `IsolatedClientLoader`. Such error msg is
> thrown:
> {code:java}
> java.lang.IllegalArgumentException: argument "src" is null
> at
> com.fasterxml.jackson.databind.ObjectMapper._assertNotNull(ObjectMapper.java:4413)
> at
> com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3157)
> at
> com.fasterxml.jackson.module.scala.ScalaObjectMapper.readValue(ScalaObjectMapper.scala:187)
> at
> com.fasterxml.jackson.module.scala.ScalaObjectMapper.readValue$(ScalaObjectMapper.scala:186)
> at
> org.apache.spark.sql.catalyst.util.RebaseDateTime$$anon$1.readValue(RebaseDateTime.scala:267)
> at
> org.apache.spark.sql.catalyst.util.RebaseDateTime$.loadRebaseRecords(RebaseDateTime.scala:269)
> at
> org.apache.spark.sql.catalyst.util.RebaseDateTime$.<init>(RebaseDateTime.scala:291)
> at
> org.apache.spark.sql.catalyst.util.RebaseDateTime$.<clinit>(RebaseDateTime.scala)
> at
> org.apache.spark.sql.catalyst.util.DateTimeUtils$.toJavaDate(DateTimeUtils.scala:109)
> at
> org.apache.spark.sql.catalyst.util.LegacyDateFormatter.format(DateFormatter.scala:95)
> at
> org.apache.spark.sql.catalyst.util.LegacyDateFormatter.format$(DateFormatter.scala:94)
> at
> org.apache.spark.sql.catalyst.util.LegacySimpleDateFormatter.format(DateFormatter.scala:138)
> at
> org.apache.spark.sql.hive.client.Shim_v0_13$ExtractableLiteral$1$.unapply(HiveShim.scala:661)
> at org.apache.spark.sql.hive.client.Shim_v0_13.convert$1(HiveShim.scala:785)
> at
> org.apache.spark.sql.hive.client.Shim_v0_13.$anonfun$convertFilters$4(HiveShim.scala:826)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
