[
https://issues.apache.org/jira/browse/HUDI-3957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17532238#comment-17532238
]
sivabalan narayanan commented on HUDI-3957:
-------------------------------------------
currently, integ test bundle does not work for spark2 and scala12. Spark
session initialization (enableHive()) expects some hive classes which are
missing.
{code:java}
22/04/22 20:26:08 WARN testsuite.HoodieTestSuiteJob: Running spark job w/ app
id local-1650673568081
22/04/22 20:26:08 INFO fs.FSUtils: Resolving file /tmp/test.propertiesto be a
remote file.
Exception in thread "main" java.lang.IllegalArgumentException: Unable to
instantiate SparkSession with Hive support because Hive classes are not found.
at
org.apache.spark.sql.SparkSession$Builder.enableHiveSupport(SparkSession.scala:871)
at
org.apache.hudi.integ.testsuite.HoodieTestSuiteJob.<init>(HoodieTestSuiteJob.java:110)
at
org.apache.hudi.integ.testsuite.HoodieTestSuiteJob.main(HoodieTestSuiteJob.java:180)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at
org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:855)
at
org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
at
org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:930)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:939)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
22/04/22 20:26:08 INFO spark.SparkContext: Invoking stop() from shutdown hook
22/04/22 20:26:08 INFO server.AbstractConnector: Stopped
Spark@889d9e8{HTTP/1.1, (http/1.1)}{0.0.0.0:8090} {code}
> Evaluate Support for spark2 and scala12
> ----------------------------------------
>
> Key: HUDI-3957
> URL: https://issues.apache.org/jira/browse/HUDI-3957
> Project: Apache Hudi
> Issue Type: Test
> Components: tests-ci
> Reporter: sivabalan narayanan
> Assignee: sivabalan narayanan
> Priority: Minor
> Fix For: 0.12.0
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
>
> We may need to evaluate the need for supporting spark2 and scala 12 and
> deprecate if there is not much usage.
>
>
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
