Abhishek Chennaka has uploaded this change for review. ( http://gerrit.cloudera.org:8080/23605
Change subject: [Java] Resolve ClassNotFoundException: kudu.DefaultSource ...................................................................... [Java] Resolve ClassNotFoundException: kudu.DefaultSource In kudu-spark-tools module, having the minimize() removes the needed class, org/apache/kudu/spark/kudu/DefaultSource.class. Along with that we exclude 'META-INF/services/**' in kudu-spark-tools and kudu-spark. But without META-INF/services/org.apache.spark.sql.sources.DataSourceRegister Spark cannot discover DefaultSource, even though the class itself is there. This commit addresses these issues and has been tested manually. ERROR yarn.Client: [main]: Application diagnostics message: User class threw exception: org.apache.spark.SparkClassNotFoundException: [DATA_SOURCE_NOT_FOUND] Failed to find the data source: kudu. Please find packages at `https://spark.apache.org/third-party-projects.html`. at org.apache.spark.sql.errors.QueryExecutionErrors$.dataSourceNotFoundError(QueryExecutionErrors.scala:725) at org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSource(DataSource.scala:649) at org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSourceV2(DataSource.scala:699) at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:208) at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:172) at org.apache.kudu.spark.tools.Verifier$.run(IntegrationTestBigLinkedList.scala:371) at org.apache.kudu.spark.tools.Verifier$.main(IntegrationTestBigLinkedList.scala:451) at org.apache.kudu.spark.tools.IntegrationTestBigLinkedList$.main(IntegrationTestBigLinkedList.scala:107) at org.apache.kudu.spark.tools.IntegrationTestBigLinkedList.main(IntegrationTestBigLinkedList.scala) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:568) at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:748) Caused by: java.lang.ClassNotFoundException: kudu.DefaultSource at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:445) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:592) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:525) at org.apache.spark.sql.execution.datasources.DataSource$.$anonfun$lookupDataSource$5(DataSource.scala:635) at scala.util.Try$.apply(Try.scala:213) at org.apache.spark.sql.execution.datasources.DataSource$.$anonfun$lookupDataSource$4(DataSource.scala:635) at scala.util.Failure.orElse(Try.scala:224) at org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSource(DataSource.scala:635) ... 12 more Change-Id: Ic0b8f07ea46759dc92d5ed2105a5480a0cf56464 --- M java/kudu-spark-tools/build.gradle M java/kudu-spark/build.gradle 2 files changed, 1 insertion(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/05/23605/1 -- To view, visit http://gerrit.cloudera.org:8080/23605 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ic0b8f07ea46759dc92d5ed2105a5480a0cf56464 Gerrit-Change-Number: 23605 Gerrit-PatchSet: 1 Gerrit-Owner: Abhishek Chennaka <[email protected]>
