[ https://issues.apache.org/jira/browse/SPARK-24279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16504246#comment-16504246 ]
Prashant Sharma commented on SPARK-24279: ----------------------------------------- Thanks a lot, that was the mistake. > Incompatible byte code errors, when using test-jar of spark sql. > ---------------------------------------------------------------- > > Key: SPARK-24279 > URL: https://issues.apache.org/jira/browse/SPARK-24279 > Project: Spark > Issue Type: Bug > Components: SQL, Structured Streaming, Tests > Affects Versions: 2.3.0, 2.4.0 > Reporter: Prashant Sharma > Priority: Major > > Using test libraries available with spark sql streaming, produces weird > incompatible byte code errors. It is already tested on a different virtual > box instance to make sure, that it is not related to my system environment. A > reproducer is uploaded to github. > [https://github.com/ScrapCodes/spark-bug-reproducer] > > Doing a clean build reproduces the error. > > Verbatim paste of the error. > {code:java} > [INFO] Compiling 1 source files to > /home/prashant/work/test/target/test-classes at 1526380360990 > [ERROR] error: missing or invalid dependency detected while loading class > file 'QueryTest.class'. > [INFO] Could not access type PlanTest in package > org.apache.spark.sql.catalyst.plans, > [INFO] because it (or its dependencies) are missing. Check your build > definition for > [INFO] missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to > see the problematic classpath.) > [INFO] A full rebuild may help if 'QueryTest.class' was compiled against an > incompatible version of org.apache.spark.sql.catalyst.plans. > [ERROR] error: missing or invalid dependency detected while loading class > file 'SQLTestUtilsBase.class'. > [INFO] Could not access type PlanTestBase in package > org.apache.spark.sql.catalyst.plans, > [INFO] because it (or its dependencies) are missing. Check your build > definition for > [INFO] missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to > see the problematic classpath.) > [INFO] A full rebuild may help if 'SQLTestUtilsBase.class' was compiled > against an incompatible version of org.apache.spark.sql.catalyst.plans. > [ERROR] error: missing or invalid dependency detected while loading class > file 'SQLTestUtils.class'. > [INFO] Could not access type PlanTest in package > org.apache.spark.sql.catalyst.plans, > [INFO] because it (or its dependencies) are missing. Check your build > definition for > [INFO] missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to > see the problematic classpath.) > [INFO] A full rebuild may help if 'SQLTestUtils.class' was compiled against > an incompatible version of org.apache.spark.sql.catalyst.plans. > [ERROR] /home/prashant/work/test/src/test/scala/SparkStreamingTests.scala:25: > error: Unable to find encoder for type stored in a Dataset. Primitive types > (Int, String, etc) and Product types (case classes) are supported by > importing spark.implicits._ Support for serializing other types will be added > in future releases. > [ERROR] val inputData = MemoryStream[Int] > [ERROR] ^ > [ERROR] /home/prashant/work/test/src/test/scala/SparkStreamingTests.scala:30: > error: Unable to find encoder for type stored in a Dataset. Primitive types > (Int, String, etc) and Product types (case classes) are supported by > importing spark.implicits._ Support for serializing other types will be added > in future releases. > [ERROR] CheckAnswer(2, 3, 4)) > [ERROR] ^ > [ERROR] 5 errors found > {code} > -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org