[
https://issues.apache.org/jira/browse/SPARK-24279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prashant Sharma updated SPARK-24279:
------------------------------------
Description:
Using test libraries available with spark sql streaming, produces weird
incompatible byte code errors. It is already tested on a different virtual box
instance to make sure, that it is not related to my system environment. A
reproducer is uploaded to github.
[https://github.com/ScrapCodes/spark-bug-reproducer]
Doing a clean build reproduces the error.
Verbatim paste of the error.
{code:java}
[INFO] Compiling 1 source files to /home/prashant/work/test/target/test-classes
at 1526380360990
[ERROR] error: missing or invalid dependency detected while loading class file
'QueryTest.class'.
[INFO] Could not access type PlanTest in package
org.apache.spark.sql.catalyst.plans,
[INFO] because it (or its dependencies) are missing. Check your build
definition for
[INFO] missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to
see the problematic classpath.)
[INFO] A full rebuild may help if 'QueryTest.class' was compiled against an
incompatible version of org.apache.spark.sql.catalyst.plans.
[ERROR] error: missing or invalid dependency detected while loading class file
'SQLTestUtilsBase.class'.
[INFO] Could not access type PlanTestBase in package
org.apache.spark.sql.catalyst.plans,
[INFO] because it (or its dependencies) are missing. Check your build
definition for
[INFO] missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to
see the problematic classpath.)
[INFO] A full rebuild may help if 'SQLTestUtilsBase.class' was compiled against
an incompatible version of org.apache.spark.sql.catalyst.plans.
[ERROR] error: missing or invalid dependency detected while loading class file
'SQLTestUtils.class'.
[INFO] Could not access type PlanTest in package
org.apache.spark.sql.catalyst.plans,
[INFO] because it (or its dependencies) are missing. Check your build
definition for
[INFO] missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to
see the problematic classpath.)
[INFO] A full rebuild may help if 'SQLTestUtils.class' was compiled against an
incompatible version of org.apache.spark.sql.catalyst.plans.
[ERROR] /home/prashant/work/test/src/test/scala/SparkStreamingTests.scala:25:
error: Unable to find encoder for type stored in a Dataset. Primitive types
(Int, String, etc) and Product types (case classes) are supported by importing
spark.implicits._ Support for serializing other types will be added in future
releases.
[ERROR] val inputData = MemoryStream[Int]
[ERROR] ^
[ERROR] /home/prashant/work/test/src/test/scala/SparkStreamingTests.scala:30:
error: Unable to find encoder for type stored in a Dataset. Primitive types
(Int, String, etc) and Product types (case classes) are supported by importing
spark.implicits._ Support for serializing other types will be added in future
releases.
[ERROR] CheckAnswer(2, 3, 4))
[ERROR] ^
[ERROR] 5 errors found
{code}
was:
Using test libraries available with spark sql streaming, produces weird
incompatible byte code errors. It is already tested on a different virtual box
instance to make sure, that it is not related to my system environment. A
reproducer is uploaded to github.
[https://github.com/ScrapCodes/spark-bug-reproducer]
Doing a clean build reproduces the error.
> Incompatible byte code errors, when using test-jar of spark sql.
> ----------------------------------------------------------------
>
> Key: SPARK-24279
> URL: https://issues.apache.org/jira/browse/SPARK-24279
> Project: Spark
> Issue Type: Bug
> Components: SQL, Structured Streaming
> Affects Versions: 2.3.0, 2.4.0
> Reporter: Prashant Sharma
> Priority: Major
>
> Using test libraries available with spark sql streaming, produces weird
> incompatible byte code errors. It is already tested on a different virtual
> box instance to make sure, that it is not related to my system environment. A
> reproducer is uploaded to github.
> [https://github.com/ScrapCodes/spark-bug-reproducer]
>
> Doing a clean build reproduces the error.
>
> Verbatim paste of the error.
> {code:java}
> [INFO] Compiling 1 source files to
> /home/prashant/work/test/target/test-classes at 1526380360990
> [ERROR] error: missing or invalid dependency detected while loading class
> file 'QueryTest.class'.
> [INFO] Could not access type PlanTest in package
> org.apache.spark.sql.catalyst.plans,
> [INFO] because it (or its dependencies) are missing. Check your build
> definition for
> [INFO] missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to
> see the problematic classpath.)
> [INFO] A full rebuild may help if 'QueryTest.class' was compiled against an
> incompatible version of org.apache.spark.sql.catalyst.plans.
> [ERROR] error: missing or invalid dependency detected while loading class
> file 'SQLTestUtilsBase.class'.
> [INFO] Could not access type PlanTestBase in package
> org.apache.spark.sql.catalyst.plans,
> [INFO] because it (or its dependencies) are missing. Check your build
> definition for
> [INFO] missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to
> see the problematic classpath.)
> [INFO] A full rebuild may help if 'SQLTestUtilsBase.class' was compiled
> against an incompatible version of org.apache.spark.sql.catalyst.plans.
> [ERROR] error: missing or invalid dependency detected while loading class
> file 'SQLTestUtils.class'.
> [INFO] Could not access type PlanTest in package
> org.apache.spark.sql.catalyst.plans,
> [INFO] because it (or its dependencies) are missing. Check your build
> definition for
> [INFO] missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to
> see the problematic classpath.)
> [INFO] A full rebuild may help if 'SQLTestUtils.class' was compiled against
> an incompatible version of org.apache.spark.sql.catalyst.plans.
> [ERROR] /home/prashant/work/test/src/test/scala/SparkStreamingTests.scala:25:
> error: Unable to find encoder for type stored in a Dataset. Primitive types
> (Int, String, etc) and Product types (case classes) are supported by
> importing spark.implicits._ Support for serializing other types will be added
> in future releases.
> [ERROR] val inputData = MemoryStream[Int]
> [ERROR] ^
> [ERROR] /home/prashant/work/test/src/test/scala/SparkStreamingTests.scala:30:
> error: Unable to find encoder for type stored in a Dataset. Primitive types
> (Int, String, etc) and Product types (case classes) are supported by
> importing spark.implicits._ Support for serializing other types will be added
> in future releases.
> [ERROR] CheckAnswer(2, 3, 4))
> [ERROR] ^
> [ERROR] 5 errors found
> {code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
