[
https://issues.apache.org/jira/browse/SPARK-23776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16411854#comment-16411854
]
Bruce Robbins commented on SPARK-23776:
---------------------------------------
As it turns out, the building-spark page does have maven instructions to build
with hive support before running pyspark tests. However, it does not include
instructions if you are building with sbt, which needs more than just building
with hive support (you need to also compile the test classes).
> pyspark-sql tests should display build instructions when components are
> missing
> -------------------------------------------------------------------------------
>
> Key: SPARK-23776
> URL: https://issues.apache.org/jira/browse/SPARK-23776
> Project: Spark
> Issue Type: Improvement
> Components: PySpark
> Affects Versions: 2.3.0
> Reporter: Bruce Robbins
> Priority: Minor
>
> This is a follow up to SPARK-23417.
> The pyspark-streaming tests print useful build instructions when certain
> components are missing in the build.
> pyspark-sql's udf and readwrite tests also have specific build requirements:
> the build must compile test scala files, and the build must also create the
> Hive assembly. When those class or jar files are not created, the tests throw
> only partially helpful exceptions, e.g.:
> {noformat}
> AnalysisException: u'Can not load class
> test.org.apache.spark.sql.JavaStringLength, please make sure it is on the
> classpath;'
> {noformat}
> or
> {noformat}
> IllegalArgumentException: u"Error while instantiating
> 'org.apache.spark.sql.hive.HiveExternalCatalog':"
> {noformat}
> You end up in this situation when you follow Spark's build instructions and
> then attempt to run the pyspark tests.
> It would be nice if pyspark-sql tests provide helpful build instructions in
> these cases.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
