[ 
https://issues.apache.org/jira/browse/SPARK-24483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shaul Lahav updated SPARK-24483:
--------------------------------
    Description: 
I run a spark job on an EMR cluster when using the SparkSession.Builder to 
create a spark Session.
 The job includes querying a Hive table, and when it is executed Spark throws "

org.apache.spark.sql.AnalysisException: Table or view not found".

I printed all the config options in Spark context and noticed that 
"spark.sql.catalogImplementation" is missing.

If I provide this option as a "-conf" in "spark-submit" then it is added and 
everything works.

This is the code I use to instantiate the SparkSession: 
 val spark: SparkSession = SparkSession

.builder
 .config("mapreduce.fileoutputcommitter.algorithm.version", "2")
 .enableHiveSupport()
 .getOrCreate()

Also note that I added the following two debugging statements just before the 
code that creates the session to see if there was already a session in place, 
and they both returned "false":

SparkSession.getActiveSession.isDefined

SparkSession.getDefaultSession.isDefined

 

  was:
I run a spark job on an EMR cluster when using the SparkSession.Builder to 
create a spark Session.
The job includes querying a Hive table, and when it is executed Spark throws "

org.apache.spark.sql.AnalysisException: Table or view not found".

I printed all the config options in Spark context and noticed that 
"spark.sql.catalogImplementation" is missing.

If I provide this option as a "-conf" in "spark-submit" then it is added and 
everything works.

This is the code I use to instantiate the SparkSession: 
val spark: SparkSession = SparkSession

.builder
 .config("mapreduce.fileoutputcommitter.algorithm.version", "2")
 .enableHiveSupport()
 .getOrCreate()

 


> enableHiveSupport doesn't work with Spark 2.3 on EMR
> ----------------------------------------------------
>
>                 Key: SPARK-24483
>                 URL: https://issues.apache.org/jira/browse/SPARK-24483
>             Project: Spark
>          Issue Type: Bug
>          Components: Project Infra
>    Affects Versions: 2.3.0
>         Environment: EMR v5.13 (Spark 2.3.0)
>            Reporter: Shaul Lahav
>            Priority: Major
>
> I run a spark job on an EMR cluster when using the SparkSession.Builder to 
> create a spark Session.
>  The job includes querying a Hive table, and when it is executed Spark throws 
> "
> org.apache.spark.sql.AnalysisException: Table or view not found".
> I printed all the config options in Spark context and noticed that 
> "spark.sql.catalogImplementation" is missing.
> If I provide this option as a "-conf" in "spark-submit" then it is added and 
> everything works.
> This is the code I use to instantiate the SparkSession: 
>  val spark: SparkSession = SparkSession
> .builder
>  .config("mapreduce.fileoutputcommitter.algorithm.version", "2")
>  .enableHiveSupport()
>  .getOrCreate()
> Also note that I added the following two debugging statements just before the 
> code that creates the session to see if there was already a session in place, 
> and they both returned "false":
> SparkSession.getActiveSession.isDefined
> SparkSession.getDefaultSession.isDefined
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

Reply via email to