Ran Mingxuan created SPARK-22560:
------------------------------------
Summary: Must create spark session directly to connect to hive
Key: SPARK-22560
URL: https://issues.apache.org/jira/browse/SPARK-22560
Project: Spark
Issue Type: Bug
Components: Java API, SQL
Affects Versions: 2.1.0
Reporter: Ran Mingxuan
I have built a spark job like below:
{code:java}
// wrong code
public void main(String[] args)
{
SparkConf sparkConf = new SparkConf().setAppName("testApp");
JavaSparkContext sc = new JavaSparkContext(sparkConf);
sc.hadoopConfiguration().setBoolean("mapreduce.fileoutputcommitter.marksuccessfuljobs",
false);
sc.hadoopConfiguration().setBoolean("parquet.enable.summary-metadata",
false);//
SparkSession spark =
SparkSession.builder().sparkContext(sc.sc()).enableHiveSupport().getOrCreate();
spark.sql("show databases").show();
}
{code}
and with this code spark job will not be able to find hive meta-store even it
can discover correct warehouse.
I have to use code like below to make things work:
{code:java}
// correct code
public String main(String[] args)
{
SparkConf sparkConf = new SparkConf().setAppName("nice_clean");
SparkSession spark =
SparkSession.builder().config(sparkConf).enableHiveSupport().getOrCreate();
SparkContext sparkContext = spark.sparkContext();
JavaSparkContext sc = JavaSparkContext.fromSparkContext(sparkContext);
sc.hadoopConfiguration().setBoolean("mapreduce.fileoutputcommitter.marksuccessfuljobs",
false);
sc.hadoopConfiguration().setBoolean("parquet.enable.summary-metadata",
false);
spark.sql("show databases").show();
}
{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
