Re: Can we execute create table and load data commands against Hive inside HiveContext?
org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdConfOnlyAuthorizerFactory
was introduced in Hive 0.14 and Spark SQL only supports Hive 0.12 and
0.13.1. Can you change the setting of hive.security.authorization.manager
to someone accepted by 0.12 or 0.13.1?
On Thu, Feb 5, 2015 at 11:40 PM, guxiaobo1982 guxiaobo1...@qq.com wrote:
Hi,
I am playing with the following example code:
public class SparkTest {
public static void main(String[] args){
String appName= This is a test application;
String master=spark://lix1.bh.com:7077;
SparkConf conf = new SparkConf().setAppName(appName).setMaster(master);
JavaSparkContext sc = new JavaSparkContext(conf);
JavaHiveContext sqlCtx = new
org.apache.spark.sql.hive.api.java.JavaHiveContext(sc);
//sqlCtx.sql(CREATE TABLE IF NOT EXISTS src (key INT, value STRING));
//sqlCtx.sql(LOAD DATA LOCAL INPATH
'/opt/spark/examples/src/main/resources/kv1.txt' INTO TABLE src);
// Queries are expressed in HiveQL.
ListRow rows = sqlCtx.sql(FROM src SELECT key, value).collect();
//ListRow rows = sqlCtx.sql(show tables).collect();
System.out.print(I got + rows.size() + rows \r\n);
sc.close();
}}
With the create table and load data commands commented out, the query
command can be executed successfully, but I come to ClassNotFoundExceptions
if these two commands are executed inside HiveContext, even with different
error messages,
The create table command will cause the following:
Exception in thread main
org.apache.spark.sql.execution.QueryExecutionException: FAILED: Hive
Internal Error:
java.lang.ClassNotFoundException(org.apache.hadoop.hive.ql.hooks.ATSHook)
at org.apache.spark.sql.hive.HiveContext.runHive(HiveContext.scala:309)
at org.apache.spark.sql.hive.HiveContext.runSqlHive(HiveContext.scala:276)
at
org.apache.spark.sql.hive.execution.NativeCommand.sideEffectResult$lzycompute(
NativeCommand.scala:35)
at org.apache.spark.sql.hive.execution.NativeCommand.sideEffectResult(
NativeCommand.scala:35)
at org.apache.spark.sql.execution.Command$class.execute(commands.scala:46)
at org.apache.spark.sql.hive.execution.NativeCommand.execute(
NativeCommand.scala:30)
at org.apache.spark.sql.SQLContext$QueryExecution.toRdd$lzycompute(
SQLContext.scala:425)
at org.apache.spark.sql.SQLContext$QueryExecution.toRdd(
SQLContext.scala:425)
at org.apache.spark.sql.SchemaRDDLike$class.$init$(SchemaRDDLike.scala:58)
at org.apache.spark.sql.api.java.JavaSchemaRDD.init(
JavaSchemaRDD.scala:42)
at org.apache.spark.sql.hive.api.java.JavaHiveContext.sql(
JavaHiveContext.scala:37)
at com.blackhorse.SparkTest.main(SparkTest.java:24)
[delete Spark temp dirs] DEBUG org.apache.spark.util.Utils - Shutdown hook
called
[delete Spark local dirs] DEBUG org.apache.spark.storage.DiskBlockManager
- Shutdown hook called
The load data command will cause the following:
Exception in thread main
org.apache.spark.sql.execution.QueryExecutionException: FAILED:
RuntimeException org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.ClassNotFoundException:
org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdConfOnlyAuthorizerFactory
at org.apache.spark.sql.hive.HiveContext.runHive(HiveContext.scala:309)
at org.apache.spark.sql.hive.HiveContext.runSqlHive(HiveContext.scala:276)
at
org.apache.spark.sql.hive.execution.NativeCommand.sideEffectResult$lzycompute(NativeCommand.scala:35)
at
org.apache.spark.sql.hive.execution.NativeCommand.sideEffectResult(NativeCommand.scala:35)
at org.apache.spark.sql.execution.Command$class.execute(commands.scala:46)
at
org.apache.spark.sql.hive.execution.NativeCommand.execute(NativeCommand.scala:30)
at
org.apache.spark.sql.SQLContext$QueryExecution.toRdd$lzycompute(SQLContext.scala:425)
at
org.apache.spark.sql.SQLContext$QueryExecution.toRdd(SQLContext.scala:425)
at org.apache.spark.sql.SchemaRDDLike$class.$init$(SchemaRDDLike.scala:58)
at
org.apache.spark.sql.api.java.JavaSchemaRDD.init(JavaSchemaRDD.scala:42)
at
org.apache.spark.sql.hive.api.java.JavaHiveContext.sql(JavaHiveContext.scala:37)
at com.blackhorse.SparkTest.main(SparkTest.java:25)
[delete Spark local dirs] DEBUG org.apache.spark.storage.DiskBlockManager
- Shutdown hook called
[delete Spark temp dirs] DEBUG org.apache.spark.util.Utils - Shutdown hook
called
Can we execute create table and load data commands against Hive inside HiveContext?
Hi,
I am playing with the following example code:
public class SparkTest {
public static void main(String[] args){
String appName= This is a test application;
String master=spark://lix1.bh.com:7077;
SparkConf conf = new
SparkConf().setAppName(appName).setMaster(master);
JavaSparkContext sc = new JavaSparkContext(conf);
JavaHiveContext sqlCtx = new
org.apache.spark.sql.hive.api.java.JavaHiveContext(sc);
//sqlCtx.sql(CREATE TABLE IF NOT EXISTS src (key INT, value
STRING));
//sqlCtx.sql(LOAD DATA LOCAL INPATH
'/opt/spark/examples/src/main/resources/kv1.txt' INTO TABLE src);
// Queries are expressed in HiveQL.
ListRow rows = sqlCtx.sql(FROM src SELECT key,
value).collect();
//ListRow rows = sqlCtx.sql(show tables).collect();
System.out.print(I got + rows.size() + rows \r\n);
sc.close();
}}
With the create table and load data commands commented out, the query command
can be executed successfully, but I come to ClassNotFoundExceptions if these
two commands are executed inside HiveContext, even with different error
messages,
The create table command will cause the following:
Exception in thread main
org.apache.spark.sql.execution.QueryExecutionException: FAILED: Hive Internal
Error: java.lang.ClassNotFoundException(org.apache.hadoop.hive.ql.hooks.ATSHook)
at org.apache.spark.sql.hive.HiveContext.runHive(HiveContext.scala:309)
at
org.apache.spark.sql.hive.HiveContext.runSqlHive(HiveContext.scala:276)
at
org.apache.spark.sql.hive.execution.NativeCommand.sideEffectResult$lzycompute(NativeCommand.scala:35)
at
org.apache.spark.sql.hive.execution.NativeCommand.sideEffectResult(NativeCommand.scala:35)
at
org.apache.spark.sql.execution.Command$class.execute(commands.scala:46)
at
org.apache.spark.sql.hive.execution.NativeCommand.execute(NativeCommand.scala:30)
at
org.apache.spark.sql.SQLContext$QueryExecution.toRdd$lzycompute(SQLContext.scala:425)
at
org.apache.spark.sql.SQLContext$QueryExecution.toRdd(SQLContext.scala:425)
at
org.apache.spark.sql.SchemaRDDLike$class.$init$(SchemaRDDLike.scala:58)
at
org.apache.spark.sql.api.java.JavaSchemaRDD.init(JavaSchemaRDD.scala:42)
at
org.apache.spark.sql.hive.api.java.JavaHiveContext.sql(JavaHiveContext.scala:37)
at com.blackhorse.SparkTest.main(SparkTest.java:24)
[delete Spark temp dirs] DEBUG org.apache.spark.util.Utils - Shutdown hook
called
[delete Spark local dirs] DEBUG org.apache.spark.storage.DiskBlockManager -
Shutdown hook called
The load data command will cause the following:
Exception in thread main
org.apache.spark.sql.execution.QueryExecutionException: FAILED:
RuntimeException org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.ClassNotFoundException:
org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdConfOnlyAuthorizerFactory
at org.apache.spark.sql.hive.HiveContext.runHive(HiveContext.scala:309)
at org.apache.spark.sql.hive.HiveContext.runSqlHive(HiveContext.scala:276)
at
org.apache.spark.sql.hive.execution.NativeCommand.sideEffectResult$lzycompute(NativeCommand.scala:35)
at
org.apache.spark.sql.hive.execution.NativeCommand.sideEffectResult(NativeCommand.scala:35)
at org.apache.spark.sql.execution.Command$class.execute(commands.scala:46)
at
org.apache.spark.sql.hive.execution.NativeCommand.execute(NativeCommand.scala:30)
at
org.apache.spark.sql.SQLContext$QueryExecution.toRdd$lzycompute(SQLContext.scala:425)
at org.apache.spark.sql.SQLContext$QueryExecution.toRdd(SQLContext.scala:425)
at org.apache.spark.sql.SchemaRDDLike$class.$init$(SchemaRDDLike.scala:58)
at org.apache.spark.sql.api.java.JavaSchemaRDD.init(JavaSchemaRDD.scala:42)
at
org.apache.spark.sql.hive.api.java.JavaHiveContext.sql(JavaHiveContext.scala:37)
at com.blackhorse.SparkTest.main(SparkTest.java:25)
[delete Spark local dirs] DEBUG org.apache.spark.storage.DiskBlockManager -
Shutdown hook called
[delete Spark temp dirs] DEBUG org.apache.spark.util.Utils - Shutdown hook
called
