Hi guys,
Trying to use the SparkSQL Thriftserver with hive metastore. It seems that hive meta impersonation works fine (when running Hive tasks). However spinning up SparkSQL thrift server, impersonation doesn't seem to work... What settings do I need to enable impersonation? I've copied the same config as in my hive-site. Here is my launch command for the spark thrift server; --hiveconf hive.server2.enable.impersonation=true --hiveconf hive.server2.enable.doAs=true --hiveconf hive.metastore.execute.setugi=true Here is my full run script: export HIVE_SERVER2_THRIFT_BIND_HOST=0.0.0.0 export HIVE_SERVER2_THRIFT_PORT=10000 export HIVE_CONF_DIR=/opt/mapr/hive/hive-0.13/conf/ export HIVE_HOME=/opt/mapr/hive/hive-0.13/ export HADOOP_HOME=/opt/mapr/hadoop/hadoop-2.5.1/ export HADOOP_CONF_DIR=/opt/mapr/hadoop/hadoop-2.5.1/etc/hadoop export EXECUTOR_MEMORY=30g export DRIVER_MEMORY=4g export EXECUTOR_CORES=15 export NUM_EXECUTORS=20 export KRYO_BUFFER=512 export SPARK_DRIVER_MAXRESULTSIZE=4096 export HIVE_METASTORE_URIS=thrift://localhost:9083 export HIVE_METASTORE_WAREHOUSE_DIR=/user/hive/warehouse export SPARK_DIST_CLASSPATH=/opt/mapr/lib/*:/opt/mapr/hadoop/hadoop-2.5.1/share/hadoop/yarn/*:/opt/mapr/hadoop/hadoop-2.5.1/share/hadoop/common/lib/*:/opt/mapr/hive/hive-current/lib/* export SPARK_LOG_DIR=/tmp/spark-log export SPARK_HISTORY_OPTS=-Dspark.history.fs.logDirectory=hdfs:///log/spark-events export SPARK_CONF_DIR=/apps/spark/global-conf/ export SPARK_HOME=/apps/spark/spark-1.3.1-bin-mapr4.0.2_yarn_j6_2.10 export SPARK_LIBRARY_PATH=/opt/mapr/lib/* export SPARK_JAVA_OPTS=-Djava.library.path=/opt/mapr/lib $SPARK_HOME/*sbin/start-thriftserver.sh* --master yarn-client --jars /opt/mapr/lib/libjpam.so --executor-memory $EXECUTOR_MEMORY --driver-memory $DRIVER_MEMORY --executor-cores $EXECUTOR_CORES --num-executors $NUM_EXECUTORS --conf spark.scheduler.mode=FAIR --conf spark.kryoserializer.buffer.mb=$KRYO_BUFFER --conf spark.serializer=org.apache.spark.serializer.KryoSerializer --conf spark.files.useFetchCache=false --conf spark.driver.maxResultSize=$SPARK_DRIVER_MAXRESULTSIZE --hiveconf hive.metastore.uris=$HIVE_METASTORE_URIS --hiveconf hive.metastore.warehouse.dir=$HIVE_METASTORE_WAREHOUSE_DIR --hiveconf hive.server2.enable.impersonation=true --hiveconf hive.server2.enable.doAs=true --hiveconf hive.metastore.execute.setugi=true Cheers, N
