Re: Does HiveContext connect to HiveServer2?
Hi Marcelo, The issue does not happen while connecting to the hive metstore, that works fine. It seems that HiveContext only uses Hive CLI to execute the queries while HiveServer2 does not support it. I dont think you can specify any configuration in hive-site.xml which can make it connect to HiveServer2. It becomes a blocking issue in case of Sentry where HiveServer2 does the translation of authenticated users to "hive" user (which is the only user that can access directories under hive/warehouse when Sentry is ON). The HiveContext is able to access the metastore and then tries to access the files under hive warehouse directory where i fails with permission error: *org.apache.hadoop.security.AccessControlException: Permission denied: user=kakn, access=READ_EXECUTE, inode="/user/hive/warehouse/rt_freewheel_mastering.db/digital_profile_cluster_in":hive:hive:drwxrwx--t* On Tue, Mar 24, 2015 at 1:43 PM, Marcelo Vanzin wrote: > spark-submit --files /path/to/hive-site.xml > > On Tue, Mar 24, 2015 at 10:31 AM, Udit Mehta wrote: > > Another question related to this, how can we propagate the hive-site.xml > to > > all workers when running in the yarn cluster mode? > > > > On Tue, Mar 24, 2015 at 10:09 AM, Marcelo Vanzin > > wrote: > >> > >> It does neither. If you provide a Hive configuration to Spark, > >> HiveContext will connect to your metastore server, otherwise it will > >> create its own metastore in the working directory (IIRC). > >> > >> On Tue, Mar 24, 2015 at 8:58 AM, nitinkak001 > >> wrote: > >> > I am wondering if HiveContext connects to HiveServer2 or does it work > >> > though > >> > Hive CLI. The reason I am asking is because Cloudera has deprecated > Hive > >> > CLI. > >> > > >> > If the connection is through HiverServer2, is there a way to specify > >> > user > >> > credentials? > >> > > >> > > >> > > >> > -- > >> > View this message in context: > >> > > http://apache-spark-user-list.1001560.n3.nabble.com/Does-HiveContext-connect-to-HiveServer2-tp22200.html > >> > Sent from the Apache Spark User List mailing list archive at > Nabble.com. > >> > > >> > - > >> > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > >> > For additional commands, e-mail: user-h...@spark.apache.org > >> > > >> > >> > >> > >> -- > >> Marcelo > >> > >> - > >> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > >> For additional commands, e-mail: user-h...@spark.apache.org > >> > > > > > > -- > Marcelo >
Re: Does HiveContext connect to HiveServer2?
Hey, I have exactly this question. Did you get an answer to it? -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Does-HiveContext-connect-to-HiveServer2-tp22200p23431.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: Does HiveContext connect to HiveServer2?
spark-submit --files /path/to/hive-site.xml On Tue, Mar 24, 2015 at 10:31 AM, Udit Mehta wrote: > Another question related to this, how can we propagate the hive-site.xml to > all workers when running in the yarn cluster mode? > > On Tue, Mar 24, 2015 at 10:09 AM, Marcelo Vanzin > wrote: >> >> It does neither. If you provide a Hive configuration to Spark, >> HiveContext will connect to your metastore server, otherwise it will >> create its own metastore in the working directory (IIRC). >> >> On Tue, Mar 24, 2015 at 8:58 AM, nitinkak001 >> wrote: >> > I am wondering if HiveContext connects to HiveServer2 or does it work >> > though >> > Hive CLI. The reason I am asking is because Cloudera has deprecated Hive >> > CLI. >> > >> > If the connection is through HiverServer2, is there a way to specify >> > user >> > credentials? >> > >> > >> > >> > -- >> > View this message in context: >> > http://apache-spark-user-list.1001560.n3.nabble.com/Does-HiveContext-connect-to-HiveServer2-tp22200.html >> > Sent from the Apache Spark User List mailing list archive at Nabble.com. >> > >> > - >> > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >> > For additional commands, e-mail: user-h...@spark.apache.org >> > >> >> >> >> -- >> Marcelo >> >> - >> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >> For additional commands, e-mail: user-h...@spark.apache.org >> > -- Marcelo - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: Does HiveContext connect to HiveServer2?
Another question related to this, how can we propagate the hive-site.xml to all workers when running in the yarn cluster mode? On Tue, Mar 24, 2015 at 10:09 AM, Marcelo Vanzin wrote: > It does neither. If you provide a Hive configuration to Spark, > HiveContext will connect to your metastore server, otherwise it will > create its own metastore in the working directory (IIRC). > > On Tue, Mar 24, 2015 at 8:58 AM, nitinkak001 > wrote: > > I am wondering if HiveContext connects to HiveServer2 or does it work > though > > Hive CLI. The reason I am asking is because Cloudera has deprecated Hive > > CLI. > > > > If the connection is through HiverServer2, is there a way to specify user > > credentials? > > > > > > > > -- > > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Does-HiveContext-connect-to-HiveServer2-tp22200.html > > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > > > - > > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > > For additional commands, e-mail: user-h...@spark.apache.org > > > > > > -- > Marcelo > > - > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > >
Re: Does HiveContext connect to HiveServer2?
It does neither. If you provide a Hive configuration to Spark, HiveContext will connect to your metastore server, otherwise it will create its own metastore in the working directory (IIRC). On Tue, Mar 24, 2015 at 8:58 AM, nitinkak001 wrote: > I am wondering if HiveContext connects to HiveServer2 or does it work though > Hive CLI. The reason I am asking is because Cloudera has deprecated Hive > CLI. > > If the connection is through HiverServer2, is there a way to specify user > credentials? > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Does-HiveContext-connect-to-HiveServer2-tp22200.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > - > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > -- Marcelo - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org