Hi Sandy, We are also going to grep data from a security enabled (with kerberos) HDFS in our Spark application. Per you answer, we have to switch Spark on YARN to achieve this. We plan to deploy a different Hadoop cluster(with YARN) only to run Spark. Is it necessary to deploy YARN with security enabled? Or is it possible to access data within a security HDFS from no-security enabled Spark on YARN?
On Wed, Jul 9, 2014 at 4:19 AM, Sandy Ryza <sandy.r...@cloudera.com> wrote: > That's correct. Only Spark on YARN supports Kerberos. > > -Sandy > > > On Tue, Jul 8, 2014 at 12:04 PM, Marcelo Vanzin <van...@cloudera.com> > wrote: > >> Someone might be able to correct me if I'm wrong, but I don't believe >> standalone mode supports kerberos. You'd have to use Yarn for that. >> >> On Tue, Jul 8, 2014 at 1:40 AM, 许晓炜 <xuxiao...@qiyi.com> wrote: >> > Hi all, >> > >> > >> > >> > I encounter a strange issue when using spark 1.0 to access hdfs with >> > Kerberos >> > >> > I just have one spark test node for spark and HADOOP_CONF_DIR is set to >> the >> > location containing the hdfs configuration files(hdfs-site.xml and >> > core-site.xml) >> > >> > When I use spark-shell with local mode, the access to hdfs is >> successfully . >> > >> > However, If I use spark-shell which connects to the stand alone cluster >> (I >> > configured the spark as standalone cluster mode with only one node). >> > >> > The access to the hdfs fails with the following error: “Can't get Master >> > Kerberos principal for use as renewer” >> > >> > >> > >> > Anyone have any ideas on this ? >> > >> > Thanks a lot. >> > >> > >> > >> > Regards, >> > Xiaowei >> > >> > >> >> >> >> -- >> Marcelo >> > >
