> On 30 Mar 2016, at 21:02, Sean Busbey <bus...@cloudera.com> wrote: > > On Wed, Mar 30, 2016 at 4:33 AM, Steve Loughran <ste...@hortonworks.com> > wrote: >> >>> On 29 Mar 2016, at 22:19, Michael Segel <msegel_had...@hotmail.com> wrote: >>> >>> Hi, >>> >>> So yeah, I know that Spark jobs running on a Hadoop cluster will inherit >>> its security from the underlying YARN job. >>> However… that’s not really saying much when you think about some use cases. >>> >>> Like using the thrift service … >>> >>> I’m wondering what else is new and what people have been thinking about how >>> to enhance spark’s security. >>> >> >> Been thinking a bit. >> >> One thing to look at is renewal of hbase and hive tokens on long-lived >> services, alongside hdfs >> >> > > I've been looking at this as well. The current work-around I'm using > is to use keytab logins on the executors, which is less than > desirable.
OK, let's work together on this ... the current spark renewal code assumes its only for HDFS (indeed, that the filesystem is HDFS and therefore the #of tokens > 0); there' s no fundamental reason why the code in YarnSparkHadoopUtils can't run in the AM too. > > Since the HBase project maintains Spark integration points, it'd be > great if there were just a hook for services to provide "here's how to > renew" to a common renewal service. > 1. Wittenauer is doing some work on a tool for doing this; I'm pushing for it to be a fairly generic API. Even if Spark has to use reflection to get at it, at least it would be consistent across services. See https://issues.apache.org/jira/browse/HADOOP-12563 2. The topic of HTTPS based acquisition/use of HDFS tokens has arisen elsewhere; needed for long-haul job submission when you don' t have a keytab to hand. This could be useful as it'd avoid actually needing hbase-*.jar on the classpath at submit time. > > > -- > busbey > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org > For additional commands, e-mail: dev-h...@spark.apache.org > >
