Can you use the principal and keytab options in Spark submit? These should circumvent this issue.
On 18 Nov 2016 1:01 p.m., "Abel Fernández" <[email protected]> wrote: > Hello, > > We are having problems with the delegation of the token in a secure > cluster: Delegation Token can be issued only with kerberos or web > authentication > > We have a spark process which is generating the hfiles to be loaded into > hbase. To generate these hfiles, (we are using a back-ported version of the > latest hbase/spark code), we are using this method HBaseRDDFunctions. > hbaseBulkLoadThinRows. > > I think the problem is in the below piece of code. This function is > executed in every partition of the rdd, when the executors are trying to > execute the code, the executors do not have a valid kerberos credential and > cannot execute anything. > > private def hbaseForeachPartition[T](configBroadcast: > Broadcast[SerializableWritable[ > Configuration]], > it: Iterator[T], > f: (Iterator[T], Connection) => > Unit) = { > > val config = getConf(configBroadcast) > > applyCreds > // specify that this is a proxy user > val smartConn = HBaseConnectionCache.getConnection(config) > f(it, smartConn.connection) > smartConn.close() > } > > I have attached the spark-submit and the complete error log trace. Has > anyone faced this problem before? > > Thanks in advance. > > Regards, > Abel. > -- > Un saludo - Best Regards. > Abel >
