Hi Everyone,
I am still trying to figure this one out. I am stuck with this error
"java.io.IOException:
Can't get Master Kerberos principal for use as renewer ". Below is my code.
Can any of you please provide any insights on this? Thanks for your time.
import java.io.{BufferedInputStream, File, FileInputStream}
import java.net.URI
import org.apache.hadoop.fs.FileSystem
import org.apache.hadoop.conf.Configuration
import org.apache.hadoop.fs.Path
import org.apache.hadoop.io.IOUtils
import org.apache.hadoop.security.UserGroupInformation
import org.apache.spark.deploy.SparkHadoopUtil
import org.apache.spark.{SparkConf, SparkContext}
object SparkHdfs {
def main(args: Array[String]): Unit = {
System.setProperty("java.security.krb5.conf", new
File("src\\main\\files\\krb5.conf").getAbsolutePath )
System.setProperty("sun.security.krb5.debug", "true")
val sparkConf = new SparkConf().setAppName("SparkHdfs").setMaster("local")
val sc = new SparkContext(sparkConf)
// Loading remote cluster configurations
sc.hadoopConfiguration.addResource(new
File("src\\main\\files\\core-site.xml").getAbsolutePath )
sc.hadoopConfiguration.addResource(new
File("src\\main\\files\\hdfs-site.xml").getAbsolutePath )
sc.hadoopConfiguration.addResource(new
File("src\\main\\files\\mapred-site.xml").getAbsolutePath )
sc.hadoopConfiguration.addResource(new
File("src\\main\\files\\yarn-site.xml").getAbsolutePath )
sc.hadoopConfiguration.addResource(new
File("src\\main\\files\\ssl-client.xml").getAbsolutePath )
sc.hadoopConfiguration.addResource(new
File("src\\main\\files\\topology.map").getAbsolutePath )
val conf = new Configuration()
// Loading remote cluster configurations
conf.addResource(new Path(new
File("src\\main\\files\\core-site.xml").getAbsolutePath ))
conf.addResource(new Path(new
File("src\\main\\files\\hdfs-site.xml").getAbsolutePath ))
conf.addResource(new Path(new
File("src\\main\\files\\mapred-site.xml").getAbsolutePath ))
conf.addResource(new Path(new
File("src\\main\\files\\yarn-site.xml").getAbsolutePath ))
conf.addResource(new Path(new
File("src\\main\\files\\ssl-client.xml").getAbsolutePath ))
conf.addResource(new Path(new
File("src\\main\\files\\topology.map").getAbsolutePath ))
conf.set("hadoop.security.authentication", "Kerberos")
UserGroupInformation.setConfiguration(conf)
UserGroupInformation.loginUserFromKeytab("my...@internal.company.com",
new File("src\\main\\files\\myusr.keytab").getAbsolutePath )
//
SparkHadoopUtil.get.loginUserFromKeytab("tsad...@internal.imsglobal.com",
// new File("src\\main\\files\\tsadusr.keytab").getAbsolutePath)
// Getting this error: java.io.IOException: Can't get Master
Kerberos principal for use as renewer
sc.textFile("hdfs://vm1.comp.com:8020/user/myusr/temp/file1").collect().foreach(println)
// Getting this error: java.io.IOException: Can't get Master
Kerberos principal for use as renewer
}
}
On Mon, Nov 7, 2016 at 9:42 PM, Ajay Chander <itsche...@gmail.com> wrote:
> Did anyone use https://www.codatlas.com/github.com/apache/spark/HEAD/
> core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala to
> interact with secured Hadoop from Spark ?
>
> Thanks,
> Ajay
>
> On Mon, Nov 7, 2016 at 4:37 PM, Ajay Chander <itsche...@gmail.com> wrote:
>
>>
>> Hi Everyone,
>>
>> I am trying to develop a simple codebase on my machine to read data from
>> secured Hadoop cluster. We have a development cluster which is secured
>> through Kerberos and I want to run a Spark job from my IntelliJ to read
>> some sample data from the cluster. Has anyone done this before ? Can you
>> point me to some sample examples?
>>
>> I understand that, if we want to talk to secured cluster, we need to have
>> keytab and principle. I tried using it through
>> UserGroupInformation.loginUserFromKeytab and
>> SparkHadoopUtil.get.loginUserFromKeytab but so far no luck.
>>
>> I have been trying to do this from quite a while ago. Please let me know
>> if you need more info. Thanks
>>
>> Regards,
>> Ajay
>>
>
>
