subject:"Re\: Access_Remote_Kerberized_Cluster_Through

Re: Access_Remote_Kerberized_Cluster_Through_Spark

2016-11-10 Thread KhajaAsmath Mohammed

Hi Ajay,

I was able to resolve it by adding yarn user principal. here is complete
code.

  def main(args: Array[String]) {
// create Spark context with Spark configuration
val cmdLine = Parse.commandLine(args)
val configFile = cmdLine.getOptionValue("c")
val propertyConfiguration = new PropertyConfiguration()
val props = propertyConfiguration.get(configFile)
   // val
fs=com.yourcompany.telematics.fs.FileSystem.getHdfsFileSystem(props);
val fs=com.yourcompany.telematics.fs.FileSystem getHdfsFileSystem props;
var sparkConfig = propertyConfiguration.initConfiguration()
val sc = new SparkContext(sparkConfig)
val
configuration:org.apache.hadoop.conf.Configuration=sc.hadoopConfiguration

java.lang.System.setProperty("javax.security.auth.useSubjectCredsOnly","true");
java.lang.System.setProperty("java.security.krb5.conf",
"C:\\krb5.conf");
System.setProperty("sun.security.krb5.debug", "true")
configuration.set("hadoop.security.authentication", "Kerberos");
configuration.set("hdfs.namenode.kerberos.principal","hdfs/_
h...@ad.yourcompany.com")
configuration.set("hdfs.datanode.kerberos.principal.pattern","hdfs/*@
AD.yourcompany.COM")
configuration.set("hdfs.master.kerberos.principal","hdfs/*@
AD.yourcompany.COM")
configuration.set("yarn.nodemanager.principal","yarn/*@
AD.yourcompany.COM")
configuration.set("yarn.resourcemanager.principal","yarn/*@
AD.yourcompany.COM")
val hadoopConf="C:\\devtools\\hadoop\\hadoop-2.2.0\\hadoop-2.2.0\\conf"

configuration.addResource(new Path(hadoopConf + "core-site.xml"));
configuration.addResource(new Path(hadoopConf + "hdfs-site.xml"));
configuration.addResource(new Path(hadoopConf + "mapred-site.xml"));
configuration.addResource(new Path(hadoopConf + "yarn-site.xml"));
configuration.addResource(new Path(hadoopConf + "hadoop-policy.xml"));
configuration.set("hadoop.security.authentication", "kerberos");
UserGroupInformation.setConfiguration(configuration);
UserGroupInformation.loginUserFromKeytab("va_d...@ad.yourcompany.com",
"C:\\va_dflt.keytab");


// get threshold
   // val threshold = args(1).toInt

// read in text file and split each document into words
val lineRdd =
sc.textFile("hdfs://XX:8020/user/yyy1k78/vehscanxmltext")
val tokenized=lineRdd.flatMap(_.split(" "))
//System.out.println(tokenized.collect().mkString(", "))
// count the occurrence of each word
val wordCounts = tokenized.map((_, 1)).reduceByKey(_ + _)

// filter out words with fewer than threshold occurrences
//val filtered = wordCounts.filter(_._2 >= threshold)

// count characters
//val charCounts = filtered.flatMap(_._1.toCharArray).map((_,
1)).reduceByKey(_ + _)

System.out.println(wordCounts.collect().mkString(", "))
  }
}

Thanks,
Asmath.

On Wed, Nov 9, 2016 at 7:44 PM, Ajay Chander  wrote:

> Hi Everyone,
>
> I am still trying to figure this one out. I am stuck with this error 
> "java.io.IOException:
> Can't get Master Kerberos principal for use as renewer ". Below is my code.
> Can any of you please provide any insights on this? Thanks for your time.
>
>
> import java.io.{BufferedInputStream, File, FileInputStream}
> import java.net.URI
>
> import org.apache.hadoop.fs.FileSystem
> import org.apache.hadoop.conf.Configuration
> import org.apache.hadoop.fs.Path
> import org.apache.hadoop.io.IOUtils
> import org.apache.hadoop.security.UserGroupInformation
> import org.apache.spark.deploy.SparkHadoopUtil
> import org.apache.spark.{SparkConf, SparkContext}
>
>
> object SparkHdfs {
>
>   def main(args: Array[String]): Unit = {
>
> System.setProperty("java.security.krb5.conf", new 
> File("src\\main\\files\\krb5.conf").getAbsolutePath )
> System.setProperty("sun.security.krb5.debug", "true")
>
> val sparkConf = new SparkConf().setAppName("SparkHdfs").setMaster("local")
> val sc = new SparkContext(sparkConf)
> //Loading remote cluster configurations
> sc.hadoopConfiguration.addResource(new 
> File("src\\main\\files\\core-site.xml").getAbsolutePath )
> sc.hadoopConfiguration.addResource(new 
> File("src\\main\\files\\hdfs-site.xml").getAbsolutePath )
> sc.hadoopConfiguration.addResource(new 
> File("src\\main\\files\\mapred-site.xml").getAbsolutePath )
> sc.hadoopConfiguration.addResource(new 
> File("src\\main\\files\\yarn-site.xml").getAbsolutePath )
> sc.hadoopConfiguration.addResource(new 
> File("src\\main\\files\\ssl-client.xml").getAbsolutePath )
> sc.hadoopConfiguration.addResource(new 
> File("src\\main\\files\\topology.map").getAbsolutePath )
>
> val conf = new Configuration()
> //Loading remote cluster configurations
> conf.addResource(new Path(new 
> File("src\\main\\files\\core-site.xml").getAbsolutePath ))
> conf.addResource(new Path(new 
> File("src\\main\\files\\hdfs-site.xml").getAbsolutePath ))
> conf.addResource(new Path(new 
>

Re: Access_Remote_Kerberized_Cluster_Through_Spark

2016-11-09 Thread Ajay Chander

Hi Everyone,

I am still trying to figure this one out. I am stuck with this error
"java.io.IOException:
Can't get Master Kerberos principal for use as renewer ". Below is my code.
Can any of you please provide any insights on this? Thanks for your time.


import java.io.{BufferedInputStream, File, FileInputStream}
import java.net.URI

import org.apache.hadoop.fs.FileSystem
import org.apache.hadoop.conf.Configuration
import org.apache.hadoop.fs.Path
import org.apache.hadoop.io.IOUtils
import org.apache.hadoop.security.UserGroupInformation
import org.apache.spark.deploy.SparkHadoopUtil
import org.apache.spark.{SparkConf, SparkContext}


object SparkHdfs {

  def main(args: Array[String]): Unit = {

System.setProperty("java.security.krb5.conf", new
File("src\\main\\files\\krb5.conf").getAbsolutePath )
System.setProperty("sun.security.krb5.debug", "true")

val sparkConf = new SparkConf().setAppName("SparkHdfs").setMaster("local")
val sc = new SparkContext(sparkConf)
//Loading remote cluster configurations
sc.hadoopConfiguration.addResource(new
File("src\\main\\files\\core-site.xml").getAbsolutePath )
sc.hadoopConfiguration.addResource(new
File("src\\main\\files\\hdfs-site.xml").getAbsolutePath )
sc.hadoopConfiguration.addResource(new
File("src\\main\\files\\mapred-site.xml").getAbsolutePath )
sc.hadoopConfiguration.addResource(new
File("src\\main\\files\\yarn-site.xml").getAbsolutePath )
sc.hadoopConfiguration.addResource(new
File("src\\main\\files\\ssl-client.xml").getAbsolutePath )
sc.hadoopConfiguration.addResource(new
File("src\\main\\files\\topology.map").getAbsolutePath )

val conf = new Configuration()
//Loading remote cluster configurations
conf.addResource(new Path(new
File("src\\main\\files\\core-site.xml").getAbsolutePath ))
conf.addResource(new Path(new
File("src\\main\\files\\hdfs-site.xml").getAbsolutePath ))
conf.addResource(new Path(new
File("src\\main\\files\\mapred-site.xml").getAbsolutePath ))
conf.addResource(new Path(new
File("src\\main\\files\\yarn-site.xml").getAbsolutePath ))
conf.addResource(new Path(new
File("src\\main\\files\\ssl-client.xml").getAbsolutePath ))
conf.addResource(new Path(new
File("src\\main\\files\\topology.map").getAbsolutePath ))

conf.set("hadoop.security.authentication", "Kerberos")

UserGroupInformation.setConfiguration(conf)

UserGroupInformation.loginUserFromKeytab("my...@internal.company.com",
  new File("src\\main\\files\\myusr.keytab").getAbsolutePath )

//   
SparkHadoopUtil.get.loginUserFromKeytab("tsad...@internal.imsglobal.com",
//  new File("src\\main\\files\\tsadusr.keytab").getAbsolutePath)
//Getting this error: java.io.IOException: Can't get Master
Kerberos principal for use as renewer


sc.textFile("hdfs://vm1.comp.com:8020/user/myusr/temp/file1").collect().foreach(println)
//Getting this error: java.io.IOException: Can't get Master
Kerberos principal for use as renewer

  }
}




On Mon, Nov 7, 2016 at 9:42 PM, Ajay Chander  wrote:

> Did anyone use https://www.codatlas.com/github.com/apache/spark/HEAD/
> core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala to
> interact with secured Hadoop from Spark ?
>
> Thanks,
> Ajay
>
> On Mon, Nov 7, 2016 at 4:37 PM, Ajay Chander  wrote:
>
>>
>> Hi Everyone,
>>
>> I am trying to develop a simple codebase on my machine to read data from
>> secured Hadoop cluster. We have a development cluster which is secured
>> through Kerberos and I want to run a Spark job from my IntelliJ to read
>> some sample data from the cluster. Has anyone done this before ? Can you
>> point me to some sample examples?
>>
>> I understand that, if we want to talk to secured cluster, we need to have
>> keytab and principle. I tried using it through
>> UserGroupInformation.loginUserFromKeytab and
>> SparkHadoopUtil.get.loginUserFromKeytab but so far no luck.
>>
>> I have been trying to do this from quite a while ago. Please let me know
>> if you need more info. Thanks
>>
>> Regards,
>> Ajay
>>
>
>

Re: Access_Remote_Kerberized_Cluster_Through_Spark

2016-11-07 Thread Ajay Chander

Did anyone use
https://www.codatlas.com/github.com/apache/spark/HEAD/core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala
to interact with secured Hadoop from Spark ?

Thanks,
Ajay

On Mon, Nov 7, 2016 at 4:37 PM, Ajay Chander  wrote:

>
> Hi Everyone,
>
> I am trying to develop a simple codebase on my machine to read data from
> secured Hadoop cluster. We have a development cluster which is secured
> through Kerberos and I want to run a Spark job from my IntelliJ to read
> some sample data from the cluster. Has anyone done this before ? Can you
> point me to some sample examples?
>
> I understand that, if we want to talk to secured cluster, we need to have
> keytab and principle. I tried using it through 
> UserGroupInformation.loginUserFromKeytab
> and SparkHadoopUtil.get.loginUserFromKeytab but so far no luck.
>
> I have been trying to do this from quite a while ago. Please let me know
> if you need more info. Thanks
>
> Regards,
> Ajay
>

Re: Access_Remote_Kerberized_Cluster_Through_Spark

Re: Access_Remote_Kerberized_Cluster_Through_Spark

Re: Access_Remote_Kerberized_Cluster_Through_Spark

3 matches

Site Navigation

Mail list logo

Footer information