The SparkConf doesn't allow you to set arbitrary variables. You can use
SparkContext's HadoopRDD and create a JobConf (with whatever variables you
want), and then grab them out of the JobConf in your RecordReader.
On Sun, Feb 22, 2015 at 4:28 PM, hnahak <harihar1...@gmail.com> wrote:
> Hi,
>
> I have written custom InputFormat and RecordReader for Spark, I need to
> use
> user variables from spark client program.
>
> I added them in SparkConf
>
> val sparkConf = new
> SparkConf().setAppName(args(0)).set("developer","MyName")
>
> *and in InputFormat class*
>
> protected boolean isSplitable(JobContext context, Path filename) {
>
>
> System.out.println("######################################### Developer "
> + context.getConfiguration().get("developer") );
> return false;
> }
>
> but its return me *null* , is there any way I can pass user variables to my
> custom code?
>
> Thanks !!
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/How-to-send-user-variables-from-Spark-client-to-custom-InputFormat-or-RecordReader-tp21755.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
> For additional commands, e-mail: user-h...@spark.apache.org
>
>
