Hi ,
   I am looking a way to pass configuration parameters to spark job.
In general I have quite simple PySpark job.

  def process_model(k, vc):
       ....
       do something
       ....


 sc = SparkContext(appName="TAD")
    lines = sc.textFile(input_job_files)
    result = lines.map(doSplit).groupByKey().map(lambda (k,vc):
process_model(k,vc))

Question:
    In case I need to pass to process_model function additional metadata ,
parameters , etc ...

   I tried to do something like
   param = 'param1'
  result = lines.map(doSplit).groupByKey().map(lambda (param,k,vc):
process_model(param1,k,vc)) ,

but job stops to work , also it looks like not elegant solution.
Is there a way to have access to SparkContext from my custom functions?
I found that there are methods setLocalProperty/getLocalProperty   but I
didn't find example how to use it for my requirements (from my function).

It would be great to have short example how to pass parameters.

Thanks
Oleg.

Reply via email to