Re: How to pass sparkSession from driver to executor

Hi, the other thing that you may try doing is use the following in your SQL and then based on regular expressions filter out records based on which directory they came from. But I would be very interested to know the details which I have asked for in my earlier email. input_file_name() Regards,

Re: How to pass sparkSession from driver to executor

Hi, I think that what you are facing is documented in SPARK: http://spark.apache.org/docs/latest/rdd-programming-guide.html#understanding-closures- May I ask what are you trying to achieve here? From what I understand, you have a list of JSON files which you want to read separately, as they have

Re: How to pass sparkSession from driver to executor

The point here is - spark session is not available in executors. So, you have to use appropriate storage clients. On Fri, Sep 22, 2017 at 1:44 AM, lucas.g...@gmail.com wrote: > I'm not sure what you're doing. But I have in the past used spark to > consume a manifest file and then execute a .map

Re: How to pass sparkSession from driver to executor

I'm not sure what you're doing. But I have in the past used spark to consume a manifest file and then execute a .mapPartition on the result like this: def map_key_to_event(s3_events_data_lake): def _map_key_to_event(event_key_list, s3_client=test_stub): print("Events in list")

Re: How to pass sparkSession from driver to executor

Spark do not allow executor code using `sparkSession`. But I think you can move all json files to one directory, and them run: ``` spark.read.json("/path/to/jsonFileDir") ``` But if you want to get filename at the same time, you can use ``` spark.sparkContext.wholeTextFiles("/path/to/jsonFileDir")

Re: How to pass sparkSession from driver to executor

Depends on your use-case however broadcasting could be a better option. On Thu, Sep 21, 2017 at 2:03 PM, Chackravarthy Esakkimuthu < chaku.mi...@gmail.com> wrote: > Hi, > > I want to know how to pass sparkSession