It may not be as easy as you think. The rest call will happen in driver but the reads will be in the executors.
On Sat, 19 Aug 2017 at 11:42 am, Imtiaz Ahmed <emtiazah...@gmail.com> wrote: > Hi All, > > I am building a spark library which developers will use when writing their > spark jobs to get access to data on Azure Data Lake. But the authentication > will depend on the dataset they ask for. I need to call a rest API from > within spark job to get credentials and authenticate to read data from > ADLS. Is that even possible? I am new to spark. > E.g, from inside a spark job a user will say: > > MyCredentials myCredentials = MyLibrary.getCredentialsForPath(userId, > "/some/path/on/azure/datalake"); > > then before spark.read.json( > "adl://examples/src/main/resources/people.json") > I need to authenticate the user to be able to read that path using the > credentials fetched above. > > Any help is appreciated. > > Thanks, > Imtiaz > -- Best Regards, Ayan Guha
