Hi All,
I am building a spark library which developers will use when writing their
spark jobs to get access to data on Azure Data Lake. But the authentication
will depend on the dataset they ask for. I need to call a rest API from
within spark job to get credentials and authenticate to read data from
ADLS. Is that even possible? I am new to spark.
E.g, from inside a spark job a user will say:
MyCredentials myCredentials = MyLibrary.getCredentialsForPath(userId,
"/some/path/on/azure/datalake");
then before spark.read.json("adl://examples/src/main/resources/people.json")
I need to authenticate the user to be able to read that path using the
credentials fetched above.
Any help is appreciated.
Thanks,
Imtiaz
