I have a continuous rest api stream which keeps spitting out data in form
I access the stream using python requests.get(url, stream=True,
I want to receive them using spark and do further processing. I am not sure
which is best way to receive it in spark.
What are the options i have.
Some options i can think of
1. push data from rest api stream in to kakfa queue and use spark kafka
streaming utilities to capture data and further process.
2. push data from rest api stream to a local socket and use spark socket
stream utilities to capture data and further process.
3. is there any other way to receive it?