I'm using Spark Streaming to process a large number of files (10s of millions) from a single directory in S3. Using sparkContext.textFile or wholeTextFile takes ages and doesn't do anything. Pointing Structured Streaming to that location seems to work, but after processing all the input, it waits for more. Is there a way to terminate the Streaming app once all input has been exhausted?
