My Spark Structured Streaming job works fine when I set "startingOffsets" to "latest". When I simply change it to "earliest" & specify a new "check point directory", the job doesn't work. The states don't get timed out after 10 minutes.
While debugging I noticed that my 'state' logic is indeed getting executed but states just don't time out - as they do when I use "latest". Any reason why? Is this a known issue? *Note*: I've tried this under Spark 2.3 & 2.4
