Which Spark version are you using? If you are using 2.1.0, could you use the monitoring APIs ( http://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#monitoring-streaming-queries) to check the input rate and the processing rate? One possible issue is that the Kafka source launched a pretty large batch and it took too long to finish it. If so, you can use "maxOffsetsPerTrigger" option to limit the data size in a batch in order to observe the progress.
On Sun, Jan 22, 2017 at 10:22 AM, Timothy Chan <tc...@lumoslabs.com> wrote: > I'm running my structured streaming jobs in EMR. We were thinking a worst > case scenario recovery situation would be to spin up another cluster and > set startingOffsets to earliest (our Kafka cluster has a retention policy > of 7 days). > > My observation is that the job never catches up to latest. This is not > acceptable. I've set the number of partitions for the topic to 6. I've > tried using a cluster of 4 in EMR. > > The producer rate for this topic is 4 events/second. Does anyone have any > suggestions on what I can do to have my consumer catch up to latest faster? >
