Re: Structured Streaming Kafka change maxOffsetsPerTrigger won't apply

2019-11-20 Thread Gabor Somogyi
Hi Roland, Not much shared apart from it's not working. Latest partition offset is used when the size of a TopicPartition is negative. This can be found out by checking the following log entry in the logs: logDebug(s"rateLimit $tp size is $size") If you've double checked and still think it's an

Structured Streaming Kafka change maxOffsetsPerTrigger won't apply

2019-11-20 Thread Roland Johann
Hi All, changing maxOffsetsPerTrigger and restarting the job won’t apply to the batch size. This is somehow bad as we currently use a trigger duration of 5minutes which consumes only 100k messages with an offset lag in the billions. Decreasing trigger duration affects also micro batch size - bu