Hi
The max bytes setting should be enough, because if the tasks fail, it read
the data from kafka very fast as normal.
The request.timeout.ms I set is 180 seconds.
I think it should be time out setting or max bandwidth setting because of
the reason that it recoveries and read the same
Sorry message sent as incomplete.
To better debug the issue, please check the below config properties:
- At Kafka consumer properties
- max.partition.fetch.bytes within spark kafka consumer. If not set
for consumer then the global config at broker level.
- request.timeout.ms
Hi,
To better debug the issue, please check the below config properties:
- max.partition.fetch.bytes within spark kafka consumer. If not set for
consumer then the global config at broker level.
- spark.streaming.kafka.consumer.poll.ms
- spark.network.timeout (If the above is not
Would be better if you share some code block to understand it better.
Else would be difficult to provide answer.
~Shyam
On Wed, Mar 6, 2019 at 8:38 AM JF Chen wrote:
> When my kafka executor reads data from kafka, sometimes it throws the
> error "java.lang.AssertionError: assertion failed:
When my kafka executor reads data from kafka, sometimes it throws the error
"java.lang.AssertionError: assertion failed: Failed to get records for
after polling for 18" , which after 3 minutes of executing.
The data waiting for read is not so huge, which is about 1GB. And other
partitions
