GitHub user jose-torres opened a pull request:
https://github.com/apache/spark/pull/20253
[SPARK-22908][SS] fix continuous Kafka data reader
## What changes were proposed in this pull request?
The Kafka reader is now interruptible and can close itself.
Note that this means we no longer have full failOnDataLoss support. We
should not block on that, since this is causing serious test flakiness.
## How was this patch tested?
I locally ran one of the ContinuousKafkaSourceSuite tests in a tight loop.
Before the fix, my machine ran out of open file descriptors a few iterations
in; now it works fine.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jose-torres/spark fix-data-reader
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/20253.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #20253
----
commit 0cb70f66fd64fb935a54a9098f9161368843b146
Author: Jose Torres <jose@...>
Date: 2018-01-12T22:40:45Z
fix data reader
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
