Github user koeninger commented on the issue:
https://github.com/apache/spark/pull/15737
In most cases poll should be returning prefetched data from the buffer, not
waiting to talk to kafka over the network.
I could see increasing it a little bit, but I don't think it should be
increased to 10 seconds by default. That's a really long time for a 1
second batch streaming job to silently wait, and the only thing you'll be
able to observe is that your job starts running slow for some unknown
reason. At least with a smaller default any network or Kafka issues will
be obvious, and you can either diagnose them or tune upwards.
On Wed, Nov 2, 2016 at 1:06 PM, Michael Armbrust <notificati...@github.com>
wrote:
> Is there are reason to not change the user facing default? Its failing our
> tests and it was failing for me for real with the current default. When we
> are poling on the executors, the offsets should always be available, so I
> don't think increasing the timeout will hurt anything.
>
> â
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/spark/pull/15737#issuecomment-257949815>, or
mute
> the thread
>
<https://github.com/notifications/unsubscribe-auth/AAGAB6lwUbgWwgpfkaOgyimftB_nYvpUks5q6NFCgaJpZM4KnW2M>
> .
>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
