[GitHub] spark issue #21038: [SPARK-22968][DStream] Fix Kafka partition revoked issue

[jerryshao]

Github user jerryshao commented on the issue:

https://github.com/apache/spark/pull/21038
  
Thanks @koeninger , then I will just improve the exception message.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21038: [SPARK-22968][DStream] Fix Kafka partition revoked issue

[koeninger]

Github user koeninger commented on the issue:

https://github.com/apache/spark/pull/21038
  
I can't think of a valid reason to create a configuration to allow it.  It
just fundamentally doesn't make sense to run different apps with the same
group id.

Trying to catch and rethrow the exception with more information might make
sense.

On Wed, Apr 11, 2018, 20:05 Saisai Shao  wrote:

> Thanks @koeninger  for your comments. I
> think your suggestion is valid, the log here is just pasted from JIRA, but
> we also got the same issue from customer's report.
>
> Here in the PR description, I mentioned that using two apps with same
> group id to mimic this issue. But I'm not sure the real use case from our
> customer, maybe in their scenario such usage is valid.
>
> So I'm wondering if we can add a configuration to control whether it
> should be fail or just warning. Also I think exception/warning log should
> be improved to directly tell user about consumer rebalance issue, rather
> than throwing from Kafka as "no current assignment for partition xxx".
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> , or 
mute
> the thread
> 

> .
>



---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21038: [SPARK-22968][DStream] Fix Kafka partition revoked issue

[jerryshao]

Github user jerryshao commented on the issue:

https://github.com/apache/spark/pull/21038
  
Thanks @koeninger for your comments. I think your suggestion is valid, the 
log here is just pasted from JIRA, but we also got the same issue from 
customer's report. 

Here in the PR description, I mentioned that using two apps with same group 
id to mimic this issue. But I'm not sure the real use case from our customer, 
maybe in their scenario such usage is valid.

So I'm wondering if we can add a configuration to control whether it should 
be fail or just warning. Also I think exception/warning log should be improved 
to directly tell user about consumer rebalance issue, rather than throwing from 
Kafka as "no current assignment for partition xxx".


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21038: [SPARK-22968][DStream] Fix Kafka partition revoked issue

[koeninger]

Github user koeninger commented on the issue:

https://github.com/apache/spark/pull/21038
  
The log in the jira looks like it's from a consumer rebalance, i.e. more 
than one driver consumer was running with the same group id.

Isn't the underlying problem here that the user is creating multiple 
streams with the same group id, despite what the documentation says?  The log 
even says s/he copy-pasted the documentation group id "group 
use_a_separate_group_id_for_each_stream"

I don't think we should silently "fix" that.  As a user, I wouldn't expect 
app A to suddenly start processing only half of the partitions just because 
entirely different app B started with the (misconfigured) same group id.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21038: [SPARK-22968][DStream] Fix Kafka partition revoked issue

[jerryshao]

Github user jerryshao commented on the issue:

https://github.com/apache/spark/pull/21038
  
@koeninger would you please help to review, thanks!


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21038: [SPARK-22968][DStream] Fix Kafka partition revoked issue

[Hackeruncle]

Github user Hackeruncle commented on the issue:

https://github.com/apache/spark/pull/21038
  
@jerryshao Thank you very much for this issue.
I go to compile and test.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21038: [SPARK-22968][DStream] Fix Kafka partition revoked issue

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/21038
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21038: [SPARK-22968][DStream] Fix Kafka partition revoked issue

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/21038
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89177/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21038: [SPARK-22968][DStream] Fix Kafka partition revoked issue

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/21038
  
**[Test build #89177 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89177/testReport)**
 for PR 21038 at commit 
[`f317dec`](https://github.com/apache/spark/commit/f317dec0d863a717dc424707571453b11c43e700).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org