Github user jnadler commented on the issue:
https://github.com/apache/spark/pull/15132
Huge thank you to Cody Koeninger for working on this change. Looking
forward to 2.1.0.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/15132
LGTM. Thanks! Merging to master and 2.1.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/15132
@zsxwing if you're comfortable with it would you like to merge?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15132
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15132
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68383/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15132
**[Test build #68383 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68383/consoleFull)**
for PR 15132 at commit
Github user koeninger commented on the issue:
https://github.com/apache/spark/pull/15132
@zsxwing @rxin the per-partition rate limit probably won't overflow, but
the overall backpressure rate limit was being cast to an int, which definitely
can overflow. I changed it in this latest
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15132
**[Test build #68383 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68383/consoleFull)**
for PR 15132 at commit
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/15132
@koeninger Looks good to me once you address the rest comments.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/15132
This looks OK to me, but I really don't know Kafka.
cc @zsxwing and @marmbrus to take a look too.
---
If your project is set up for it, you can reply to this email and have your
reply appear
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15132
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15132
**[Test build #68179 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68179/consoleFull)**
for PR 15132 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15132
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68179/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15132
**[Test build #68179 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68179/consoleFull)**
for PR 15132 at commit
Github user koeninger commented on the issue:
https://github.com/apache/spark/pull/15132
@rxin thanks, changed to abstract class. If you think that's sufficient
future proofing I otherwise think this is a worthwhile change, seems like it
meets a real user need.
---
If your project
Github user koeninger commented on the issue:
https://github.com/apache/spark/pull/15132
I sent a message to d...@spark.apache.org, if you're not already subscribed
I'd say subscribe and follow up in case there's any discussion there rather
than on the pr / jira. Someone may want to
Github user jnadler commented on the issue:
https://github.com/apache/spark/pull/15132
Anything I can do to help get the right person's attention?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does
Github user koeninger commented on the issue:
https://github.com/apache/spark/pull/15132
Ok... so the question at this point is whether it's worth making the API
change, which ultimately we'll have to track down a committer to decide.
As the PR stands, it shouldn't break
Github user jnadler commented on the issue:
https://github.com/apache/spark/pull/15132
Testing has gone well. It's clearly successful in applying separate
maxrates to our 2 topics. Works as expected with backpressure enabled;
backpressure throttles both streams proportionally, no
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15132
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68024/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15132
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15132
**[Test build #68024 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68024/consoleFull)**
for PR 15132 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15132
**[Test build #68024 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68024/consoleFull)**
for PR 15132 at commit
Github user jnadler commented on the issue:
https://github.com/apache/spark/pull/15132
Just had first run that is basically working. The change to stop shading
Guava in Spark 2.x was a challenge. One thought, maybe PerPartitionConfig
should be Serializable. I had to add
Github user jnadler commented on the issue:
https://github.com/apache/spark/pull/15132
Building a test environment for this right now. Sorry for the delays.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user jnadler commented on the issue:
https://github.com/apache/spark/pull/15132
Not yet. Got Kafka 0.10 running and mirroring our production data streams
in prep for this test. I hope to get it tested next week.
On Thu, Oct 6, 2016 at 1:40 PM, Cody Koeninger
Github user koeninger commented on the issue:
https://github.com/apache/spark/pull/15132
@jnadler Have you had a chance to try this out and see whether it addresses
your issue?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as
Github user koeninger commented on the issue:
https://github.com/apache/spark/pull/15132
I personally wouldn't encourage people to use receivers unless they had a
very specific reason to. The kafka-0-10 submodule doesn't even have a way
to create receiver based streams, so I'm
Github user jnadler commented on the issue:
https://github.com/apache/spark/pull/15132
Are you comfortable with the inconsistency this creates between Direct
Stream and Receivers? I don't need a receiver-based solution personally. It
does occur to me that it might be desirable
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15132
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15132
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65544/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15132
**[Test build #65544 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65544/consoleFull)**
for PR 15132 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15132
**[Test build #65544 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65544/consoleFull)**
for PR 15132 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15132
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15132
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65543/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15132
**[Test build #65543 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65543/consoleFull)**
for PR 15132 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15132
**[Test build #65543 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65543/consoleFull)**
for PR 15132 at commit
37 matches
Mail list logo
