Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/18143
@ScrapCodes sorry for the delay. I think @tdas has fixed the issue. Please
close the PR.
---
-
To unsubscribe, e-mail:
Github user ScrapCodes commented on the issue:
https://github.com/apache/spark/pull/18143
@brkyvz and @zsxwing No comments on whether this will be useful or not, so
far. Should I consider closing it ?
---
-
To
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18143
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18143
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86529/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18143
**[Test build #86529 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86529/testReport)**
for PR 18143 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18143
**[Test build #86529 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86529/testReport)**
for PR 18143 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18143
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18143
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/139/
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18143
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18143
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86270/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18143
**[Test build #86270 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86270/testReport)**
for PR 18143 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18143
**[Test build #86270 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86270/testReport)**
for PR 18143 at commit
Github user ScrapCodes commented on the issue:
https://github.com/apache/spark/pull/18143
I am currently, trying to run some performance tests and see how this
change impacts performance in any case. Meanwhile, if I could get an idea if
things are moving in the right direction that
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18143
**[Test build #79909 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79909/testReport)**
for PR 18143 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18143
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79909/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18143
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18143
**[Test build #79909 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79909/testReport)**
for PR 18143 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18143
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18143
**[Test build #79908 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79908/testReport)**
for PR 18143 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18143
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79908/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18143
**[Test build #79907 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79907/testReport)**
for PR 18143 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18143
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18143
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79907/
Test PASSed.
---
Github user ScrapCodes commented on the issue:
https://github.com/apache/spark/pull/18143
@zsxwing, can you please take a look.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18143
**[Test build #79908 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79908/testReport)**
for PR 18143 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18143
**[Test build #79907 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79907/testReport)**
for PR 18143 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18143
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18143
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79711/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18143
**[Test build #79711 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79711/testReport)**
for PR 18143 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18143
**[Test build #79711 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79711/testReport)**
for PR 18143 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18143
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18143
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79710/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18143
**[Test build #79710 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79710/testReport)**
for PR 18143 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18143
**[Test build #79710 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79710/testReport)**
for PR 18143 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18143
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79708/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18143
**[Test build #79708 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79708/testReport)**
for PR 18143 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18143
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18143
**[Test build #79708 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79708/testReport)**
for PR 18143 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18143
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18143
**[Test build #79707 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79707/testReport)**
for PR 18143 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18143
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79707/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18143
**[Test build #79707 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79707/testReport)**
for PR 18143 at commit
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/18143
@ScrapCodes I think it should be bounded by
`spark.sql.kafkaConsumerCache.capacity`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well.
Github user ScrapCodes commented on the issue:
https://github.com/apache/spark/pull/18143
@brkyvz and @zsxwing, do you think this object pool should be bounded ? Or
they can become weak reference values, incase the object pool is unbounded ?
---
If your project is set up for it,
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/18143
@brkyvz I think consumers and producers have different cache strategy and
seem sharing the same interface is weird. We can share the same producer in
multiple tasks at the same time, but that's not
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18143
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18143
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/7/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18143
**[Test build #7 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/7/testReport)**
for PR 18143 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18143
**[Test build #7 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/7/testReport)**
for PR 18143 at commit
Github user ScrapCodes commented on the issue:
https://github.com/apache/spark/pull/18143
Thanks @koeninger for helping me.
Thanks @brkyvz, please take a look again and see if what I have done is
along the lines the change you wanted.
---
If your project is set up for it,
Github user brkyvz commented on the issue:
https://github.com/apache/spark/pull/18143
Thanks @ScrapCodes for this PR. I was envisioning using a common cache
interface for both the producer and the consumer, which kinda looked like:
```scala
trait KafkaClientCache[K, V] {
Github user koeninger commented on the issue:
https://github.com/apache/spark/pull/18143
Sorry, noticed a couple more minor configuration related things. Otherwise
LGTM.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18143
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18143
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77640/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18143
**[Test build #77640 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77640/testReport)**
for PR 18143 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18143
**[Test build #77640 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77640/testReport)**
for PR 18143 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18143
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18143
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77590/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18143
**[Test build #77590 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77590/testReport)**
for PR 18143 at commit
Github user ScrapCodes commented on the issue:
https://github.com/apache/spark/pull/18143
Thanks @koeninger for taking a look. I did some testing to see if
performance was impacted, and made corrections.
---
If your project is set up for it, you can reply to this email and have
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18143
**[Test build #77590 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77590/testReport)**
for PR 18143 at commit
Github user koeninger commented on the issue:
https://github.com/apache/spark/pull/18143
Generally looks ok to me, thanks.
2 questions -
Did you do any testing on workloads to see if performance stayed the same?
Is there a reason not to do the same thing to
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18143
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18143
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77532/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18143
**[Test build #77532 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77532/testReport)**
for PR 18143 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18143
**[Test build #77532 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77532/testReport)**
for PR 18143 at commit
Github user ScrapCodes commented on the issue:
https://github.com/apache/spark/pull/18143
Thanks @srowen for taking a look. I have tried to address your comments.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18143
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18143
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77522/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18143
**[Test build #77522 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77522/testReport)**
for PR 18143 at commit
Github user ScrapCodes commented on the issue:
https://github.com/apache/spark/pull/18143
Hi, @brkyvz as we discussed on the PR for CachedKafkaProducer that Guava
cache can be used for consumer as well. May be you could take a look at this as
well ?
---
If your project is set up
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18143
**[Test build #77522 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77522/testReport)**
for PR 18143 at commit
72 matches
Mail list logo
