[GitHub] spark issue #19789: [SPARK-22562][Streaming] CachedKafkaConsumer unsafe evic...

2018-07-22 Thread HyukjinKwon 
Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/19789
  
@daroo thanks for checking it.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19789: [SPARK-22562][Streaming] CachedKafkaConsumer unsafe evic...

2018-07-21 Thread daroo 
Github user daroo commented on the issue:

https://github.com/apache/spark/pull/19789
  
@HyukjinKwon I've looked at the latest changes in the code and I think this 
problem shouldn't happen (at least in practice) anymore. So the JIRA can be 
closed.
One minor issue I've noticed is that the cache size can keep growing beyond 
configured limit in certain cases.



---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19789: [SPARK-22562][Streaming] CachedKafkaConsumer unsafe evic...

2018-07-17 Thread daroo 
Github user daroo commented on the issue:

https://github.com/apache/spark/pull/19789
  
sure :-)


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19789: [SPARK-22562][Streaming] CachedKafkaConsumer unsafe evic...

2018-07-13 Thread HyukjinKwon 
Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/19789
  
@daroo, mind reopening this if you have some time to update?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19789: [SPARK-22562][Streaming] CachedKafkaConsumer unsafe evic...

2018-07-13 Thread HyukjinKwon 
Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/19789
  
ok to test


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19789: [SPARK-22562][Streaming] CachedKafkaConsumer unsafe evic...

2018-06-08 Thread AmplabJenkins 
Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19789
  
Can one of the admins verify this patch?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19789: [SPARK-22562][Streaming] CachedKafkaConsumer unsafe evic...

2018-05-14 Thread cloud-fan 
Github user cloud-fan commented on the issue:

https://github.com/apache/spark/pull/19789
  
cc @marmbrus @zsxwing 


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19789: [SPARK-22562][Streaming] CachedKafkaConsumer unsafe evic...

2018-05-13 Thread daroo 
Github user daroo commented on the issue:

https://github.com/apache/spark/pull/19789
  
Hi @HyukjinKwon

It seems that @zsxwing is not really interested in this PR. Should I cancel 
it and close https://issues.apache.org/jira/browse/SPARK-22562 ?

BTW. Why AppVeyor CI build takes so long time? The build has failed due 
to reaching max allowed time (1,5h) :-(



---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19789: [SPARK-22562][Streaming] CachedKafkaConsumer unsafe evic...

2018-05-13 Thread AmplabJenkins 
Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19789
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19789: [SPARK-22562][Streaming] CachedKafkaConsumer unsafe evic...

2018-05-13 Thread SparkQA 
Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19789
  
**[Test build #90551 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90551/testReport)**
 for PR 19789 at commit 
[`639f11c`](https://github.com/apache/spark/commit/639f11c9fd856f58d067036f9b9131ea9efdc13d).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19789: [SPARK-22562][Streaming] CachedKafkaConsumer unsafe evic...

2018-05-13 Thread AmplabJenkins 
Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19789
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90551/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19789: [SPARK-22562][Streaming] CachedKafkaConsumer unsafe evic...

2018-05-13 Thread SparkQA 
Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19789
  
**[Test build #90551 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90551/testReport)**
 for PR 19789 at commit 
[`639f11c`](https://github.com/apache/spark/commit/639f11c9fd856f58d067036f9b9131ea9efdc13d).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19789: [SPARK-22562][Streaming] CachedKafkaConsumer unsafe evic...

2018-01-18 Thread AmplabJenkins 
Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19789
  
Can one of the admins verify this patch?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19789: [SPARK-22562][Streaming] CachedKafkaConsumer unsafe evic...

2018-01-18 Thread AmplabJenkins 
Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19789
  
Can one of the admins verify this patch?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19789: [SPARK-22562][Streaming] CachedKafkaConsumer unsafe evic...

2017-12-31 Thread AmplabJenkins 
Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19789
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19789: [SPARK-22562][Streaming] CachedKafkaConsumer unsafe evic...

2017-12-31 Thread SparkQA 
Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19789
  
**[Test build #85575 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85575/testReport)**
 for PR 19789 at commit 
[`7700bf0`](https://github.com/apache/spark/commit/7700bf02c06d6fe3909053a1e146569a67b21de3).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19789: [SPARK-22562][Streaming] CachedKafkaConsumer unsafe evic...

2017-12-31 Thread AmplabJenkins 
Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19789
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85575/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19789: [SPARK-22562][Streaming] CachedKafkaConsumer unsafe evic...

2017-12-31 Thread SparkQA 
Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19789
  
**[Test build #85575 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85575/testReport)**
 for PR 19789 at commit 
[`7700bf0`](https://github.com/apache/spark/commit/7700bf02c06d6fe3909053a1e146569a67b21de3).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19789: [SPARK-22562][Streaming] CachedKafkaConsumer unsafe evic...

2017-12-31 Thread HyukjinKwon 
Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/19789
  
ok to test


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19789: [SPARK-22562][Streaming] CachedKafkaConsumer unsafe evic...

2017-12-19 Thread daroo 
Github user daroo commented on the issue:

https://github.com/apache/spark/pull/19789
  
Hey @zsxwing

Could you please have a look at my PR? It's been a month almost since I 
opened it. @koeninger said that it looks good to him, but everyone else is 
waiting for your feedback.

Thanks a lot,
Darek


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19789: [SPARK-22562][Streaming] CachedKafkaConsumer unsafe evic...

2017-12-14 Thread AmplabJenkins 
Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19789
  
Can one of the admins verify this patch?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19789: [SPARK-22562][Streaming] CachedKafkaConsumer unsafe evic...

2017-12-04 Thread HyukjinKwon 
Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/19789
  
I think we should wait for @zsxwing's feedback.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19789: [SPARK-22562][Streaming] CachedKafkaConsumer unsafe evic...

2017-12-01 Thread daroo 
Github user daroo commented on the issue:

https://github.com/apache/spark/pull/19789
  
@HyukjinKwon
@zsxwing 

Any chance to go ahead with this?



---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19789: [SPARK-22562][Streaming] CachedKafkaConsumer unsafe evic...

2017-11-24 Thread daroo 
Github user daroo commented on the issue:

https://github.com/apache/spark/pull/19789
  
I don't know what are the plans for the structured streaming which @zsxwing 
has in mind, but before I created the PR I actually had seen how this problem 
is currently solved in kafka-0-10-sql module and I didn't like it. My main 
concern is that in certain cases the cache size my grow well beyond configured 
spark.sql.kafkaConsumerCache.capacity. That’s why I’ve chosen to do it in a 
bit different way.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19789: [SPARK-22562][Streaming] CachedKafkaConsumer unsafe evic...

2017-11-24 Thread koeninger 
Github user koeninger commented on the issue:

https://github.com/apache/spark/pull/19789
  
Seems reasonable to me but you should probably ask zsxwing if it fits in
with plans for the structured streaming kafka code.

On Thu, Nov 23, 2017 at 10:23 PM, Hyukjin Kwon 
wrote:

> *@HyukjinKwon* commented on this pull request.
>
> Does this look good to you @koeninger ?
> --
>
> In external/kafka-0-10/src/main/scala/org/apache/spark/
> streaming/kafka010/KafkaRDD.scala
> :
>
> > @@ -211,8 +211,8 @@ private[spark] class KafkaRDD[K, V](
>  var requestOffset = part.fromOffset
>
>  def closeIfNeeded(): Unit = {
> -  if (!useConsumerCache && consumer != null) {
> -consumer.close
> +  if (consumer != null) {
> +  consumer.close()
>
> I think this should be double spaced
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> ,
> or mute the thread
> 

> .
>



---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19789: [SPARK-22562][Streaming] CachedKafkaConsumer unsafe evic...

2017-11-24 Thread AmplabJenkins 
Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19789
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84162/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19789: [SPARK-22562][Streaming] CachedKafkaConsumer unsafe evic...

2017-11-24 Thread AmplabJenkins 
Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19789
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19789: [SPARK-22562][Streaming] CachedKafkaConsumer unsafe evic...

2017-11-24 Thread SparkQA 
Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19789
  
**[Test build #84162 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84162/testReport)**
 for PR 19789 at commit 
[`7700bf0`](https://github.com/apache/spark/commit/7700bf02c06d6fe3909053a1e146569a67b21de3).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19789: [SPARK-22562][Streaming] CachedKafkaConsumer unsafe evic...

2017-11-24 Thread SparkQA 
Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19789
  
**[Test build #84162 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84162/testReport)**
 for PR 19789 at commit 
[`7700bf0`](https://github.com/apache/spark/commit/7700bf02c06d6fe3909053a1e146569a67b21de3).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19789: [SPARK-22562][Streaming] CachedKafkaConsumer unsafe evic...

2017-11-24 Thread cloud-fan 
Github user cloud-fan commented on the issue:

https://github.com/apache/spark/pull/19789
  
cc @zsxwing 


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19789: [SPARK-22562][Streaming] CachedKafkaConsumer unsafe evic...

2017-11-23 Thread AmplabJenkins 
Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19789
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19789: [SPARK-22562][Streaming] CachedKafkaConsumer unsafe evic...

2017-11-23 Thread AmplabJenkins 
Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19789
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84143/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19789: [SPARK-22562][Streaming] CachedKafkaConsumer unsafe evic...

2017-11-23 Thread SparkQA 
Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19789
  
**[Test build #84143 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84143/testReport)**
 for PR 19789 at commit 
[`c2c3ed9`](https://github.com/apache/spark/commit/c2c3ed9e2e672b3b9205a824b8cdd2f9ca2131eb).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `class CachedKafkaConsumerSuite extends SparkFunSuite with 
BeforeAndAfterAll `


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19789: [SPARK-22562][Streaming] CachedKafkaConsumer unsafe evic...

2017-11-23 Thread SparkQA 
Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19789
  
**[Test build #84143 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84143/testReport)**
 for PR 19789 at commit 
[`c2c3ed9`](https://github.com/apache/spark/commit/c2c3ed9e2e672b3b9205a824b8cdd2f9ca2131eb).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19789: [SPARK-22562][Streaming] CachedKafkaConsumer unsafe evic...

2017-11-23 Thread HyukjinKwon 
Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/19789
  
ok to test


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19789: [SPARK-22562][Streaming] CachedKafkaConsumer unsafe evic...

2017-11-23 Thread daroo 
Github user daroo commented on the issue:

https://github.com/apache/spark/pull/19789
  
Hey Spark commiters @jiangxb1987 @srowen @cloud-fan @HyukjinKwon

Could you please authorize my PR for testing?

Thanks
Darek


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19789: [SPARK-22562][Streaming] CachedKafkaConsumer unsafe evic...

2017-11-23 Thread koeninger 
Github user koeninger commented on the issue:

https://github.com/apache/spark/pull/19789
  
You'll need to get a commiter's attention to merge it anyway

On Nov 23, 2017 01:48, "Daroo"  wrote:

It seems that your "magic spell" didn't work. No build was triggered

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
, or mute
the thread


.



---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19789: [SPARK-22562][Streaming] CachedKafkaConsumer unsafe evic...

2017-11-22 Thread daroo 
Github user daroo commented on the issue:

https://github.com/apache/spark/pull/19789
  
It seems that your "magic spell" didn't work. No build was triggered 


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19789: [SPARK-22562][Streaming] CachedKafkaConsumer unsafe evic...

2017-11-22 Thread koeninger 
Github user koeninger commented on the issue:

https://github.com/apache/spark/pull/19789
  
ok to test

On Wed, Nov 22, 2017 at 2:49 PM, Daroo  wrote:

> Cool. Could you please authorize it for testing?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> , or 
mute
> the thread
> 

> .
>



---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19789: [SPARK-22562][Streaming] CachedKafkaConsumer unsafe evic...

2017-11-22 Thread daroo 
Github user daroo commented on the issue:

https://github.com/apache/spark/pull/19789
  
Cool. Could you please authorize it for testing?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19789: [SPARK-22562][Streaming] CachedKafkaConsumer unsafe evic...

2017-11-22 Thread koeninger 
Github user koeninger commented on the issue:

https://github.com/apache/spark/pull/19789
  
Seems reasonable.

On Wed, Nov 22, 2017 at 1:52 PM, Daroo  wrote:

> It fails on the current master branch and doesn't after the patch
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> , or 
mute
> the thread
> 

> .
>



---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19789: [SPARK-22562][Streaming] CachedKafkaConsumer unsafe evic...

2017-11-22 Thread daroo 
Github user daroo commented on the issue:

https://github.com/apache/spark/pull/19789
  
It fails on the current master branch and doesn't after the patch


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19789: [SPARK-22562][Streaming] CachedKafkaConsumer unsafe evic...

2017-11-22 Thread koeninger 
Github user koeninger commented on the issue:

https://github.com/apache/spark/pull/19789
  
What are you actually asserting in that test and/or does it reliably fail
if run on the version of your code before the patch?

On Wed, Nov 22, 2017 at 1:33 PM, Daroo  wrote:

> I've added a test. @koeninger  is it
> something you had in mind?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> , or 
mute
> the thread
> 

> .
>



---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19789: [SPARK-22562][Streaming] CachedKafkaConsumer unsafe evic...

2017-11-22 Thread daroo 
Github user daroo commented on the issue:

https://github.com/apache/spark/pull/19789
  
I've added a test. @koeninger is it something you had in mind?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19789: [SPARK-22562][Streaming] CachedKafkaConsumer unsafe evic...

2017-11-20 Thread koeninger 
Github user koeninger commented on the issue:

https://github.com/apache/spark/pull/19789
  
Yeah, subscribepattern could definitely be an issue.

As far as unit testing, have you tried anything along the lines of setting
the cache size artificially low and then introducing new topicpartitions?

On Mon, Nov 20, 2017 at 11:43 AM, Daroo  wrote:

> I see you point, but it's kind of difficult to size it properly when you
> use SubscribePattern (i.e. dynamic number of topics/partitions) consumer
> strategy
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> , or 
mute
> the thread
> 

> .
>



---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19789: [SPARK-22562][Streaming] CachedKafkaConsumer unsafe evic...

2017-11-20 Thread daroo 
Github user daroo commented on the issue:

https://github.com/apache/spark/pull/19789
  
I see you point, but it's kind of difficult to size it properly when you 
use SubscribePattern (i.e. dynamic number of topics/partitions) consumer 
strategy


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19789: [SPARK-22562][Streaming] CachedKafkaConsumer unsafe evic...

2017-11-20 Thread koeninger 
Github user koeninger commented on the issue:

https://github.com/apache/spark/pull/19789
  
My main comment is that if you have a situation where there's actually 
contention on the size of the cache, chances are things are going to be screwed 
up anyway due to consumers being recreated and losing any benefit of buffering.

Are you running into this issue in situations where the cache is 
appropriately sized given the number of topicpartitions and executors?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19789: [SPARK-22562][Streaming] CachedKafkaConsumer unsafe evic...

2017-11-20 Thread daroo 
Github user daroo commented on the issue:

https://github.com/apache/spark/pull/19789
  
Hey @koeninger ,
could you please have a look at my PR?

Thanx a lot,
Darek


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19789: [SPARK-22562][Streaming] CachedKafkaConsumer unsafe evic...

2017-11-20 Thread AmplabJenkins 
Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19789
  
Can one of the admins verify this patch?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org