[GitHub] spark issue #21690: [SPARK-24713]AppMatser of spark streaming kafka OOM if t...

2018-07-13 Thread yuanboliu 
Github user yuanboliu commented on the issue:

https://github.com/apache/spark/pull/21690
  
Thanks very much


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21690: [SPARK-24713]AppMatser of spark streaming kafka OOM if t...

2018-07-13 Thread koeninger 
Github user koeninger commented on the issue:

https://github.com/apache/spark/pull/21690
  
LGTM, merging to master.  Thanks!


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21690: [SPARK-24713]AppMatser of spark streaming kafka OOM if t...

2018-07-12 Thread yuanboliu 
Github user yuanboliu commented on the issue:

https://github.com/apache/spark/pull/21690
  
After applying this patch, my application can be running successfully. This 
issue could happen in the case of many topics(hundreds of ) consumed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21690: [SPARK-24713]AppMatser of spark streaming kafka OOM if t...

2018-07-12 Thread koeninger 
Github user koeninger commented on the issue:

https://github.com/apache/spark/pull/21690
  
What results are you seeing?

On Thu, Jul 12, 2018, 6:53 AM Yuanbo Liu  wrote:

> @koeninger  Sorry to interrupt, could you
> take a look at my patch?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> , or 
mute
> the thread
> 

> .
>



---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21690: [SPARK-24713]AppMatser of spark streaming kafka OOM if t...

2018-07-12 Thread yuanboliu 
Github user yuanboliu commented on the issue:

https://github.com/apache/spark/pull/21690
  
@koeninger Sorry to interrupt, could you take a look at my patch?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21690: [SPARK-24713]AppMatser of spark streaming kafka OOM if t...

2018-07-07 Thread AmplabJenkins 
Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/21690
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92709/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21690: [SPARK-24713]AppMatser of spark streaming kafka OOM if t...

2018-07-07 Thread AmplabJenkins 
Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/21690
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21690: [SPARK-24713]AppMatser of spark streaming kafka OOM if t...

2018-07-07 Thread SparkQA 
Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/21690
  
**[Test build #92709 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92709/testReport)**
 for PR 21690 at commit 
[`d1a8c60`](https://github.com/apache/spark/commit/d1a8c605e163bc09d1329cbd90560cc5165de555).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21690: [SPARK-24713]AppMatser of spark streaming kafka OOM if t...

2018-07-07 Thread SparkQA 
Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/21690
  
**[Test build #92709 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92709/testReport)**
 for PR 21690 at commit 
[`d1a8c60`](https://github.com/apache/spark/commit/d1a8c605e163bc09d1329cbd90560cc5165de555).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21690: [SPARK-24713]AppMatser of spark streaming kafka OOM if t...

2018-07-07 Thread yuanboliu 
Github user yuanboliu commented on the issue:

https://github.com/apache/spark/pull/21690
  
@koeninger Thanks for your reply. Agree with you. there is no need to to 
use pause repeatedly. 
This is my test without any pause, and the app master stuck for a long time 
without any process




![wechatworkscreenshot_abb443bd-97db-48f9-88a2-e45a65617f80](https://user-images.githubusercontent.com/5643344/42409693-b324d45e-8210-11e8-96eb-39fc359b1b42.png)

I will update my patch shortly.



---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21690: [SPARK-24713]AppMatser of spark streaming kafka OOM if t...

2018-07-06 Thread koeninger 
Github user koeninger commented on the issue:

https://github.com/apache/spark/pull/21690
  
@yuanboliu What I'm suggesting is more like this:


https://github.com/apache/spark/compare/master...koeninger:SPARK-24713?expand=1


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21690: [SPARK-24713]AppMatser of spark streaming kafka OOM if t...

2018-07-05 Thread yuanboliu 
Github user yuanboliu commented on the issue:

https://github.com/apache/spark/pull/21690
  
@koeninger  Thanks for your details.  Sorry quite busy this week. I will 
delete the last pause, test the patch on my own cluster this weekend and give 
feedback asap.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21690: [SPARK-24713]AppMatser of spark streaming kafka OOM if t...

2018-07-03 Thread koeninger 
Github user koeninger commented on the issue:

https://github.com/apache/spark/pull/21690
  
@yuanboliu From reading KafkaConsumer code, and from testing, I don't see 
where consumer.position() alone would un-pause topicpartitions.  See below.  
Can you give a counter-example?

I am seeing poll() reset the paused state.  When you are having the 
problem, are you seeing the info level log messages "poll(0) returned messages"?

If that's what's happening, I think the best we can do is call pause() in 
only one place, the first line of paranoidPoll, e.g.

`c.pause(c.assignment)
val msgs = c.poll(0)
`



Here's what I saw in testing:

`scala> c.paused
res34: java.util.Set[org.apache.kafka.common.TopicPartition] = []

scala> c.assignment
res35: java.util.Set[org.apache.kafka.common.TopicPartition] = [test-0]

scala> c.pause(topics)

scala> c.paused
res37: java.util.Set[org.apache.kafka.common.TopicPartition] = [test-0]

scala> c.position(tp)
res38: Long = 248

scala> c.paused
res39: java.util.Set[org.apache.kafka.common.TopicPartition] = [test-0]

scala> c.poll(0)
res40: org.apache.kafka.clients.consumer.ConsumerRecords[String,String] = 
org.apache.kafka.clients.consumer.ConsumerRecords@20d7efbe

scala> c.paused
res41: java.util.Set[org.apache.kafka.common.TopicPartition] = [test-0]

scala> c.position(tp)
res42: Long = 248

scala> c.paused
res43: java.util.Set[org.apache.kafka.common.TopicPartition] = [test-0]

scala> c.poll(1)
res44: org.apache.kafka.clients.consumer.ConsumerRecords[String,String] = 
org.apache.kafka.clients.consumer.ConsumerRecords@20d7efbe

scala> c.paused
res45: java.util.Set[org.apache.kafka.common.TopicPartition] = [test-0]

scala> c.poll(100)
res46: org.apache.kafka.clients.consumer.ConsumerRecords[String,String] = 
org.apache.kafka.clients.consumer.ConsumerRecords@28e4439b

scala> c.paused
res47: java.util.Set[org.apache.kafka.common.TopicPartition] = []
`


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21690: [SPARK-24713]AppMatser of spark streaming kafka OOM if t...

2018-07-02 Thread yuanboliu 
Github user yuanboliu commented on the issue:

https://github.com/apache/spark/pull/21690
  
The first pause is used to stop poll() in the method paranoidPoll
The second one is attached because of p.partition().
I'm not sure whether the state of pause will be rewritten after these 
methods are called, so I use pause repeatedly.  


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21690: [SPARK-24713]AppMatser of spark streaming kafka OOM if t...

2018-07-02 Thread AmplabJenkins 
Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/21690
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21690: [SPARK-24713]AppMatser of spark streaming kafka OOM if t...

2018-07-02 Thread AmplabJenkins 
Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/21690
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92528/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21690: [SPARK-24713]AppMatser of spark streaming kafka OOM if t...

2018-07-02 Thread SparkQA 
Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/21690
  
**[Test build #92528 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92528/testReport)**
 for PR 21690 at commit 
[`15d03e6`](https://github.com/apache/spark/commit/15d03e63453e72675390c65ddf72492caefdadee).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21690: [SPARK-24713]AppMatser of spark streaming kafka OOM if t...

2018-07-02 Thread koeninger 
Github user koeninger commented on the issue:

https://github.com/apache/spark/pull/21690
  
@yuanboliu Can you clarify why repeated pause is necessary?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21690: [SPARK-24713]AppMatser of spark streaming kafka OOM if t...

2018-07-02 Thread SparkQA 
Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/21690
  
**[Test build #92528 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92528/testReport)**
 for PR 21690 at commit 
[`15d03e6`](https://github.com/apache/spark/commit/15d03e63453e72675390c65ddf72492caefdadee).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21690: [SPARK-24713]AppMatser of spark streaming kafka OOM if t...

2018-07-02 Thread koeninger 
Github user koeninger commented on the issue:

https://github.com/apache/spark/pull/21690
  
Jenkins, ok to test


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21690: [SPARK-24713]AppMatser of spark streaming kafka OOM if t...

2018-07-02 Thread yuanboliu 
Github user yuanboliu commented on the issue:

https://github.com/apache/spark/pull/21690
  
@koeninger Sorry to interrupt, would you please review my patch? Thanks in 
advance.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21690: [SPARK-24713]AppMatser of spark streaming kafka OOM if t...

2018-07-01 Thread AmplabJenkins 
Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/21690
  
Can one of the admins verify this patch?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21690: [SPARK-24713]AppMatser of spark streaming kafka OOM if t...

2018-07-01 Thread AmplabJenkins 
Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/21690
  
Can one of the admins verify this patch?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21690: [SPARK-24713]AppMatser of spark streaming kafka OOM if t...

2018-07-01 Thread AmplabJenkins 
Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/21690
  
Can one of the admins verify this patch?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21690: [SPARK-24713]AppMatser of spark streaming kafka OOM if t...

2018-07-01 Thread yuanboliu 
Github user yuanboliu commented on the issue:

https://github.com/apache/spark/pull/21690
  
Can one of the admins verify this patch?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org