[GitHub] spark issue #15397: [SPARK-17834][SQL]Fetch the earliest offsets manually in...

2016-10-13 Thread zsxwing
Github user zsxwing commented on the issue:

https://github.com/apache/spark/pull/15397
  
Thanks! Merging to master and 2.0. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark issue #15397: [SPARK-17834][SQL]Fetch the earliest offsets manually in...

2016-10-13 Thread koeninger
Github user koeninger commented on the issue:

https://github.com/apache/spark/pull/15397
  
LGTM, thanks for talking it through


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark issue #15397: [SPARK-17834][SQL]Fetch the earliest offsets manually in...

2016-10-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15397
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark issue #15397: [SPARK-17834][SQL]Fetch the earliest offsets manually in...

2016-10-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15397
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66908/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark issue #15397: [SPARK-17834][SQL]Fetch the earliest offsets manually in...

2016-10-13 Thread SparkQA
Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15397
  
**[Test build #66908 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66908/consoleFull)**
 for PR 15397 at commit 
[`7986f18`](https://github.com/apache/spark/commit/7986f185649ad474b7f22bb0930e0d4a9e992a91).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark issue #15397: [SPARK-17834][SQL]Fetch the earliest offsets manually in...

2016-10-13 Thread SparkQA
Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15397
  
**[Test build #66908 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66908/consoleFull)**
 for PR 15397 at commit 
[`7986f18`](https://github.com/apache/spark/commit/7986f185649ad474b7f22bb0930e0d4a9e992a91).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark issue #15397: [SPARK-17834][SQL]Fetch the earliest offsets manually in...

2016-10-13 Thread zsxwing
Github user zsxwing commented on the issue:

https://github.com/apache/spark/pull/15397
  
> My main point is that whoever implements SPARK-17812 is going to have to 
deal with the issue shown in SPARK-17782, which means much of this patch is 
going to need to be changed anyway.

@koeninger I agreed that this patch will be changed. However, this PR does 
fix a known issue for the current supported features and there is not user 
facing changes. Considering 2.0.2 may come out soon and I don't think 
SPARK-17812 will be done soon, I would like to merge this to fix issues for 
2.0.2. What do you think?





---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark issue #15397: [SPARK-17834][SQL]Fetch the earliest offsets manually in...

2016-10-12 Thread koeninger
Github user koeninger commented on the issue:

https://github.com/apache/spark/pull/15397
  
My main point is that whoever implements SPARK-17812 is going to have to 
deal with the issue shown in SPARK-17782, which means much of this patch is 
going to need to be changed anyway.

But It's not just about external group id.  Committed offsets would 
actually make the issue in SPARK-17782 less of a problem, because they would 
take precedence over auto.offset.reset


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark issue #15397: [SPARK-17834][SQL]Fetch the earliest offsets manually in...

2016-10-12 Thread SparkQA
Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15397
  
**[Test build #66849 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66849/consoleFull)**
 for PR 15397 at commit 
[`9578555`](https://github.com/apache/spark/commit/957855512083741e0421c3124c1904479fe5f9ce).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark issue #15397: [SPARK-17834][SQL]Fetch the earliest offsets manually in...

2016-10-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15397
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark issue #15397: [SPARK-17834][SQL]Fetch the earliest offsets manually in...

2016-10-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15397
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66849/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark issue #15397: [SPARK-17834][SQL]Fetch the earliest offsets manually in...

2016-10-12 Thread zsxwing
Github user zsxwing commented on the issue:

https://github.com/apache/spark/pull/15397
  
@koeninger sorry for the delay. Right now KafkaSource doesn't support 
external group id, so we don't need to concern about how to fetching committed 
offsets. Any other cases that I'm missing? 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark issue #15397: [SPARK-17834][SQL]Fetch the earliest offsets manually in...

2016-10-12 Thread SparkQA
Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15397
  
**[Test build #66849 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66849/consoleFull)**
 for PR 15397 at commit 
[`9578555`](https://github.com/apache/spark/commit/957855512083741e0421c3124c1904479fe5f9ce).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark issue #15397: [SPARK-17834][SQL]Fetch the earliest offsets manually in...

2016-10-10 Thread koeninger
Github user koeninger commented on the issue:

https://github.com/apache/spark/pull/15397
  
Look at the poll/seek implementation in the DStream's subscribe and
subscribe pattern when user offsets are provided, i.e. the problem that
triggered this ticket to begin with. You're going to have to solve the same
problem there with the structured stream, unless the structured stream
somehow wants to limit assigning specific partitions only to the assign
strategy, which eliminates lots of valid use cases.


On Sun, Oct 9, 2016 at 11:41 PM, Shixiong Zhu 
wrote:

> How is this going to work with assign? It seems like it's just avoiding
> the problem, not fixing it.
>
> We can seek to the offsets provided by the user.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> , or 
mute
> the thread
> 

> .
>



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark issue #15397: [SPARK-17834][SQL]Fetch the earliest offsets manually in...

2016-10-09 Thread zsxwing
Github user zsxwing commented on the issue:

https://github.com/apache/spark/pull/15397
  
> How is this going to work with assign? It seems like it's just avoiding 
the problem, not fixing it.

We can seek to the offsets provided by the user.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark issue #15397: [SPARK-17834][SQL]Fetch the earliest offsets manually in...

2016-10-07 Thread koeninger
Github user koeninger commented on the issue:

https://github.com/apache/spark/pull/15397
  
How is this going to work with assign?  It seems like it's just avoiding 
the problem, not fixing it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark issue #15397: [SPARK-17834][SQL]Fetch the earliest offsets manually in...

2016-10-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15397
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark issue #15397: [SPARK-17834][SQL]Fetch the earliest offsets manually in...

2016-10-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15397
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66543/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark issue #15397: [SPARK-17834][SQL]Fetch the earliest offsets manually in...

2016-10-07 Thread SparkQA
Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15397
  
**[Test build #66543 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66543/consoleFull)**
 for PR 15397 at commit 
[`95a0c96`](https://github.com/apache/spark/commit/95a0c96f040530f5eec270a2b53b46d195a94b8c).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark issue #15397: [SPARK-17834][SQL]Fetch the earliest offsets manually in...

2016-10-07 Thread SparkQA
Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15397
  
**[Test build #66543 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66543/consoleFull)**
 for PR 15397 at commit 
[`95a0c96`](https://github.com/apache/spark/commit/95a0c96f040530f5eec270a2b53b46d195a94b8c).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark issue #15397: [SPARK-17834][SQL]Fetch the earliest offsets manually in...

2016-10-07 Thread zsxwing
Github user zsxwing commented on the issue:

https://github.com/apache/spark/pull/15397
  
/cc @tdas @koeninger


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org