Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/15102
> Merged this to master, @zsxwing do you have another PR for 2.0.
See #15367
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well.
Github user tdas commented on the issue:
https://github.com/apache/spark/pull/15102
Merged this to master, @zsxwing do you have another PR for 2.0.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15102
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66398/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15102
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
e
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15102
**[Test build #66398 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66398/consoleFull)**
for PR 15102 at commit
[`4754125`](https://github.com/apache/spark/commit/
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15102
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
e
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15102
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66397/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15102
**[Test build #66397 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66397/consoleFull)**
for PR 15102 at commit
[`7d658f1`](https://github.com/apache/spark/commit/
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15102
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66396/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15102
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
e
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15102
**[Test build #66396 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66396/consoleFull)**
for PR 15102 at commit
[`d9d848c`](https://github.com/apache/spark/commit/
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15102
**[Test build #66398 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66398/consoleFull)**
for PR 15102 at commit
[`4754125`](https://github.com/apache/spark/commit/4
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15102
**[Test build #66397 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66397/consoleFull)**
for PR 15102 at commit
[`7d658f1`](https://github.com/apache/spark/commit/7
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15102
**[Test build #66396 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66396/consoleFull)**
for PR 15102 at commit
[`d9d848c`](https://github.com/apache/spark/commit/d
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15102
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66345/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15102
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
e
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15102
**[Test build #66345 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66345/consoleFull)**
for PR 15102 at commit
[`4316906`](https://github.com/apache/spark/commit/
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15102
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
e
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15102
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66338/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15102
**[Test build #66338 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66338/consoleFull)**
for PR 15102 at commit
[`d50a05e`](https://github.com/apache/spark/commit/
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15102
**[Test build #66345 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66345/consoleFull)**
for PR 15102 at commit
[`4316906`](https://github.com/apache/spark/commit/4
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15102
**[Test build #66338 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66338/consoleFull)**
for PR 15102 at commit
[`d50a05e`](https://github.com/apache/spark/commit/d
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15102
**[Test build #3294 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3294/consoleFull)**
for PR 15102 at commit
[`a6c4970`](https://github.com/apache/spark/commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15102
**[Test build #3294 has
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3294/consoleFull)**
for PR 15102 at commit
[`a6c4970`](https://github.com/apache/spark/commit/
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15102
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66289/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15102
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
e
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15102
**[Test build #66289 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66289/consoleFull)**
for PR 15102 at commit
[`a6c4970`](https://github.com/apache/spark/commit/
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15102
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
e
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15102
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66288/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15102
**[Test build #66288 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66288/consoleFull)**
for PR 15102 at commit
[`7ff1059`](https://github.com/apache/spark/commit/
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15102
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66285/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15102
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
e
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15102
**[Test build #66285 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66285/consoleFull)**
for PR 15102 at commit
[`ccadd81`](https://github.com/apache/spark/commit/
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15102
**[Test build #66289 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66289/consoleFull)**
for PR 15102 at commit
[`a6c4970`](https://github.com/apache/spark/commit/a
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15102
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
e
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15102
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66283/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15102
**[Test build #66283 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66283/consoleFull)**
for PR 15102 at commit
[`77208d1`](https://github.com/apache/spark/commit/
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15102
**[Test build #66288 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66288/consoleFull)**
for PR 15102 at commit
[`7ff1059`](https://github.com/apache/spark/commit/7
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15102
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
e
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15102
**[Test build #66287 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66287/consoleFull)**
for PR 15102 at commit
[`f78c990`](https://github.com/apache/spark/commit/
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15102
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66287/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15102
**[Test build #66287 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66287/consoleFull)**
for PR 15102 at commit
[`f78c990`](https://github.com/apache/spark/commit/f
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15102
**[Test build #66285 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66285/consoleFull)**
for PR 15102 at commit
[`ccadd81`](https://github.com/apache/spark/commit/c
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15102
**[Test build #66283 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66283/consoleFull)**
for PR 15102 at commit
[`77208d1`](https://github.com/apache/spark/commit/7
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15102
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
e
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15102
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66274/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15102
**[Test build #66274 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66274/consoleFull)**
for PR 15102 at commit
[`d154532`](https://github.com/apache/spark/commit/
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15102
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66265/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15102
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
e
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15102
**[Test build #66265 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66265/consoleFull)**
for PR 15102 at commit
[`e883062`](https://github.com/apache/spark/commit/
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15102
**[Test build #66274 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66274/consoleFull)**
for PR 15102 at commit
[`d154532`](https://github.com/apache/spark/commit/d
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15102
**[Test build #66265 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66265/consoleFull)**
for PR 15102 at commit
[`e883062`](https://github.com/apache/spark/commit/e
Github user koeninger commented on the issue:
https://github.com/apache/spark/pull/15102
> It would be nice to be able to do something other than earliest/latest.
That's what Assign and the starting offset arguments to the Subscribe
strategies are for. The implementation
Github user marmbrus commented on the issue:
https://github.com/apache/spark/pull/15102
I spent a while playing around with this today on a real cluster, and
overall it is pretty cool! I have a few suggestions we should implement in the
long run, but these can probably be done in fol
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15102
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
e
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15102
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66047/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15102
**[Test build #66047 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66047/consoleFull)**
for PR 15102 at commit
[`9d95d52`](https://github.com/apache/spark/commit/
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15102
**[Test build #66047 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66047/consoleFull)**
for PR 15102 at commit
[`9d95d52`](https://github.com/apache/spark/commit/9
Github user marmbrus commented on the issue:
https://github.com/apache/spark/pull/15102
FYI: #15274 adds support for parsing JSON from the key/value into a Spark
SQL `StructType`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15102
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
e
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15102
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66005/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15102
**[Test build #66005 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66005/consoleFull)**
for PR 15102 at commit
[`59a93a5`](https://github.com/apache/spark/commit/
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15102
**[Test build #66005 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66005/consoleFull)**
for PR 15102 at commit
[`59a93a5`](https://github.com/apache/spark/commit/5
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/15102
Just pushed to verify if the workaround for
https://issues.apache.org/jira/browse/KAFKA-1894 does work on Jenkins. Not
ready for another round of review.
---
If your project is set up for it, you
Github user koeninger commented on the issue:
https://github.com/apache/spark/pull/15102
Ok, so this kind of thing is why I was concerned about the copy, paste,
randomly change things approach to developing this module.
> (5) Topics are deleted when a Spark job is runinng, whi
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/15102
We can add an option to allow the user failing the query instead of just
logging the warning.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/15102
Right now since there are some arguments about how to handle various
failures, I'm listing what I found via stress test to discuss:
(1) Kafka APIs fail because we cannot connect to Kafka clu
Github user koeninger commented on the issue:
https://github.com/apache/spark/pull/15102
Ok, finished a line-by-line compare + comment.
The biggest thing I'm having trouble reconciling is the stated emphasis on
limiting user options in order to give guarantees, yet throwing th
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15102
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65937/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15102
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
e
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15102
**[Test build #65937 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65937/consoleFull)**
for PR 15102 at commit
[`852f607`](https://github.com/apache/spark/commit/
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15102
**[Test build #65937 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65937/consoleFull)**
for PR 15102 at commit
[`852f607`](https://github.com/apache/spark/commit/8
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15102
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
e
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15102
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65926/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15102
**[Test build #65926 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65926/consoleFull)**
for PR 15102 at commit
[`755ceaa`](https://github.com/apache/spark/commit/
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15102
**[Test build #65926 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65926/consoleFull)**
for PR 15102 at commit
[`755ceaa`](https://github.com/apache/spark/commit/7
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15102
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65845/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15102
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
e
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15102
**[Test build #65845 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65845/consoleFull)**
for PR 15102 at commit
[`5f33eb4`](https://github.com/apache/spark/commit/
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15102
**[Test build #65845 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65845/consoleFull)**
for PR 15102 at commit
[`5f33eb4`](https://github.com/apache/spark/commit/5
Github user koeninger commented on the issue:
https://github.com/apache/spark/pull/15102
Source.getOffset returns an sql Offset, not a kafka offset. At this point
the current plan is to remove the ordering requirement for sql Offset,
which makes the whole discussion of comparin
Github user jwbear commented on the issue:
https://github.com/apache/spark/pull/15102
Which offset does getOffset() return one from the partition or something
you created? Because it looked like when you were hashing them, you were
returning partition offsets.
---
If your project is
Github user koeninger commented on the issue:
https://github.com/apache/spark/pull/15102
We aren't comparing ordering of offsets across partitions, and I don't
think that was ever in consideration. At this point the most likely
candidate for the global ordering is implicit in t
Github user jwbear commented on the issue:
https://github.com/apache/spark/pull/15102
Just curious looking at this, if you are comparing "sequential" offsets
across partitions a rebalance would definitely affect this and, unless
something has changed, it probably not a good idea to co
Github user koeninger commented on the issue:
https://github.com/apache/spark/pull/15102
> I agree that if/when we add that ability to add existing partitions
midstream we'd probably need to add two offsets in to the SQL offset for new
partitions.
It's not just existing parti
Github user marmbrus commented on the issue:
https://github.com/apache/spark/pull/15102
> "I want to be able to add a topicpartition mid stream, but I don't want
to start it from the beginning."
I see, I was thinking only of new topics that appear that match your
pattern. I
Github user koeninger commented on the issue:
https://github.com/apache/spark/pull/15102
@tdas I think as long as marmbrus' PR to remove comparable from the
interface works for sane variations of subscription changes it's the best way
to go. I'm honestly fine with someone getting wha
Github user marmbrus commented on the issue:
https://github.com/apache/spark/pull/15102
Comparable requirement removed in #15207.
> I think in the absence of prior information about the position in a
topicpartition, you start a new batch on topic B starting from wherever the
Github user tdas commented on the issue:
https://github.com/apache/spark/pull/15102
@koeninger
I did some independent brainstorming with @zsxwing on topic deletion, and
yeah I agree with you that attempting to account for deleted topics in the
offset in the KafkaSourceOffset such
Github user koeninger commented on the issue:
https://github.com/apache/spark/pull/15102
> For streaming you already know what the global order is, because you know
when you asked for A and B. I agree that we should probably remove the
comparable requirement from Offset in favor of ju
Github user marmbrus commented on the issue:
https://github.com/apache/spark/pull/15102
For streaming you already know what the global order is, because you know
when you asked for A and B. I agree that we should probably remove the
comparable requirement from `Offset` in favor of ju
Github user koeninger commented on the issue:
https://github.com/apache/spark/pull/15102
@tdas moving this conversation back to the PR that's linked from the public
jira
> yeah, i am trying to figure out all the options and write up something to
so that we are clear on the pr
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15102
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65785/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15102
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
e
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15102
**[Test build #65785 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65785/consoleFull)**
for PR 15102 at commit
[`786af2f`](https://github.com/apache/spark/commit/
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15102
**[Test build #65785 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65785/consoleFull)**
for PR 15102 at commit
[`786af2f`](https://github.com/apache/spark/commit/7
Github user koeninger commented on the issue:
https://github.com/apache/spark/pull/15102
This is pretty much the fundamental issue. Kafka offsets alone aren't
capable of meeting the SQL Offset interface as defined. I think that means
the Offset interface needs to be reconsider
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/15102
> PR with failing test indicating at least one reason why it's wrong from
an end-user perspective:
@koeninger Thanks for writing the test. Yes, we are aware of this issue.
However, it's unl
Github user koeninger commented on the issue:
https://github.com/apache/spark/pull/15102
> I'd want to see some test cases though that show why the current
implementation is wrong from an end-user perspective if it needs to block
merging initial kafka support.
PR with failing
Github user marmbrus commented on the issue:
https://github.com/apache/spark/pull/15102
I asked @koeninger to clarify the specific suggestions he is referring to
above, here's my response:
> [Comments here and on JIRA relating to concerns with the `Offset`
implementation]
1 - 100 of 142 matches
Mail list logo