[jira] [Commented] (SPARK-18386) Batch mode SQL source for Kafka

[ https://issues.apache.org/jira/browse/SPARK-18386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15653388#comment-15653388 ] Ofir Manor commented on SPARK-18386: This would be really useful for me right now! We

[jira] [Commented] (SPARK-18057) Update structured streaming kafka from 10.0.1 to 10.1.0

[ https://issues.apache.org/jira/browse/SPARK-18057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15653415#comment-15653415 ] Ofir Manor commented on SPARK-18057: This is great, especially if we follow up by add

[jira] [Commented] (SPARK-18386) Batch mode SQL source for Kafka

[ https://issues.apache.org/jira/browse/SPARK-18386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15653821#comment-15653821 ] Ofir Manor commented on SPARK-18386: BTW [~c...@koeninger.org] - I think that filteri

[jira] [Commented] (SPARK-18475) Be able to provide higher parallelization for StructuredStreaming Kafka Source

[ https://issues.apache.org/jira/browse/SPARK-18475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15674092#comment-15674092 ] Ofir Manor commented on SPARK-18475: Are you sure this is working? Having a visible p

[jira] [Commented] (SPARK-18475) Be able to provide higher parallelization for StructuredStreaming Kafka Source

[ https://issues.apache.org/jira/browse/SPARK-18475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15674628#comment-15674628 ] Ofir Manor commented on SPARK-18475: I was just wondering if it actually works, but i

[jira] [Commented] (SPARK-18475) Be able to provide higher parallelization for StructuredStreaming Kafka Source

[ https://issues.apache.org/jira/browse/SPARK-18475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15680780#comment-15680780 ] Ofir Manor commented on SPARK-18475: I understand, but I disagree with you. I think y

[jira] [Commented] (SPARK-18475) Be able to provide higher parallelization for StructuredStreaming Kafka Source

[ https://issues.apache.org/jira/browse/SPARK-18475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15688246#comment-15688246 ] Ofir Manor commented on SPARK-18475: Cody, for me your are the main gatekeeper for ev

[jira] [Commented] (SPARK-18057) Update structured streaming kafka from 10.0.1 to 10.2.0

[ https://issues.apache.org/jira/browse/SPARK-18057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15884619#comment-15884619 ] Ofir Manor commented on SPARK-18057: [~c...@koeninger.org] I think my previous issue

[jira] [Commented] (SPARK-18057) Update structured streaming kafka from 10.0.1 to 10.2.0

[ https://issues.apache.org/jira/browse/SPARK-18057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15884790#comment-15884790 ] Ofir Manor commented on SPARK-18057: Sure - [~marmbrus] - what do you think? it resol

[jira] [Commented] (SPARK-15406) Structured streaming support for consuming from Kafka

[ https://issues.apache.org/jira/browse/SPARK-15406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15488487#comment-15488487 ] Ofir Manor commented on SPARK-15406: Cody, I think you are right. Now is the right ti

[jira] [Commented] (SPARK-15406) Structured streaming support for consuming from Kafka

[ https://issues.apache.org/jira/browse/SPARK-15406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15548350#comment-15548350 ] Ofir Manor commented on SPARK-15406: Checking back - is there any progress on the API

[jira] [Commented] (SPARK-15406) Structured streaming support for consuming from Kafka

[ https://issues.apache.org/jira/browse/SPARK-15406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15553076#comment-15553076 ] Ofir Manor commented on SPARK-15406: Thanks Cody! I missed the activity in the subtas

[jira] [Commented] (SPARK-15406) Structured streaming support for consuming from Kafka

[ https://issues.apache.org/jira/browse/SPARK-15406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15553161#comment-15553161 ] Ofir Manor commented on SPARK-15406: Alright, I do hope you all find a way to resolve

[jira] [Commented] (SPARK-15406) Structured streaming support for consuming from Kafka

[ https://issues.apache.org/jira/browse/SPARK-15406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15553372#comment-15553372 ] Ofir Manor commented on SPARK-15406: Thanks Michael, I'll check the JIRAs, and will a

[jira] [Commented] (SPARK-15406) Structured streaming support for consuming from Kafka

[ https://issues.apache.org/jira/browse/SPARK-15406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15553613#comment-15553613 ] Ofir Manor commented on SPARK-15406: I see three somewhat-related issues: 1. More fle

[jira] [Commented] (SPARK-17815) Report committed offsets

[ https://issues.apache.org/jira/browse/SPARK-17815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15553662#comment-15553662 ] Ofir Manor commented on SPARK-17815: I think this is a good idea.There is a minor con

[jira] [Commented] (SPARK-15406) Structured streaming support for consuming from Kafka

[ https://issues.apache.org/jira/browse/SPARK-15406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15554426#comment-15554426 ] Ofir Manor commented on SPARK-15406: Thanks Micahel for the checkpoint location expla

[jira] [Commented] (SPARK-17815) Report committed offsets

[ https://issues.apache.org/jira/browse/SPARK-17815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15559541#comment-15559541 ] Ofir Manor commented on SPARK-17815: As far as I understand, there is a clear single-

[jira] [Commented] (SPARK-17815) Report committed offsets

[ https://issues.apache.org/jira/browse/SPARK-17815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15559549#comment-15559549 ] Ofir Manor commented on SPARK-17815: Good, we are on the same page, no argument that

[jira] [Commented] (SPARK-17815) Report committed offsets

[ https://issues.apache.org/jira/browse/SPARK-17815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15560661#comment-15560661 ] Ofir Manor commented on SPARK-17815: I hope others with better understanding of Spark

[jira] [Commented] (SPARK-17812) More granular control of starting offsets

[ https://issues.apache.org/jira/browse/SPARK-17812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15560696#comment-15560696 ] Ofir Manor commented on SPARK-17812: Cody, you are absolutely right that the option n

[jira] [Comment Edited] (SPARK-17812) More granular control of starting offsets

[ https://issues.apache.org/jira/browse/SPARK-17812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15560696#comment-15560696 ] Ofir Manor edited comment on SPARK-17812 at 10/9/16 10:00 PM: -

[jira] [Commented] (SPARK-17815) Report committed offsets

[ https://issues.apache.org/jira/browse/SPARK-17815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15560749#comment-15560749 ] Ofir Manor commented on SPARK-17815: Thanks Cody, this is much clearer. (BTW - I've b

[jira] [Commented] (SPARK-17812) More granular control of starting offsets

[ https://issues.apache.org/jira/browse/SPARK-17812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15560820#comment-15560820 ] Ofir Manor commented on SPARK-17812: I'll have to think about it - not sure if it eve

[jira] [Commented] (SPARK-17812) More granular control of starting offsets

[ https://issues.apache.org/jira/browse/SPARK-17812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15560865#comment-15560865 ] Ofir Manor commented on SPARK-17812: Why? If you choose oldest, latest or by timestam

[jira] [Commented] (SPARK-17815) Report committed offsets

[ https://issues.apache.org/jira/browse/SPARK-17815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15560906#comment-15560906 ] Ofir Manor commented on SPARK-17815: I totally understand your concerns, and yes, I h

[jira] [Commented] (SPARK-17812) More granular control of starting offsets

[ https://issues.apache.org/jira/browse/SPARK-17812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15563273#comment-15563273 ] Ofir Manor commented on SPARK-17812: If you think it is useful, you should definitely

[jira] [Commented] (SPARK-17812) More granular control of starting offsets (assign)

[ https://issues.apache.org/jira/browse/SPARK-17812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15573109#comment-15573109 ] Ofir Manor commented on SPARK-17812: Regarding (1) - of course it is *all* data in th

[jira] [Commented] (SPARK-17812) More granular control of starting offsets (assign)

[ https://issues.apache.org/jira/browse/SPARK-17812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15573341#comment-15573341 ] Ofir Manor commented on SPARK-17812: Thanks Cody! great to have a concrete example. I

[jira] [Commented] (SPARK-17812) More granular control of starting offsets (assign)

[ https://issues.apache.org/jira/browse/SPARK-17812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15573457#comment-15573457 ] Ofir Manor commented on SPARK-17812: I'm with you - I warned you it is bikeshedding..

[jira] [Commented] (SPARK-17812) More granular control of starting offsets (assign)

[ https://issues.apache.org/jira/browse/SPARK-17812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15578699#comment-15578699 ] Ofir Manor commented on SPARK-17812: I think Michael suggest that if you use {{starti

[jira] [Commented] (SPARK-15406) Structured streaming support for consuming from Kafka

[ https://issues.apache.org/jira/browse/SPARK-15406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15453289#comment-15453289 ] Ofir Manor commented on SPARK-15406: Cody, why do you think Structured Streaming supp

[jira] [Commented] (SPARK-15406) Structured streaming support for consuming from Kafka

[ https://issues.apache.org/jira/browse/SPARK-15406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15453412#comment-15453412 ] Ofir Manor commented on SPARK-15406: For me - structured streaming is currently all a