[jira] [Commented] (KAFKA-8037) KTable restore may load bad data

2020-07-28 Thread Guozhang Wang (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-8037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17166762#comment-17166762 ] Guozhang Wang commented on KAFKA-8037: -- {quote} Sure, this would hold if a user specifies the new

[jira] [Commented] (KAFKA-8037) KTable restore may load bad data

2020-07-28 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-8037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17166703#comment-17166703 ] Matthias J. Sax commented on KAFKA-8037: {quote}Per the interaction: I'm actually suggesting that

[jira] [Commented] (KAFKA-8037) KTable restore may load bad data

2020-07-28 Thread Guozhang Wang (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-8037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17166602#comment-17166602 ] Guozhang Wang commented on KAFKA-8037: -- Per the interaction: I'm actually suggesting that the

[jira] [Commented] (KAFKA-8037) KTable restore may load bad data

2020-07-28 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-8037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17166592#comment-17166592 ] Matthias J. Sax commented on KAFKA-8037: Thanks [~guozhang]. Overall, SGTM. {quote}During IQ the

[jira] [Commented] (KAFKA-8037) KTable restore may load bad data

2020-07-27 Thread Guozhang Wang (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-8037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17166090#comment-17166090 ] Guozhang Wang commented on KAFKA-8037: -- Thank you all for providing your thoughts! In consolidating

[jira] [Commented] (KAFKA-8037) KTable restore may load bad data

2020-07-23 Thread Bruno Cadonna (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-8037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17163485#comment-17163485 ] Bruno Cadonna commented on KAFKA-8037: -- I find the idea of putting records in its original byte

[jira] [Commented] (KAFKA-8037) KTable restore may load bad data

2020-07-22 Thread Guozhang Wang (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-8037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17163204#comment-17163204 ] Guozhang Wang commented on KAFKA-8037: -- Here's a wild thought following [~agavra]'s last comment:

[jira] [Commented] (KAFKA-8037) KTable restore may load bad data

2020-07-22 Thread Sophie Blee-Goldman (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-8037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17163142#comment-17163142 ] Sophie Blee-Goldman commented on KAFKA-8037: Ah, thanks for bringing us back to the question

[jira] [Commented] (KAFKA-8037) KTable restore may load bad data

2020-07-22 Thread Almog Gavra (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-8037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17163135#comment-17163135 ] Almog Gavra commented on KAFKA-8037: > That said...in my (admiteddly anecdotal) experience, the

[jira] [Commented] (KAFKA-8037) KTable restore may load bad data

2020-07-22 Thread Guozhang Wang (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-8037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17163134#comment-17163134 ] Guozhang Wang commented on KAFKA-8037: -- Great comments from all of you, thank you so much! I'd like

[jira] [Commented] (KAFKA-8037) KTable restore may load bad data

2020-07-22 Thread Almog Gavra (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-8037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17163133#comment-17163133 ] Almog Gavra commented on KAFKA-8037: > Can you give a specific example of how things would break due

[jira] [Commented] (KAFKA-8037) KTable restore may load bad data

2020-07-22 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-8037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17163126#comment-17163126 ] Matthias J. Sax commented on KAFKA-8037: {quote}It is currently true that this is _only_ used to

[jira] [Commented] (KAFKA-8037) KTable restore may load bad data

2020-07-22 Thread Sophie Blee-Goldman (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-8037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17163096#comment-17163096 ] Sophie Blee-Goldman commented on KAFKA-8037: By the way, I'm not trying to advocate strongly

[jira] [Commented] (KAFKA-8037) KTable restore may load bad data

2020-07-22 Thread Sophie Blee-Goldman (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-8037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17163090#comment-17163090 ] Sophie Blee-Goldman commented on KAFKA-8037: [~vvcephei]I think you're referring to the

[jira] [Commented] (KAFKA-8037) KTable restore may load bad data

2020-07-22 Thread Sophie Blee-Goldman (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-8037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17163085#comment-17163085 ] Sophie Blee-Goldman commented on KAFKA-8037: > many users might not even know if their serde

[jira] [Commented] (KAFKA-8037) KTable restore may load bad data

2020-07-22 Thread Almog Gavra (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-8037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17163021#comment-17163021 ] Almog Gavra commented on KAFKA-8037: There are lots of threads going on in this discussion, but re:

[jira] [Commented] (KAFKA-8037) KTable restore may load bad data

2020-07-22 Thread John Roesler (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-8037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17163020#comment-17163020 ] John Roesler commented on KAFKA-8037: - Thanks [~ableegoldman] for the thoughts. It seems like we're

[jira] [Commented] (KAFKA-8037) KTable restore may load bad data

2020-07-22 Thread John Roesler (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-8037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17163012#comment-17163012 ] John Roesler commented on KAFKA-8037: - Thanks for the reply [~guozhang] . I think we're on the same

[jira] [Commented] (KAFKA-8037) KTable restore may load bad data

2020-07-22 Thread Sophie Blee-Goldman (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-8037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17163011#comment-17163011 ] Sophie Blee-Goldman commented on KAFKA-8037: I'm tempted to agree with Matthias that

[jira] [Commented] (KAFKA-8037) KTable restore may load bad data

2020-07-22 Thread Guozhang Wang (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-8037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17163002#comment-17163002 ] Guozhang Wang commented on KAFKA-8037: -- Hey guys thanks for your input. I think [~vvcephei] has a

[jira] [Commented] (KAFKA-8037) KTable restore may load bad data

2020-07-21 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-8037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17161796#comment-17161796 ] Matthias J. Sax commented on KAFKA-8037: I believe that the optimization is still worth to have.

[jira] [Commented] (KAFKA-8037) KTable restore may load bad data

2020-07-20 Thread John Roesler (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-8037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17161675#comment-17161675 ] John Roesler commented on KAFKA-8037: - Thanks for your thoughts, [~guozhang] . It really seems like

[jira] [Commented] (KAFKA-8037) KTable restore may load bad data

2020-07-20 Thread Guozhang Wang (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-8037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17161565#comment-17161565 ] Guozhang Wang commented on KAFKA-8037: -- There are some more discussions on related issues in

[jira] [Commented] (KAFKA-8037) KTable restore may load bad data

2020-07-20 Thread Almog Gavra (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-8037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17161440#comment-17161440 ] Almog Gavra commented on KAFKA-8037: I'm not totally sure I'm understanding the solutions, but

[jira] [Commented] (KAFKA-8037) KTable restore may load bad data

2020-02-21 Thread Sophie Blee-Goldman (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-8037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17042347#comment-17042347 ] Sophie Blee-Goldman commented on KAFKA-8037: Right, duh, we definitely wouldn't want the

[jira] [Commented] (KAFKA-8037) KTable restore may load bad data

2020-02-21 Thread Guozhang Wang (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-8037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17042315#comment-17042315 ] Guozhang Wang commented on KAFKA-8037: -- Not sure I fully understand... the committed offset records

[jira] [Commented] (KAFKA-8037) KTable restore may load bad data

2020-02-21 Thread Sophie Blee-Goldman (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-8037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17042181#comment-17042181 ] Sophie Blee-Goldman commented on KAFKA-8037: That's a good thought actually – I'm not sure

[jira] [Commented] (KAFKA-8037) KTable restore may load bad data

2020-02-21 Thread Guozhang Wang (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-8037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17042172#comment-17042172 ] Guozhang Wang commented on KAFKA-8037: -- This is an interesting idea. As for whether that blocks on

[jira] [Commented] (KAFKA-8037) KTable restore may load bad data

2020-02-12 Thread Sophie Blee-Goldman (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-8037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17035541#comment-17035541 ] Sophie Blee-Goldman commented on KAFKA-8037: Ha, yes good catch I'll fix the typo in 2)

[jira] [Commented] (KAFKA-8037) KTable restore may load bad data

2020-02-12 Thread Patrik Kleindl (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-8037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17035215#comment-17035215 ] Patrik Kleindl commented on KAFKA-8037: --- [~ableegoldman] Interesting proposal. At 2. you mean

[jira] [Commented] (KAFKA-8037) KTable restore may load bad data

2020-02-11 Thread Sophie Blee-Goldman (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-8037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17034746#comment-17034746 ] Sophie Blee-Goldman commented on KAFKA-8037: [~pkleindl] [~vvcephei] [~mjsax] [~guozhang] I

[jira] [Commented] (KAFKA-8037) KTable restore may load bad data

2020-01-13 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-8037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17014149#comment-17014149 ] ASF GitHub Bot commented on KAFKA-8037: --- pkleindl commented on pull request #7945: KAFKA-8037: WIP

[jira] [Commented] (KAFKA-8037) KTable restore may load bad data

2020-01-09 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-8037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17012268#comment-17012268 ] ASF GitHub Bot commented on KAFKA-8037: --- pkleindl commented on pull request #7923: KAFKA-8037:

[jira] [Commented] (KAFKA-8037) KTable restore may load bad data

2020-01-09 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-8037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17012264#comment-17012264 ] ASF GitHub Bot commented on KAFKA-8037: --- pkleindl commented on pull request #6398: KAFKA-8037:

[jira] [Commented] (KAFKA-8037) KTable restore may load bad data

2020-01-04 Thread John Roesler (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-8037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17008094#comment-17008094 ] John Roesler commented on KAFKA-8037: - Hey, all. If I understood, the proposal is something like:

[jira] [Commented] (KAFKA-8037) KTable restore may load bad data

2020-01-02 Thread Guozhang Wang (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-8037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17007093#comment-17007093 ] Guozhang Wang commented on KAFKA-8037: -- [~pkleindl] for case 2) all the records before the committed

[jira] [Commented] (KAFKA-8037) KTable restore may load bad data

2020-01-02 Thread Sophie Blee-Goldman (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-8037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17007062#comment-17007062 ] Sophie Blee-Goldman commented on KAFKA-8037: > Or should we just transit to normal processing

[jira] [Commented] (KAFKA-8037) KTable restore may load bad data

2020-01-02 Thread Patrik Kleindl (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-8037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17007020#comment-17007020 ] Patrik Kleindl commented on KAFKA-8037: --- [~guozhang] I was about to ask in the thread of KAFKA-7663

[jira] [Commented] (KAFKA-8037) KTable restore may load bad data

2020-01-02 Thread Guozhang Wang (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-8037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17006992#comment-17006992 ] Guozhang Wang commented on KAFKA-8037: -- For non-global KTables, it now makes me thinking if our

[jira] [Commented] (KAFKA-8037) KTable restore may load bad data

2019-03-07 Thread Patrik Kleindl (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-8037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16787319#comment-16787319 ] Patrik Kleindl commented on KAFKA-8037: --- Sorry for the mess, I thought I had to re-create the PR

[jira] [Commented] (KAFKA-8037) KTable restore may load bad data

2019-03-07 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-8037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16787305#comment-16787305 ] ASF GitHub Bot commented on KAFKA-8037: --- pkleindl commented on pull request #6394: KAFKA-8037:

[jira] [Commented] (KAFKA-8037) KTable restore may load bad data

2019-03-07 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-8037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16787302#comment-16787302 ] ASF GitHub Bot commented on KAFKA-8037: --- pkleindl commented on pull request #6394: KAFKA-8037:

[jira] [Commented] (KAFKA-8037) KTable restore may load bad data

2019-03-07 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-8037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16787301#comment-16787301 ] ASF GitHub Bot commented on KAFKA-8037: --- pkleindl commented on pull request #6394: KAFKA-8037:

[jira] [Commented] (KAFKA-8037) KTable restore may load bad data

2019-03-07 Thread Patrik Kleindl (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-8037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16787240#comment-16787240 ] Patrik Kleindl commented on KAFKA-8037: --- [~mjsax] [~guozhang] Added 

[jira] [Commented] (KAFKA-8037) KTable restore may load bad data

2019-03-06 Thread Patrik Kleindl (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-8037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16785420#comment-16785420 ] Patrik Kleindl commented on KAFKA-8037: --- [~guozhang] Does that work too if

[jira] [Commented] (KAFKA-8037) KTable restore may load bad data

2019-03-05 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-8037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16785127#comment-16785127 ] Guozhang Wang commented on KAFKA-8037: -- We use `offsetLimit` in ProcessorStateManager for source