[jira] [Commented] (HUDI-184) Integrate Hudi with Apache Flink

2019-11-27 Thread vinoyang (Jira)
[ https://issues.apache.org/jira/browse/HUDI-184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16983313#comment-16983313 ] vinoyang commented on HUDI-184: --- [~vinoth] I wrote more thought in a Google doc, please see h

[jira] [Commented] (HUDI-184) Integrate Hudi with Apache Flink

2019-11-25 Thread Vinoth Chandar (Jira)
[ https://issues.apache.org/jira/browse/HUDI-184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16982107#comment-16982107 ] Vinoth Chandar commented on HUDI-184: - Okay cool then. On using Flink’s state as index,

[jira] [Commented] (HUDI-184) Integrate Hudi with Apache Flink

2019-11-25 Thread vinoyang (Jira)
[ https://issues.apache.org/jira/browse/HUDI-184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16982081#comment-16982081 ] vinoyang commented on HUDI-184: --- {quote}In spark, we can run two spark jobs  (i.e the jobs ta

[jira] [Commented] (HUDI-184) Integrate Hudi with Apache Flink

2019-11-25 Thread Vinoth Chandar (Jira)
[ https://issues.apache.org/jira/browse/HUDI-184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16981964#comment-16981964 ] Vinoth Chandar commented on HUDI-184: - >> Yes, It seems the ingestion and compaction st

[jira] [Commented] (HUDI-184) Integrate Hudi with Apache Flink

2019-11-25 Thread vinoyang (Jira)
[ https://issues.apache.org/jira/browse/HUDI-184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16981649#comment-16981649 ] vinoyang commented on HUDI-184: --- [~vinoth] Based on limited observation, answer your two ques

[jira] [Commented] (HUDI-184) Integrate Hudi with Apache Flink

2019-11-22 Thread vinoyang (Jira)
[ https://issues.apache.org/jira/browse/HUDI-184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16980247#comment-16980247 ] vinoyang commented on HUDI-184: --- [~vinoth] I have started to think about some questions and v

[jira] [Commented] (HUDI-184) Integrate Hudi with Apache Flink

2019-11-22 Thread Vinoth Chandar (Jira)
[ https://issues.apache.org/jira/browse/HUDI-184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16980117#comment-16980117 ] Vinoth Chandar commented on HUDI-184: - We can start with streaming APIs IMO. I think Ba

[jira] [Commented] (HUDI-184) Integrate Hudi with Apache Flink

2019-11-22 Thread vinoyang (Jira)
[ https://issues.apache.org/jira/browse/HUDI-184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16979954#comment-16979954 ] vinoyang commented on HUDI-184: --- [~taherk77] Thanks for sharing your thought. I agree that we

[jira] [Commented] (HUDI-184) Integrate Hudi with Apache Flink

2019-11-21 Thread Taher Koitawala (Jira)
[ https://issues.apache.org/jira/browse/HUDI-184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16979838#comment-16979838 ] Taher Koitawala commented on HUDI-184: -- [~yanghua] In terms of Streaming I do not have

[jira] [Commented] (HUDI-184) Integrate Hudi with Apache Flink

2019-11-20 Thread vinoyang (Jira)
[ https://issues.apache.org/jira/browse/HUDI-184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16978185#comment-16978185 ] vinoyang commented on HUDI-184: --- [~taherk77] Aligning with you, have you got a original plan

[jira] [Commented] (HUDI-184) Integrate Hudi with Apache Flink

2019-11-18 Thread vinoyang (Jira)
[ https://issues.apache.org/jira/browse/HUDI-184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16977141#comment-16977141 ] vinoyang commented on HUDI-184: --- [~taherk77] I know you have provided some thoughts before. M

[jira] [Commented] (HUDI-184) Integrate Hudi with Apache Flink

2019-11-18 Thread Taher Koitawala (Jira)
[ https://issues.apache.org/jira/browse/HUDI-184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16977123#comment-16977123 ] Taher Koitawala commented on HUDI-184: -- Should we like try a small mock up of it? > I

[jira] [Commented] (HUDI-184) Integrate Hudi with Apache Flink

2019-11-18 Thread Vinoth Chandar (Jira)
[ https://issues.apache.org/jira/browse/HUDI-184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16977120#comment-16977120 ] Vinoth Chandar commented on HUDI-184: - Yes. I remember. But I was and is still unclear

[jira] [Commented] (HUDI-184) Integrate Hudi with Apache Flink

2019-11-18 Thread Taher Koitawala (Jira)
[ https://issues.apache.org/jira/browse/HUDI-184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16977101#comment-16977101 ] Taher Koitawala commented on HUDI-184: -- [~vinoth] The index checkpointed to dfs was so

[jira] [Commented] (HUDI-184) Integrate Hudi with Apache Flink

2019-11-18 Thread Vinoth Chandar (Jira)
[ https://issues.apache.org/jira/browse/HUDI-184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16977098#comment-16977098 ] Vinoth Chandar commented on HUDI-184: - Interesting idea. [~yanghua] so IIUC, we can map

[jira] [Commented] (HUDI-184) Integrate Hudi with Apache Flink

2019-11-18 Thread Taher Koitawala (Jira)
[ https://issues.apache.org/jira/browse/HUDI-184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16977092#comment-16977092 ] Taher Koitawala commented on HUDI-184: -- [~vinoth] and [~vinoyang] I agree we can do

[jira] [Commented] (HUDI-184) Integrate Hudi with Apache Flink

2019-11-18 Thread vinoyang (Jira)
[ https://issues.apache.org/jira/browse/HUDI-184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16977090#comment-16977090 ] vinoyang commented on HUDI-184: --- bq. Also there are probably some bigger questions to answer

[jira] [Commented] (HUDI-184) Integrate Hudi with Apache Flink

2019-11-18 Thread Vinoth Chandar (Jira)
[ https://issues.apache.org/jira/browse/HUDI-184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16976791#comment-16976791 ] Vinoth Chandar commented on HUDI-184: - [~yanghua]  been thinking a bit around what we c

[jira] [Commented] (HUDI-184) Integrate Hudi with Apache Flink

2019-08-22 Thread vinoyang (Jira)
[ https://issues.apache.org/jira/browse/HUDI-184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16913145#comment-16913145 ] vinoyang commented on HUDI-184: --- Hi [~grapebaba] I am still investigating the matching of fea

[jira] [Commented] (HUDI-184) Integrate Hudi with Apache Flink

2019-08-22 Thread Kai Chen (Jira)
[ https://issues.apache.org/jira/browse/HUDI-184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16913121#comment-16913121 ] Kai Chen commented on HUDI-184: --- [~taherk77] [~yanghua] What's the progress for this task?Any