[jira] [Commented] (OOZIE-3594) Support Flink batch action
[ https://issues.apache.org/jira/browse/OOZIE-3594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17375669#comment-17375669 ] Andras Salamon commented on OOZIE-3594: --- [~zuston] This is the Jira about the hadoop-3 update: OOZIE-2973 I've already opened a Jira about the possibility of pig removal: OOZIE-3514 > Support Flink batch action > -- > > Key: OOZIE-3594 > URL: https://issues.apache.org/jira/browse/OOZIE-3594 > Project: Oozie > Issue Type: New Feature >Affects Versions: 5.1.0 >Reporter: Junfan Zhang >Assignee: Junfan Zhang >Priority: Major > > Spark and Hive action have been supported on Oozie since a long time ago. Now > Flink batch is productive, and i think it's necessary to integerate Flink > batch action on Oozie as a workflow node. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (OOZIE-3594) Support Flink batch action
[ https://issues.apache.org/jira/browse/OOZIE-3594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17375664#comment-17375664 ] Junfan Zhang commented on OOZIE-3594: - Agree more [~asalamon74] [~dionusos]. I will submit a proposal design doc later, and then we can talk more details. Besides, do you have some plan to remove sqooq or pig action in newer Oozie version, because they may block Oozie campatible with Hadoop3.x. > Support Flink batch action > -- > > Key: OOZIE-3594 > URL: https://issues.apache.org/jira/browse/OOZIE-3594 > Project: Oozie > Issue Type: New Feature >Affects Versions: 5.1.0 >Reporter: Junfan Zhang >Assignee: Junfan Zhang >Priority: Major > > Spark and Hive action have been supported on Oozie since a long time ago. Now > Flink batch is productive, and i think it's necessary to integerate Flink > batch action on Oozie as a workflow node. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (OOZIE-3594) Support Flink batch action
[ https://issues.apache.org/jira/browse/OOZIE-3594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17375472#comment-17375472 ] Andras Salamon commented on OOZIE-3594: --- Thanks [~zuston] for the proposal. I'm also not too familiar with Flink, but it is an active and popular project, I think a flink action would be a good improvement. We have a few actions which are a bit obsolete, pig is not too active, sqoop has retired, we need more active actions for Oozie. Oozie has a few 3rd party dependencies where we use rather old versions (hadoop, guava,...) this might cause problems here. As [~dionusos] mentioned earlier, this is a big task, we need not only the action, but documentation, example... > Support Flink batch action > -- > > Key: OOZIE-3594 > URL: https://issues.apache.org/jira/browse/OOZIE-3594 > Project: Oozie > Issue Type: New Feature >Affects Versions: 5.1.0 >Reporter: Junfan Zhang >Assignee: Junfan Zhang >Priority: Major > > Spark and Hive action have been supported on Oozie since a long time ago. Now > Flink batch is productive, and i think it's necessary to integerate Flink > batch action on Oozie as a workflow node. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (OOZIE-3594) Support Flink batch action
[ https://issues.apache.org/jira/browse/OOZIE-3594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17375471#comment-17375471 ] Dénes Bodó commented on OOZIE-3594: --- [~zuston] sorry for the late answer and thank you for the reminder. I would be happy to see Flink action in Oozie. +1 Thanks, Denes > Support Flink batch action > -- > > Key: OOZIE-3594 > URL: https://issues.apache.org/jira/browse/OOZIE-3594 > Project: Oozie > Issue Type: New Feature >Affects Versions: 5.1.0 >Reporter: Junfan Zhang >Assignee: Junfan Zhang >Priority: Major > > Spark and Hive action have been supported on Oozie since a long time ago. Now > Flink batch is productive, and i think it's necessary to integerate Flink > batch action on Oozie as a workflow node. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (OOZIE-3594) Support Flink batch action
[ https://issues.apache.org/jira/browse/OOZIE-3594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17371889#comment-17371889 ] Junfan Zhang commented on OOZIE-3594: - Thanks for your quick reply. [~dionusos] To support authentication mechanism with delegation token, i have submitted serveral PR which have been all merged. The prerequisites for Flink integration into Oozie have been met If the community needs to support flink batch action, i will submit the proposal about Flink batch action laterly. > Support Flink batch action > -- > > Key: OOZIE-3594 > URL: https://issues.apache.org/jira/browse/OOZIE-3594 > Project: Oozie > Issue Type: New Feature >Affects Versions: 5.1.0 >Reporter: Junfan Zhang >Assignee: Junfan Zhang >Priority: Major > > Spark and Hive action have been supported on Oozie since a long time ago. Now > Flink batch is productive, and i think it's necessary to integerate Flink > batch action on Oozie as a workflow node. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (OOZIE-3594) Support Flink batch action
[ https://issues.apache.org/jira/browse/OOZIE-3594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17371307#comment-17371307 ] Dénes Bodó commented on OOZIE-3594: --- [~zuston] Thank for your activity. Could you please explain in the "Description" of this Jira why would it good for Oozie, what would be the use cases? I am not really familiar with Flink either. Could you please also fetch the necessary changes, like workflow definition (XSD), new type of credential class/definition(XSD), what new would be introduced, what part of code would be changed? Do you have any draft or proposal which you could share? I am pretty sure adding a wanted functionality to Oozie is great but we need the above. If the community then accepts we should create sub tasks for code change, documentation, testing, etc. Thanks, Denes > Support Flink batch action > -- > > Key: OOZIE-3594 > URL: https://issues.apache.org/jira/browse/OOZIE-3594 > Project: Oozie > Issue Type: New Feature >Reporter: Junfan Zhang >Assignee: Junfan Zhang >Priority: Major > > Plan to do it? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (OOZIE-3594) Support Flink batch action
[ https://issues.apache.org/jira/browse/OOZIE-3594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17371028#comment-17371028 ] Junfan Zhang commented on OOZIE-3594: - Update: Another Flink & Hive PR has been merged, https://issues.apache.org/jira/browse/FLINK-22329 Now Flink has supported delegation tokens, we have extend Flink batch action in our production env. Any ideas on it? If ok, i will submit proposal to support it. [~dionusos] [~asalamon74] > Support Flink batch action > -- > > Key: OOZIE-3594 > URL: https://issues.apache.org/jira/browse/OOZIE-3594 > Project: Oozie > Issue Type: New Feature >Reporter: Junfan Zhang >Assignee: Junfan Zhang >Priority: Major > > Plan to do it? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (OOZIE-3594) Support Flink batch action
[ https://issues.apache.org/jira/browse/OOZIE-3594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17344291#comment-17344291 ] Junfan Zhang commented on OOZIE-3594: - [~gezapeti] Update: To support Flink batch Action on Oozie, Flink need to support submmission with delegation token credentials. So I submitted some PR to Flink, which have beed merged. 1. https://issues.apache.org/jira/browse/FLINK-21700# 2. https://issues.apache.org/jira/browse/FLINK-22534 3. https://issues.apache.org/jira/browse/FLINK-21768 > Support Flink batch action > -- > > Key: OOZIE-3594 > URL: https://issues.apache.org/jira/browse/OOZIE-3594 > Project: Oozie > Issue Type: New Feature >Reporter: Junfan Zhang >Assignee: Junfan Zhang >Priority: Major > > Plan to do it? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (OOZIE-3594) Support flink batch action
[ https://issues.apache.org/jira/browse/OOZIE-3594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17089791#comment-17089791 ] Junfan Zhang commented on OOZIE-3594: - We have such a plan in my company, and progress will be synchronized here. We are very happy to support Flink batch with the community > Support flink batch action > -- > > Key: OOZIE-3594 > URL: https://issues.apache.org/jira/browse/OOZIE-3594 > Project: Oozie > Issue Type: New Feature >Reporter: Junfan Zhang >Priority: Major > > Plan to do it? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (OOZIE-3594) Support flink batch action
[ https://issues.apache.org/jira/browse/OOZIE-3594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17058660#comment-17058660 ] Gézapeti commented on OOZIE-3594: - I'm not familiar with Flink. What's your proposal for a batch action? > Support flink batch action > -- > > Key: OOZIE-3594 > URL: https://issues.apache.org/jira/browse/OOZIE-3594 > Project: Oozie > Issue Type: New Feature >Reporter: Junfan Zhang >Priority: Major > > Plan to do it? -- This message was sent by Atlassian Jira (v8.3.4#803005)
