[jira] [Commented] (FLINK-31425) Support submitting a job with streamgraph
[ https://issues.apache.org/jira/browse/FLINK-31425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17702472#comment-17702472 ] Jeff commented on FLINK-31425: -- Thanks [~JunRuiLi] and [~chesnay] for the insight. For item 2 'Make StreamGraph public so that users can get streamGraph. I'm not sure if this part has enough meaning to expose an internal interface to users, which may bring additional burden' This is proposed to serve as alternative interface to JobGraph based api. Agree it's sort of internal interface, but it's a relative higher level abstraction compared with JobGraph and should be more developer friendly. [~chesnay] some existing integration like Beam Flink Runner has to leverage on JobGraph interface for job submission, [https://github.com/apache/beam/blob/master/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkPipelineExecutionEnvironment.java#L136,] where it appears like a blackbox and not feasible for some runtime check or debug log maybe. That makes me think whether a StreamGraph based interface can make more sense. By the way, do we have some FLIP or mail thread that tracks discussion of 'language-agnostic way to submit jobs'. > Support submitting a job with streamgraph > -- > > Key: FLINK-31425 > URL: https://issues.apache.org/jira/browse/FLINK-31425 > Project: Flink > Issue Type: New Feature > Components: API / DataStream >Reporter: Jeff >Priority: Major > > Currently, we have rest api to submit a job via jobgraph, which is aligned to > the way of flink cli running the entry class locally and submit the compiled > binary to remote cluster for execution. > This is convenient in its own right. However it also seems to bring in some > confusion and 'blackbox' feeling in that the payload of rest api is a binary > object and thus not self-descriptive and it's relative a low-level > presentation of the job executions whose interface is more likely to change > as version evolves. > Do you think it make more sense to build an api that accepts streamgraph as > input which may be presented with a json(just like visualizer did for an > execution plan visualization) plus additional runtime related configs and > resources? This may make the rest interface more descriptive. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-31425) Support submitting a job with streamgraph
[ https://issues.apache.org/jira/browse/FLINK-31425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17700078#comment-17700078 ] Chesnay Schepler commented on FLINK-31425: -- ??However it also seems to bring in some confusion and 'blackbox' feeling in that the payload of rest api is a binary object?? It's not intended to be used by anything but the client. ??Do you think it make more sense to build an api that accepts streamgraph as input which may be presented with a json?? That JSON would still have to contain many binary blobs for the user-code. While there has been a desire for a more language-agnostic way to submit jobs I don't see this as being the way to do it. Expanding the API more without a particularly strong benefit doesn't make much sense to me. > Support submitting a job with streamgraph > -- > > Key: FLINK-31425 > URL: https://issues.apache.org/jira/browse/FLINK-31425 > Project: Flink > Issue Type: New Feature > Components: API / DataStream >Reporter: Jeff >Priority: Major > > Currently, we have rest api to submit a job via jobgraph, which is aligned to > the way of flink cli running the entry class locally and submit the compiled > binary to remote cluster for execution. > This is convenient in its own right. However it also seems to bring in some > confusion and 'blackbox' feeling in that the payload of rest api is a binary > object and thus not self-descriptive and it's relative a low-level > presentation of the job executions whose interface is more likely to change > as version evolves. > Do you think it make more sense to build an api that accepts streamgraph as > input which may be presented with a json(just like visualizer did for an > execution plan visualization) plus additional runtime related configs and > resources? This may make the rest interface more descriptive. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-31425) Support submitting a job with streamgraph
[ https://issues.apache.org/jira/browse/FLINK-31425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17699925#comment-17699925 ] Junrui Li commented on FLINK-31425: --- [~tophei] Thanks for creating this issue. IIUC, according to the description, I think this issue can be divided into two parts: 1. Submit and run jobs by submitting a streamGraph, which is very meaningful. Submitting and running jobs through streamGraph can expand the dynamic adjustment capability of flink runtime. This is a relatively large and very meaningful change, and we are currently doing related research. 2. Make StreamGraph public so that users can get streamGraph. I'm not sure if this part has enough meaning to expose an internal interface to users, which may bring additional burden. > Support submitting a job with streamgraph > -- > > Key: FLINK-31425 > URL: https://issues.apache.org/jira/browse/FLINK-31425 > Project: Flink > Issue Type: New Feature > Components: API / DataStream >Reporter: Jeff >Priority: Major > > Currently, we have rest api to submit a job via jobgraph, which is aligned to > the way of flink cli running the entry class locally and submit the compiled > binary to remote cluster for execution. > This is convenient in its own right. However it also seems to bring in some > confusion and 'blackbox' feeling in that the payload of rest api is a binary > object and thus not self-descriptive and it's relative a low-level > presentation of the job executions whose interface is more likely to change > as version evolves. > Do you think it make more sense to build an api that accepts streamgraph as > input which may be presented with a json(just like visualizer did for an > execution plan visualization) plus additional runtime related configs and > resources? This may make the rest interface more descriptive. -- This message was sent by Atlassian Jira (v8.20.10#820010)
