Hi, eveyone. The discussion doesn't receive any response for a while. I will close the discussion and start the vote tomorrow if the discussion doesn't receive any response today. Thanks all yours response!
Best, Shengkai Shengkai Fang <fskm...@gmail.com> 于2024年11月5日周二 14:00写道： > Hi, Lincoln. Thanks for your response. > > > since both scriptPath and script(statements) can be null, we need to > clarify the behavior when both are empty, such as throwing an error > > Yes, you are correct. I have updated the FLIP about this. When these > fields are both empty, the server throws an exception to notify users. > > > use a unified name for script vs statements, like 'script'? > > Updated. > > > Regarding Python UDFs, should we change it to a description of > "Additional Python resources," corresponding to "Additional Jar Resources" > > Updated. > > Best, > Shengkai > > > > Lincoln Lee <lincoln.8...@gmail.com> 于2024年11月5日周二 11:17写道： > >> Thanks Shengkai for driving this! Overall, looks good!\ >> >> I have two minor questions: >> 1. Regarding the interface parameters (including REST API >> & Java interfaces), since both scriptPath and script(statements) >> can be null, we need to clarify the behavior when both are >> empty, such as throwing an error? >> Also use a unified name for script vs statements, like 'script'? >> >> 2. Regarding Python UDFs, should we change it to a >> description of "Additional Python resources," corresponding >> to "Additional Jar Resources"? >> >> >> Best, >> Lincoln Lee >> >> >> Shengkai Fang <fskm...@gmail.com> 于2024年11月5日周二 10:16写道： >> >> > Hi, Ferenc. >> > >> > Thanks for your clarification. We can hard code these different options >> in >> > the sql-gateway module. I have updated the FLIP and PoC branch about >> this >> > part. But I think we should provide a unified API to ship artifacts to >> > different deployment. >> > >> > Best, >> > Shengkai >> > >> > >> > >> > Ferenc Csaky <ferenc.cs...@pm.me.invalid> 于2024年11月4日周一 21:05写道： >> > >> > > Hi Shengkai, >> > > >> > > Thank you for driving this FLIP! I think this is a good way to >> > > close this gap on the short-term until FLIP-316 can be finished. >> > > >> > > I would only like to add one thing: YARN has a `yarn.ship-files` >> > > config option that ships local or DFS files/directories to the >> > > YARN cluster [1]. >> > > >> > > Best, >> > > Ferenc >> > > >> > > [1] >> > > >> > >> https://nightlies.apache.org/flink/flink-docs-release-1.20/docs/deployment/config/ >> > > >> > > >> > > >> > > On Monday, November 4th, 2024 at 10:11, Xuyang <xyzhong...@163.com> >> > wrote: >> > > >> > > > >> > > > >> > > > Hi, Shegnkai. >> > > > >> > > > Thank you for your answer. I have no further questions. >> > > > >> > > > >> > > > >> > > > >> > > > -- >> > > > >> > > > Best！ >> > > > Xuyang >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > At 2024-11-04 10:00:32, "Shengkai Fang" fskm...@gmail.com wrote: >> > > > >> > > > > Hi, Xuyang. Thanks a lot for your response! >> > > > > >> > > > > > Does that means we will support multi DMLs, multi DQLs, mixed >> DMLs >> > & >> > > DQLs >> > > > > > in one sql script? >> > > > > >> > > > > According to the doc[1], application mode only supports one job >> in ha >> > > > > mode[2]. If users submit multiple jobs, dispatcher throws a >> > > > > DuplicateJobSubmissionException to notify users. >> > > > > >> > > > > In non-ha mode, the application mode doesn't have job number >> > > limitation. >> > > > > The SQL driver runs statement one by one and it is similar to >> > > submitting >> > > > > job to a session cluster. But just as the doc says, when any of >> > > multiple >> > > > > running jobs in Application Mode (submitted for example using >> > > > > executeAsync()) gets cancelled, all jobs will be stopped and the >> > > JobManager >> > > > > will shut down. >> > > > > >> > > > > [1] >> > > > > >> > > >> > >> https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/overview/#application-mode >> > > > > [2] >> > > > > >> > > >> > >> https://github.com/apache/flink/blob/master/flink-clients/src/main/java/org/apache/flink/client/deployment/application/ApplicationDispatcherBootstrap.java#L218 >> > > > > >> > > > > Best, >> > > > > Shengkai >> > > > > >> > > > > Xuyang xyzhong...@163.com 于2024年10月31日周四 17:10写道： >> > > > > >> > > > > > Hi, Shengkai. >> > > > > > >> > > > > > Thanks for driving this great work. LGTM overall, I just have >> one >> > > > > > question. >> > > > > > >> > > > > > IIUC, application mode supports to run multi-execute in a single >> > > `main` >> > > > > > function[1]. Does that means >> > > > > > >> > > > > > we will support multi DMLs, multi DQLs, mixed DMLs & DQLs in one >> > sql >> > > > > > script? If yes, can you explain >> > > > > > >> > > > > > a little about how do they work? >> > > > > > >> > > > > > [1] >> > > > > > >> > > >> > >> https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/overview/#application-mode >> > > > > > >> > > > > > -- >> > > > > > >> > > > > > Best！ >> > > > > > Xuyang >> > > > > > >> > > > > > 在 2024-10-31 10:18:13，"Ron Liu" ron9....@gmail.com 写道： >> > > > > > >> > > > > > > Hi, Shengkai >> > > > > > > >> > > > > > > Thanks for your quick response. It looks good to me. >> > > > > > > >> > > > > > > Best >> > > > > > > Ron >> > > > > > > >> > > > > > > Shengkai Fang fskm...@gmail.com 于2024年10月31日周四 10:08写道： >> > > > > > > >> > > > > > > > Hi, Ron! >> > > > > > > > >> > > > > > > > > I noticed that you say this FLIP focuses on supporting >> deploy >> > > sql >> > > > > > > > > scripts to the application cluster, does it mean that it >> only >> > > supports >> > > > > > > > > non-interactive gateway mode? >> > > > > > > > >> > > > > > > > Yes. This FLIP only supports to deploy a script in >> > > non-interactive mode. >> > > > > > > > >> > > > > > > > > Whether all SQL commands such as DDL & DML & SELECT are >> > > supported. >> > > > > > > > >> > > > > > > > We supports all SQL commands and the execution results are >> > > visible in >> > > > > > > > the >> > > > > > > > JM log. But application cluster has some limitations that >> only >> > > one job >> > > > > > > > is >> > > > > > > > allowed to run in the dedicated cluster. >> > > > > > > > >> > > > > > > > > How to dynamically download the JAR specified by the user >> > when >> > > > > > > > > submitting the sql script, and whether it is possible to >> > > specify a local >> > > > > > > > > jar? >> > > > > > > > >> > > > > > > > This is a good question. I think it's totally up to the >> > > deployment api. >> > > > > > > > For >> > > > > > > > example, kubernetes deployment provides the option >> > > > > > > > `kubernetes-artifacts-local-upload-enabled`[1] to upload the >> > > artifact to >> > > > > > > > the DFS but yarn deployment doesn't support to ship the >> > > artifacts to >> > > > > > > > DFS in >> > > > > > > > application mode. If runtime API can provide unified >> interface, >> > > I think >> > > > > > > > we >> > > > > > > > can use the unified API to upload local artifacts. >> > > Alternatively, we can >> > > > > > > > provide a special service that allows sql-gateway to support >> > > pulling >> > > > > > > > jar. >> > > > > > > > You can read the future work for more details. >> > > > > > > > >> > > > > > > > [1] >> > > > > > >> > > > > > >> > > >> > >> https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/#kubernetes-artifacts-local-upload-enabled >> > > > > > >> > > > > > > > Shengkai Fang fskm...@gmail.com 于2024年10月31日周四 09:30写道： >> > > > > > > > >> > > > > > > > > Hi, Feng! >> > > > > > > > > >> > > > > > > > > > if only clusterID is available, it may not be very >> > > convenient to >> > > > > > > > > > connect >> > > > > > > > > > to this application later on. >> > > > > > > > > >> > > > > > > > > If FLIP-479 is accepted, I think we can just adapt the >> > > sql-gateway >> > > > > > > > > behaviour to the behaviour that FLIP-479 mentioned. >> > > > > > > > > >> > > > > > > > > Best, >> > > > > > > > > Shengkai >> > > >> > >> >
