Re: Re: Running Flink SQL in production
That makes sense, thank you. I found FLIP-316 [1] and will keep an eye on it too. Thanks, Robin. [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-316%3A+Support+application+mode+for+SQL+Gateway On Fri, 8 Mar 2024 at 13:56, Zhanghao Chen wrote: > Hi Robin, > > It's better to use Application mode [1] for mission-critical long-running > SQL jobs as it provides better isolation, you can utilize the table API to > package a jar as suggested by Feng to do so. Neither SQL client nor SQL > gateway supports submitting SQL in Application mode for now, but there's > some on-going effort [2]. Hopefully, it would be much easier to do so in > the future. > > [1] > https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/overview/#application-mode > > [2] https://issues.apache.org/jira/browse/FLINK-26541 > > Best, > Zhanghao Chen > -- > *From:* Feng Jin > *Sent:* Friday, March 8, 2024 9:46 > *To:* Xuyang > *Cc:* Robin Moffatt ; user@flink.apache.org < > user@flink.apache.org> > *Subject:* Re: Re: Running Flink SQL in production > > Hi, > > If you need to use Flink SQL in a production environment, I think it would > be better to use the Table API [1] and package it into a jar. > Then submit the jar to the cluster environment. > > [1] > https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/dev/table/common/#sql > > Best, > Feng > > On Thu, Mar 7, 2024 at 9:56 PM Xuyang wrote: > > Hi. > Hmm, if I'm mistaken, please correct me. Using a SQL client might not be > very convenient for those who need to verify the > results of submissions, such as checking for exceptions related to > submission failures, and so on. > > > -- > Best！ > Xuyang > > > 在 2024-03-07 17:32:07，"Robin Moffatt" 写道： > > Thanks for the reply. > In terms of production, my thinking is you'll have your SQL in a file > under code control. Whether that SQL ends up getting submitted via an > invocation of SQL Client with -f or via REST API seems moot. WDYT? > > > > On Thu, 7 Mar 2024 at 01:53, Xuyang wrote: > > Hi, IMO, both the SQL Client and the Restful API can provide connections > to the SQL Gateway service for submitting jobs. A slight difference is that > the SQL Client also offers a command-line visual interface for users to > view results. > In your production scenes, placing the SQL to be submitted into a file and > then using the '-f' command in SQL Client to submit the file sounds a bit > roundabout. You can just use the Restful API to submit them directly? > > > -- > Best！ > Xuyang > > > At 2024-03-07 04:11:01, "Robin Moffatt via user" > wrote: > > I'm reading the deployment guide[1] and wanted to check my understanding. > For deploying a SQL job into production, would the pattern be to write the > SQL in a file that's under source control, and pass that file as an > argument to SQL Client with -f argument (as in this docs example[2])? > Or script a call to the SQL Gateway's REST API? > > Are there pros and cons to each approach? > > thanks, Robin > > [1]: > https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/deployment/overview/ > [2]: > https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/dev/table/sqlclient/#execute-sql-files > >
Re: Re: Running Flink SQL in production
Hi Robin, It's better to use Application mode [1] for mission-critical long-running SQL jobs as it provides better isolation, you can utilize the table API to package a jar as suggested by Feng to do so. Neither SQL client nor SQL gateway supports submitting SQL in Application mode for now, but there's some on-going effort [2]. Hopefully, it would be much easier to do so in the future. [1] https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/overview/#application-mode [2] https://issues.apache.org/jira/browse/FLINK-26541 Best, Zhanghao Chen From: Feng Jin Sent: Friday, March 8, 2024 9:46 To: Xuyang Cc: Robin Moffatt ; user@flink.apache.org Subject: Re: Re: Running Flink SQL in production Hi, If you need to use Flink SQL in a production environment, I think it would be better to use the Table API [1] and package it into a jar. Then submit the jar to the cluster environment. [1] https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/dev/table/common/#sql Best, Feng On Thu, Mar 7, 2024 at 9:56 PM Xuyang mailto:xyzhong...@163.com>> wrote: Hi. Hmm, if I'm mistaken, please correct me. Using a SQL client might not be very convenient for those who need to verify the results of submissions, such as checking for exceptions related to submission failures, and so on. -- Best！ Xuyang 在 2024-03-07 17:32:07，"Robin Moffatt" mailto:ro...@decodable.co>> 写道： Thanks for the reply. In terms of production, my thinking is you'll have your SQL in a file under code control. Whether that SQL ends up getting submitted via an invocation of SQL Client with -f or via REST API seems moot. WDYT? On Thu, 7 Mar 2024 at 01:53, Xuyang mailto:xyzhong...@163.com>> wrote: Hi, IMO, both the SQL Client and the Restful API can provide connections to the SQL Gateway service for submitting jobs. A slight difference is that the SQL Client also offers a command-line visual interface for users to view results. In your production scenes, placing the SQL to be submitted into a file and then using the '-f' command in SQL Client to submit the file sounds a bit roundabout. You can just use the Restful API to submit them directly? -- Best！ Xuyang At 2024-03-07 04:11:01, "Robin Moffatt via user" mailto:user@flink.apache.org>> wrote: I'm reading the deployment guide[1] and wanted to check my understanding. For deploying a SQL job into production, would the pattern be to write the SQL in a file that's under source control, and pass that file as an argument to SQL Client with -f argument (as in this docs example[2])? Or script a call to the SQL Gateway's REST API? Are there pros and cons to each approach? thanks, Robin [1]: https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/deployment/overview/ [2]: https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/dev/table/sqlclient/#execute-sql-files
Re: Re: Running Flink SQL in production
Hi, If you need to use Flink SQL in a production environment, I think it would be better to use the Table API [1] and package it into a jar. Then submit the jar to the cluster environment. [1] https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/dev/table/common/#sql Best, Feng On Thu, Mar 7, 2024 at 9:56 PM Xuyang wrote: > Hi. > Hmm, if I'm mistaken, please correct me. Using a SQL client might not be > very convenient for those who need to verify the > results of submissions, such as checking for exceptions related to > submission failures, and so on. > > > -- > Best！ > Xuyang > > > 在 2024-03-07 17:32:07，"Robin Moffatt" 写道： > > Thanks for the reply. > In terms of production, my thinking is you'll have your SQL in a file > under code control. Whether that SQL ends up getting submitted via an > invocation of SQL Client with -f or via REST API seems moot. WDYT? > > > > On Thu, 7 Mar 2024 at 01:53, Xuyang wrote: > >> Hi, IMO, both the SQL Client and the Restful API can provide connections >> to the SQL Gateway service for submitting jobs. A slight difference is that >> the SQL Client also offers a command-line visual interface for users to >> view results. >> In your production scenes, placing the SQL to be submitted into a file >> and then using the '-f' command in SQL Client to submit the file sounds a >> bit roundabout. You can just use the Restful API to submit them directly? >> >> >> -- >> Best！ >> Xuyang >> >> >> At 2024-03-07 04:11:01, "Robin Moffatt via user" >> wrote: >> >> I'm reading the deployment guide[1] and wanted to check my understanding. >> For deploying a SQL job into production, would the pattern be to write the >> SQL in a file that's under source control, and pass that file as an >> argument to SQL Client with -f argument (as in this docs example[2])? >> Or script a call to the SQL Gateway's REST API? >> >> Are there pros and cons to each approach? >> >> thanks, Robin >> >> [1]: >> https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/deployment/overview/ >> [2]: >> https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/dev/table/sqlclient/#execute-sql-files >> >>
