Thanks for bringing this up, Dongwoo. Flink SQL Gateway is also a key component for OLAP scenarios.
@Gyula How about add sql gateway as an optional component to Session Cluster Deployments. User can specify the resource / instance number and ports of the sql gateway. I think that would help a lot for OLAP and batch user. Best, Yangze Guo On Fri, Sep 15, 2023 at 3:19 PM ConradJam <jam.gz...@gmail.com> wrote: > > If we start from the crd direction, I think this mode is more like a > sidecar of the session cluster, which is submitted to the session cluster > by sending sql commands to the sql gateway. I don't know if my statement is > accurate. > > Xiaolong Wang <xiaolong.w...@smartnews.com.invalid> 于2023年9月15日周五 13:27写道： > > > Hi, Dongwoo, > > > > Since Flink SQL gateway should run upon a Flink session cluster, I think > > it'd be easier to add more fields to the CRD of `FlinkSessionJob`. > > > > e.g. > > > > apiVersion: flink.apache.org/v1beta1 > > kind: FlinkSessionJob > > metadata: > > name: sql-gateway > > spec: > > sqlGateway: > > endpoint: "hiveserver2" > > mode: "streaming" > > hiveConf: > > configMap: > > name: hive-config > > items: > > - key: hive-site.xml > > path: hive-site.xml > > > > > > On Fri, Sep 15, 2023 at 12:56 PM Dongwoo Kim <dongwoo7....@gmail.com> > > wrote: > > > > > Hi all, > > > > > > *@Gyula* > > > Thanks for the consideration Gyula. My initial idea for the CR was > > roughly > > > like below. > > > I focused on simplifying the setup in k8s environment, but I agree with > > > your opinion that for the sql gateway > > > we don't need custom operator logic to handle and most of the > > requirements > > > can be met by existing k8s resources. > > > So maybe helm chart that bundles all resources needed should be enough. > > > > > > apiVersion: flink.apache.org/v1beta1 > > > kind: FlinkSqlGateway > > > metadata: > > > name: flink-sql-gateway-example > > > namespace: default > > > spec: > > > clusterName: flink-session-cluster-example > > > exposeServiceType: LoadBalancer > > > flinkSqlGatewayConfiguration: > > > sql-gateway.endpoint.type: "hiveserver2" > > > sql-gateway.endpoint.hiveserver2.catalog.name: "hive" > > > hiveConf: > > > configMap: > > > name: hive-config > > > items: > > > - key: hive-site.xml > > > path: hive-site.xml > > > > > > > > > *@xiaolong, @Shammon* > > > Hi xiaolong and Shammon. > > > Thanks for taking the time to share. > > > I'd also like to add my experience with setting up flink sql gateway on > > > k8s. > > > Without building a new Docker image, I've added a separate container to > > the > > > existing JobManager pod and started the sql gateway using the > > > "sql-gateway.sh start-foreground" command. > > > I haven't explored deploying the sql gateway as an independent deployment > > > yet, but that's something I'm considering after modifying JM's address to > > > desired session cluster. > > > > > > Thanks all > > > > > > Best > > > Dongwoo > > > > > > 2023년 9월 15일 (금) 오전 11:55, Xiaolong Wang > > > <xiaolong.w...@smartnews.com.invalid>님이 작성: > > > > > > > Hi, Shammon, > > > > > > > > Yes, I want to create a Flink SQL-gateway in a job-manager. > > > > > > > > Currently, the above script is generally a work-around and allows me to > > > > start a Flink session job manager with a SQL gateway running upon. > > > > > > > > I agree that it'd be more elegant that we create a new job type and > > > write a > > > > script, which is much easier for the user to use (since they do not > > need > > > to > > > > build a separate Flink image any more). > > > > > > > > > > > > > > > > On Fri, Sep 15, 2023 at 10:29 AM Shammon FY <zjur...@gmail.com> wrote: > > > > > > > > > Hi, > > > > > > > > > > Currently `sql-gateway` can be started with the script > > `sql-gateway.sh` > > > > in > > > > > an existing node, it is more like a simple "standalone" node. I think > > > > it's > > > > > valuable if we can do more work to start it in k8s. > > > > > > > > > > For xiaolong: > > > > > Do you want to start a sql-gateway instance in the jobmanager pod? I > > > > think > > > > > maybe we need a script like `kubernetes-sql-gatewah.sh` to start > > > > > `sql-gateway` pods with a flink image, what do you think? > > > > > > > > > > Best, > > > > > Shammon FY > > > > > > > > > > > > > > > On Fri, Sep 15, 2023 at 10:02 AM Xiaolong Wang > > > > > <xiaolong.w...@smartnews.com.invalid> wrote: > > > > > > > > > > > Hi, I've experiment this feature on K8S recently, here is some of > > my > > > > > trial: > > > > > > > > > > > > > > > > > > 1. Create a new kubernetes-jobmanager.sh script with the following > > > > > content > > > > > > > > > > > > #!/usr/bin/env bash > > > > > > $FLINK_HOME/bin/sql-gateway.sh start > > > > > > $FLINK_HOME/bin/kubernetes-jobmanager1.sh kubernetes-session > > > > > > > > > > > > 2. Build your own Flink docker image something like this > > > > > > FROM flink:1.17.1-scala_2.12-java11 > > > > > > > > > > > > RUN mv $FLINK_HOME/bin/kubernetes-jobmanager.sh $FLINK_HOME/bin/ > > > > > > kubernetes-jobmanager1.sh > > > > > > COPY ./kubernetes-jobmanager.sh > > > > $FLINK_HOME/bin/kubernetes-jobmanager.sh > > > > > > > > > > > > RUN chmod +x $FLINK_HOME/bin/*.sh > > > > > > USER flink > > > > > > > > > > > > 3. Create a Flink session job with the operator using the above > > > image. > > > > > > > > > > > > On Thu, Sep 14, 2023 at 9:49 PM Gyula Fóra <gyula.f...@gmail.com> > > > > wrote: > > > > > > > > > > > > > Hi! > > > > > > > > > > > > > > I don't completely understand what would be a content of such > > CRD, > > > > > could > > > > > > > you give a minimal example how the Flink SQL Gateway CR yaml > > would > > > > look > > > > > > > like? > > > > > > > > > > > > > > Adding a CRD would mean you need to add some operator/controller > > > > logic > > > > > as > > > > > > > well. Why not simply use a Deployment / StatefulSet in > > Kubernetes? > > > > > > > > > > > > > > Or a Helm chart if you want to make it more user friendly? > > > > > > > > > > > > > > Cheers, > > > > > > > Gyula > > > > > > > > > > > > > > On Thu, Sep 14, 2023 at 12:57 PM Dongwoo Kim < > > > dongwoo7....@gmail.com > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > > > I've been working on setting up a flink SQL gateway in a k8s > > > > > > environment > > > > > > > > and it got me thinking — what if we had a CRD for this? > > > > > > > > > > > > > > > > So I have quick questions below. > > > > > > > > 1. Is there ongoing work to create a CRD for the Flink SQL > > > Gateway? > > > > > > > > 2. If not, would the community be open to considering a CRD for > > > > this? > > > > > > > > > > > > > > > > I've noticed a growing demand for simplified setup of the flink > > > sql > > > > > > > gateway > > > > > > > > in flink's slack channel. > > > > > > > > Implementing a CRD could make deployments easier and offer > > better > > > > > > > > integration with k8s. > > > > > > > > > > > > > > > > If this idea is accepted, I'm open to drafting a FLIP for > > further > > > > > > > > discussion > > > > > > > > > > > > > > > > Thanks for your time and looking forward to your thoughts! > > > > > > > > > > > > > > > > Best regards, > > > > > > > > Dongwoo > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > Best > > ConradJam
