Hi, Dongwoo,
Since Flink SQL gateway should run upon a Flink session cluster, I think
it'd be easier to add more fields to the CRD of `FlinkSessionJob`.
e.g.
apiVersion: flink.apache.org/v1beta1
kind: FlinkSessionJob
metadata:
name: sql-gateway
spec:
sqlGateway:
endpoint: "hiveserver2"
mode: "streaming"
hiveConf:
configMap:
name: hive-config
items:
- key: hive-site.xml
path: hive-site.xml
On Fri, Sep 15, 2023 at 12:56 PM Dongwoo Kim <dongwoo7....@gmail.com> wrote:
> Hi all,
>
> *@Gyula*
> Thanks for the consideration Gyula. My initial idea for the CR was roughly
> like below.
> I focused on simplifying the setup in k8s environment, but I agree with
> your opinion that for the sql gateway
> we don't need custom operator logic to handle and most of the requirements
> can be met by existing k8s resources.
> So maybe helm chart that bundles all resources needed should be enough.
>
> apiVersion: flink.apache.org/v1beta1
> kind: FlinkSqlGateway
> metadata:
> name: flink-sql-gateway-example
> namespace: default
> spec:
> clusterName: flink-session-cluster-example
> exposeServiceType: LoadBalancer
> flinkSqlGatewayConfiguration:
> sql-gateway.endpoint.type: "hiveserver2"
> sql-gateway.endpoint.hiveserver2.catalog.name: "hive"
> hiveConf:
> configMap:
> name: hive-config
> items:
> - key: hive-site.xml
> path: hive-site.xml
>
>
> *@xiaolong, @Shammon*
> Hi xiaolong and Shammon.
> Thanks for taking the time to share.
> I'd also like to add my experience with setting up flink sql gateway on
> k8s.
> Without building a new Docker image, I've added a separate container to the
> existing JobManager pod and started the sql gateway using the
> "sql-gateway.sh start-foreground" command.
> I haven't explored deploying the sql gateway as an independent deployment
> yet, but that's something I'm considering after modifying JM's address to
> desired session cluster.
>
> Thanks all
>
> Best
> Dongwoo
>
> 2023년 9월 15일 (금) 오전 11:55, Xiaolong Wang
> <xiaolong.w...@smartnews.com.invalid>님이 작성:
>
> > Hi, Shammon,
> >
> > Yes, I want to create a Flink SQL-gateway in a job-manager.
> >
> > Currently, the above script is generally a work-around and allows me to
> > start a Flink session job manager with a SQL gateway running upon.
> >
> > I agree that it'd be more elegant that we create a new job type and
> write a
> > script, which is much easier for the user to use (since they do not need
> to
> > build a separate Flink image any more).
> >
> >
> >
> > On Fri, Sep 15, 2023 at 10:29 AM Shammon FY <zjur...@gmail.com> wrote:
> >
> > > Hi,
> > >
> > > Currently `sql-gateway` can be started with the script `sql-gateway.sh`
> > in
> > > an existing node, it is more like a simple "standalone" node. I think
> > it's
> > > valuable if we can do more work to start it in k8s.
> > >
> > > For xiaolong:
> > > Do you want to start a sql-gateway instance in the jobmanager pod? I
> > think
> > > maybe we need a script like `kubernetes-sql-gatewah.sh` to start
> > > `sql-gateway` pods with a flink image, what do you think?
> > >
> > > Best,
> > > Shammon FY
> > >
> > >
> > > On Fri, Sep 15, 2023 at 10:02 AM Xiaolong Wang
> > > <xiaolong.w...@smartnews.com.invalid> wrote:
> > >
> > > > Hi, I've experiment this feature on K8S recently, here is some of my
> > > trial:
> > > >
> > > >
> > > > 1. Create a new kubernetes-jobmanager.sh script with the following
> > > content
> > > >
> > > > #!/usr/bin/env bash
> > > > $FLINK_HOME/bin/sql-gateway.sh start
> > > > $FLINK_HOME/bin/kubernetes-jobmanager1.sh kubernetes-session
> > > >
> > > > 2. Build your own Flink docker image something like this
> > > > FROM flink:1.17.1-scala_2.12-java11
> > > >
> > > > RUN mv $FLINK_HOME/bin/kubernetes-jobmanager.sh $FLINK_HOME/bin/
> > > > kubernetes-jobmanager1.sh
> > > > COPY ./kubernetes-jobmanager.sh
> > $FLINK_HOME/bin/kubernetes-jobmanager.sh
> > > >
> > > > RUN chmod +x $FLINK_HOME/bin/*.sh
> > > > USER flink
> > > >
> > > > 3. Create a Flink session job with the operator using the above
> image.
> > > >
> > > > On Thu, Sep 14, 2023 at 9:49 PM Gyula Fóra <gyula.f...@gmail.com>
> > wrote:
> > > >
> > > > > Hi!
> > > > >
> > > > > I don't completely understand what would be a content of such CRD,
> > > could
> > > > > you give a minimal example how the Flink SQL Gateway CR yaml would
> > look
> > > > > like?
> > > > >
> > > > > Adding a CRD would mean you need to add some operator/controller
> > logic
> > > as
> > > > > well. Why not simply use a Deployment / StatefulSet in Kubernetes?
> > > > >
> > > > > Or a Helm chart if you want to make it more user friendly?
> > > > >
> > > > > Cheers,
> > > > > Gyula
> > > > >
> > > > > On Thu, Sep 14, 2023 at 12:57 PM Dongwoo Kim <
> dongwoo7....@gmail.com
> > >
> > > > > wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > I've been working on setting up a flink SQL gateway in a k8s
> > > > environment
> > > > > > and it got me thinking — what if we had a CRD for this?
> > > > > >
> > > > > > So I have quick questions below.
> > > > > > 1. Is there ongoing work to create a CRD for the Flink SQL
> Gateway?
> > > > > > 2. If not, would the community be open to considering a CRD for
> > this?
> > > > > >
> > > > > > I've noticed a growing demand for simplified setup of the flink
> sql
> > > > > gateway
> > > > > > in flink's slack channel.
> > > > > > Implementing a CRD could make deployments easier and offer better
> > > > > > integration with k8s.
> > > > > >
> > > > > > If this idea is accepted, I'm open to drafting a FLIP for further
> > > > > > discussion
> > > > > >
> > > > > > Thanks for your time and looking forward to your thoughts!
> > > > > >
> > > > > > Best regards,
> > > > > > Dongwoo
> > > > > >
> > > > >
> > > >
> > >
> >
>
