[
https://issues.apache.org/jira/browse/FLINK-38377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18021115#comment-18021115
]
yuanfenghu commented on FLINK-38377:
------------------------------------
[~gyfora] Do you have any opinions or suggestions on this? If I can, I can
finish this work ~
> SQL Job Submission in the Flink Kubernetes Operator via SQL Gateway
> --------------------------------------------------------------------
>
> Key: FLINK-38377
> URL: https://issues.apache.org/jira/browse/FLINK-38377
> Project: Flink
> Issue Type: New Feature
> Components: Kubernetes Operator, Table SQL / Gateway
> Reporter: yuanfenghu
> Priority: Minor
>
> 1. Abstract
> This proposal outlines a new, integrated approach for submitting Flink SQL
> jobs on Kubernetes. We propose enhancing the Flink Kubernetes Operator to
> natively support SQL job submission by leveraging the REST API of the Flink
> SQL Gateway. This change will provide a standardized, API-driven, and
> user-friendly experience, moving away from the current reliance on
> custom-built runner JARs.
> 2. Motivation
> Currently, submitting a pure SQL job through the Flink Kubernetes Operator
> is not a first-class experience. The common approach, demonstrated by the
> flink-sql-runner-example, has several significant drawbacks:
> * Requires Custom Tooling: Users must build and maintain a specialized JAR
> file (flink-sql-runner.jar) to package and execute their SQL scripts. This
> adds an extra layer of complexity to the development and deployment
> workflow.
> * Complex Dependency Management: Managing SQL connectors, UDFs, and other
> dependencies is cumbersome. Users often need to bundle all dependencies into a
> "fat JAR" or manage them through complex classpath configurations,
> which is error-prone.
> * Non-Standard Workflow: This method feels like a workaround rather than a
> native feature. It deviates from the declarative, API-centric philosophy of
> Kubernetes and the operator pattern.
> * Poor User Experience: The process is indirect and not intuitive for
> users who simply want to run a SQL script against Flink on Kubernetes.
> Starting with Flink 2.0, the Flink SQL Gateway has introduced the
> capability to directly submit SQL application jobs to a Kubernetes cluster
> [FLIP-480|https://cwiki.apache.org/confluence/display/FLINK/FLIP-480%3A+Support+to+deploy+SQL+script+in+application+mode].
> This
> presents a perfect opportunity to modernize and streamline the SQL
> submission process within the operator ecosystem.
> 3. Proposed Change
> We propose to integrate the Flink Kubernetes Operator with the Flink SQL
> Gateway to create a seamless SQL job submission workflow. This approach
> effectively turns the operator into an orchestrator that translates a
> declarative Kubernetes resource into an imperative API call against the SQL
> Gateway.
> 4. Key Advantages
> This integration would deliver substantial benefits:
> * Standardized, API-Driven Submission: Eliminates the need for custom
> runner JARs, providing a clean, official, and maintainable submission process.
> * Simplified Dependency Management: Leverages the SQL Gateway's robust
> mechanisms for dynamically loading connectors and dependencies, which can be
> achieved through the ADD JAR syntax.
> * Improved User Experience: Users can define and manage Flink SQL jobs
> using familiar kubectl and YAML files, creating a true Kubernetes-native
> experience.
> * Decoupling of Logic and Packaging: SQL developers can focus on writing
> SQL without worrying about Java build systems or container image
> customization.
> * Alignment with Flink Project Direction: This approach aligns the
> operator with the strategic investment being made in the Flink SQL Gateway as
> the
> primary entry point for SQL interactions.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
