Do you mean a new interface just like *ApplicationDeployer* for
flink-jar-application submission?
The *ApplicationDeployer*[1] interface has already been used in the
flink-kubernetes-operator project.
[1].
Hi Yang,
It would be better to have an interface to submitting application mode
jobs. It's a little tricky for SQL CLI/Gateway process to invoke a shell
script to submit jobs.
Best,
Jark
On Thu, 17 Feb 2022 at 15:20, Yang Wang wrote:
> I am not sure whether the sql script could also be
I am not sure whether the sql script could also be submitted like python.
We will need a sql-runner jar, which plays as the user jar and has the sql
script as the argument.
./bin/flink run-application \ --target kubernetes-application \
-Dkubernetes.cluster-id= \
I think this mode is still limited and maybe not easy to extend.
Could the application mode provide an interface to execute?
So that clients can implement the interface and pass arbitrary parameters
(e.g. SQL scripts) ?
Best,
Jark
On Wed, 16 Feb 2022 at 18:54, Konstantin Knauf wrote:
> Hi
Hi Jark,
I think you are raising a very good point. I think we need an application
mode for SQL that would work along the lines of executing a SQL script
(incl. init scripts) located in a particular directory in the Docker Image.
Details to be discussed.
Do you think Zeppelin/SQL CLI could work
Hi David,
Zeppelin and SQL CLI also support submitting long-running streaming SQL
jobs. So the session cluster is not a fit mode.
Best,
Jark
On Fri, 11 Feb 2022 at 22:42, David Morávek wrote:
> Hi Jark, can you please elaborate about the current need of the per-job
> mode for interactive
Hi Jark, can you please elaborate about the current need of the per-job
mode for interactive clients (eg. Zeppelin that you've mentioned)? Aren't
these a natural fit for the session cluster?
D.
On Fri, Feb 11, 2022 at 3:25 PM Jark Wu wrote:
> Hi Konstantin,
>
> I'm not very familiar with the
Hi Konstantin,
I'm not very familiar with the implementation of per-job mode and
application mode.
But is there any instruction for users abou how to migrate platforms/jobs
to application mode?
IIUC, the biggest difference between the two modes is where the main()
method is executed.
However, SQL
Hi everyone,
Thank you for sharing your perspectives. I was not aware of
these limitations of per-job mode on YARN. It seems that there is a general
agreement to deprecate per-job mode and to drop it once the limitations
around YARN are resolved. I've started a corresponding vote in [1].
Thanks
Hi Yang,
Thank you for the clarification. In general I think we will have time to
experiment with this until it will be removed totally and migrate our solution
to use application mode.
Regards,
F
On 2022/01/26 02:42:24 Yang Wang wrote:
> Hi all,
>
> I remember the application mode was
Hi all,
I remember the application mode was initially named "cluster mode". As a
contrast, the per-job mode is the "client mode".
So I believe application mode should cover all the functionalities of
per-job except where we are running the user main code.
In the containerized or the Kubernetes
Hi Konstantin,
First of all, sorry for the delay. We at Cloudera are currently relying on
per-job mode deploying Flink applications over YARN.
Specifically, we allow users to upload connector jars and other artifacts.
There are also some default jars that we need to ship. These are all stored
Hi all,
I agree with Xintong's comment: Reducing the number of deployment modes
would help users. There is a clearer distinction between session mode and
the two other deployment modes (i.e. application and job mode). The
difference between application and job mode is not that easy to grasp for
Sorry for joining the discussion late.
I'm leaning towards deprecating the per-job mode soonish, and eventually
dropping it in the long-term.
- One less deployment mode makes it easier for users (especially newcomers)
to understand. Deprecating the per-job mode sends the signal that it is
legacy,
Thanks Thomas & Biao for your feedback.
Any additional opinions on how we should proceed with per job-mode? As you
might have guessed, I am leaning towards proposing to deprecate per-job
mode.
On Thu, Jan 13, 2022 at 5:11 PM Thomas Weise wrote:
> Regarding session mode:
>
> ## Session Mode
> *
Regarding session mode:
## Session Mode
* main() method executed in client
Session mode also supports execution of the main method on Jobmanager
with submission through REST API. That's how Flinkk k8s operators like
[1] work. It's actually an important capability because it allows for
allocation
Hi Konstantin,
Thanks a lot for starting this discussion! I hope my thoughts and
experiences why users use Per-Job Mode, especially in YARN can help:
#1. Per-job mode makes managing dependencies easier: I have met some
customers who used Per-Job Mode to submit jobs with a lot of local
Hi everyone,
I would like to discuss and understand if the benefits of having Per-Job
Mode in Apache Flink outweigh its drawbacks.
*# Background: Flink's Deployment Modes*
Flink currently has three deployment modes. They differ in the following
dimensions:
* main() method executed on Jobmanager
18 matches
Mail list logo
