Hi Ferenc, But the job lifecycle doesn't tie to the SQL Gateway session. Even if the session is closed, all the running jobs are not affected.
Best, Jark On Tue, 27 Jun 2023 at 04:14, Ferenc Csaky <ferenc.cs...@pm.me.invalid> wrote: > Hi Jark, > > Thank you for pointing out FLIP-295 abouth catalog persistence, I was not > aware the current state. Although as far as I see, that persistent catalogs > are necessary, but not sufficient achieving a "persistent gateway". > > The current implementation ties the job lifecycle to the SQL gateway > session, so if it gets closed, it will cancel all the jobs. So that would > be the next step I think. Any work or thought regarding this aspect? We are > definitely willing to help out on this front. > > Cheers, > F > > > ------- Original Message ------- > On Sunday, June 25th, 2023 at 06:23, Jark Wu <imj...@gmail.com> wrote: > > > > > > > > Hi Ferenc, > > > > Making SQL Gateway to be an easy-to-use platform infrastructure of Flink > > SQL > > is one of the important roadmaps [1]. > > > > The persistence ability of the SQL Gateway is a major work in 1.18 > release. > > One of the persistence demand is that the registered catalogs are > currently > > kept in memory and lost when Gateway restarts. There is an accepted FLIP > > (FLIP-295)[2] target to resolve this issue and make Gateway can persist > the > > registered catalogs information into files or databases. > > > > I'm not sure whether this is something you are looking for? > > > > Best, > > Jark > > > > > > [1]: https://flink.apache.org/roadmap/#a-unified-sql-platform > > [2]: > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations > > > > On Fri, 23 Jun 2023 at 00:25, Ferenc Csaky ferenc.cs...@pm.me.invalid > > > > wrote: > > > > > Hello devs, > > > > > > I would like to open a discussion about persistence possibilitis for > the > > > SQL Gateway. At Cloudera, we are happy to see the work already done on > this > > > project and looking for ways to utilize it on our platform as well, but > > > currently it lacks some features that would be essential in our case, > where > > > we could help out. > > > > > > I am not sure if any thought went into gateway persistence specifics > > > already, and this feature could be implemented in fundamentally > differnt > > > ways, so I think the frist step could be to agree on the basics. > > > > > > First, in my opinion, persistence should be an optional feature of the > > > gateway, that can be enabled if desired. There can be a lot of > > > implementation details, but there can be some major directions to > follow: > > > > > > - Utilize Hive catalog: The Hive catalog can already be used to have > > > persistenct meta-objects, so the crucial thing that would be missing in > > > this case is other catalogs. Personally, I would not pursue this > option, > > > because in my opinion it would limit the usability of this feature too > much. > > > - Serialize the session as is: Saving the whole session (or its > context) > > > [1] as is to durable storage, so it can be kept and picked up again. > > > - Serialize the required elements (catalogs, tables, functions, etc.), > not > > > necessarily as a whole: The main point here would be to serialize a > > > different object, so the persistent data will not be that sensitive to > > > changes of the session (or its context). There can be numerous factors > > > here, like try to keep the model close to the session itself, so the > > > boilerplate required for the mapping can be kept to minimal, or focus > on > > > saving what is actually necessary, making the persistent storage more > > > portable. > > > > > > WDYT? > > > > > > Cheers, > > > F > > > > > > [1] > > > > https://github.com/apache/flink/blob/master/flink-table/flink-sql-gateway/src/main/java/org/apache/flink/table/gateway/service/session/Session.java >
