Are there plans to address all or few of the above apart from the "JM LB not possible" which seems understandable ?
On Mon, Mar 19, 2018 at 9:58 AM, Fabian Hueske <fhue...@gmail.com> wrote: > Queryable state is "just" an additional feature on regular keyed state. > i.e., the only difference is that you can read the state from an outside > application. > Besides that it behaves exactly like regular application state > > Queryable state is (at the moment) designed to be accessible if a job runs. > If the job fails (and recovers) or is manually taken down for maintenance, > the state cannot be queried. > It's not possible to put a load balancer in front of a JobManager. Only > one JM is the active master that maintains a running job. > State is also not replicated. > > Best, Fabian > > > 2018-03-19 14:24 GMT+01:00 Vishal Santoshi <vishal.santo...@gmail.com>: > >> Those are understandable. I am more interested in a few things ( and may >> be more that could be added ) >> >> * As far as I can understand JM is the SPOF. Does HA become a necessity ? >> * If there are 2 or more JM could we theoretically have a LB fronting >> them ? Thus it is a peer to peer access ( Cassandra ) or a master slave >> setup for JM HA specifically for Queryable access ( For flink jobs it is >> master slave ) >> * Do we replicate state to other TMs for read optimization ( specifically >> to avoid Hot Node issues ) ? >> * If the job goes down it seems the state is not accessible. What plans >> to we have to "separate concerns" for Queryable state. >> >> We consider Queryable State significant a feature Flink provides and >> would do the necessary leg work if there are certain gaps in it being >> trully considered a Highly Available Key Value store. >> >> Regards. >> >> >> >> >> >> On Mon, Mar 19, 2018 at 5:53 AM, Fabian Hueske <fhue...@gmail.com> wrote: >> >>> Hi Vishal, >>> >>> In general, Queryable State should be ready to use. >>> There are a few things to consider though: >>> >>> - State queries are not synchronized with the application code, i.e., >>> they can happen at the same time. Therefore, the Flink application should >>> not modify objects that have been put into or read from the state if you >>> are not using the RocksDBStatebackend (which creates copies by >>> deserialization). >>> - State will be rolled back after a failure. Hence, you can read writes >>> that are not "committed by a checkpoint". >>> >>> @Kostas, did I forget something? >>> >>> Best, Fabian >>> >>> >>> >>> 2018-03-18 16:50 GMT+01:00 Vishal Santoshi <vishal.santo...@gmail.com>: >>> >>>> To be more precise, is anything thing similar to >>>> https://engineering.linkedin.com/blog/2018/03/air-traffic >>>> -controller--member-first-notifications-at-linkedin . done in Samza, >>>> can be replicated with production level guarantees with Flink Queryable >>>> state ( as it stands currently version 1.5 ) ? >>>> >>>> On Fri, Mar 16, 2018 at 5:10 PM, Vishal Santoshi < >>>> vishal.santo...@gmail.com> wrote: >>>> >>>>> We are making few decisions on use cases where Queryable state is a >>>>> natural fit https://ci.apache.org/projects >>>>> /flink/flink-docs-release-1.4/dev/stream/state/queryable_state.html >>>>> >>>>> Is Queryable state production ready ? We will go to 1.5 flnk if that >>>>> helps to make the case for the usage. >>>>> >>>> >>>> >>> >> >
