Re: Stateful functions presentation code (UI part)
Thanks Igal for the detailed explanantion. I know that this was only a demo, I just wanted to reason a bit on the pros and cons of sending data to an UI from Flink. Best, Flavio Il Ven 1 Nov 2019, 12:21 Igal Shilman ha scritto: > Hi Flavio, let me try to clarify: > > The intention of this example is to demonstrate how > different entities (drivers, passengers, etc') participates in a protocol > (ride matching). For that we have the stateful functions application, and a > standalone java application that just generates the events to trigger the > simulation (i.e. a passenger requests a ride would be an event that the > simulator emits). > The visualization aspect of this example is not a part of what we are > trying to demonstrate but rather, an out-of-band peek into the live system > (websocket that duplicates the events directly to the UI) which we thought > is a nice addition to the example that visualizes how cars are moving on > the grid :-) > > If your goal is to create a scalable visualization of car locations, then > I guess your suggestion to break the grid into distinct geo-regions can > work. For that you would need: > 1. An additional egress backed by a Kafka topic. > 2. Modify the driver function to send location changes to that egress, > where you'd use the region id (geohash with a resolution to your liking) as > key [1]. > 3. At the consuming side subscribe to the partition that holds the region > of interest. > > But, realistically speaking for this kind of a visualization you would be > better off consuming from the locations update topic (from-driver topic) > and populating an optimized geo-indexed store, > and querying it directly. > > I hope this clarifies things, > Igal > > [1] > https://github.com/ververica/stateful-functions/blob/master/stateful-functions-examples/stateful-functions-ridesharing-example/stateful-functions-ridesharing-example-functions/src/main/java/com/ververica/statefun/examples/ridesharing/KafkaSpecs.java#L121 > > On Thu, Oct 31, 2019 at 6:21 PM Flavio Pompermaier > wrote: > >> Thanks Igal, this is more or less what I was expecting..this implies that >> ALL events are received on the UI side. >> I was concerned about the tradeoffs of this choice: when I zoom on the >> map I could simply ignore messages outside the boundaries (but I still >> spend many cpu resource in the reading of useless messages). >> >> In the case of a worldwide company (eg uber or simolar) it's probably better >> to create a topic per geographical arean..but also in this case then the UI >> should know when to attach or detach from queue topics when it reach a >> section of the map served by different topics..in another case I could just >> have too many events also in the single map section (let's think about some >> big city with many user). >> >> Is there any talk at FF ir someone else that faced those issues too? >> >> Il Gio 31 Ott 2019, 17:44 Igal Shilman ha scritto: >> >>> For that particular example, the simulator [1] is responsible for >>> simulating physical drivers and passengers that interact with their >>> corresponding >>> stateful functions [2]. >>> The interaction between the simulator and the stateful functions is >>> happening via four Kafka topics: >>> * to-driver - messages that are sent from the physical drivers (the >>> simulator [1]) to the stateful functions. The messages always carry a >>> driver-id which acts as a routing key (I think that this is what you mean >>> by correlation id) to a specific driver stateful function (FnDriver) >>> * from-driver - messages that are sent from a stateful function with a >>> specific driver id to the simulator >>> * to-passenger - symmetric to to-driver >>> * from-passenger - symmetric to from-driver. >>> The ingress and egress definition are specified here [3], and you may >>> want to checkout how to router is defined as well [4][5]. >>> >>> In addition the simulator is also feeding the UI directly by duplicating >>> the messages to a web socket (see [6]) >>> >>> I hope this clarifies the examples. >>> >>> Igal. >>> >>> [1] >>> https://github.com/ververica/stateful-functions/tree/master/stateful-functions-examples/stateful-functions-ridesharing-example/stateful-functions-ridesharing-example-simulator/src/main/java/com/ververica/statefun/examples/ridesharing/simulator >>> [2] >>> https://github.com/ververica/stateful-functions/tree/master/stateful-functions-examples/stateful-functions-ridesharing-example/stateful-functions-ridesharing-example-functions >>> [3] >>> https://github.com/ververica/stateful-functions/blob/master/stateful-functions-examples/stateful-functions-ridesharing-example/stateful-functions-ridesharing-example-functions/src/main/java/com/ververica/statefun/examples/ridesharing/KafkaSpecs.java#L43 >>> [4] >>>
Re: Stateful functions presentation code (UI part)
Hi Flavio, let me try to clarify: The intention of this example is to demonstrate how different entities (drivers, passengers, etc') participates in a protocol (ride matching). For that we have the stateful functions application, and a standalone java application that just generates the events to trigger the simulation (i.e. a passenger requests a ride would be an event that the simulator emits). The visualization aspect of this example is not a part of what we are trying to demonstrate but rather, an out-of-band peek into the live system (websocket that duplicates the events directly to the UI) which we thought is a nice addition to the example that visualizes how cars are moving on the grid :-) If your goal is to create a scalable visualization of car locations, then I guess your suggestion to break the grid into distinct geo-regions can work. For that you would need: 1. An additional egress backed by a Kafka topic. 2. Modify the driver function to send location changes to that egress, where you'd use the region id (geohash with a resolution to your liking) as key [1]. 3. At the consuming side subscribe to the partition that holds the region of interest. But, realistically speaking for this kind of a visualization you would be better off consuming from the locations update topic (from-driver topic) and populating an optimized geo-indexed store, and querying it directly. I hope this clarifies things, Igal [1] https://github.com/ververica/stateful-functions/blob/master/stateful-functions-examples/stateful-functions-ridesharing-example/stateful-functions-ridesharing-example-functions/src/main/java/com/ververica/statefun/examples/ridesharing/KafkaSpecs.java#L121 On Thu, Oct 31, 2019 at 6:21 PM Flavio Pompermaier wrote: > Thanks Igal, this is more or less what I was expecting..this implies that > ALL events are received on the UI side. > I was concerned about the tradeoffs of this choice: when I zoom on the map > I could simply ignore messages outside the boundaries (but I still spend > many cpu resource in the reading of useless messages). > > In the case of a worldwide company (eg uber or simolar) it's probably better > to create a topic per geographical arean..but also in this case then the UI > should know when to attach or detach from queue topics when it reach a > section of the map served by different topics..in another case I could just > have too many events also in the single map section (let's think about some > big city with many user). > > Is there any talk at FF ir someone else that faced those issues too? > > Il Gio 31 Ott 2019, 17:44 Igal Shilman ha scritto: > >> For that particular example, the simulator [1] is responsible for >> simulating physical drivers and passengers that interact with their >> corresponding >> stateful functions [2]. >> The interaction between the simulator and the stateful functions is >> happening via four Kafka topics: >> * to-driver - messages that are sent from the physical drivers (the >> simulator [1]) to the stateful functions. The messages always carry a >> driver-id which acts as a routing key (I think that this is what you mean >> by correlation id) to a specific driver stateful function (FnDriver) >> * from-driver - messages that are sent from a stateful function with a >> specific driver id to the simulator >> * to-passenger - symmetric to to-driver >> * from-passenger - symmetric to from-driver. >> The ingress and egress definition are specified here [3], and you may >> want to checkout how to router is defined as well [4][5]. >> >> In addition the simulator is also feeding the UI directly by duplicating >> the messages to a web socket (see [6]) >> >> I hope this clarifies the examples. >> >> Igal. >> >> [1] >> https://github.com/ververica/stateful-functions/tree/master/stateful-functions-examples/stateful-functions-ridesharing-example/stateful-functions-ridesharing-example-simulator/src/main/java/com/ververica/statefun/examples/ridesharing/simulator >> [2] >> https://github.com/ververica/stateful-functions/tree/master/stateful-functions-examples/stateful-functions-ridesharing-example/stateful-functions-ridesharing-example-functions >> [3] >> https://github.com/ververica/stateful-functions/blob/master/stateful-functions-examples/stateful-functions-ridesharing-example/stateful-functions-ridesharing-example-functions/src/main/java/com/ververica/statefun/examples/ridesharing/KafkaSpecs.java#L43 >> [4] >> https://github.com/ververica/stateful-functions/blob/master/stateful-functions-examples/stateful-functions-ridesharing-example/stateful-functions-ridesharing-example-functions/src/main/java/com/ververica/statefun/examples/ridesharing/Module.java#L33 >> [5] >> https://github.com/ververica/stateful-functions/blob/master/stateful-functions-examples/stateful-functions-ridesharing-example/stateful-functions-ridesharing-example-functions/src/main/java/com/ververica/statefun/examples/ridesharing/InboundDriverRouter.java#L26 >> [6] >>
Re: Stateful functions presentation code (UI part)
Thanks Igal, this is more or less what I was expecting..this implies that ALL events are received on the UI side. I was concerned about the tradeoffs of this choice: when I zoom on the map I could simply ignore messages outside the boundaries (but I still spend many cpu resource in the reading of useless messages). In the case of a worldwide company (eg uber or simolar) it's probably better to create a topic per geographical arean..but also in this case then the UI should know when to attach or detach from queue topics when it reach a section of the map served by different topics..in another case I could just have too many events also in the single map section (let's think about some big city with many user). Is there any talk at FF ir someone else that faced those issues too? Il Gio 31 Ott 2019, 17:44 Igal Shilman ha scritto: > For that particular example, the simulator [1] is responsible for > simulating physical drivers and passengers that interact with their > corresponding > stateful functions [2]. > The interaction between the simulator and the stateful functions is > happening via four Kafka topics: > * to-driver - messages that are sent from the physical drivers (the > simulator [1]) to the stateful functions. The messages always carry a > driver-id which acts as a routing key (I think that this is what you mean > by correlation id) to a specific driver stateful function (FnDriver) > * from-driver - messages that are sent from a stateful function with a > specific driver id to the simulator > * to-passenger - symmetric to to-driver > * from-passenger - symmetric to from-driver. > The ingress and egress definition are specified here [3], and you may want > to checkout how to router is defined as well [4][5]. > > In addition the simulator is also feeding the UI directly by duplicating > the messages to a web socket (see [6]) > > I hope this clarifies the examples. > > Igal. > > [1] > https://github.com/ververica/stateful-functions/tree/master/stateful-functions-examples/stateful-functions-ridesharing-example/stateful-functions-ridesharing-example-simulator/src/main/java/com/ververica/statefun/examples/ridesharing/simulator > [2] > https://github.com/ververica/stateful-functions/tree/master/stateful-functions-examples/stateful-functions-ridesharing-example/stateful-functions-ridesharing-example-functions > [3] > https://github.com/ververica/stateful-functions/blob/master/stateful-functions-examples/stateful-functions-ridesharing-example/stateful-functions-ridesharing-example-functions/src/main/java/com/ververica/statefun/examples/ridesharing/KafkaSpecs.java#L43 > [4] > https://github.com/ververica/stateful-functions/blob/master/stateful-functions-examples/stateful-functions-ridesharing-example/stateful-functions-ridesharing-example-functions/src/main/java/com/ververica/statefun/examples/ridesharing/Module.java#L33 > [5] > https://github.com/ververica/stateful-functions/blob/master/stateful-functions-examples/stateful-functions-ridesharing-example/stateful-functions-ridesharing-example-functions/src/main/java/com/ververica/statefun/examples/ridesharing/InboundDriverRouter.java#L26 > [6] > https://github.com/ververica/stateful-functions/blob/master/stateful-functions-examples/stateful-functions-ridesharing-example/stateful-functions-ridesharing-example-simulator/src/main/java/com/ververica/statefun/examples/ridesharing/simulator/simulation/Driver.java#L70 > > On Thu, Oct 31, 2019 at 4:15 PM Flavio Pompermaier > wrote: > >> Yes, I'm interested in how to read data from a UI..which egress should I >> use? If we use a kafka queue, how to filter data received in the topic? >> Should I use a correlation id or use a new topic per user? >> >> Il Gio 31 Ott 2019, 16:08 Igal Shilman ha scritto: >> >>> Hi Flavio, >>> >>> We haven't included the UI source code just yet, we've only used it for >>> demos and talks. >>> >>> The reason is that (1) we didn't put a lot of effort and time there (2) >>> didn't check the time to go through the individual dependencies and >>> licences. >>> But we will add that very soon. >>> >>> Would having the UI code there would improve your understanding? or is >>> there another reason? >>> >>> Thanks, >>> Igal >>> >>> On Thu, Oct 31, 2019 at 5:44 AM Flavio Pompermaier >>> wrote: >>> Hi Vino, I already checked that code but I can't find the UI part :( On Thu, Oct 31, 2019 at 12:32 PM vino yang wrote: > Hi Flavio, > > Please see this link.[1] > > Best, > Vino > > [1]: > https://github.com/ververica/stateful-functions/tree/master/stateful-functions-examples/stateful-functions-ridesharing-example > > Flavio Pompermaier 于2019年10月31日周四 下午4:53写道： > >> Hi to all, >> yould it be possible to provide also the source code of the UI part >> of the ride sharing example? It would be interesting to me how the UI is >> reading the data from the Kafka egress. >> >> Best, >> Flavio >>
Re: Stateful functions presentation code (UI part)
For that particular example, the simulator [1] is responsible for simulating physical drivers and passengers that interact with their corresponding stateful functions [2]. The interaction between the simulator and the stateful functions is happening via four Kafka topics: * to-driver - messages that are sent from the physical drivers (the simulator [1]) to the stateful functions. The messages always carry a driver-id which acts as a routing key (I think that this is what you mean by correlation id) to a specific driver stateful function (FnDriver) * from-driver - messages that are sent from a stateful function with a specific driver id to the simulator * to-passenger - symmetric to to-driver * from-passenger - symmetric to from-driver. The ingress and egress definition are specified here [3], and you may want to checkout how to router is defined as well [4][5]. In addition the simulator is also feeding the UI directly by duplicating the messages to a web socket (see [6]) I hope this clarifies the examples. Igal. [1] https://github.com/ververica/stateful-functions/tree/master/stateful-functions-examples/stateful-functions-ridesharing-example/stateful-functions-ridesharing-example-simulator/src/main/java/com/ververica/statefun/examples/ridesharing/simulator [2] https://github.com/ververica/stateful-functions/tree/master/stateful-functions-examples/stateful-functions-ridesharing-example/stateful-functions-ridesharing-example-functions [3] https://github.com/ververica/stateful-functions/blob/master/stateful-functions-examples/stateful-functions-ridesharing-example/stateful-functions-ridesharing-example-functions/src/main/java/com/ververica/statefun/examples/ridesharing/KafkaSpecs.java#L43 [4] https://github.com/ververica/stateful-functions/blob/master/stateful-functions-examples/stateful-functions-ridesharing-example/stateful-functions-ridesharing-example-functions/src/main/java/com/ververica/statefun/examples/ridesharing/Module.java#L33 [5] https://github.com/ververica/stateful-functions/blob/master/stateful-functions-examples/stateful-functions-ridesharing-example/stateful-functions-ridesharing-example-functions/src/main/java/com/ververica/statefun/examples/ridesharing/InboundDriverRouter.java#L26 [6] https://github.com/ververica/stateful-functions/blob/master/stateful-functions-examples/stateful-functions-ridesharing-example/stateful-functions-ridesharing-example-simulator/src/main/java/com/ververica/statefun/examples/ridesharing/simulator/simulation/Driver.java#L70 On Thu, Oct 31, 2019 at 4:15 PM Flavio Pompermaier wrote: > Yes, I'm interested in how to read data from a UI..which egress should I > use? If we use a kafka queue, how to filter data received in the topic? > Should I use a correlation id or use a new topic per user? > > Il Gio 31 Ott 2019, 16:08 Igal Shilman ha scritto: > >> Hi Flavio, >> >> We haven't included the UI source code just yet, we've only used it for >> demos and talks. >> >> The reason is that (1) we didn't put a lot of effort and time there (2) >> didn't check the time to go through the individual dependencies and >> licences. >> But we will add that very soon. >> >> Would having the UI code there would improve your understanding? or is >> there another reason? >> >> Thanks, >> Igal >> >> On Thu, Oct 31, 2019 at 5:44 AM Flavio Pompermaier >> wrote: >> >>> Hi Vino, >>> I already checked that code but I can't find the UI part :( >>> >>> On Thu, Oct 31, 2019 at 12:32 PM vino yang >>> wrote: >>> Hi Flavio, Please see this link.[1] Best, Vino [1]: https://github.com/ververica/stateful-functions/tree/master/stateful-functions-examples/stateful-functions-ridesharing-example Flavio Pompermaier 于2019年10月31日周四 下午4:53写道： > Hi to all, > yould it be possible to provide also the source code of the UI part of > the ride sharing example? It would be interesting to me how the UI is > reading the data from the Kafka egress. > > Best, > Flavio > >>>
Re: Stateful functions presentation code (UI part)
Yes, I'm interested in how to read data from a UI..which egress should I use? If we use a kafka queue, how to filter data received in the topic? Should I use a correlation id or use a new topic per user? Il Gio 31 Ott 2019, 16:08 Igal Shilman ha scritto: > Hi Flavio, > > We haven't included the UI source code just yet, we've only used it for > demos and talks. > > The reason is that (1) we didn't put a lot of effort and time there (2) > didn't check the time to go through the individual dependencies and > licences. > But we will add that very soon. > > Would having the UI code there would improve your understanding? or is > there another reason? > > Thanks, > Igal > > On Thu, Oct 31, 2019 at 5:44 AM Flavio Pompermaier > wrote: > >> Hi Vino, >> I already checked that code but I can't find the UI part :( >> >> On Thu, Oct 31, 2019 at 12:32 PM vino yang wrote: >> >>> Hi Flavio, >>> >>> Please see this link.[1] >>> >>> Best, >>> Vino >>> >>> [1]: >>> https://github.com/ververica/stateful-functions/tree/master/stateful-functions-examples/stateful-functions-ridesharing-example >>> >>> Flavio Pompermaier 于2019年10月31日周四 下午4:53写道： >>> Hi to all, yould it be possible to provide also the source code of the UI part of the ride sharing example? It would be interesting to me how the UI is reading the data from the Kafka egress. Best, Flavio >>> >>
Re: Stateful functions presentation code (UI part)
Hi Flavio, We haven't included the UI source code just yet, we've only used it for demos and talks. The reason is that (1) we didn't put a lot of effort and time there (2) didn't check the time to go through the individual dependencies and licences. But we will add that very soon. Would having the UI code there would improve your understanding? or is there another reason? Thanks, Igal On Thu, Oct 31, 2019 at 5:44 AM Flavio Pompermaier wrote: > Hi Vino, > I already checked that code but I can't find the UI part :( > > On Thu, Oct 31, 2019 at 12:32 PM vino yang wrote: > >> Hi Flavio, >> >> Please see this link.[1] >> >> Best, >> Vino >> >> [1]: >> https://github.com/ververica/stateful-functions/tree/master/stateful-functions-examples/stateful-functions-ridesharing-example >> >> Flavio Pompermaier 于2019年10月31日周四 下午4:53写道： >> >>> Hi to all, >>> yould it be possible to provide also the source code of the UI part of >>> the ride sharing example? It would be interesting to me how the UI is >>> reading the data from the Kafka egress. >>> >>> Best, >>> Flavio >>> >> >
Re: Stateful functions presentation code (UI part)
Hi Vino, I already checked that code but I can't find the UI part :( On Thu, Oct 31, 2019 at 12:32 PM vino yang wrote: > Hi Flavio, > > Please see this link.[1] > > Best, > Vino > > [1]: > https://github.com/ververica/stateful-functions/tree/master/stateful-functions-examples/stateful-functions-ridesharing-example > > Flavio Pompermaier 于2019年10月31日周四 下午4:53写道： > >> Hi to all, >> yould it be possible to provide also the source code of the UI part of >> the ride sharing example? It would be interesting to me how the UI is >> reading the data from the Kafka egress. >> >> Best, >> Flavio >> >
Re: Stateful functions presentation code (UI part)
Hi Flavio, Please see this link.[1] Best, Vino [1]: https://github.com/ververica/stateful-functions/tree/master/stateful-functions-examples/stateful-functions-ridesharing-example Flavio Pompermaier 于2019年10月31日周四 下午4:53写道： > Hi to all, > yould it be possible to provide also the source code of the UI part of the > ride sharing example? It would be interesting to me how the UI is reading > the data from the Kafka egress. > > Best, > Flavio >
Stateful functions presentation code (UI part)
Hi to all, yould it be possible to provide also the source code of the UI part of the ride sharing example? It would be interesting to me how the UI is reading the data from the Kafka egress. Best, Flavio
