Thanks Gyula, Another question i have..
> ... while external model updates would be *tricky *to keep consistent. Is that still the case if the Operator treat the external model as read-only ? We create another stream that will update the external model separately. Could you please elaborate more about this one ? Cheers On Wed, Aug 19, 2015 at 2:52 PM, Gyula Fóra <[email protected]> wrote: > In that case I would apply a map to wrap in some common type, like a n > Either<t1,t2> before the union. > > And then in the coflatmap you can unwrap it. > On Wed, Aug 19, 2015 at 9:50 AM Welly Tambunan <[email protected]> wrote: > >> Hi Gyula, >> >> Thanks. >> >> However update1 and update2 have a different type. Based on my >> understanding, i don't think we can use union. How can we handle this one ? >> >> We like to create our event strongly type to get the domain language >> captured. >> >> >> Cheers >> >> On Wed, Aug 19, 2015 at 2:37 PM, Gyula Fóra <[email protected]> wrote: >> >>> Hey, >>> >>> One input of your co-flatmap would be model updates and the other input >>> would be events to check against the model if I understand correctly. >>> >>> This means that if your model updates come from more than one stream you >>> need to union them into a single stream before connecting them with the >>> event stream and applying the coatmap. >>> >>> DataStream updates1 = .... >>> DataStream updates2 = .... >>> DataStream events = .... >>> >>> events.connect(updates1.union(updates2).broadcast()).flatMap(...) >>> >>> Does this answer your question? >>> >>> Gyula >>> >>> >>> On Wednesday, August 19, 2015, Welly Tambunan <[email protected]> wrote: >>> >>>> Hi Gyula, >>>> >>>> Thanks for your response. >>>> >>>> However the model can received multiple event for update. How can we do >>>> that with co-flatmap as i can see the connect API only received single >>>> datastream ? >>>> >>>> >>>> > ... while external model updates would be tricky to keep consistent. >>>> Is that still the case if the Operator treat the external model as >>>> read-only ? We create another stream that will update the external model >>>> separately. >>>> >>>> Cheers >>>> >>>> On Wed, Aug 19, 2015 at 2:05 PM, Gyula Fóra <[email protected]> wrote: >>>> >>>>> Hey! >>>>> >>>>> I think it is safe to say that the best approach in this case is >>>>> creating a co-flatmap that will receive updates on one input. The events >>>>> should probably be broadcasted in this case so you can check in parallel. >>>>> >>>>> This approach can be used effectively with Flink's checkpoint >>>>> mechanism, while external model updates would be tricky to keep >>>>> consistent. >>>>> >>>>> Cheers, >>>>> Gyula >>>>> >>>>> >>>>> >>>>> >>>>> On Wed, Aug 19, 2015 at 8:44 AM Welly Tambunan <[email protected]> >>>>> wrote: >>>>> >>>>>> Hi All, >>>>>> >>>>>> We have a streaming computation that required to validate the data >>>>>> stream against the model provided by the user. >>>>>> >>>>>> Right now what I have done is to load the model into flink operator >>>>>> and then validate against it. However the model can be updated and >>>>>> changed >>>>>> frequently. Fortunately we always publish this event to RabbitMQ. >>>>>> >>>>>> I think we can >>>>>> >>>>>> >>>>>> 1. Create RabbitMq listener for model changed event from inside >>>>>> the operator, then update the model if event arrived. >>>>>> >>>>>> But i think this will create race condition if not handle >>>>>> correctly and it seems odd to keep this >>>>>> >>>>>> 2. We can move the model into external in external memory cache >>>>>> storage and keep the model up to date using flink. So the operator >>>>>> will >>>>>> retrieve that from memory cache >>>>>> >>>>>> 3. Create two stream and using co operator for managing the >>>>>> shared state. >>>>>> >>>>>> >>>>>> What is your suggestion on keeping the state up to date from external >>>>>> event ? Is there some kind of best practice for maintaining model up to >>>>>> date on streaming operator ? >>>>>> >>>>>> Thanks a lot >>>>>> >>>>>> >>>>>> Cheers >>>>>> >>>>>> >>>>>> -- >>>>>> Welly Tambunan >>>>>> Triplelands >>>>>> >>>>>> http://weltam.wordpress.com >>>>>> http://www.triplelands.com <http://www.triplelands.com/blog/> >>>>>> >>>>> >>>> >>>> >>>> -- >>>> Welly Tambunan >>>> Triplelands >>>> >>>> http://weltam.wordpress.com >>>> http://www.triplelands.com <http://www.triplelands.com/blog/> >>>> >>> >> >> >> -- >> Welly Tambunan >> Triplelands >> >> http://weltam.wordpress.com >> http://www.triplelands.com <http://www.triplelands.com/blog/> >> > -- Welly Tambunan Triplelands http://weltam.wordpress.com http://www.triplelands.com <http://www.triplelands.com/blog/>
