Re: DataStream request / response
I will dig deeper into Statefun. Also, yes for now I also can try the Spring/Kafka solution if Statefun doesn't fit. Austin - as far rewriting our microservices in Flink here are some things I was looking for: - We need to be able to easily share/transform data with other teams. Flink SQL seems really nice for this. We also have use cases for real-time analytics within our own application. - If a Flink job is down temporarily due to redeployment, it can just pick up where it left off. With microservices, data gets lost/corrupted. - I'm trying to help improve developer productivity, have better auditing and logging, improve testing, etc. An event driven architecture obviously isn't required to have these things, but it should help. - My intuition is that Flink will have lower hosting costs, but I haven't tested this yet. Thanks everyone for the help, I really appreciate it! -Jason On Fri, Apr 8, 2022 at 2:34 PM Roman Khachatryan wrote: > It seems to be possible now with RequestReplyHandlers from Java SDK > [1] (or other SDKs) unless I'm missing something. > > [1] > > https://nightlies.apache.org/flink/flink-statefun-docs-release-3.2/docs/sdk/java/#serving-functions > > Regards, > Roman > > On Fri, Apr 8, 2022 at 7:45 PM Austin Cawley-Edwards > wrote: > > > > Good suggestion – though a common misconception with Statefun is that > HTTP ingestion is possible. Last time I checked it was still under > theoretical discussion. Do you know the current state there? > > > > Austin > > > > On Fri, Apr 8, 2022 at 1:19 PM Roman Khachatryan > wrote: > >> > >> Hi, > >> > >> Besides the solution suggested by Austing, you might also want to look > >> at Stateful Functions [1]. They provide a more convenient programming > >> model for the use-case I think, while DataStream is a relatively > >> low-level API. > >> > >> [1] > >> https://nightlies.apache.org/flink/flink-statefun-docs-stable/ > >> > >> Regards, > >> Roman > >> > >> On Fri, Apr 8, 2022 at 6:56 PM Austin Cawley-Edwards > >> wrote: > >> > > >> > Hi Jason, > >> > > >> > No, there is no HTTP source/ sink support that I know of for Flink. > Would running the Spring + Kafka solution in front of Flink work for you? > >> > > >> > On a higher level, what drew you to migrating the microservice to > Flink? > >> > > >> > Best, > >> > Austin > >> > > >> > On Fri, Apr 8, 2022 at 12:35 PM Jason Thomas < > katsoftware...@gmail.com> wrote: > >> >> > >> >> I'm taking an existing REST based microservice application and > moving all of the logic into Flink DataStreams. > >> >> > >> >> Is there an easy way to get a request/response from a Flink > DataStream so I can 'call' into it from a REST service? For example, > something similar to this Kafka streams example that uses Spring > ReplyingKafkaTemplate - https://stackoverflow.com/a/58202587. > >> >> > >> >> Thanks for any help! > >> >> > >> >> -Jason > >> >> >
Re: DataStream request / response
It seems to be possible now with RequestReplyHandlers from Java SDK [1] (or other SDKs) unless I'm missing something. [1] https://nightlies.apache.org/flink/flink-statefun-docs-release-3.2/docs/sdk/java/#serving-functions Regards, Roman On Fri, Apr 8, 2022 at 7:45 PM Austin Cawley-Edwards wrote: > > Good suggestion – though a common misconception with Statefun is that HTTP > ingestion is possible. Last time I checked it was still under theoretical > discussion. Do you know the current state there? > > Austin > > On Fri, Apr 8, 2022 at 1:19 PM Roman Khachatryan wrote: >> >> Hi, >> >> Besides the solution suggested by Austing, you might also want to look >> at Stateful Functions [1]. They provide a more convenient programming >> model for the use-case I think, while DataStream is a relatively >> low-level API. >> >> [1] >> https://nightlies.apache.org/flink/flink-statefun-docs-stable/ >> >> Regards, >> Roman >> >> On Fri, Apr 8, 2022 at 6:56 PM Austin Cawley-Edwards >> wrote: >> > >> > Hi Jason, >> > >> > No, there is no HTTP source/ sink support that I know of for Flink. Would >> > running the Spring + Kafka solution in front of Flink work for you? >> > >> > On a higher level, what drew you to migrating the microservice to Flink? >> > >> > Best, >> > Austin >> > >> > On Fri, Apr 8, 2022 at 12:35 PM Jason Thomas >> > wrote: >> >> >> >> I'm taking an existing REST based microservice application and moving all >> >> of the logic into Flink DataStreams. >> >> >> >> Is there an easy way to get a request/response from a Flink DataStream so >> >> I can 'call' into it from a REST service? For example, something >> >> similar to this Kafka streams example that uses Spring >> >> ReplyingKafkaTemplate - https://stackoverflow.com/a/58202587. >> >> >> >> Thanks for any help! >> >> >> >> -Jason >> >>
Re: DataStream request / response
Good suggestion – though a common misconception with Statefun is that HTTP ingestion is possible. Last time I checked it was still under theoretical discussion. Do you know the current state there? Austin On Fri, Apr 8, 2022 at 1:19 PM Roman Khachatryan wrote: > Hi, > > Besides the solution suggested by Austing, you might also want to look > at Stateful Functions [1]. They provide a more convenient programming > model for the use-case I think, while DataStream is a relatively > low-level API. > > [1] > https://nightlies.apache.org/flink/flink-statefun-docs-stable/ > > Regards, > Roman > > On Fri, Apr 8, 2022 at 6:56 PM Austin Cawley-Edwards > wrote: > > > > Hi Jason, > > > > No, there is no HTTP source/ sink support that I know of for Flink. > Would running the Spring + Kafka solution in front of Flink work for you? > > > > On a higher level, what drew you to migrating the microservice to Flink? > > > > Best, > > Austin > > > > On Fri, Apr 8, 2022 at 12:35 PM Jason Thomas > wrote: > >> > >> I'm taking an existing REST based microservice application and moving > all of the logic into Flink DataStreams. > >> > >> Is there an easy way to get a request/response from a Flink DataStream > so I can 'call' into it from a REST service? For example, something > similar to this Kafka streams example that uses Spring > ReplyingKafkaTemplate - https://stackoverflow.com/a/58202587. > >> > >> Thanks for any help! > >> > >> -Jason > >> >
Re: DataStream request / response
Hi, Besides the solution suggested by Austing, you might also want to look at Stateful Functions [1]. They provide a more convenient programming model for the use-case I think, while DataStream is a relatively low-level API. [1] https://nightlies.apache.org/flink/flink-statefun-docs-stable/ Regards, Roman On Fri, Apr 8, 2022 at 6:56 PM Austin Cawley-Edwards wrote: > > Hi Jason, > > No, there is no HTTP source/ sink support that I know of for Flink. Would > running the Spring + Kafka solution in front of Flink work for you? > > On a higher level, what drew you to migrating the microservice to Flink? > > Best, > Austin > > On Fri, Apr 8, 2022 at 12:35 PM Jason Thomas wrote: >> >> I'm taking an existing REST based microservice application and moving all of >> the logic into Flink DataStreams. >> >> Is there an easy way to get a request/response from a Flink DataStream so I >> can 'call' into it from a REST service? For example, something similar to >> this Kafka streams example that uses Spring ReplyingKafkaTemplate - >> https://stackoverflow.com/a/58202587. >> >> Thanks for any help! >> >> -Jason >>
Re: DataStream request / response
Hi Jason, No, there is no HTTP source/ sink support that I know of for Flink. Would running the Spring + Kafka solution in front of Flink work for you? On a higher level, what drew you to migrating the microservice to Flink? Best, Austin On Fri, Apr 8, 2022 at 12:35 PM Jason Thomas wrote: > I'm taking an existing REST based microservice application and moving all > of the logic into Flink DataStreams. > > Is there an easy way to get a request/response from a Flink DataStream > so I can 'call' into it from a REST service? For example, something > similar to this Kafka streams example that uses Spring > ReplyingKafkaTemplate - https://stackoverflow.com/a/58202587. > > Thanks for any help! > > -Jason > >
DataStream request / response
I'm taking an existing REST based microservice application and moving all of the logic into Flink DataStreams. Is there an easy way to get a request/response from a Flink DataStream so I can 'call' into it from a REST service? For example, something similar to this Kafka streams example that uses Spring ReplyingKafkaTemplate - https://stackoverflow.com/a/58202587. Thanks for any help! -Jason
