Hi Victor,
thank you for your reply and here are my remarks :
Point 1. You have a source of "events" and the only way you know if there
are any events, is if you ask for events
exactly
Point 2.You want to consume events from this source
Yes
Point 3) You only need to consume events from the source if there is
demand from "downstream"
It is the other way round. The source logically pushes the events
downstream.
Ideally as soon as theses Events arrive I would like them to be pushed
downstream. So its more a push scenario.
The source pushes as many Events as the downstream processes can handle.
(But I have to limit myself for polling the database at a rate of 2-5
seconds because of hardware reasons)
Point 4) Does the source of events ever end, and how do you know?
No the source never ends. External Systems always insert new Event records
into the database
Point 5. can you ask for a specified number of events or does polling
imply reading all available events?
No I can ask for a specified number of events.
Here is a concrete business case:
1) External systems want Email Messages to be emailed to Users.
2) They insert Email Events (Records with email-address, content,
user_id,etc) into a Email Table (log-table)
3) Email-Send-Service sends these Emails over the wire
4) Acknowledge-Service logs the Email-Send Event and cleans up the Email
Table (log-table).
So I can set up a Stream: Email-Source -> Email-Send-Service -
Acknowledge-Service .
I am now considering two ways of doing it and are hoping for your advice (-:
1) create the following
Email-Source (select * from Email limit 1000) -> Email-Send-Service
-> Acknowledge-Service
and materialize this stream every two seconds
2) Set up a custom source which polls forever
Polling-Neverending-Email-Source (select * from Email limit 1000 every
two seconds inside the Actor) -> Email-Send-Service ->
Acknowledge-Service
Background: I am building up with Akka Streams a inhouse toolkit which
deals with many ESB typical processes.
The above usecase profits not so much from throughput (actually the users
could live with some delay) but more from
reusing stream components and having a nice clear Flow DSL.
Many Greetings
John
Am Sonntag, 7. Februar 2016 13:05:59 UTC+1 schrieb √:
>
> Hi John!
>
> I think I can help you, but I have some follow-on questions :)
>
> On Sun, Feb 7, 2016 at 12:52 PM, <[email protected] <javascript:>> wrote:
>
>> Hi Victor,
>> I know you love simple elegant code but
>> I looked at unfoldAsync and I don't see that it solves the usecase I
>> have in mind.
>>
>> I'll try to explain why because I am not sure if I am overlooking
>> something obvious?
>>
>> An external system writes events into a log table. Since this is a
>> non-reactive-sql-database the consumer needs to poll the log table (for
>> example every 2 seconds) for new events.
>>
>
> (Why) does it need to poll the log table if there is no demand for events?
>
>
>>
>> To convert this scenario into a streaming source using unfoldAsync
>> I need to implement a polling loop inside the future so that
>> Future.success gets only called when new events are inserted in the log
>> table.
>>
>
> This seems much like a technical aspect rather than a requirement.
>
>
>>
>> If I don't use polling the future would send 0 Events upstream and the
>> stream would come to an end?
>>
>
> Events go downstream, or did I misunderstand something?
>
>
>>
>> That's why I like using instead of unfoldAsync an Actor like JobManager
>> <http://doc.akka.io/docs/akka-stream-and-http-experimental/2.0.3/java/stream-integrations.html>.
>>
>> as an Source. Within the Actor I can be more fine grained and use the
>> scheduler to implement the polling logic.
>>
>
> Let's take a step back, what are the actual requirements?
>
> 1. You have a source of "events" and the only way you know if there are
> any events, is if you ask for events
> 2. You want to consume events from this source
> 3. You only need to consume events from the source if there is demand from
> "downstream"
> 4. Does the source of events ever end, and how do you know?
> 5. can you ask for a specified number of events or does polling imply
> reading all available events?
>
>
>>
>> Many Greetings
>> John
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> Am Mittwoch, 3. Februar 2016 23:10:15 UTC+1 schrieb √:
>>>
>>>
>>> http://doc.akka.io/api/akka-stream-and-http-experimental/2.0.3/?_ga=1.45749860.1579561034.1353497989#akka.stream.scaladsl.Source$
>>>
>>> On Wed, Feb 3, 2016 at 11:06 PM, <[email protected]> wrote:
>>>
>>>> where do I find unfold/unfoldAsync ? I looked at
>>>> http://doc.akka.io/docs/akka-stream-and-http-experimental/2.0.3/stages-overview.html
>>>> ?
>>>> Many Greetings
>>>> John
>>>>
>>>>
>>>> Am Mittwoch, 3. Februar 2016 20:51:17 UTC+1 schrieb √:
>>>>>
>>>>> I don't see why you'd need to write a custom GraphStage for this.
>>>>>
>>>>> unfold/unfoldAsync paired with a buffer with an appropriate dropping
>>>>> policy should work, no?
>>>>>
>>>>> On Wed, Feb 3, 2016 at 8:47 PM, clca <[email protected]> wrote:
>>>>>
>>>>>> Yes this is close to what I need to do.
>>>>>> The code in the onPull method is something like
>>>>>>
>>>>>> val m = ReadFromService(...)
>>>>>> if(m != null)
>>>>>> push(out, m)
>>>>>>
>>>>>> in a traditional app reading would be done in a loop
>>>>>>
>>>>>> while(true) {
>>>>>> val m = ReadFromService(...)
>>>>>> //do something with the new message
>>>>>> }
>>>>>>
>>>>>> So I'll add a mechanism in the Source to keep polling the external
>>>>>> service.
>>>>>>
>>>>>> Thanks!
>>>>>>
>>>>>>
>>>>>> On Wednesday, February 3, 2016 at 12:55:16 AM UTC-8,
>>>>>> [email protected] wrote:
>>>>>>>
>>>>>>> I've done something similar.
>>>>>>> I adapted this JobManager
>>>>>>> <http://doc.akka.io/docs/akka-stream-and-http-experimental/2.0.3/java/stream-integrations.html>.
>>>>>>>
>>>>>>> When no data is available( for example when it recieves a Request(16)
>>>>>>> Messag) it starts a" polling Actor" which polls an external Database
>>>>>>> for
>>>>>>> more data.
>>>>>>> Does this help?
>>>>>>>
>>>>>>> Am Mittwoch, 3. Februar 2016 08:33:16 UTC+1 schrieb clca:
>>>>>>>>
>>>>>>>> I searched through the topics but I could not find any example on
>>>>>>>> how to pull from a Source in the case where data is coming in bursts.
>>>>>>>> I
>>>>>>>> built a customized Source that read from an external source. Data is
>>>>>>>> coming
>>>>>>>> in bursts, the flow can stop for a while so I need to keep pulling the
>>>>>>>> Source in such a way I can keep reading data (the actual read process
>>>>>>>> is
>>>>>>>> done in the old poll fashion). I guess I need something like a
>>>>>>>> KeepAlive
>>>>>>>> type of mechanism for the data stream.
>>>>>>>>
>>>>>>>> BTW: Fantastic job with Akka, Akka Stream & Akka HTTP!
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>> Claudio
>>>>>>>>
>>>>>>>>
>>>>>>> --
>>>>>> >>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>> >>>>>>>>>> Check the FAQ:
>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>> >>>>>>>>>> Search the archives:
>>>>>> https://groups.google.com/group/akka-user
>>>>>> ---
>>>>>> You received this message because you are subscribed to the Google
>>>>>> Groups "Akka User List" group.
>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>> send an email to [email protected].
>>>>>> To post to this group, send email to [email protected].
>>>>>> Visit this group at https://groups.google.com/group/akka-user.
>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Cheers,
>>>>> √
>>>>>
>>>> --
>>>> >>>>>>>>>> Read the docs: http://akka.io/docs/
>>>> >>>>>>>>>> Check the FAQ:
>>>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>>> >>>>>>>>>> Search the archives:
>>>> https://groups.google.com/group/akka-user
>>>> ---
>>>> You received this message because you are subscribed to the Google
>>>> Groups "Akka User List" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to [email protected].
>>>> To post to this group, send email to [email protected].
>>>> Visit this group at https://groups.google.com/group/akka-user.
>>>> For more options, visit https://groups.google.com/d/optout.
>>>>
>>>
>>>
>>>
>>> --
>>> Cheers,
>>> √
>>>
>> --
>> >>>>>>>>>> Read the docs: http://akka.io/docs/
>> >>>>>>>>>> Check the FAQ:
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>> >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "Akka User List" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to [email protected] <javascript:>.
>> To post to this group, send email to [email protected]
>> <javascript:>.
>> Visit this group at https://groups.google.com/group/akka-user.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
>
> --
> Cheers,
> √
>
--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ:
>>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
