On Wed, Mar 18, 2015 at 5:21 PM, Matthew Pocock < [email protected]> wrote:
> Hi, > > I was wondering how easy it would be to implement a reactive-stream with > back-pressure for parsing files. I have some apps that chew on a large > number of large files, each file containing many data items. Threads can > block on the IO. > > There may be a whole down-stream pipeline of processing going on from the > data in from the IO, often consuming data from several of these files but > also often with data processing per data item. > > Ideally, there would be some non-blocking code that it processing a file. > It would produce a stream of data items. While these are being eaten, it > continues to chew on the file. When the downstream processing has backed > up, the parsing suspends and that thread is handed off for other work. When > it is no longer backed up, it would regain a thread and unsuspend and chew > some more on the file. > > Does this sound sensible, and is it something I can implement today with > akka? > You can implement this today with Akka streams. Easiest is probably to use a iterator based Source for reading the file. We have plans for providing better support for files, see https://github.com/akka/akka/issues/15588. Cheers, Patrik > > Cheers, > Matthew > > -- > Dr Matthew Pocock > Turing ate my hamster LTD > mailto: [email protected] > > Integrative Bioinformatics Group, School of Computing Science, Newcastle > University > mailto: [email protected] > > gchat: [email protected] > msn: [email protected] > irc.freenode.net: drdozer > skype: matthew.pocock > tel: (0191) 2566550 > mob: +447535664143 > > -- > >>>>>>>>>> Read the docs: http://akka.io/docs/ > >>>>>>>>>> Check the FAQ: > http://doc.akka.io/docs/akka/current/additional/faq.html > >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user > --- > You received this message because you are subscribed to the Google Groups > "Akka User List" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at http://groups.google.com/group/akka-user. > For more options, visit https://groups.google.com/d/optout. > -- Patrik Nordwall Typesafe <http://typesafe.com/> - Reactive apps on the JVM Twitter: @patriknw -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
