Hi there, nice blog and example app! Please use SynchronousFileSource instead of using getLines from the scala io.Source - it will be much faster (much): Here's docs on how to use it: http://doc.akka.io/docs/akka-stream-and-http-experimental/1.0-RC2/scala/stream-io.html#Streaming_File_IO
On Tue, Jun 23, 2015 at 3:08 PM, Endre Varga <endre.va...@typesafe.com> wrote: > Hi, > > On Tue, Jun 23, 2015 at 12:29 PM, BigAl <albert.pastr...@intenthq.com> > wrote: > >> Hi all, >> >> In my company we've been running some tests to evaluate if Akka Streams >> would be useful for us to process both the Wikidata and Wikipedia dumps. >> >> I have to say that both PoC were very satisfactory and we found Akka >> Streams a really nice framework to play with :-). >> > > I think what you do is actually a sweet spot for Akka Streams, these kind > of use-cases match well with the underlying assumptions. > > >> >> After that, we wanted to share our findings with the rest of the world :) >> so we've created a small repo that contains part of the PoC, we have >> structured in a way that can be explain in a single (and simple) blog post: >> Github repo: https://github.com/intenthq/wikidata-akka-streams >> Blog post: http://engineering.intenthq.com/2015/06/wikidata-akka-streams/ >> >> We would really appreciate any comments about it. >> > > There is an InputStreamSource available, so you can use it directly where > you read the file: > http://doc.akka.io/api/akka-stream-and-http-experimental/1.0-RC3/#akka.stream.io.InputStreamSource$ > > The above does not do line parsing, but the upcoming RC4 contains a simple > line parsing (or other delimiter based parsing) stage: > https://github.com/akka/akka/blob/release-2.3-dev/akka-stream/src/main/scala/akka/stream/io/Framing.scala#L53 > > You can try that combination after RC4 comes out. > > -Endre > > >> >> Thanks, >> >> BigAl >> >> -- >> >>>>>>>>>> Read the docs: http://akka.io/docs/ >> >>>>>>>>>> Check the FAQ: >> http://doc.akka.io/docs/akka/current/additional/faq.html >> >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user >> --- >> You received this message because you are subscribed to the Google Groups >> "Akka User List" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to akka-user+unsubscr...@googlegroups.com. >> To post to this group, send email to akka-user@googlegroups.com. >> Visit this group at http://groups.google.com/group/akka-user. >> For more options, visit https://groups.google.com/d/optout. >> > > -- > >>>>>>>>>> Read the docs: http://akka.io/docs/ > >>>>>>>>>> Check the FAQ: > http://doc.akka.io/docs/akka/current/additional/faq.html > >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user > --- > You received this message because you are subscribed to the Google Groups > "Akka User List" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to akka-user+unsubscr...@googlegroups.com. > To post to this group, send email to akka-user@googlegroups.com. > Visit this group at http://groups.google.com/group/akka-user. > For more options, visit https://groups.google.com/d/optout. > -- Cheers, Konrad 'ktoso' Malawski Akka <http://akka.io/> @ Typesafe <http://typesafe.com/> -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com. To post to this group, send email to akka-user@googlegroups.com. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
