> On Jun 7, 2015, at 3:19 AM, Akka Team <[email protected]> wrote: > > Hi Jason, > > On Mon, Jun 1, 2015 at 8:24 PM, Jason Martens <[email protected] > <mailto:[email protected]>> wrote: > Hello All, > I deal with a lot of large files on Amazon S3 and disk, which have been > indexed into chunks that can be used individually. What I would like to do is > have a Source of offsets and sizes which can be filtered to then pass into > another stream element which could read that particular file from S3 or from > a local disk. However, my attempts thus far to make this work have failed, > since the stream element which does the reading seems to block after the > first read. > > What do you mean it is blocked? What is the code that fails? Can you give a > small example?
I’m hoping I have time to work on a small example I can make public… We’ll see. I realize it’s hard to understand what I’m describing without it. Sorry! > > In effect, I guess this is equivalent to having an upstream and downstream > source, which is perhaps not supported. > > Umm, what does a downstream Source mean? Sources have exactly one output > port, so they cannot be downstream. If you want to have a stream processing > element that takes input, then it must be a Sink, Flow, BidiFlow (or a more > complex graph) that has at least one input port. I think what I was trying to describe is having a Flow element which was actually reading data from S3 or a file as a side-effect. It would look something like this: OffsetSource ~> Filter ~> FileReaderFlow ~> Sink I considered having a Merge element which would have an OffsetSource and FileReaderSource, but for this to work, FileReaderSource would have to get the offsets requested as input, perhaps from the demand requested? It doesn’t look like that’s supported now. > > Alternatively, you can take the stream of indices and offsets, map that > stream to a Sources (so now you have a stream of Sources) then use flatten to > create one contiguous stream from them. I think this is exactly the solution I’m looking for, but couldn’t think of, thanks! I’ll give this a try. Jason -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
