At the end, everything is OK. There was one element not show in my Gist that broke the flow. I called produceTo and onComplete (to close input buffer) on the same flow instance (which materialized twice the flow), and thus created concurrency problem. Now, with a TEE it works perfectly.
I still had a problem of transfer (missing some bytes for large files). Upgraded with akka 2.3.5, everything works perfectly. Low memory consumption and no leaks. Wonderful tool :) Thanks for you help Viktor Le mercredi 20 août 2014 17:19:36 UTC+2, Xavier Bucchiotty a écrit : > > Thanks I check this, > > I also compute a checksum on the fly with a closed value inside the map. > I've seend it's a bad practice and will refactored it in this way: > https://groups.google.com/forum/#!topic/akka-user/i21YLDj5YTw (post about > lost data in TCP Echo) > > > I keep you informed asap > > Le mardi 19 août 2014 16:08:54 UTC+2, Xavier Bucchiotty a écrit : >> >> Hello HakKers, >> >> I need to transfer larges files (~GB) between 2 distants VM. I first used >> akka.io module which works great. >> I currently take a look a akka-stream-experimental module to benefit from >> precious asynchronous backpressure. >> >> But when I create a Flow from a Stream[ByteString], it keeps a reference >> on it. And because of Scala Stream memoization ==> OutOfMemory. >> I tried with an iterator instead. But then my integrations tests fails. >> Seems to be that some chunk have disappeared or some concurrent access to >> the datasource. >> >> From reading the documentation of Flow in scaladsl package, I begin to >> think that streaming a file from 1 point to another is not a use case >> covered by reactive streams. Am I correct? >> Can I expect some improvement in next release on this way? >> >> >> To help, here is the code: >> >> // streamer is an implements Iterable[Byte] and reads the file byte by >> byte. >> val byteStream = new Streamer(buffStream) >> >> // toStream is needed there to pass from Iterator[Iterable[Byte]] to >> Iterable[Iterable[Byte]] >> // because we need a flow of an Iterable[ByteString] >> byteStream.grouped(chunkSize).toStream.map(_.toArray) >> .map(ByteString.fromArray) >> >> Flow(bytes).produceTo(materializer, client.outputStream) >> >> Thanks for your help >> > -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
