On Tue, Apr 22, 2014 at 1:42 PM, Patrik Nordwall <[email protected]>wrote:
> Yes, uncompressing processor must be able to buffer and send smaller > chunks downstream. If it has a compression ratio of 10:1, and it gets > demand from downstream of 1000 chunks it should request 100 chunks from its > upstream. It can not know exactly, but it can do a fairly good guess, and > even adapt dynamically. > IIUC, you expect each processor to be explicitly configured to know the desired chunk size, for the entire stream or for each producer/consumer pair separately. Or at least to try to preserve the input chunk size in its output (which is not always the correct behavior). What worries me is the lack of a generic interface to handle this setup. If I get a Producer<ByteString> as a parameter, I have no way to communicate to it what size chunks I need; I have to trust that whoever created the actual producer implementation instance configured it correctly. I can't enforce the constraints that the reactive streams SPI is supposed to guarantee, such as not getting too much data as input. It's also bad for separation of mechanism (processors) and policy (chunk size). One day we'll have support for deploying an akka-stream on top of akka-remoting, across physical nodes. The stream (materializer?) will want to control chunk size and buffering policy, which should be different between local and remote links. Without a standard interface, the materializer wouldn't be able to tell a black-box Producer to give it chunks of size X. I expect projects will start creating local solutions to this problem (I think akka-http will need one), and I hope eventually one solution will become a standard library, but it surprises me that the Reactive Streams designers don't think this is important enough to be part of the API . (As an optional, but standardized, extra interface a Producer/Consumer can implement.) Thanks, Daniel Armak -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
