Hi, How would I use reactive streams in general, and/or akka streams in particular, to represent a stream of binary data? I'm thinking of all binary streams that don't have a natural division into chunks of a fixed (small) size; you can divide them into chunks any way you want and all processors in the pipeline will still work. Like downloading a big file over HTTP, decompressing and saving it to disk.
The natural representation is a stream of ByteString (or byte[] in Java). But each ByteString can be arbitrarily large. It's no good to tell the producer I'm willing to accept one more element, if I have no idea what size it's going to be. Maybe the producer is reading from a 100MB ByteString it already has in memory, and the easiest thing for it to do (i.e. the easiest way for a programmer to code the producer) is to send all of the data as one element. What I really want is to tell it how many more bytes (or characters, etc) I'm willing to accept, but in the current reactive streams API that would require sending each byte in a separate call to onNext. Some of the implementations might address this scenario, but it seems to me that this will be a common use case and so standardization and interoperability would be of value. I'm sure this has all been thought of. What's the recommended usage pattern? ----- Also, some processors do have a natural chunk size (e.g. compression, encryption, network packet transmission). This size will rarely match the size of the incoming chunks (if only because different processors in a pipeline have different chunk sizes). To maintain efficiency it might be desirable for some processors to buffer their output and only forward chunks of discrete sizes. This, too, might benefit from typed declarations of the chunk sizes each processor prefers or requires as input. Otherwise the programmer will have to configure each processor manually to do the right amount of buffering, because the processor won't be automatically aware of the preferences of the next processor in the pipeline. This is a more complex scenario, so I expect it will be left to each implementation to introduce its own patterns. Daniel Armak -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
