On Monday, 21 April 2014 12:32:57 UTC+1, Patrik Nordwall wrote: > > I intend to read the documentation fully, but I was a little disappointed > that the activator examples did not have a simple example with an > (effectively) infinite data source that can only be polled in serial, with > parallel (but controllably finite) consumers. > > > Isn't that demonstrated with the random number generator source, and its > slow consumers? >
I missed that one. How many consumers are there at any given moment? Is it in here somewhere? https://github.com/typesafehub/activator-akka-stream-scala/tree/master/src/main/scala/sample/stream My example is trying to simulate real world examples of: * parsing loads of data coming from a single data source (e.g. indexing a multi-TB database with Lucene, running in under 1GB) * parallel finite element calculations, where there are a lot more elements than bytes of RAM so they have to be batched (and with minimal object churn) > BasicTransformation defines the input text in code (to make it simple), > but the iterator next() is not called more than what can be consumed > downstream. > > Isn't the log file sample more similar to your text file input? It does > not read the whole file (if it was large) into memory. > Right, so you pass an Iterator[String] to the flow. Yes, that looks good, sorry I missed it. But Iterator[T] is a little too ill-defined near the end of the stream (that's why I created my own Producer in the RxJava playground). For example, does it block on hasNext or on next if it knows there are more elements that are not yet available, or does it close up and go home? Traditional Java APIs (such as Queues) would actually return early if a queue was exhausted, instead of waiting for a poison pill. In any case, if Flow can handle an Iterator that blocks (e.g. for I/O), it's probably good enough for most situations. It would be even better if it knew how often to poll the data source... for example I have an EEG (brain scanner) library which has to poll the device at 57Hz. If it does it too quickly, there are inadequacies in the underlying hardware which result in busy spinning (yes, it's insane, and it really does eat the whole CPU)... but if I don't poll quickly enough then data can be lost. Relevant code (and my non-stream hack) here: https://github.com/fommil/emokit-java/blob/master/src/main/java/com/github/fommil/emokit/EmotivHid.java#L84 Best regards, Sam -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com. To post to this group, send email to akka-user@googlegroups.com. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
