Hi Roland, Thank you for the reply and sorry for the delay.
The way I'd like to look at stream materialization is as a facility for runtime information discovery and action execution. This mechanism sits next to the main information flow and at times has the ability to influence that flow. A management sub-channel is probably not technically accurate, but I hope you get my meaning. The flexibility of this discovery/management facility in its current form is what I'm worried about. There are good examples in the documentation for how to expose both discovery and management operations via materialization. val (promise, cancellable, future) = r11.run() promise.success(0) cancellable.cancel() future.map(_ + 3) This does a pretty good job of getting these "monitoring and management" concerns across the stream boundary. Issue 16168 will hopefully address access to these values from within a stream. I'm tempted to use this facility for JMX-like functionality on a stream. But if you aren't careful you'll run into a problem with shape. Let's say that version 1 of my lib ships out with a throttler stage. That stage, like in the docs, throttles at 1/s and exposes a cancellable "management" object. That's great, until folks start asking for the ability to change the throttle rate at runtime as well. This functionality could of course be implemented in v2, however the shape of my stage's materialized object then needs to change. So I replace my Cancellable object with an Operations object that has both cancel() and updateRate(newRate) methods. I push this out in v2, but this is a breaking change for everyone in v1 because the materialized shape is part of the interface and used or not is still there. The folks that make vanilla use of my stage are annoyed because they're forced to recompile against a feature they weren't even using. Even if I have the foresight to build stages with an easier to evolve materialization shape (maybe an object that exposes Operations and Informations in a more flexible way) I don't have the ability to benefit from foundational enhancements by the stream library itself. For example, even when I finish rolling my own JMX concept, I can't ask Akka to help me add to this functionality with their own materialization details (where would such information be placed?). Currently (AFAIK) Akka Streams doesn't expose the ability to discover or manipulate per-stage runtime materialization, as in the materialization provided by the materializer. Can't discover the current default internal buffer size, can't modify this size, update thread pool settings, access stats metrics of message flow through the internal actors, or execution time taken. And the lack of this functionality today is fine. Over time more and more base operations and informations could be exposed by the runtime, and developers could add their own operations and informations on top of this base. But this only works if there is a base and this base is of some flexible shape. I guess what I'm asking for is for the Akka team to look at the materialization value system as more of a per-stage command and control system. Something that would benefit from Akka prescribing a management methodology instead of leaving it completely open. IMHO the current form, while open and type safe, ends up being brittle. There isn't a methodology or framework by which Akka can raise all ships in the future (by providing certain standard ops/infos) and developers without management framework development experience will accidentally find themselves painted into a corner wrt to the shape of their materialization values. I think I'll stop here, but next up would be a discussion on the proper way to interact with materialization values. Even if I get everything that I ask for and the shape turns into some standardized object exposing information and actions, this still ends up being shared mutable state bleeding out of the Stream with a now undefined concurrency model. Wouldn't it be better to expose some sort of action sink and information source instead of a single shared object? Thanks for your time, Jim On Sunday, March 8, 2015 at 11:24:58 AM UTC-7, rkuhn wrote: > > Hi Jim, > > thanks for starting the discussion! Replies inline. > > 5 mar 2015 kl. 21:30 skrev Jim Hazen <jimhaz...@gmail.com <javascript:>>: > > I've been following Akka Streams for bit. I've noticed the changes in M4 > and of course read the materialization sections of: > http://doc.akka.io/docs/akka-stream-and-http-experimental/1.0-M4/scala/stream-flows-and-basics.html > > I have tremendous respect for the Akka team and understand that getting > this right is hard. But, are we there yet? > > > Basically yes, but one thing is still missing > <https://github.com/akka/akka/issues/16168>. I hope that we’ll be able to > fix the remaining things without further disruptive changes. > > Originally the concept was Map based, with the ability for stages to set > keys within the map and eventually the client had access to this map. Not > really type safe, or clear as to what keys were available, or what their > types were, etc. So that's not great. But on the flip side (and why I > think it was reached for first) it was extensible. > > > That’s what we thought, but we discovered that this was not actually the > case: the MaterializedMap may be extensible in a certain fashion, but it > lacks distinction in the case of component reuse—the problem boils down to > needing multiple values for a single key, and that would obviously be very > confusing and non-deterministic. > > Now with M4, we are much more clear and explicit about what > materialization each stage offers. This looks a lot better. But I'm > worried about what happens when you attempt to evolve these materialization > types later, especially for stages provided by the core library. > > > A component that is offered by a library (like StreamTcp’s incoming > connection source in case of a server socket) will have two kinds of types: > the elements that it processes, and the materialization results. To me > these behave identically wrt. evolution, changing the materialized value in > an incompatible way is just as breaking as changing the type of elements > produced. But then again we have APIs for TCP sockets that have returned > the same types for decades, so I am not overly worried that we will have to > change things—unless we make a mess now (which we are trying hard to avoid, > hence the long development time for this feature). > > I'm thinking down the road to Monk, where the goal is to provide a nice > tracing abstraction and SPI. And if folks are interested in tracing Actors > today, with Streams, they'll be interested in tracing materialized flows > tomorrow. > > Given this current mechanism for exposing a per-stage materialization > value, how later would we switch all of our Unit materialization values to > materialization values that contain "standard metrics"? It isn't there > today, but in the future a lot of runtime metrics could be gathered and > exposed. > > > As far as I am aware tracing does not interact with materialized values, > in which way should that be the case? All tracing libraries that I know > instrument the code—either before or after compilation—and siphon off usage > information to some side channel behind the scenes. The tracing data is > never part of the public API, and for good reason, it is a completely > separate concern. > > With M4, I don't see a clean way of evolving or exposing new "secondary" > materialization values over time. "Secondary" is in the eye of the > beholder, but we have preferredMerge stages, and things like Try and Option > that have success based affinity within flow processing. > > > I don’t know what kind of “secondary” materialization values you have in > mind, but the process for adding features to existing classes is the same > as always: add some methods that make the new feature accessible, keep all > the old methods working in the same fashion as before. As a concrete > example imagine that the ServerBinding class, which currently holds the > localAddress and the unbind() functionality, learns some new trick in the > future. We can just add a method “doIt()” on that class and everything > keeps working. > > I can't prescribe a complete solution in this post but I would like to > talk this out. > > Down the road how do we intend to support the exposure of new > materialization based information and/or patterns? How would "adaptive > flows", flows that adapt to runtime realities, be implemented? How would > one get access to a Stream of materialization values of a running Stream? > > > Adaptive flows will adapt, but that happens after materialization, so I’m > not sure I understand the exact use-case you have in mind. It could well be > interesting to expose the tuned parameters as additional Outlets on the > given processing stages, but that is obviously just part of the model and > requires no additions. > > I hope this is food for positive thought and a positive discussion. > > PS: This particular post uses tracing and metrics as an example, but I > think there's a larger fundamental problem to be solved. Should each stage > need to evolve from a materialization of Unit -> tuple/wrapper -> new > tuple/wrapper. All the while likely breaking backward compatibility in the > process. > > > The current model uses normal functional composition, and it is exactly as > easy/hard to evolve as anything in functional programming, unless I am > overlooking something. > > Thanks again for opening the discussion; me saying that I think we have it > all figured out does by no means imply correctness, and it is important to > put our thinking to the test, so if you can point out flaws we’ll be happy > to fix them! > > Regards, > > *Dr. Roland Kuhn* > *Akka Tech Lead* > Typesafe <http://typesafe.com/> – Reactive apps on the JVM. > twitter: @rolandkuhn > <http://twitter.com/#!/rolandkuhn> > > -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com. To post to this group, send email to akka-user@googlegroups.com. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
