Re: A tiny producer-consumer framework
Ken Wesson kwess...@gmail.com writes: On Sat, Jan 22, 2011 at 11:26 AM, Eric Schulte schulte.e...@gmail.com wrote: Nice concise example, Thanks. A while back I implemented something similar; a propagator system using agents fed with `send', coming in at a slightly more verbose ~35 lines of code. http://cs.unm.edu/~eschulte/research/propagator/ Interesting. But I have some questions ... (defn set-cell Set the value of a cell [cell value] (send cell (fn [_] value))) Why (fn [_] value) instead of (constantly value)? OK, actually (constantly value) is textually longer, but I'd argue it might be clearer. And it works; (constantly foo) accepts all arities. It's something like (fn [ _] foo). I agree constantly would have been a better choice, had I known it existed. A question for Clojure's developers: why does restart-agent require the agent to be in a failed state? That seems to be a superfluous requirement. Atoms have reset! and refs ref-set to clobber the old value; agents would have something analogous if restart-agent worked on agents that weren't failed too. (defmacro propagator Define a new propagator. [name in-cells out-cells body] `(do (defn ~(with-meta name (assoc (meta name) :in-cells in-cells :out-cells out-cells)) Whoa. Why not (defmacro propagator Define a new propagator. [name in-cells out-cells body] `(do (defn ~(vary-meta name assoc :in-cells in-cells :out-cells out-cells) ? Again, I work with the functions I am aware of. Thanks for these pointers, they are very helpful especially as it can be difficult to learn new functions once a sufficient subset has been discovered. (doseq [cell# ~in-cells] (add-neighbor cell# ~name)) There's no forward-declaration of add-neighbor before this. ~name)) Usually a deffoo form returns the new var, not just a symbol. This can be fixed with (declare add-neighbor) (defmacro propagator Define a new propagator. [name in-cells out-cells body] `(let [v# (defn ~(vary-meta name assoc :in-cells in-cells :out-cells out-cells) ~in-cells ~@body) (doseq [cell# ~in-cells] (add-neighbor cell# ~name)) v#)) which captures the return value of the defn and returns it after the doseq. Agreed, that is an improvement. (defmacro run-propagator Run a propagator, first collect the most recent values from all cells associated with the propagator, then evaluate. [propagator] `(let [results# (apply ~propagator (map deref (:in-cells ^#'~propagator)))] (doseq [cell# (:out-cells ^#'~propagator)] (when (not (= @cell# results#)) (send cell# (fn [_#] results# results#)) Why not use your already-defined set-cell function on that second-to-last-line? Also your syntax for accessing the metadata seems a bit odd. Why not (meta ~propagator)? Agreed (meta ~propagator) is much more clear. Good use of metadata, though. (defmacro add-neighbor Add a neighbor to the given cell. [cell neighbor] `(add-watcher ~cell :send (agent nil :validator (fn [_#] (do (future (run-propagator ~neighbor)) true))) (fn [_# _#]))) What is add-watcher? I'm somewhat familiar with add-watch, whose syntax would be somewhat different. There'd be nothing between the watcher key :send and the (fn ...), and that function would take four arguments. This was an old project (last spring) and it's version of clojure (org.clojure/clojure 1.1.0-alpha-SNAPSHOT) is not even resolved any longer by lein. It appears that the add-watcher function has been deprecated and replaced by add-watch, which doesn't require that the function return true allowing the simpler construction you suggest below. I'd have probably used something like: (defmacro add-neighbor Add a neighbor to the given cell. [cell neighbor] `(add-watch ~cell :send (fn [_# _# _# _#] (future (run-propagator ~neighbor You could also get rid of the need to forward-declare add-neighbor by moving the propagator macro last. Convention also is that you'd call cell defcell and propagator defpropagator, since they expand to def forms. Ah, this is a big oversight on my part. Thanks for mentioning. I've implemented your suggestions, and committed the results to http://gitweb.adaptive.cs.unm.edu/propagator.git Now some more general notes: Having multiple out cells is interesting, but maybe redundant. Even with only a single out cell, the same effect can be had if you have an in cell, a bunch of out cells, and for each out cell a propagator that just returns the in cell's value. At that point the distinction between cells and propagators can go away: every cell has a propagator that sets its own value if any of its in-cells change. A cell that's intended to be an input can have an empty set of in-cells, which will result in it never doing anything since
Re: A tiny producer-consumer framework
On Mon, Jan 24, 2011 at 3:22 AM, Eric Schulte schulte.e...@gmail.com wrote: Ken Wesson kwess...@gmail.com writes: Why (fn [_] value) instead of (constantly value)? OK, actually (constantly value) is textually longer, but I'd argue it might be clearer. And it works; (constantly foo) accepts all arities. It's something like (fn [ _] foo). I agree constantly would have been a better choice, had I known it existed. Ah. I thought you might have known about it, but not that the function it emitted would accept arities other than zero. Whoa. Why not (defmacro propagator Define a new propagator. [name in-cells out-cells body] `(do (defn ~(vary-meta name assoc :in-cells in-cells :out-cells out-cells) ? Again, I work with the functions I am aware of. Ah. Funnily enough it wasn't long ago that I was caught actually reimplementing vary-meta because I didn't know about it. Thanks for these pointers, they are very helpful especially as it can be difficult to learn new functions once a sufficient subset has been discovered. True, that. Though I'd phrase it more as difficult to discover new functions. When you don't know how to do X or feel it should be a one-liner, if not a single function call, in core, you go trawling through http://clojure.github.com/clojure/clojure.core-api.html or use find-doc and apropos at the REPL to try and find something that fits, or at least simplifies the job. On the other hand when you can slap X together in two seconds flat it may not occur to you to look. The more functions you do know and the more adept you've become at combining them in various ways, the less likely it is that you *won't* think of at least one way to build X quickly, unless X is something particularly big and complicated like a major new piece of GUI functionality or an FTP server or something, and then you're more likely to look for a third-party library (and, probably, a Java rather than a Clojure library at that). I find the cheatsheet at http://clojure.org/cheatsheet useful sometimes for this, if I want to check if core has a function related to a category of use (seqs, say, or atoms) that might be useful. The cheatsheet has a couple of problems. For example, vary-meta isn't in there. ;) In fact there's no section on metadata at all, just a few scattered related things like *print-meta* and the reader macros (which are out of date -- ^ for meta and #^ for with-meta rather than just ^ for with-meta, nothing for meta). Also: * deref/@ is listed under refs, instead of somewhere general or repeated also for atoms and agents. * delay is listed under laziness but force under other/misc instead of with delay * the special forms list is incomplete, missing at least . and set!. The Java interop section lists set! but not . and the vars section doesn't list set!. Nonetheless, keeping those caveats in mind I find it useful. What is add-watcher? I'm somewhat familiar with add-watch, whose syntax would be somewhat different. There'd be nothing between the watcher key :send and the (fn ...), and that function would take four arguments. This was an old project (last spring) and it's version of clojure (org.clojure/clojure 1.1.0-alpha-SNAPSHOT) is not even resolved any longer by lein. It appears that the add-watcher function has been deprecated and replaced by add-watch, which doesn't require that the function return true allowing the simpler construction you suggest below. What, it had combined validator and watch functions in one back then? Interesting point. I think your suggestion would lead to simpler code, but IMO the existing design is simpler conceptually, I like the divide of propagators as functions and cells as data, rather than every function having an associated piece of data (namely its output). That's OK. I'm not sure which implementation would lead to a more efficient running system. Having fewer larger propagators which can set the values of many cells at once, or more smaller propagators each of which only sets the value for a single cell. It's likely to depend on how many distinct cells (may sometimes not all have the same value) you sometimes need to set to identical values all at once. If you have few such, you don't lose much by having a one-propagator-one-output-cell rule. If you have many such, you probably do. (If you have multiple cells that never have distinct values, they can be replaced by a single cell.) That would be a great application for this system. Each cell of the spreadsheet could be a cell, and each formula could be a propagator. I've implemented this and it seems to work well, I've committed this to the repo above, and posted the spreadsheet code with example usage into a gist at https://gist.github.com/792968 Cool. I notice the page at the original URL you posted hasn't been updated though. I hope you don't think all of that was too critical. It's very interesting work. Not at all, I
Re: A tiny producer-consumer framework
That would be a great application for this system. Each cell of the spreadsheet could be a cell, and each formula could be a propagator. I've implemented this and it seems to work well, I've committed this to the repo above, and posted the spreadsheet code with example usage into a gist at https://gist.github.com/792968 Cool. I think this example is just a couple of lines of Swing code away from being a generally useful spreadsheet application. I notice the page at the original URL you posted hasn't been updated though. I've moved everything (repo and documentation) over to repo.or.cz http://repo.or.cz/w/propagator.git Thanks again for the feedback -- Eric -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: A tiny producer-consumer framework
Nice concise example, A while back I implemented something similar; a propagator system using agents fed with `send', coming in at a slightly more verbose ~35 lines of code. http://cs.unm.edu/~eschulte/research/propagator/ Cheers -- Eric Ken Wesson kwess...@gmail.com writes: (defmacro consumer [[item] body] `(agent (fn c# [~item] ~@body c#))) (defmacro defconsumer [name item body] `(def ~name (consumer ~item ~@body))) (defn feed [consumer values] (doseq [v values] (send-off consumer apply [v]))) Nine lines of code. user= (defconsumer foo [x] (println x)) #'user/foo user= (feed foo 7) #Agent #user$c__1481__auto__ user$c__1481__auto__@1b030d8 user= (feed foo 42 196) #Agent #user$c__1481__auto__ user$c__1481__auto__@1b030d8 (and appearing at System/out) 7 42 196 I used send-off because your typical producer/consumer queue context involves I/O- rather than CPU-bound work. Since agent sends from the same thread are processed in the order sent, the whole thing works as expected with one producer thread; with multiple producer threads the inputs from separate threads will get interleaved but those from any single one will be in sequence. -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: A tiny producer-consumer framework
On Sat, Jan 22, 2011 at 11:26 AM, Eric Schulte schulte.e...@gmail.com wrote: Nice concise example, Thanks. A while back I implemented something similar; a propagator system using agents fed with `send', coming in at a slightly more verbose ~35 lines of code. http://cs.unm.edu/~eschulte/research/propagator/ Interesting. But I have some questions ... (defn set-cell Set the value of a cell [cell value] (send cell (fn [_] value))) Why (fn [_] value) instead of (constantly value)? OK, actually (constantly value) is textually longer, but I'd argue it might be clearer. And it works; (constantly foo) accepts all arities. It's something like (fn [ _] foo). A question for Clojure's developers: why does restart-agent require the agent to be in a failed state? That seems to be a superfluous requirement. Atoms have reset! and refs ref-set to clobber the old value; agents would have something analogous if restart-agent worked on agents that weren't failed too. (defmacro propagator Define a new propagator. [name in-cells out-cells body] `(do (defn ~(with-meta name (assoc (meta name) :in-cells in-cells :out-cells out-cells)) Whoa. Why not (defmacro propagator Define a new propagator. [name in-cells out-cells body] `(do (defn ~(vary-meta name assoc :in-cells in-cells :out-cells out-cells) ? (doseq [cell# ~in-cells] (add-neighbor cell# ~name)) There's no forward-declaration of add-neighbor before this. ~name)) Usually a deffoo form returns the new var, not just a symbol. This can be fixed with (declare add-neighbor) (defmacro propagator Define a new propagator. [name in-cells out-cells body] `(let [v# (defn ~(vary-meta name assoc :in-cells in-cells :out-cells out-cells) ~in-cells ~@body) (doseq [cell# ~in-cells] (add-neighbor cell# ~name)) v#)) which captures the return value of the defn and returns it after the doseq. (defmacro run-propagator Run a propagator, first collect the most recent values from all cells associated with the propagator, then evaluate. [propagator] `(let [results# (apply ~propagator (map deref (:in-cells ^#'~propagator)))] (doseq [cell# (:out-cells ^#'~propagator)] (when (not (= @cell# results#)) (send cell# (fn [_#] results# results#)) Why not use your already-defined set-cell function on that second-to-last-line? Also your syntax for accessing the metadata seems a bit odd. Why not (meta ~propagator)? Good use of metadata, though. (defmacro add-neighbor Add a neighbor to the given cell. [cell neighbor] `(add-watcher ~cell :send (agent nil :validator (fn [_#] (do (future (run-propagator ~neighbor)) true))) (fn [_# _#]))) What is add-watcher? I'm somewhat familiar with add-watch, whose syntax would be somewhat different. There'd be nothing between the watcher key :send and the (fn ...), and that function would take four arguments. I'd have probably used something like: (defmacro add-neighbor Add a neighbor to the given cell. [cell neighbor] `(add-watch ~cell :send (fn [_# _# _# _#] (future (run-propagator ~neighbor You could also get rid of the need to forward-declare add-neighbor by moving the propagator macro last. Convention also is that you'd call cell defcell and propagator defpropagator, since they expand to def forms. Now some more general notes: Having multiple out cells is interesting, but maybe redundant. Even with only a single out cell, the same effect can be had if you have an in cell, a bunch of out cells, and for each out cell a propagator that just returns the in cell's value. At that point the distinction between cells and propagators can go away: every cell has a propagator that sets its own value if any of its in-cells change. A cell that's intended to be an input can have an empty set of in-cells, which will result in it never doing anything since it will never be triggered. Of course the effect of that is to turn it into an almost-spreadsheet. Giving the cells a spatial organization is the last step to making it a spreadsheet. :) I hope you don't think all of that was too critical. It's very interesting work. -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
