Re: Memoize in the real world
It looks very similar to the pattern I was trying to avoid in the first
place. I've also got the problem of multiple threads (and its been pointed
out that my original solution was not thread safe). In my experience bugs
of an 'extremely rare but could conceivably happen' nature are the sort of
thing that appear years after deployment when nobody understands the fine
detail of a system any more. I'm speculating of course but its the kind of
thing that brought down the UKs Air Traffic Control system with a 'never
before seen' bug just last week.
'
On Saturday, December 13, 2014 12:37:28 PM UTC, Sergey Didenko wrote:
Some people use vars for seldom changing things. What do you think
about this VS atoms?
For example:
(declare ^:dynamic *server*)
(defn get-possibly-unbound-var [v]
(try (var-get v)
(catch Exception e
nil)))
(defn start-server! []
(if (get-possibly-unbound-var *server*)
(.start *server*)
(def ^:dynamic *server*
(run-jetty #'app {:port 8000 :join? false}
--
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
Clojure group.
To unsubscribe from this group and stop receiving emails from it, send an email
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Re: Memoize in the real world
As I understand this var approach is used for development purposes, so in theory it should not occur in production. I wonder though why someone would prefer it to atoms. -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Memoize in the real world
Some people use vars for seldom changing things. What do you think
about this VS atoms?
For example:
(declare ^:dynamic *server*)
(defn get-possibly-unbound-var [v]
(try (var-get v)
(catch Exception e
nil)))
(defn start-server! []
(if (get-possibly-unbound-var *server*)
(.start *server*)
(def ^:dynamic *server*
(run-jetty #'app {:port 8000 :join? false}
--
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
Clojure group.
To unsubscribe from this group and stop receiving emails from it, send an email
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Re: Memoize in the real world
Thanks to everyone for the responses. I was completely unaware of core.memoize although I think there is a very convincing case for delay; I certainly accept the thread safety problem with the existing code. Action this day on that one! I'm inclined to go down the @@ syntactic route, simply in order to enjoy the look of horror on my co-worker's faces when I check it in. On Wednesday, December 10, 2014 4:45:16 AM UTC, Steven Yi wrote: No reason really, just ended up writing it that way for the example. (Maybe subconsciously I didn't want to think about pointer pointers and wanted to break up the notation. :P) On Tue, Dec 9, 2014 at 11:18 PM, Fluid Dynamics a209...@trbvm.com javascript: wrote: On Tuesday, December 9, 2014 7:19:27 PM UTC-5, Steven Yi wrote: Hi Andy, Just my two cents here, but I don't think memoize makes sense here myself due to this not being referentially transparent, and I think Andrey's comment on delays makes more sense to me. You could do something like this: (def conf (atom (delay (read-and-expensively-parse somefile.xml (defn config [] @(deref conf)) Why not just @@conf? It seems to work at my REPL, at least, to unpeel two onion layers of indirection. -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clo...@googlegroups.com javascript: Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+u...@googlegroups.com javascript: For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to a topic in the Google Groups Clojure group. To unsubscribe from this topic, visit https://groups.google.com/d/topic/clojure/FaPliPRgJBI/unsubscribe. To unsubscribe from this group and all its topics, send an email to clojure+u...@googlegroups.com javascript:. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Memoize in the real world
Although if you do not expect the config to change, you do not need to wrap the delay in an atom. The atom in Steven Yi's solution is only there to allow the combination of (delay) executing the operation strictly once but also (atom) allow for changing the underlying value (i.e. clear the cache). On Wednesday, 10 December 2014, Andy Dwelly andydwe...@gmail.com wrote: Thanks to everyone for the responses. I was completely unaware of core.memoize although I think there is a very convincing case for delay; I certainly accept the thread safety problem with the existing code. Action this day on that one! I'm inclined to go down the @@ syntactic route, simply in order to enjoy the look of horror on my co-worker's faces when I check it in. On Wednesday, December 10, 2014 4:45:16 AM UTC, Steven Yi wrote: No reason really, just ended up writing it that way for the example. (Maybe subconsciously I didn't want to think about pointer pointers and wanted to break up the notation. :P) On Tue, Dec 9, 2014 at 11:18 PM, Fluid Dynamics a209...@trbvm.com wrote: On Tuesday, December 9, 2014 7:19:27 PM UTC-5, Steven Yi wrote: Hi Andy, Just my two cents here, but I don't think memoize makes sense here myself due to this not being referentially transparent, and I think Andrey's comment on delays makes more sense to me. You could do something like this: (def conf (atom (delay (read-and-expensively-parse somefile.xml (defn config [] @(deref conf)) Why not just @@conf? It seems to work at my REPL, at least, to unpeel two onion layers of indirection. -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clo...@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+u...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to a topic in the Google Groups Clojure group. To unsubscribe from this topic, visit https://groups.google.com/d/topic/clojure/FaPliPRgJBI/unsubscribe. To unsubscribe from this group and all its topics, send an email to clojure+u...@googlegroups.com. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com javascript:_e(%7B%7D,'cvml','clojure@googlegroups.com'); Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com javascript:_e(%7B%7D,'cvml','clojure%2bunsubscr...@googlegroups.com'); For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com javascript:_e(%7B%7D,'cvml','clojure%2bunsubscr...@googlegroups.com');. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Memoize in the real world
Understood. I was using a configuration example as a kind of shorthand for a larger set of problems - most of which are multithreaded and will be dealing with occasionally changing results. As usual, its a tricky area, but not as difficult as achieving the same result in some other environments, fortunately. On Wednesday, December 10, 2014 11:22:39 AM UTC, Gary Verhaegen wrote: Although if you do not expect the config to change, you do not need to wrap the delay in an atom. The atom in Steven Yi's solution is only there to allow the combination of (delay) executing the operation strictly once but also (atom) allow for changing the underlying value (i.e. clear the cache). On Wednesday, 10 December 2014, Andy Dwelly andyd...@gmail.com javascript: wrote: Thanks to everyone for the responses. I was completely unaware of core.memoize although I think there is a very convincing case for delay; I certainly accept the thread safety problem with the existing code. Action this day on that one! I'm inclined to go down the @@ syntactic route, simply in order to enjoy the look of horror on my co-worker's faces when I check it in. On Wednesday, December 10, 2014 4:45:16 AM UTC, Steven Yi wrote: No reason really, just ended up writing it that way for the example. (Maybe subconsciously I didn't want to think about pointer pointers and wanted to break up the notation. :P) On Tue, Dec 9, 2014 at 11:18 PM, Fluid Dynamics a209...@trbvm.com wrote: On Tuesday, December 9, 2014 7:19:27 PM UTC-5, Steven Yi wrote: Hi Andy, Just my two cents here, but I don't think memoize makes sense here myself due to this not being referentially transparent, and I think Andrey's comment on delays makes more sense to me. You could do something like this: (def conf (atom (delay (read-and-expensively-parse somefile.xml (defn config [] @(deref conf)) Why not just @@conf? It seems to work at my REPL, at least, to unpeel two onion layers of indirection. -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clo...@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+u...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to a topic in the Google Groups Clojure group. To unsubscribe from this topic, visit https://groups.google.com/d/topic/clojure/FaPliPRgJBI/unsubscribe. To unsubscribe from this group and all its topics, send an email to clojure+u...@googlegroups.com. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Memoize in the real world
Looking through my recent work I see that a number of atoms, swap! and reset! calls have snuck into my work, usually when there's an expensive operation like reading and parsing a large file or connecting to a database. I find I'm doing things like (def conf (atom nil)) (defn config [] (if (nil? @conf) (reset! conf (read-and-expensively-parse somefile.xml)) @conf)) So that the first time (config) is called it will do the operation, but future attempts will simply deref the atom. Recent discussions here alerted me to memoize which will do this without the atom, but the documentation mentions that the functions that are memoized should be referentially transparent. Although I'm pretty sure that while my server is running, the config will not change - it can't be guaranteed in a formal sense. Reading a file or connecting to an external process is doing io of course, in theory someone could change the file while my back was turned. Questions: if I was prepared to live with the consequences of having to restart the server if the conf file had been changed, would it be more idiomatic to use use memoize to avoid mutating state? If I really had to change a conf file while the server was running, is there any way a memoized function could have it's cache cleared? I suspect the answer to that one is no - but you never know -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Memoize in the real world
Hi! 2014-12-09 18:08 GMT+01:00 Andy Dwelly andydwe...@gmail.com: Looking through my recent work I see that a number of atoms, swap! and reset! calls have snuck into my work, usually when there's an expensive operation like reading and parsing a large file or connecting to a database. I find I'm doing things like (def conf (atom nil)) (defn config [] (if (nil? @conf) (reset! conf (read-and-expensively-parse somefile.xml)) @conf)) So that the first time (config) is called it will do the operation, but future attempts will simply deref the atom. Recent discussions here alerted me to memoize which will do this without the atom, but the documentation mentions that the functions that are memoized should be referentially transparent. Although I'm pretty sure that while my server is running, the config will not change - it can't be guaranteed in a formal sense. Reading a file or connecting to an external process is doing io of course, in theory someone could change the file while my back was turned. Questions: if I was prepared to live with the consequences of having to restart the server if the conf file had been changed, would it be more idiomatic to use use memoize to avoid mutating state? If I really had to change a conf file while the server was running, is there any way a memoized function could have it's cache cleared? I suspect the answer to that one is no - but you never know In my opinion, if you assumes the consequences of having to restart the server, delay is the best solution for it. As far as I know, it guarantees one unique execution regardless how many threads are dereferencing it, and subsequently it always returns the computed result. My two cents. Andrey -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout. -- Andrey Antukh - Андрей Антух - andrei.anto...@kaleidos.net / n...@niwi.be http://www.niwi.be http://www.niwi.be/page/about/ https://github.com/niwibe -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Memoize in the real world
If you want to use memoize, then the clojure.core.memoize provides a more tunable version, which supports clearing the cache if required. https://github.com/clojure/core.memoize/blob/master/docs/Using.md -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Memoize in the real world
Hi Andy, Just my two cents here, but I don't think memoize makes sense here myself due to this not being referentially transparent, and I think Andrey's comment on delays makes more sense to me. You could do something like this: (def conf (atom (delay (read-and-expensively-parse somefile.xml (defn config [] @(deref conf)) (defn reload-conf! [] (reset! conf (delay (read-and-expensively-parse somefile.xml))) This would ensure that there's a valid delay in there for the first time anyone hits config, and still let you reload by replacing the delay in the conf atom with a new one. Also, just FYI, for this code: (defn config [] (if (nil? @conf) (reset! conf (read-and-expensively-parse somefile.xml)) @conf)) There's a danger here in that the time between the first @conf and the reset! is not protected, i.e. this is not an atomic operation. So if n threads hit config at the same time, it's entirely possible that n calls to read-and-expensively-parse could happen. I don't know if that's a possibility to have multiple callers to config with the codebase you have, but I think that general code pattern is one to be careful about. Cheers! steven On Tuesday, December 9, 2014 12:08:25 PM UTC-5, Andy Dwelly wrote: Looking through my recent work I see that a number of atoms, swap! and reset! calls have snuck into my work, usually when there's an expensive operation like reading and parsing a large file or connecting to a database. I find I'm doing things like (def conf (atom nil)) (defn config [] (if (nil? @conf) (reset! conf (read-and-expensively-parse somefile.xml)) @conf)) So that the first time (config) is called it will do the operation, but future attempts will simply deref the atom. Recent discussions here alerted me to memoize which will do this without the atom, but the documentation mentions that the functions that are memoized should be referentially transparent. Although I'm pretty sure that while my server is running, the config will not change - it can't be guaranteed in a formal sense. Reading a file or connecting to an external process is doing io of course, in theory someone could change the file while my back was turned. Questions: if I was prepared to live with the consequences of having to restart the server if the conf file had been changed, would it be more idiomatic to use use memoize to avoid mutating state? If I really had to change a conf file while the server was running, is there any way a memoized function could have it's cache cleared? I suspect the answer to that one is no - but you never know -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Memoize in the real world
On Tuesday, December 9, 2014 7:19:27 PM UTC-5, Steven Yi wrote: Hi Andy, Just my two cents here, but I don't think memoize makes sense here myself due to this not being referentially transparent, and I think Andrey's comment on delays makes more sense to me. You could do something like this: (def conf (atom (delay (read-and-expensively-parse somefile.xml (defn config [] @(deref conf)) Why not just @@conf? It seems to work at my REPL, at least, to unpeel two onion layers of indirection. -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Memoize in the real world
No reason really, just ended up writing it that way for the example. (Maybe subconsciously I didn't want to think about pointer pointers and wanted to break up the notation. :P) On Tue, Dec 9, 2014 at 11:18 PM, Fluid Dynamics a2093...@trbvm.com wrote: On Tuesday, December 9, 2014 7:19:27 PM UTC-5, Steven Yi wrote: Hi Andy, Just my two cents here, but I don't think memoize makes sense here myself due to this not being referentially transparent, and I think Andrey's comment on delays makes more sense to me. You could do something like this: (def conf (atom (delay (read-and-expensively-parse somefile.xml (defn config [] @(deref conf)) Why not just @@conf? It seems to work at my REPL, at least, to unpeel two onion layers of indirection. -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to a topic in the Google Groups Clojure group. To unsubscribe from this topic, visit https://groups.google.com/d/topic/clojure/FaPliPRgJBI/unsubscribe. To unsubscribe from this group and all its topics, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
