Re: ANN: Specter 0.11.0, performance without the tradeoffs
Following the convention not to use `def` in non-top-level positions, you should use `intern` instead of `def`. -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: ANN: Specter 0.11.0, performance without the tradeoffs
I'm glad it helped! -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: ANN: Specter 0.11.0, performance without the tradeoffs
Great idea on interning a var at macro-time and then using it later! I did
something similar for the ClojureScript implementation of inline caching
but for some odd reason didn't think of doing it for the Clojure version.
I'm seeing a big performance improvement for the [:a :b :c] benchmark from
my post - about 15% faster than before and now very close to full manual
precompilation.
Here's the
commit:
https://github.com/nathanmarz/specter/commit/fbca7ab99c84d93a28f7773f1c56be12e1a939a3
On Tuesday, May 31, 2016 at 9:33:29 PM UTC-4, puzzler wrote:
>
> I think this is an interesting problem, so here are some additional
> brainstorms on the issue that may or may not be useful...
>
> One strategy to create a global mutable variable from inside the function
> would be to use def at macroexpansion time to create a var, and then just
> refer to it in the expansion, like this:
>
> (defmacro expand-to-something-with-mutable-global []
> ; At macroexpansion time, create a (private) var containing a volatile.
> (let [global-name (gensym "cache")]
> (eval `(def ^:private ~global-name (volatile! nil)))
> ; Now macro can refer to this var, can use as a cache, etc.
> `(if-let [cache-contents# (deref ~global-name)]
>(do-something-with cache-contents#)
>(reset! ~global-name init-value-for-cache
>
> Not sure if this would be any faster than ConcurrentHashMap, but I would
> guess that it would be if Clojure resolves the location of the var in
> memory once, at compile-time.
>
> A related technique would be for Specter to maintain an ArrayList of
> volatiles. At macroexpansion time, you add a fresh volatile to the end of
> the ArrayList, noting the index of its location, and then inside the macro
> you hardcode a lookup in the ArrayList at the specific index to retrieve
> the volatile containing the cache for this particular expansion. I would
> expect this to be faster than looking up in a hash map, although there'd be
> some additional concurrency/locking details to worry about that I assume
> ConcurrentHashMap handles for you.
>
> Or just use an ArrayList's slot directly as the cache (rather than storing
> a volatile to add a level of indirection), but then the concurrency
> logistics would be even more complicated.
>
> On Tue, May 31, 2016 at 12:50 PM, Nathan Marz > wrote:
>
>> No, because that global mutable variable would need to be specifiable by
>> Specter on usage of the library. For example, if you wrote:
>>
>> (defn foo []
>> (select [:a :b] {:a {:b 1}}))
>>
>> `select` has no ability to control anything outside the context of its
>> form. It certainly can't wrap `foo` to put a volatile field in its closure.
>> So for the static-field idea, it would expand to something like:
>>
>> (defn foo []
>> (static-field [pathcache]
>> (if-not pathcache (set! pathcache ...))
>> ...
>> ))
>>
>>
>>
>> On Tuesday, May 31, 2016 at 3:15:26 PM UTC-4, puzzler wrote:
>>>
>>> In your writeup, you say that there would be further speed benefits if
>>> you could have a global mutable variable within the context of a function
>>> (like a static field).
>>>
>>> Can't you effectively accomplish that already in Clojure like this?:
>>>
>>> (let [mycache (volatile! nil)]
>>> (defn foo []
>>> ...)))
>>>
>> --
>> You received this message because you are subscribed to the Google
>> Groups "Clojure" group.
>> To post to this group, send email to clo...@googlegroups.com
>>
>> Note that posts from new members are moderated - please be patient with
>> your first post.
>> To unsubscribe from this group, send email to
>> clojure+u...@googlegroups.com
>> For more options, visit this group at
>> http://groups.google.com/group/clojure?hl=en
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "Clojure" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to clojure+u...@googlegroups.com .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Re: ANN: Specter 0.11.0, performance without the tradeoffs
I think this is an interesting problem, so here are some additional
brainstorms on the issue that may or may not be useful...
One strategy to create a global mutable variable from inside the function
would be to use def at macroexpansion time to create a var, and then just
refer to it in the expansion, like this:
(defmacro expand-to-something-with-mutable-global []
; At macroexpansion time, create a (private) var containing a volatile.
(let [global-name (gensym "cache")]
(eval `(def ^:private ~global-name (volatile! nil)))
; Now macro can refer to this var, can use as a cache, etc.
`(if-let [cache-contents# (deref ~global-name)]
(do-something-with cache-contents#)
(reset! ~global-name init-value-for-cache
Not sure if this would be any faster than ConcurrentHashMap, but I would
guess that it would be if Clojure resolves the location of the var in
memory once, at compile-time.
A related technique would be for Specter to maintain an ArrayList of
volatiles. At macroexpansion time, you add a fresh volatile to the end of
the ArrayList, noting the index of its location, and then inside the macro
you hardcode a lookup in the ArrayList at the specific index to retrieve
the volatile containing the cache for this particular expansion. I would
expect this to be faster than looking up in a hash map, although there'd be
some additional concurrency/locking details to worry about that I assume
ConcurrentHashMap handles for you.
Or just use an ArrayList's slot directly as the cache (rather than storing
a volatile to add a level of indirection), but then the concurrency
logistics would be even more complicated.
On Tue, May 31, 2016 at 12:50 PM, Nathan Marz wrote:
> No, because that global mutable variable would need to be specifiable by
> Specter on usage of the library. For example, if you wrote:
>
> (defn foo []
> (select [:a :b] {:a {:b 1}}))
>
> `select` has no ability to control anything outside the context of its
> form. It certainly can't wrap `foo` to put a volatile field in its closure.
> So for the static-field idea, it would expand to something like:
>
> (defn foo []
> (static-field [pathcache]
> (if-not pathcache (set! pathcache ...))
> ...
> ))
>
>
>
> On Tuesday, May 31, 2016 at 3:15:26 PM UTC-4, puzzler wrote:
>>
>> In your writeup, you say that there would be further speed benefits if
>> you could have a global mutable variable within the context of a function
>> (like a static field).
>>
>> Can't you effectively accomplish that already in Clojure like this?:
>>
>> (let [mycache (volatile! nil)]
>> (defn foo []
>> ...)))
>>
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Re: ANN: Specter 0.11.0, performance without the tradeoffs
No, because that global mutable variable would need to be specifiable by
Specter on usage of the library. For example, if you wrote:
(defn foo []
(select [:a :b] {:a {:b 1}}))
`select` has no ability to control anything outside the context of its
form. It certainly can't wrap `foo` to put a volatile field in its closure.
So for the static-field idea, it would expand to something like:
(defn foo []
(static-field [pathcache]
(if-not pathcache (set! pathcache ...))
...
))
On Tuesday, May 31, 2016 at 3:15:26 PM UTC-4, puzzler wrote:
>
> In your writeup, you say that there would be further speed benefits if you
> could have a global mutable variable within the context of a function (like
> a static field).
>
> Can't you effectively accomplish that already in Clojure like this?:
>
> (let [mycache (volatile! nil)]
> (defn foo []
> ...)))
>
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Re: ANN: Specter 0.11.0, performance without the tradeoffs
In your writeup, you say that there would be further speed benefits if you could have a global mutable variable within the context of a function (like a static field). Can't you effectively accomplish that already in Clojure like this?: (let [mycache (volatile! nil)] (defn foo [] ...))) -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
ANN: Specter 0.11.0, performance without the tradeoffs
Specter is a library for querying and transforming nested data structures concisely and efficiently. The 0.11.0 release is a huge milestone that implements something which for the better part of the past year I wasn't sure was possible. In summary: now you don't have to do anything special to make Specter run fast. The prior work of manually precompiling your paths for performance is now seamlessly automated. As part of the release, wrote up a detailed guide to how Specter achieves its high performance. Should also be an interesting read for anyone curious about the design of a very powerful Clojure library. https://github.com/nathanmarz/specter/wiki/Specter-0.11.0:-Performance-without-the-tradeoffs Note that there are some backwards incompatible changes in this release, so please read the changelog below. The core select/transform/etc. functions have changed to macros, and there have been some name updates to clean up the terminology. Updating your projects to the new changes should be very easy. Changes: * New `path` macro does intelligent inline caching of the provided path. The path is factored into a static portion and into params which may change on each usage of the path (e.g. local parameters). The static part is factored and compiled on the first run-through, and then re-used for all subsequent invocations. As an example, `[ALL (keypath k)]` is factored into `[ALL keypath]`, which is compiled and cached, and `[k]`, which is provided on each execution. If it is not possible to precompile the path (e.g. [ALL some-local-variable]), nothing is cached and the path will be compiled on each run-through. * BREAKING CHANGE: all `select/transform/setval/replace-in` functions changed to macros and moved to com.rpl.specter.macros namespace. The new macros now automatically wrap the provided path in `path` to enable inline caching. Expect up to a 100x performance improvement without using explicit precompilation, and to be within 2% to 15% of the performance of explicitly precompiled usage. * Added `select*/transform*/setval*/replace-in*/etc.` functions that have the same functionality as the old `select/transform/setval/replace-in` functions. * Added `must-cache-paths!` function to throw an error if it is not possible to factor a path into a static portion and dynamic parameters. * BREAKING CHANGE: `defpath` renamed to `defnav` * BREAKING CHANGE: `path` renamed to `nav` * BREAKING CHANGE: `fixed-pathed-path` and `variable-pathed-path` renamed to `fixed-pathed-nav` and `variabled-pathed-nav` * Added `must` navigator to navigate to a key if and only if it exists in the structure * Added `continous-subseqs` navigator * Added `ATOM` navigator (thanks @rakeshp) * Added "navigator constructors" that can be defined via `defnavconstructor`. These allow defining a flexible function to parameterize a defnav, and the function integrates with inline caching for high performance. -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
