Re: Transient Data Structures
I really, really like this feature. My only complaint is that you have to use different names for the modifying functions. If the function signatures will be identical to their non-transient variants, then I guess the primary arguments would be: * Clojure convention for names of functions with side-effects, * An additional interface check in conj/assoc? But if after calling (conj v 1), you can't use the 'v' reference anymore, then did you really cause a side effect? Its another tree falling in the woods situation. Thanks, Stu On Wed, Aug 5, 2009 at 9:05 AM, Rich Hickey richhic...@gmail.com wrote: On Tue, Aug 4, 2009 at 10:13 PM, John Harropjharrop...@gmail.com wrote: On Tue, Aug 4, 2009 at 5:50 PM, Rich Hickey richhic...@gmail.com wrote: On Aug 4, 4:31 pm, John Harrop jharrop...@gmail.com wrote: What about things like: (persistent! (reduce (fn [x [i v]] (assoc! x i v)) (transient (vec (repeat 0 (reduce max (map first coll-of-index-val-pairs) coll-of-index-val-pairs)) Yes, that's completely fine intended usage, as the return value of the reducing fn becomes an argument to the next call. which is just the transientification of Nice word - transientification. Thanks. Of course, this makes me think of ways to possibly speed up other operations. For example: (defn vmap* [fun vect] (let [c (count vect)] (loop [out (transient []) i 0] (if (= i c) out (recur (conj! out (fun (nth vect i))) (inc i)) (defn vmap [fun vect] (persistent! (vmap* fun vect))) Vector in, vector out, conj'd up using a transient. Mapping into vectors and similar ops are coming, although nothing like vmap* would ever be exposed. And, of course: (defn vpmap [fun vect] (loop [out (vmap* #(future (fun %)) vect) i (dec (count vect))] (let [o2 (assoc! out i @(nth out i))] (if (zero? i) (persistent! o2) (recur o2 (dec i) Note that this last manipulates a transient vector in a single thread, though other threads (from the agent pool) calculate a bunch of futures that are stored in it. The transient vector of futures is generated using the vmap* from above, and then the futures are replaced with their values in-place by the loop in vpmap, before this persistentizes and returns that vector. The future-dereferencing loop works backwards from the end of the vector in case zero? is a quicker test than a general equality test. This is likely on hardware that implements an equality test as a subtraction followed by a zero test, because it eliminates the subtraction. On the other hand, hardware with a 1-cycle equality test of 32-bit ints is plausible, as is hardware that optimizes forward traversal of vectors, so I can't vouch for that being an optimization. The first loop has to go forward to conj up the output vector without reversing it, though. There is already a very nice pvmap in the par branch that uses ForkJoin. I strongly recommend against trying to write parallel ops with transients - that's not what they are for. What's nice about pvmap is that, just like transients, it also takes, manipulates, and returns ordinary Clojure vectors (unlike the old parallel lib which copied into and out of ForkJoin's ParallelArrays). The goal is to make using the normal data structures as powerful as possible, and not needing to switch to something else for performance. Rich --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
Re: why is io! not enforced?
This won't work unfortunately, because it means that the in-memory transaction has already commited before the disk write is performed by the agent. If the application crashed at that point, your write was not durable. -- Sent from my Palm Pre ataggart wrote: On Jun 20, 4:59 pm, Rowdy Rednose wrote: On a side-note: I actually think it can make sense to do io in transactions in Clojure, and I believe (knowing that transactions can be replayed) it is possible to use that to e.g. implement a transaction log written to disk that could be used to rebuild the data in case of a crash. Use agents. When inside a running transaction, all agent dispatches are held until the transaction commits. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
Metadata for Any Object
Hey gang,
Metadata support for pure Java objects is not currently supported, because
it requires a modification to the object. Additionally, only some Clojure
objects support metadata, due to the necessity of implementing the IMeta
interface. This can be confusing for new users, and eliminates some of the
utility.
Instead of attaching the metadata directly to the object, what if the
metadata was stored outside the object, in a global map of {object metadata,
...}? In order to handle garbage collection, something similar to Java's
WeakHashMap could be used, with the object itself as the key.
I know global state is Considered Harmful, but I think this would clarify
the usage of metadata, and remove duplication in the implementation of
Clojure.
Thoughts?
Stu
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
Clojure group.
To post to this group, send email to clojure@googlegroups.com
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
-~--~~~~--~~--~--~---
Re: [Discuss] Contribs with dependencies
I agree. I think the breaking into modules approach is the only scalable solution. Someone else mentioned that clojure-contrib is/shouldbe an incubating area for core, which seems reasonable. There should be a little more pushback when a project wants to make it into contrib, and it should already be organized such that its dependencies would be well known if it wanted to make it out of 'incubation' and into core. On Tue, Apr 14, 2009 at 5:02 PM, Howard Lewis Ship hls...@gmail.com wrote: I'd say to refactor clojure-contrib into a number of seperate modules; individual modules (each with its own pom) could have their own dependencies. Thus if you choose clojure-contrib-freechart, you get that JAR (or compiled Clojure sources) plus the jfreechart dependency. In this way you are using the good part of Maven: transitive dependency management. On Tue, Apr 14, 2009 at 5:19 AM, Rich Hickey richhic...@gmail.com wrote: I've been thinking recently about contribs with dependencies. I think it's very important to have layers - e.g. core depends only on JDK 1.5, contrib only on core. Lately there have been some ideas centering around Joda Time, [Parallel]Colt, AWS, JFreeChart, Fork/Join etc. I'd like to start a discussion about how best to support the contributions of libraries that depend on things not in the JDK. Obviously, without care and rules it could get crazy quickly, and I want to avoid the kitchen-sink effect. It is very important that things remain [truly, not just apparently] simple. Looking for suggestions, Rich -- Howard M. Lewis Ship Creator of Apache Tapestry Director of Open Source Technology at Formos --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
Re: advice needed converting large files to xml
If you write your CSV - XML processing as a function, you could pmap ( http://clojure.org/api#pmap) that function across the list of input files. pmap will transparently create the threads as needed, and it will probably be enough to saturate your disk. Thanks, Stu On Mon, Mar 16, 2009 at 5:56 PM, Brian Doyle brianpdo...@gmail.com wrote: I've been using Clojure for about 6 months now and really like it. I am somewhat new to multi-threading and using any of the parallel features in Clojure though. I have a situation where I need to convert 7 files from CSV to XML. Each one of these files is about 180MB apiece in size. I have dual core machine with 2GB of RAM and would like some advice on the best strategy for processing these files in a way that really utilizes both cores and my memory to really speed up the processing.I'm sure this isn't the best way, but I've only come up with starting up two threads at first, having each thread open up a file, call line-seq on that file, write out the XML for each line and then go to the next file when it's complete. Any advice would be great. Thanks. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
Re: Bytecode optimization
Rich has done a lot of work to make sure that when you are working with primitives, the JVM bytecode ends up being very similar to what Java would generate. See http://clojure.org/java_interop#toc36 Thanks, Stu On Thu, Mar 12, 2009 at 6:20 PM, Howard Lewis Ship hls...@gmail.com wrote: Well, at some point I'll open up the code and check. Though I'll be overly tempted to write some comments, fix the indentation and write some tests if I do! On Thu, Mar 12, 2009 at 2:36 PM, Jarkko Oranen chous...@gmail.com wrote: On Mar 12, 11:15 pm, Howard Lewis Ship hls...@gmail.com wrote: I have to wonder a bit about the ability to optimize. Everything boils down to one of the seven or so basic forms. That's a lot of function calls to do even small things, like adding numbers. You might think that simple math would be optimized and inlined, but it isn't: Clojure user= (doc +) - clojure.core/+ ([] [x] [x y] [x y more]) Returns the sum of nums. (+) returns 0. nil user= (ns clojure.core) #Namespace clojure.core clojure.core= (defn + [x y] (* x y)) #'clojure.core/+ clojure.core= (+ 3 5) 15 clojure.core= This implies, to me, that Clojure is doing a full functional call, with dynamic lookup via the namespace, even for +. I think this is mistaken. Sure, for (+ foo bar) calls in the repl Clojure will do a full lookup, but if you have addition in a tight loop, the JVM should notice this and optimise it further. (I think Clojure also does some kind of caching.) You can also use type casts to make clojure use primitive instead of boxed math, bringing performance very close to Java. Now, it's probably true that the most straightforward Clojure implementation of an algorithm is likely not quite as fast as the Java equivalent, with a few tricks, you can often get pretty close. I don't have very in-depth knowledge of how Clojure handles these performance issues, but I believe your assumption is overly pessimistic. :) -- Jarkko -- Howard M. Lewis Ship Creator Apache Tapestry and Apache HiveMind --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
Re: Performance tips for Clojure
There is a interface 'Counted' that a lot of Clojure data structures implement to indicate that they provide O(1) for (count). Thanks, Stu On Fri, Mar 13, 2009 at 4:59 AM, Christophe Grand christo...@cgrand.netwrote: Christian Vest Hansen a écrit : I think that count is O(n) for lists, no? Count is O(1) for lists but O(n) for a chain of conses. Clojure user= (let [l (apply list (range 10))] (time (dotimes [_ 100] (count l Elapsed time: 169.710116 msecs nil user= (let [l (apply list (range 40))] (time (dotimes [_ 100] (count l Elapsed time: 167.664046 msecs nil user= (let [l (reduce #(cons %2 %1) nil (range 10))] (time (dotimes [_ 100] (count l Elapsed time: 662.121862 msecs nil user= (let [l (reduce #(cons %2 %1) nil (range 100))] (time (dotimes [_ 100] (count l Elapsed time: 5316.110567 msecs nil -- Professional: http://cgrand.net/ (fr) On Clojure: http://clj-me.blogspot.com/ (en) --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
Re: Laziness madness
Laziness hit home for me in Clojure when I noticed that one of our Java library functions, looking for a Collection, had happily accepted a Clojure (map ...) object which was going to expand a small set of integers into much larger objects. Because Clojure deferred the work, they were only generated when they were actually needed, and saved memory the rest of the time. Thanks, Stu On Mon, Mar 2, 2009 at 1:17 AM, Laurent PETIT laurent.pe...@gmail.comwrote: Hello, The way I think about it is am I in a portion of code that does pure functional stuff, or am I doing side effects ? A pure functional stuff resembles a computation, but not the storage part, or the part that gets the stuff from the outside (that is, no input / output stuff). doing side effects is when your function not only responds to a call to it by reading its input data and computing a new one that it returns, but also when it reads the input data, does some stuff to change the world (by accessing functions that will permanently change the world), and also (maybe) return a value. In this case, generally, comes with the non pure stuff the need to control the flow of execution of your program. That is : function A must be called before function B, and the effects of A on the world must have been realized before function B can run. In your case, you're calling add-watch which is clearly (to me) a side-effect function because it adds watchers to a long list of watchees by modifying them in place (side-effect). Generally speaking, if this talk about side-effect free function is still a little bit unclear to you, here are some smells to guess if a given function is pure or not : - does the function return a value (different from nil) ? If no - probably a function with side effect .. or a function that really does nothing ! - does the function return newly computed values ? If no - probably a function with side effect .. or a function that really does nothing ! (please note that the inverse is not true : you can't say that a function is pure just because it returns a new computed value of something) I would like to add a third rule to this list, but currently you will not encounter it for a lot of cases, yet it is correct in a number of case and I think it applies correctly to those cases : - does the function implementation use calls to functions ending with a ! ( e.g. set!) : those functions are performing side effects by permanently changing the value of something in the world. HTH, -- Laurent 2009/3/2 max3000 maxime.lar...@gmail.com Hi, I find the laziness in clojure very hard to wrap my head around. I understand the idea and it's probably nice in theory. However, in real life it doesn't seem really useful beyond hardcore mathematical problems. Case in point, I just spent 2 hours debugging a piece of code (shown below) that seemed simple enough. This is the 3rd time this week that I've lost substantial time to laziness. I'm pretty pissed to tell the truth and I find myself wrapping things in doseq more and more just to be sure. I rarely use 'for' anymore, what's the point? Here is the code that gave me trouble: (map #(add-watch % watcher callback-fn) all-agents) This was not executing. I had to change it to the below expression: (doseq [agent all-labor-agents] (add-watch agent total-labor-agent callback-fn)) This second expression seems less elegant than the map above. Why doesn't clojure realize that an add-watch really should actually loop over all-agents? Why is it that Java calls are not made in similar expressions? Is laziness so useful that we should waste time investigating and fixing errors like this? Sure, there could be special constructs for laziness when we really need it. However, clojure shouldn't default to it IMO. At this point, laziness is a nice concept but it feels somewhat removed from reality to tell the truth. Of course I want to iterate over my collection when I'm doing an add-watch! What am I missing? Thanks, Max --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
Re: Simple math functions should support sequences
This is a common misconception: passing a seq to apply doesn't force its evaluation. Ahh, is this because the [ more] portion is itself a lazy sequence? That's very cool =) Hmm, the (reduce + ...) approach works just fine, but if it is already implemented as reduce, it seems like it would be very easy to allow (+) to take a sequence. I still think the ( (range 10)) ... use case is really worthwhile though, and I don't see a way to accomplish it with reduce. Thanks, Stu On Tue, Feb 3, 2009 at 1:36 PM, Christophe Grand christo...@cgrand.netwrote: Hello, stuhood a écrit : Functions like (+), (*), (-), (and probably more) should support sequences as parameters. The current way to accomplish this (without implementing your own sum using reduce) seems to be: (apply + (map #(. Math pow 2 %) (range 10))) ... which has to generate the sequence first. This is a common misconception: passing a seq to apply doesn't force its evaluation. You can test this fact by passing an infinite seq to a function: (defn second-arg [ args] (second args)) user= (apply second-arg (iterate inc 0)) 1 and (apply + some-seq) is really equivalent to (reduce + some-seq), see the def of + in core.clj: (defn + ... ([x y more] (reduce + (+ x y) more))) Christophe -- Professional: http://cgrand.net/ (fr) On Clojure: http://clj-me.blogspot.com/ (en) --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
Re: Simple math functions should support sequences
but apply works very well for this use case: (apply (range 10)) and it stops as soon as it can: Alright, I fold... thanks for clearing things up Christophe! On Tue, Feb 3, 2009 at 3:13 PM, Christophe Grand christo...@cgrand.netwrote: Stu Hood a écrit : I still think the ( (range 10)) ... use case is really worthwhile though, and I don't see a way to accomplish it with reduce. reduce is not a good fit since you would want to short circuit the computation at the first false. but apply works very well for this use case: (apply (range 10)) and it stops as soon as it can: user=(apply (map #(doto % println) (concat (range 10 20) (range 10 10 11 12 13 14 15 16 17 18 19 0 false Christophe -- Professional: http://cgrand.net/ (fr) On Clojure: http://clj-me.blogspot.com/ (en) --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
Re: Synchronization Benchmarks
I added a Clojure implementation based on an Atom instead of a Ref, and re-ran the tests (this time on a quad core machine). I also fixed the calculation of the percent change in both tests (ugh). It looks like using an Atom is slightly more performant than a Ref if you are making a single state change that is commutative/indempotent. Also, RWDict fell apart on a quad core machine (independent of the number of writes): I'll try with the alternative fairness setting like you suggested. Thanks, Stu On 1/16/09, Christian Vest Hansen karmazi...@gmail.com wrote: Another thing you might want to test is the fairness of the rw-lock in your RWDict, because even a couple of very active readers can easily starve out any number of writers when the rw-lock is non-fair. The reason is simple: readers can interleave but writers cannot, and writers can only get in when noone is reading :) On Fri, Jan 16, 2009 at 4:21 AM, Stu Hood stuh...@gmail.com wrote: Ah! but a mere hash table is not bi-directional :-) Right =) I got the idea in a Channel 9 video about MS' efforts with STM: http://channel9.msdn.com/shows/Going+Deep/Software-Transactional-Memory-The-Current-State-of-the-Art/ (which reminds me, the spin-lock approach they try is probably fairly close to using an Atom in Clojure). I made the changes that Christophe suggested, and added type hints for the HashMaps used in CLJDict, and the speed improvement is very impressive. To see the scalability of the different approaches, I graphed with various numbers of threads and read percentages: http://github.com/stuhood/clojure-conc/tree/master/results Two conclusions: 1. The overhead for STM with low contention is very reasonable, 2. Optimism + MVCC + persistence fall down when faced with a majority of writes. (see the 100% write case in the writes graph.) Thanks, Stu On Thu, Jan 15, 2009 at 2:52 PM, Christian Vest Hansen karmazi...@gmail.com wrote: On Thu, Jan 15, 2009 at 8:47 PM, Christian Vest Hansen karmazi...@gmail.com wrote: On Thu, Jan 15, 2009 at 8:35 PM, Mark H. mark.hoem...@gmail.com wrote: On Jan 15, 1:38 am, stuhood stuh...@gmail.com wrote: The benchmark contains 4 bi-directional dictionary implementations: ... Doesn't Java already have a more optimized thread-safe hash table that works by locking individual buckets, rather than the whole table? Maybe I'm just confused ;-P Ah! but a mere hash table is not bi-directional :-) -- Venlig hilsen / Kind regards, Christian Vest Hansen. -- Venlig hilsen / Kind regards, Christian Vest Hansen. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
Re: Synchronization Benchmarks
Ah! but a mere hash table is not bi-directional :-) Right =) I got the idea in a Channel 9 video about MS' efforts with STM: http://channel9.msdn.com/shows/Going+Deep/Software-Transactional-Memory-The-Current-State-of-the-Art/(which reminds me, the spin-lock approach they try is probably fairly close to using an Atom in Clojure). I made the changes that Christophe suggested, and added type hints for the HashMaps used in CLJDict, and the speed improvement is very impressive. To see the scalability of the different approaches, I graphed with various numbers of threads and read percentages: http://github.com/stuhood/clojure-conc/tree/master/results Two conclusions: 1. The overhead for STM with low contention is very reasonable, 2. Optimism + MVCC + persistence fall down when faced with a majority of writes. (see the 100% write case in the writes graph.) Thanks, Stu On Thu, Jan 15, 2009 at 2:52 PM, Christian Vest Hansen karmazi...@gmail.com wrote: On Thu, Jan 15, 2009 at 8:47 PM, Christian Vest Hansen karmazi...@gmail.com wrote: On Thu, Jan 15, 2009 at 8:35 PM, Mark H. mark.hoem...@gmail.com wrote: On Jan 15, 1:38 am, stuhood stuh...@gmail.com wrote: The benchmark contains 4 bi-directional dictionary implementations: ... Doesn't Java already have a more optimized thread-safe hash table that works by locking individual buckets, rather than the whole table? Maybe I'm just confused ;-P Ah! but a mere hash table is not bi-directional :-) -- Venlig hilsen / Kind regards, Christian Vest Hansen. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
Re: test-is reporting problem
You do that.
-another Stuart
On Thu, Jan 15, 2009 at 10:32 PM, Stuart Sierra the.stuart.sie...@gmail.com
wrote:
I was afraid that would happen. I'll fix it, probably tomorrow.
-the other Stuart
On Jan 15, 6:27 pm, Stuart Halloway stuart.hallo...@gmail.com wrote:
The improved error reposting in test-is breaks some tests, e.g. from
the book:
(deftest test-lazy-index-of-any-with-match
(is (with-out-str (is (zero? (index-of-any zzabyycdxx #{\z \a}
Iterating overz\n)
(is (with-out-str (is (= 3 (index-of-any zzabyycdxx #{\b \y}
Iterating overz\nIterating over z\nIterating over a\n))
The problem is that it tries to take the value of with-out-str, not
realizing that it is a macro.
Should I
(1) rewrite the test to not use a macro?
(2) take a stab at fixing this in test-is?
(3) get out of the way and let another Stuart handle it? :-)
Stuart
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
Clojure group.
To post to this group, send email to clojure@googlegroups.com
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
-~--~~~~--~~--~--~---
