Re: Running out of memory when using loop/recur and destructuring

2009-11-10 Thread John Harrop
On Tue, Nov 10, 2009 at 7:21 AM, Rich Hickey richhic...@gmail.com wrote: Right - pervasive locals clearing will definitely do the trick here. Interestingly, when I was at Microsoft and asked them about handling this issue for the CLR they stated plainly it wasn't an issue at all - their

Re: Running out of memory when using loop/recur and destructuring

2009-11-10 Thread Christophe Grand
On Wed, Nov 4, 2009 at 6:46 AM, John Harrop jharrop...@gmail.com wrote: On Tue, Nov 3, 2009 at 1:53 AM, Alex Osborne a...@meshy.org wrote: The new loop uses the outer-let to get around this: (let [G__13697 s       [x xs] G__13697       y xs]   (loop* [G__13697 G__13697           y y]    

Re: Running out of memory when using loop/recur and destructuring

2009-11-09 Thread Rock
I've been following this thread, and I must say I'm puzzled that Rich hasn't said anything at all about this issue yet. It seems important enough to hear his own opinion. On 6 Nov, 18:56, Paul Mooser taron...@gmail.com wrote: So, I've been hoping that Rich (or someone?) would weigh in on this,

Re: Running out of memory when using loop/recur and destructuring

2009-11-09 Thread John Harrop
On Mon, Nov 9, 2009 at 4:31 PM, Rock rocco.ro...@gmail.com wrote: I've been following this thread, and I must say I'm puzzled that Rich hasn't said anything at all about this issue yet. It seems important enough to hear his own opinion. My observation over the past few months is that Rich

Re: Running out of memory when using loop/recur and destructuring

2009-11-09 Thread Paul Mooser
I imagine he's just busy. At this point, I plan to create a ticket on assembla, if that's possible - I think I just need to create a login and then file it. On Nov 9, 2:07 pm, John Harrop jharrop...@gmail.com wrote: On Mon, Nov 9, 2009 at 4:31 PM, Rock rocco.ro...@gmail.com wrote: I've been

Re: Running out of memory when using loop/recur and destructuring

2009-11-05 Thread Paul Mooser
It does make me wonder, however, if having the lazy-seq cache things is sort of conflating laziness and consistency, since as you point out, not all ISeq implementations do any sort of caching. I wonder if it would be interesting to decompose it into 'lazy- seq' (uncached), and 'cached-seq'. I

Re: Running out of memory when using loop/recur and destructuring

2009-11-04 Thread Christophe Grand
On Tue, Nov 3, 2009 at 7:27 PM, Paul Mooser taron...@gmail.com wrote: Ah -- I hadn't understood that when using destructuring, that subsequent bindings could refer to the destructured elements. I should have, since clojure only has let*, and this behavior seems consistent with that, for

Re: Running out of memory when using loop/recur and destructuring

2009-11-04 Thread Paul Mooser
Well, I care (conceptually) more about the fix being made, rather than the exact timeframe. If we had to wait until clojure-in-clojure, I think I could live with that, since the issue can be readily avoided. We'll see if Rich has a chance to chime-in to acknowledge whether or not he considers

Re: Running out of memory when using loop/recur and destructuring

2009-11-04 Thread Chouser
On Tue, Nov 3, 2009 at 11:51 PM, Mark Engelberg mark.engelb...@gmail.com wrote: Clojure's built-in range function (last time I looked) essentially produces an uncached sequence.  And that makes a lot of sense. 'range' has since changed and now produces a chunked lazy seq (master branch

Re: Running out of memory when using loop/recur and destructuring

2009-11-04 Thread Paul Mooser
I completely understand the difference between the ISeq interface, and the particular implementation (lazy-seq) that results in these problems. It would be fairly straightforward, I think, to write some kind of uncached-lazy-seq which doesn't exhibit these problems, but I've felt that is

Re: Running out of memory when using loop/recur and destructuring

2009-11-03 Thread Paul Mooser
Ah -- I hadn't understood that when using destructuring, that subsequent bindings could refer to the destructured elements. I should have, since clojure only has let*, and this behavior seems consistent with that, for binding. Eeww. It seems like quite a thorny issue to solve, even if simple to

Re: Running out of memory when using loop/recur and destructuring

2009-11-03 Thread Paul Mooser
In the particular case given below, I'd assume that during the invocation of print-seq, the binding to s (the head of the sequence) would be retained, because my mental model for the execution environment of a function is that it is the environment in which they were declared, extended with the

Re: Running out of memory when using loop/recur and destructuring

2009-11-03 Thread Paul Mooser
I understand the pragmatism of your approach, but it's really unfortunate. Seqs are a really convenient abstraction, and the ability to model arbitrarily large or infinite ones (with laziness) is really useful. In my opinion, only using seqs when all of the data can be fit into memory really

Re: Running out of memory when using loop/recur and destructuring

2009-11-03 Thread Brian Hurt
On Tue, Nov 3, 2009 at 5:19 PM, Paul Mooser taron...@gmail.com wrote: I understand the pragmatism of your approach, but it's really unfortunate. Seqs are a really convenient abstraction, and the ability to model arbitrarily large or infinite ones (with laziness) is really useful. In my

Re: Running out of memory when using loop/recur and destructuring

2009-11-03 Thread Brian Hurt
We encountered similar problems at work trying to wrap I/O up into lazy seq's. The problem is that it is very easy to accidentally hold on to the head of a seq while enumerating it's elements. In addition, we had problems with not closing file descriptors. A common pattern was to open a file,

Re: Running out of memory when using loop/recur and destructuring

2009-11-03 Thread Mark Engelberg
I agree that seqs carry a large degree of risk. You have to work very hard to avoid giving your large sequences a name, lest you accidentally hang on to the head. In Clojure's early days, I complained about this and described some of my own experiments with uncached sequences. Rich said he was

Re: Running out of memory when using loop/recur and destructuring

2009-11-03 Thread John Harrop
On Tue, Nov 3, 2009 at 1:53 AM, Alex Osborne a...@meshy.org wrote: The new loop uses the outer-let to get around this: (let [G__13697 s [x xs] G__13697 y xs] (loop* [G__13697 G__13697 y y] (let [[x xs] G__13697 y y] ...)))

Re: Running out of memory when using loop/recur and destructuring

2009-11-02 Thread Paul Mooser
I'm a little surprised I haven't seen more response on this topic, since this class of bug (inadvertently holding onto the head of sequences) is pretty nasty to run into, and is sort of awful to debug. I'm wondering if there's a different way to write the loop macro so that it doesn't expand into

Re: Running out of memory when using loop/recur and destructuring

2009-11-02 Thread Christophe Grand
Hi Paul, It's indeed surprising and at first glance, it looks like a bug but after researching the logs, this let form was introduced in the following commit http://github.com/richhickey/clojure/commit/288f34dbba4a9e643dd7a7f77642d0f0088f95ad with comment fixed loop with destructuring and

Re: Running out of memory when using loop/recur and destructuring

2009-11-02 Thread John Harrop
On Mon, Nov 2, 2009 at 2:39 PM, Christophe Grand christo...@cgrand.netwrote: Right now I can't see how loop can be made to support both cases. Hopefully someone else will. In the meantime, remember that it's always worth trying to implement seq-processing in terms of map, reduce, filter, for,

Re: Running out of memory when using loop/recur and destructuring

2009-11-02 Thread Paul Mooser
This is great advice, of course. On the other hand, I feel it's important to be explicitly clear about which forms will hold on to (seemingly) transient data. Certain things are explicitly clear about this (such as the docstring for doseq), and this particular case is unfortunate because in the

Re: Running out of memory when using loop/recur and destructuring

2009-11-02 Thread Paul Mooser
Good job tracking down that diff -- upon looking at it, unfortunately, I obviously don't understand the underlying issue being fixed (the inter-binding dependencies) because the old code basically matches what I would think would be the way to avoid introducing this in an outer let form --

Re: Running out of memory when using loop/recur and destructuring

2009-11-02 Thread Alex Osborne
Paul Mooser wrote: Good job tracking down that diff -- upon looking at it, unfortunately, I obviously don't understand the underlying issue being fixed (the inter-binding dependencies) because the old code basically matches what I would think would be the way to avoid introducing this in an

Re: Running out of memory when using loop/recur and destructuring

2009-10-31 Thread Paul Mooser
I actually restructured my code (not the toy example posted here) to avoid the destructuring, and was disappointed to find it also eventually blows up on 1.6 as well. I'm reasonably certain in that case that I'm not holding on to any of the sequence (since I don't refer to it outside the

Re: Running out of memory when using loop/recur and destructuring

2009-10-31 Thread Paul Mooser
A user on IRC named hiredman had the excellent idea (which should have occurred to me, but didn't) to macroexpand my code. A macro expansion of (loop [[head tail] (repeat 1)] (recur tail)) results in: (let* [G__10 (repeat 1) vec__11 G__10 head (clojure.core/nth vec__11 0

Re: Running out of memory when using loop/recur and destructuring

2009-10-31 Thread Paul Mooser
From looking at the source code the loop macro, it looks like this might be particular to destructuring with loop, rather than being related to destructuring in general ? --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google

Running out of memory when using loop/recur and destructuring

2009-10-30 Thread Paul Mooser
I was working with a large data set earlier today, and I had written a loop/recur where I was passing in a huge seq to the first iteration, and I was surprised when I ran out of heap space, because I was very careful not to hold on to the head of the seq, and I though that loop ended up rebinding

Re: Running out of memory when using loop/recur and destructuring

2009-10-30 Thread John Harrop
On Fri, Oct 30, 2009 at 3:15 PM, Paul Mooser taron...@gmail.com wrote: Is this behavior due to some artifact of destructuring I'm not aware of (or something else I'm missing), or is there a bug? If it sounds like a bug, can anyone else reproduce? Thanks! I vaguely remember something like