Hi Stuart,
On Dec 4, 4:09 pm, Stuart Sierra <[EMAIL PROTECTED]> wrote:
> I think a lazy cons is cached only as long as you hold a pointer to
> the head. Are you sure you don't have any pointers to doc-seq hanging
> around in another piece of code? Or, alternately, pointers to the
> filter output?
You're probably right with regard to lazy-cons not being the problem,
because looking at the source code for the map function, it is built
on top of lazy-conses as well, and I'm using it fine for some
sequences with millions items.
I don't think I'm retaining either the head or the sequence it is
creating, because I only use this in the context of doseq (here's a
brief snippet):
(doseq [document (documents-from-vendors (document-seq path)
vendors)
When I'm filtering using documents-from-vendors (as above), it
actually runs out of memory quite quickly - it occurred to me that
perhaps there was some small memory overhead, but looking at the code
I could not reason about why it would blow up after a few tens of
thousands of documents, instead of many millions.
If I remove that filtering line, and instead do:
(doseq [document (document-seq path)
it doesn't blow up, and it will get through the complete sequence
without exhausting the available heap space (but of course without the
filtering I'm including things in the set of items under consideration
that I don't actually want there). I can manually filter when I'm
consuming documents from one of my other (working) sequences, but I
like the elegance of composing filtering operations on top of lazy
sequences.
Thanks for the feedback and ideas.
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Clojure" group.
To post to this group, send email to clojure@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
-~----------~----~----~----~------~----~------~--~---
