On 31 January 2018 at 18:08, James Reeves <ja...@booleanknot.com> wrote:
> On 31 January 2018 at 17:59, Jacek Grzebyta <grzebyta....@gmail.com>
> wrote:
>
>> I have application with quite intense tripe store populating ~30/40 k
>> records per chunk (139 portions). The data are wrapped within the future:
>>
>> (conj agent (future (apply task args)))
>>
>> and that all together is send-off into (agent []).
>>
>
> What is "agent"? The first line of code indicates that it's a local
> collection shadowing the code function, while the second code snippet
> indicates that you're using the core agent function.
>
> Also why are you sending off to an agent?
>
I have ~8sec computing task for each input dataset which generates those
records. After that I write it into disk (in software-specific
transaction). I just wanted to separate hard computing and io operations. I
created a side-effect method which is injected together with the dataset
into a future. The futures are async collected within a list wrapped in
agent. After the computing the main thread is waiting until all io tasks
will be finished.
>
> At the end of the main thread function I just use await-for and after that:
>>
>> (reduce + (map #(deref %) @data-loading-tasks))
>>
>
As a control, tasks return number of written records.
>
>> For some reason I see the happy collecting (see attached screenshot of
>> jconsole).
>>
>
> "happy" = "heap"?
>
Both. As you can see on attached screenshot the heap usage grows easy until
aver. ~2 1/4 G than keep that for a few minutes. In that moment I stopped.
After that starts grow till ~4G with tendency to do jumps a bit more that
4G.
>
>
>> After seeing the source code of future I suppose that the memory (data
>> are kept as #{} set) is not released. The task returns only integer so I do
>> not think that might cause the problem.
>>
>
> Can you provide more detail? You keep alluding to things that you don't
> provide code for, such as the sets of data.
>
The code is attached. However the important code is
L123 .
(let [;; keeps all data loading futures.
;; waiting until all futures are finished
;; should be done outside the main loop
data-loading-tasks (agent [])]
L128
(doseq
(let [r1 (long operation)] L133
(doseq
(let [r2 (v.v. long)] L155
L163 (send-off data-loading-task conj-task)
)
)
)
)
I guess first I will move data-loading-tasks list into one of inner lets.
Also I will create within an injecting function a separate abstract
function let inside. The task will populate tmp variable which will be
returned as a future result:
L114 (conj agent (future (apply (fn [] (let [result (apply task args)]
result)))))
>
> --
> James Reeves
> booleanknot.com
>
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
experiment_result.clj
Description: Binary data
