[job] S.F Bay Area: Enjoy using Clojure to do a bit of everyhing?

[Arthur Ulfeldt]

Yummly, Redwood City, CA

I'm looking for folks who genuinly enjoy using Clojure to do a great 
variety of things like Backend mobile APIs, Web services, Infrastructre/AWS 
automation, security scanning, and all sorts of other things. It's really a 
lot of fun.

If you're intersted please email me: art...@yummly.com or call me at 
1-219-CLOJURE

This is honestly the happiest team i have worked with (small sample size 
bias)

https://careers.jobscore.com/careers/yummly/jobs/generalist-software-engineer-dnDs0w99Kr57z9dG1ZS6tF?sid=161

and in general about working at yummly:
https://www.yummly.com/careers

most of us work remotly some of the time and everyone is in the office much 
of the time as well. It's a really nice office and friendly pet friendly. 

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: what does future do after fn finish ?

[Alan Thompson]

You may find that using the Claypoole library is the easiest way to handle
threadpools:   https://github.com/TheClimateCorporation/claypoole
Alan

On Thu, Feb 1, 2018 at 11:16 AM, Justin Smith  wrote:

> yes, that's the idea exactly
>
> also, you might want more fine grained control of how much parallelism
> occurs (eg. if every thread is writing to the same physical device, you can
> often get better throughput by not parallelizing at all, or keeping the
> parallelism quite limited - it's worth experimenting) - there are good ways
> to control that using ThreadPoolExecutor directly, or using
> clojure.lang.PersistentQueue/EMPTY as a control construct, or core.async
> channels, or ztellman's manifold library, or the claypoole threading library
>
> On Thu, Feb 1, 2018, 03:44 Jacek Grzebyta  wrote:
>
>> Thanks folks. I see now! It should be a list of agents not list of
>> futures within agent.  Also any task sent to a agent is processed
>> within a thread anyway so I do not need to add future...
>>
>> On 1 February 2018 at 02:17, John Newman  wrote:
>>
>>> Ah, he's using one agent, I see.
>>>
>>> On Jan 31, 2018 9:15 PM, "John Newman"  wrote:
>>>
 Multiple sen-doffs to one agent will serialize it's calls, but spawning
 agents on each new task will spawn threads on a bounded thread pool, I
 believe.

 On Jan 31, 2018 8:32 PM, "Justin Smith"  wrote:

> Doing all the actions via one agent means that the actions are
> serialized though - you end up with no performance improvement over doing
> them all in a doseq in one future - the right way to do this tends to be
> trickier than it looks at first glance, and depends on your requirements.
> agents, the claypoole library, and reducers are all potentially useful. If
> parallelization leads to complex coordination needs, core.async can help
> too.
>
> On Wed, Jan 31, 2018 at 5:18 PM John Newman  wrote:
>
>> Agents manage a pool of threads for you. Try doing it without the
>> future call and see if that works (unless you're trying to do something
>> else).
>>
>> John
>>
>> On Wed, Jan 31, 2018 at 7:31 PM, Jacek Grzebyta <
>> grzebyta@gmail.com> wrote:
>>
>>> Thanks a lot. I will check it tomorrow.
>>>
>>> J
>>>
>>> On 1 Feb 2018 12:12 a.m., "Justin Smith" 
>>> wrote:
>>>
 this is exactly the kind of problem code I was describing - there's
 no backpressure on existing future tasks to hold up the launching of 
 more
 futures - the work done by the agent calling conj is negligible. You 
 need
 to control the size of the pool of threads used, and you need to impose
 back-pressure.

 On Wed, Jan 31, 2018 at 3:46 PM Jacek Grzebyta <
 grzebyta@gmail.com> wrote:

> On 31 January 2018 at 18:08, James Reeves 
> wrote:
>
>> On 31 January 2018 at 17:59, Jacek Grzebyta <
>> grzebyta@gmail.com> wrote:
>>
>>> I have application with quite intense tripe store populating
>>> ~30/40 k records per chunk (139 portions). The data are wrapped 
>>> within the
>>> future:
>>>
>>> (conj agent (future (apply task args)))
>>>
>>>  and that all together is send-off into (agent []).
>>>
>>
>> What is "agent"? The first line of code indicates that it's a
>> local collection shadowing the code function, while the second code 
>> snippet
>> indicates that you're using the core agent function.
>>
>> Also why are you sending off to an agent?
>>
>
> I have ~8sec computing task for each input dataset which generates
> those records. After that I write it into disk (in software-specific
> transaction). I just wanted to separate hard computing and io 
> operations. I
> created a side-effect method which is injected together with the 
> dataset
> into a future. The futures are async collected within a list wrapped 
> in
> agent. After the computing the main thread is waiting until all io 
> tasks
> will be finished.
>
>
>>
>> At the end of the main thread function I just use await-for and
>>> after that:
>>>
>>> (reduce + (map #(deref %) @data-loading-tasks))
>>>
>>
> As a control, tasks return number of written records.
>
>
>
>>
>>> For some reason I see the happy collecting (see attached
>>> screenshot of jconsole).
>>>
>>
>> "happy" = "heap"?
>>
>
> Both. As 

Re: what does future do after fn finish ?

[Justin Smith]

yes, that's the idea exactly

also, you might want more fine grained control of how much parallelism
occurs (eg. if every thread is writing to the same physical device, you can
often get better throughput by not parallelizing at all, or keeping the
parallelism quite limited - it's worth experimenting) - there are good ways
to control that using ThreadPoolExecutor directly, or using
clojure.lang.PersistentQueue/EMPTY as a control construct, or core.async
channels, or ztellman's manifold library, or the claypoole threading library

On Thu, Feb 1, 2018, 03:44 Jacek Grzebyta  wrote:

> Thanks folks. I see now! It should be a list of agents not list of futures
> within agent.  Also any task sent to a agent is processed within a
> thread anyway so I do not need to add future...
>
> On 1 February 2018 at 02:17, John Newman  wrote:
>
>> Ah, he's using one agent, I see.
>>
>> On Jan 31, 2018 9:15 PM, "John Newman"  wrote:
>>
>>> Multiple sen-doffs to one agent will serialize it's calls, but spawning
>>> agents on each new task will spawn threads on a bounded thread pool, I
>>> believe.
>>>
>>> On Jan 31, 2018 8:32 PM, "Justin Smith"  wrote:
>>>
 Doing all the actions via one agent means that the actions are
 serialized though - you end up with no performance improvement over doing
 them all in a doseq in one future - the right way to do this tends to be
 trickier than it looks at first glance, and depends on your requirements.
 agents, the claypoole library, and reducers are all potentially useful. If
 parallelization leads to complex coordination needs, core.async can help
 too.

 On Wed, Jan 31, 2018 at 5:18 PM John Newman  wrote:

> Agents manage a pool of threads for you. Try doing it without the
> future call and see if that works (unless you're trying to do something
> else).
>
> John
>
> On Wed, Jan 31, 2018 at 7:31 PM, Jacek Grzebyta <
> grzebyta@gmail.com> wrote:
>
>> Thanks a lot. I will check it tomorrow.
>>
>> J
>>
>> On 1 Feb 2018 12:12 a.m., "Justin Smith" 
>> wrote:
>>
>>> this is exactly the kind of problem code I was describing - there's
>>> no backpressure on existing future tasks to hold up the launching of 
>>> more
>>> futures - the work done by the agent calling conj is negligible. You 
>>> need
>>> to control the size of the pool of threads used, and you need to impose
>>> back-pressure.
>>>
>>> On Wed, Jan 31, 2018 at 3:46 PM Jacek Grzebyta <
>>> grzebyta@gmail.com> wrote:
>>>
 On 31 January 2018 at 18:08, James Reeves 
 wrote:

> On 31 January 2018 at 17:59, Jacek Grzebyta <
> grzebyta@gmail.com> wrote:
>
>> I have application with quite intense tripe store populating
>> ~30/40 k records per chunk (139 portions). The data are wrapped 
>> within the
>> future:
>>
>> (conj agent (future (apply task args)))
>>
>>  and that all together is send-off into (agent []).
>>
>
> What is "agent"? The first line of code indicates that it's a
> local collection shadowing the code function, while the second code 
> snippet
> indicates that you're using the core agent function.
>
> Also why are you sending off to an agent?
>

 I have ~8sec computing task for each input dataset which generates
 those records. After that I write it into disk (in software-specific
 transaction). I just wanted to separate hard computing and io 
 operations. I
 created a side-effect method which is injected together with the 
 dataset
 into a future. The futures are async collected within a list wrapped in
 agent. After the computing the main thread is waiting until all io 
 tasks
 will be finished.


>
> At the end of the main thread function I just use await-for and
>> after that:
>>
>> (reduce + (map #(deref %) @data-loading-tasks))
>>
>
 As a control, tasks return number of written records.



>
>> For some reason I see the happy collecting (see attached
>> screenshot of jconsole).
>>
>
> "happy" = "heap"?
>

 Both. As you can see on attached screenshot the heap usage grows
 easy until aver. ~2 1/4 G than keep that  for a few minutes. In that 
 moment
 I stopped. After that starts grow till ~4G with tendency to do jumps a 
 bit
 more that 4G.


>
>
>> After seeing the source code of future I suppose 

Re: what does future do after fn finish ?

[Jacek Grzebyta]

Thanks folks. I see now! It should be a list of agents not list of futures
within agent.  Also any task sent to a agent is processed within a
thread anyway so I do not need to add future...

On 1 February 2018 at 02:17, John Newman  wrote:

> Ah, he's using one agent, I see.
>
> On Jan 31, 2018 9:15 PM, "John Newman"  wrote:
>
>> Multiple sen-doffs to one agent will serialize it's calls, but spawning
>> agents on each new task will spawn threads on a bounded thread pool, I
>> believe.
>>
>> On Jan 31, 2018 8:32 PM, "Justin Smith"  wrote:
>>
>>> Doing all the actions via one agent means that the actions are
>>> serialized though - you end up with no performance improvement over doing
>>> them all in a doseq in one future - the right way to do this tends to be
>>> trickier than it looks at first glance, and depends on your requirements.
>>> agents, the claypoole library, and reducers are all potentially useful. If
>>> parallelization leads to complex coordination needs, core.async can help
>>> too.
>>>
>>> On Wed, Jan 31, 2018 at 5:18 PM John Newman  wrote:
>>>
 Agents manage a pool of threads for you. Try doing it without the
 future call and see if that works (unless you're trying to do something
 else).

 John

 On Wed, Jan 31, 2018 at 7:31 PM, Jacek Grzebyta  wrote:

> Thanks a lot. I will check it tomorrow.
>
> J
>
> On 1 Feb 2018 12:12 a.m., "Justin Smith"  wrote:
>
>> this is exactly the kind of problem code I was describing - there's
>> no backpressure on existing future tasks to hold up the launching of more
>> futures - the work done by the agent calling conj is negligible. You need
>> to control the size of the pool of threads used, and you need to impose
>> back-pressure.
>>
>> On Wed, Jan 31, 2018 at 3:46 PM Jacek Grzebyta <
>> grzebyta@gmail.com> wrote:
>>
>>> On 31 January 2018 at 18:08, James Reeves 
>>> wrote:
>>>
 On 31 January 2018 at 17:59, Jacek Grzebyta  wrote:

> I have application with quite intense tripe store populating
> ~30/40 k records per chunk (139 portions). The data are wrapped 
> within the
> future:
>
> (conj agent (future (apply task args)))
>
>  and that all together is send-off into (agent []).
>

 What is "agent"? The first line of code indicates that it's a local
 collection shadowing the code function, while the second code snippet
 indicates that you're using the core agent function.

 Also why are you sending off to an agent?

>>>
>>> I have ~8sec computing task for each input dataset which generates
>>> those records. After that I write it into disk (in software-specific
>>> transaction). I just wanted to separate hard computing and io 
>>> operations. I
>>> created a side-effect method which is injected together with the dataset
>>> into a future. The futures are async collected within a list wrapped in
>>> agent. After the computing the main thread is waiting until all io tasks
>>> will be finished.
>>>
>>>

 At the end of the main thread function I just use await-for and
> after that:
>
> (reduce + (map #(deref %) @data-loading-tasks))
>

>>> As a control, tasks return number of written records.
>>>
>>>
>>>

> For some reason I see the happy collecting (see attached
> screenshot of jconsole).
>

 "happy" = "heap"?

>>>
>>> Both. As you can see on attached screenshot the heap usage grows
>>> easy until aver. ~2 1/4 G than keep that  for a few minutes. In that 
>>> moment
>>> I stopped. After that starts grow till ~4G with tendency to do jumps a 
>>> bit
>>> more that 4G.
>>>
>>>


> After seeing the source code of future I suppose that the memory
> (data are kept as #{} set) is not released. The task returns only 
> integer
> so I do not think that might cause the problem.
>

 Can you provide more detail? You keep alluding to things that you
 don't provide code for, such as the sets of data.

>>>
>>>
>>> The code is attached. However the important code is
>>>
>>> L123 .
>>>   (let [;; keeps all data loading futures.
>>> ;; waiting until all futures are finished
>>> ;; should be done outside the main loop
>>> data-loading-tasks (agent [])]
>>>
>>> L128
>>> (doseq
>>>  (let [r1 (long operation)]   L133
>>>  (doseq
>>> (let [r2 

Re: If Clojure is to blame for the majority of the startup time, why doesn't ClojureScript proportionally slow down the JavaScript startup time also?

[Herwig Hochleitner]

2018-02-01 3:24 GMT+01:00 scott stackelhouse :

> CRIU seems like it could actually help in this case.  In particular,
> paired with Docker, I can just restart from a known good checkpoint with
> editor loaded, repl up and connected, etc.  That would be very nice.
>

Yep, it nice. Although, with CRIU, you have to be very careful how far you
initialize your program (open files, network sockets, ) and you have to
disable java's hsperfdata files (because they get deleted after the first
run and are then missing for CRIU's restore), but I suppose the larger a
Program is, the more CRIU can save time for require / load.

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.