by the way, have you tried both Oracle and Open JDK with the same results?
Gianluca
On Tuesday, November 17, 2015 at 8:28:49 PM UTC+1, Andy Fingerhut wrote:
>
> David, you say "Based on jvisualvm monitoring, doesn't seem to be
> GC-related".
>
> What is jvisualvm showing you related to GC and/or memory allocation when
> you tried the 18-core version with 18 threads in the same process?
>
> Even memory allocation could become a point of contention, depending upon
> how the memory allocation works with many threads. e.g. Depends on whether
> a thread gets a large chunk of memory on a global lock, and then locally
> carves it up into the small pieces it needs for each individual Java 'new'
> allocation, or gets a global lock for every 'new'. The latter would give
> terrible performance as # cores increase, but I don't know how to tell
> whether that is the case, except by knowing more about how the memory
> allocator is implemented in your JVM. Maybe digging through OpenJDK source
> code in the right place would tell?
>
> Andy
>
> On Tue, Nov 17, 2015 at 2:00 AM, David Iba <davi...@gmail.com
> <javascript:>> wrote:
>
>> correction: that "do" should be a "doall". (My actual test code was a
>> bit different, but each run printed some info when it started so it doesn't
>> have to do with delayed evaluation of lazy seq's or anything).
>>
>>
>> On Tuesday, November 17, 2015 at 6:49:16 PM UTC+9, David Iba wrote:
>>>
>>> Andy: Interesting. Thanks for educating me on the fact that atom
>>> swap's don't use the STM. Your theory seems plausible... I will try those
>>> tests next time I launch the 18-core instance, but yeah, not sure how
>>> illuminating the results will be.
>>>
>>> Niels: along the lines of this (so that each thread prints its time as
>>> well as printing the overall time):
>>>
>>> 1. (time
>>> 2. (let [f f1
>>> 3. n-runs 18
>>> 4. futs (do (for [i (range n-runs)]
>>> 5. (future (time (f)))))]
>>> 6. (doseq [fut futs]
>>> 7. @fut)))
>>>
>>>
>>> On Tuesday, November 17, 2015 at 5:33:01 PM UTC+9, Niels van Klaveren
>>> wrote:
>>>>
>>>> Could you also show how you are running these functions in parallel and
>>>> time them ? The way you start the functions can have as much impact as the
>>>> functions themselves.
>>>>
>>>> Regards,
>>>> Niels
>>>>
>>>> On Tuesday, November 17, 2015 at 6:38:39 AM UTC+1, David Iba wrote:
>>>>>
>>>>> I have functions f1 and f2 below, and let's say they run in T1 and T2
>>>>> amount of time when running a single instance/thread. The issue I'm
>>>>> facing
>>>>> is that parallelizing f2 across 18 cores takes anywhere from 2-5X T2, and
>>>>> for more complex funcs takes absurdly long.
>>>>>
>>>>>
>>>>> 1. (defn f1 []
>>>>> 2. (apply + (range 2e9)))
>>>>> 3.
>>>>> 4. ;; Note: each call to (f2) makes its own x* atom, so the
>>>>> 'swap!' should never retry.
>>>>> 5. (defn f2 []
>>>>> 6. (let [x* (atom {})]
>>>>> 7. (loop [i 1e9]
>>>>> 8. (when-not (zero? i)
>>>>> 9. (swap! x* assoc :k i)
>>>>> 10. (recur (dec i))))))
>>>>>
>>>>>
>>>>> Of note:
>>>>> - On a 4-core machine, both f1 and f2 parallelize well (roungly T1 and
>>>>> T2 for 4 runs in parallel)
>>>>> - running 18 f1's in parallel on the 18-core machine also parallelizes
>>>>> well.
>>>>> - Disabling hyperthreading doesn't help.
>>>>> - Based on jvisualvm monitoring, doesn't seem to be GC-related
>>>>> - also tried on dedicated 18-core ec2 instance with same issues, so
>>>>> not shared-tenancy-related
>>>>> - if I make a jar that runs a single f2 and launch 18 in parallel, it
>>>>> parallelizes well (so I don't think it's machine/aws-related)
>>>>>
>>>>> Could it be that the 18 f2's in parallel on a single JVM instance is
>>>>> overworking the STM with all the swap's? Any other theories?
>>>>>
>>>>> Thanks!
>>>>>
>>>> --
>> You received this message because you are subscribed to the Google
>> Groups "Clojure" group.
>> To post to this group, send email to clo...@googlegroups.com
>> <javascript:>
>> Note that posts from new members are moderated - please be patient with
>> your first post.
>> To unsubscribe from this group, send email to
>> clojure+u...@googlegroups.com <javascript:>
>> For more options, visit this group at
>> http://groups.google.com/group/clojure?hl=en
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "Clojure" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to clojure+u...@googlegroups.com <javascript:>.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
