That makes sense, thanks for the explanation! It's still strange why sort
needs to allocate so much memory. There doesn't to be any place where it
allocates memory in the method apart from the initial copy.
On Sunday, July 6, 2014 6:09:02 PM UTC-4, Stefan Karpinski wrote:
>
> Tasks and produce/consume aren't really an optimization issue – coroutines
> are just the wrong abstraction for doing a lot of small, fast work. Think
> of each task as it's own stack and switching coroutines as moving the point
> of execution from one stack to another. This is not just an analogy – this
> is how it works. Switching stacks is fine for doing I/O but not great for
> producing a series of integers.
>
> On Jul 6, 2014, at 2:54 PM, Sid <tmf...@gmail.com <javascript:>> wrote:
>
> Ah ok. So memory for stack frames for instance would also count?
>
> Also -- I think an issue like this (the functions not being inlined)
> should probably be added to the Julia performance tips page. One of the
> biggest fears I have when trying a new language is that I won't know the
> main pitfalls in terms of performance so I won't realize when my code is
> running slow because of I used a language feature that's not at the moment
> fully optimized.
>
> I also saw somewhere else that @task and @produce are not fully optimized
> and that Iterators.jl is the recommended solution for iterators at the
> moment so maybe that too would be useful to add to the performance page.
>
> On Sunday, July 6, 2014 5:40:37 PM UTC-4, Stefan Karpinski wrote:
>>
>> This seems to be a frequent cause of confusion: allocated is the
>> cumulative amount of memory allocated while running something. It is not
>> the high water mark, nor is it the net increase in memory usage.
>>
>> On Jul 6, 2014, at 2:16 PM, Sid <tmf...@gmail.com> wrote:
>>
>> You mean @allocated shows the total amount of memory traversed or
>> something instead of the total amount allocated?
>>
>> Incidentally, I reran the benchmark with a 100 million size array this
>> time. I got this number from @allocation
>>
>> 101077550864
>>
>> which is 101 GB.
>>
>> Activity monitor shows a much more reasonble continuous memory usage of
>> 1.6 GB which is still much more than the 800MB needed for the array (but
>> maybe that's due to how the VM allocates memory?). Still this ratio of 2 is
>> much better than the ratio of almost 10 when sorting an array of size 1
>> million.
>>
>> On Sunday, July 6, 2014 4:28:02 PM UTC-4, Kevin Squire wrote:
>>>
>>> It's likely that much of the memory was for temporary allocations that
>>> were reused.
>>>
>>> Cheers,
>>> Kevin
>>>
>>>
>>> On Sun, Jul 6, 2014 at 10:32 AM, Sid <tmf...@gmail.com> wrote:
>>>
>>>> Regarding memory allocation -- it seems like it might be a bug in the
>>>> @allocated macro rather than an actual indication of memory usage.
>>>>
>>>> When I run
>>>>
>>>> a = rand(100000)
>>>> @allocated sort(a, by=x->x)
>>>>
>>>> I get 768403668 which is 768 MB.
>>>>
>>>> However, I was also recording the process's memory usage using %mem.
>>>> For my 16 GB machine, the usage initially is 0.5% and jumps to 1.3% after
>>>> running the above two commands which comes to around ~200MB rather than
>>>> 768MB.
>>>>
>>>> While that's still a high jump compared to the array size (which is
>>>> only ~8MB), that might be because of the way VM allocates more memory.
>>>> Perhaps
>>>> someone who's more familiar with the internals can comment.
>>>>
>>>> On Sunday, July 6, 2014 7:30:47 AM UTC-4, gentlebeldin wrote:
>>>>>
>>>>> No, that wasn't the reason, in this case. I had another, hard look at
>>>>> my own (rough and ready) implementation of factoring, found a rather
>>>>> stupid
>>>>> mistake, and now, it runs in 2.2 seconds.
>>>>> Using List<Integer> or List<Double> instead of int[] or double[] for
>>>>> numerical computations would be a silly idea, indeed, and not just
>>>>> because
>>>>> of poor performance. It would also eat up lots of memory. In this case, I
>>>>> had to sort a big array of integers (100,000,000 elements), with a custom
>>>>> comparator. That isn't possible with Java's int[], and Integer[] with so
>>>>> many elements is a bit harsh for a netbook with 2 GB of RAM. Then, I
>>>>> thought "hey, I can implement the List<Integer> interface with an int[]!"
>>>>> No such luck, the implementation of Collections.sort calls toArray,
>>>>> first, ->Integer[] :-(
>>>>> Then, they introduced streams in Java 8, including IntStream, a stream
>>>>> of ints, for better performance, and defined the corresponding functional
>>>>> interface for a custom comparator. That sounds promising, but as we know,
>>>>> most promises aren't kept. Believe it or not, the implementation of
>>>>> IntStream.sort collects the whole stream in an Integer[], first. :-(
>>>>> That's why I put some hope into Julia. Unfortunately, it may allocate
>>>>> huge amounts of memory in an unpredictable way (for me, at least), too.
>>>>>
>>>>> Am Sonntag, 6. Juli 2014 07:12:26 UTC+2 schrieb Sid:
>>>>>>
>>>>>> Are you using Integer/Double objects by any chance in Java? I've
>>>>>> found using them to be a surefire way of destroying Java's performance...
>>>>>>
>>>>>> On Saturday, July 5, 2014 3:11:47 AM UTC-4, gentlebeldin wrote:
>>>>>>>
>>>>>>> It's hard to tell for sure, I wouldn't know how to check that. I
>>>>>>> prefer to port a few more solutions. My curiosity was much increased by
>>>>>>> this one:
>>>>>>>
>>>>>>> julia> include("julia/jl/e443gcd.jl")
>>>>>>> elapsed time: 3.255950529 seconds (929628 bytes allocated)
>>>>>>>
>>>>>>>
>>>>>>> The Java original "runs" much longer, 33 seconds.
>>>>>>>
>>>>>>>
>>>>>>>>
>>>
