On Sun, Mar 13, 2016 at 10:02 AM, Tim Loderhose <[email protected]> wrote:
> I passed it as an argument from the REPL. I made a test script where I
> hardcode these things to make sure its not global, but the allocation
> remains.
> In this gist there is the code with allocations and @time result for the
> double for loop as well as the vectorised version:
> https://gist.github.com/timlod/6a2324db4d01dc07dd5a
> roft = Array{Complex64}(dims...)
Don't splat an array. Type inference have no idea how long the array may be.
Also, in general, use the various
{@}code_{lowered/typed/warntype/llvm/native} functions/macros to see
what the code is actually doing and solve the type instability before
trying to guess from benchmark result.
>
>
> On Sunday, 13 March 2016 13:36:10 UTC+1, tshort wrote:
>>
>> Where does your variable `dims` come from? As pointed out above, global
>> variables can hurt type inference.
>>
>> On Mar 13, 2016 8:29 AM, "Tim Loderhose" <[email protected]> wrote:
>>>
>>> I implemented the suggestions (see updated gist:
>>> https://gist.github.com/timlod/0f607e311d0464fd6c63).
>>> The allocation in the for loops disappeared and the time required halved
>>> to ~3s. The mmap access though still allocated a lot.
>>> And the vectorised access is still much faster and allocates less
>>> (although I want to get rid of the allocation altogether).
>>> Any other ideas?
>>>
>>> On Saturday, 12 March 2016 15:15:51 UTC+1, Dan wrote:
>>>>
>>>> Yep, `peCounters`, `paCounters` and `dims` are not type-stable. They are
>>>> one type by their default values and then assigned another. Perhaps rename
>>>> the default parameters, and copy them to `peCounters`, `paCounters` and
>>>> `dims` only if they are set to something other than `0`.
>>>>
>>>> Also, `mdhcounters` might not return a definite type (need to check that
>>>> function).
>>>> Fixing these should make the loop efficient.
>>>>
>>>> On Saturday, March 12, 2016 at 4:05:39 PM UTC+2, Tim Loderhose wrote:
>>>>>
>>>>> Here's the actual code:
>>>>> https://gist.github.com/timlod/0f607e311d0464fd6c63
>>>>>
>>>>> I am running the code from the REPL, may that be a problem? (As I read
>>>>> in the REPL everything is global). In the file nothing is global.
>>>>> Also, the counters are UInt16s, but that shouldnt matter I guess.
>>>>>
>>>>> Thanks for the help so far!
>>>>>
>>>>> On Saturday, 12 March 2016 14:22:38 UTC+1, Dan wrote:
>>>>>>
>>>>>> It's better to have code which actually runs in the post. In any case,
>>>>>> the allocations at the `for` lines is suspicious - the for should
>>>>>> basically
>>>>>> only allocate a counter. Are there any global variables? Is `counter1` or
>>>>>> `counter2` or `dims` global? Globals are always a good source of
>>>>>> confusion
>>>>>> to the type-inference engine.
>>>>>>
>>>>>> On Saturday, March 12, 2016 at 2:28:51 PM UTC+2, Tim Loderhose wrote:
>>>>>>>
>>>>>>> The code is in a function. I changed the names a bit to make it more
>>>>>>> understandable. The actual function is longer and has different variable
>>>>>>> names.
>>>>>>>
>>>>>>> On Saturday, 12 March 2016 13:01:28 UTC+1, tshort wrote:
>>>>>>>>
>>>>>>>> Is that code in a function? (It should be.) Also, one of your
>>>>>>>> variable names changed to `counter1s`. Suspect a type instability.
>>>>>>>>
>>>>>>>> On Mar 12, 2016 4:12 AM, "Tim Loderhose" <[email protected]> wrote:
>>>>>>>>>
>>>>>>>>> I tried around with that a bit, but then it gets much worse: From
>>>>>>>>> ~1s to ~6s, allocation as shown:
>>>>>>>>>
>>>>>>>>> 153710487 mat = Array{Complex64}(dims...)
>>>>>>>>> 4722450 file = Mmap.mmap(filename, Array{Complex64,2},
>>>>>>>>> (dims[2],length(counter1)))
>>>>>>>>> 9568 for i = 1:dims[2]
>>>>>>>>> 4000 for j = 1:length(counter1)
>>>>>>>>> 1690462534 mat[counter1s[j],i,counter2[j]] = file[i,j]
>>>>>>>>> - end
>>>>>>>>>
>>>>>>>>> I swapped the for loops around here, but that didn't matter. I can
>>>>>>>>> gain a little bit by indexing i into the first dimension of mat, but
>>>>>>>>> it
>>>>>>>>> still lags far behind.
>>>>>>>>> Any other ideas?
>>>>>>>>>
>>>>>>>>> On Saturday, 12 March 2016 03:15:33 UTC+1, Greg Plowman wrote:
>>>>>>>>>>
>>>>>>>>>> I think array slices (on right hand side of assignment) create new
>>>>>>>>>> arrays, hence the allocation.
>>>>>>>>>> Try writing an explicit loop instead, something like:
>>>>>>>>>>
>>>>>>>>>> for j = 1:length(counter1)
>>>>>>>>>> for i = 1:size(file,1)
>>>>>>>>>> mat[counter1[j],i,counter2[j]] = file[i,j]
>>>>>>>>>> end
>>>>>>>>>> end
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Saturday, March 12, 2016 at 12:25:00 PM UTC+11, Tim Loderhose
>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> I have a question regarding some allocation in my code I would
>>>>>>>>>>> like to get rid of.
>>>>>>>>>>> I am memory mapping a file (which could be very large) which is
>>>>>>>>>>> part of a complex 3D matrix, and then put its contents into the
>>>>>>>>>>> preallocated
>>>>>>>>>>> matrix along the second dimension. I need the counters because the
>>>>>>>>>>> contents
>>>>>>>>>>> of file are only a subset of the full matrix.
>>>>>>>>>>>
>>>>>>>>>>> Here's a profiled snippet, where the file which is loaded has
>>>>>>>>>>> 120619520 bytes.
>>>>>>>>>>>
>>>>>>>>>>> 153705063 mat = Array{Complex64}(dims...)
>>>>>>>>>>> 4721282 file = Mmap.mmap(filename, Array{Complex64,2},
>>>>>>>>>>> (dims[2],length(counter1)))
>>>>>>>>>>> 16 for i = 1:length(counter1)
>>>>>>>>>>> 148179531 mat[counter1[i],:,counter2[i]] = file[:,i]
>>>>>>>>>>> - end
>>>>>>>>>>>
>>>>>>>>>>> Why does the code allocate so much memory inside the for-loop
>>>>>>>>>>> (even more bytes than the contents of file)?
>>>>>>>>>>> It seems like this is a trivial matter, right now I just can't
>>>>>>>>>>> get my head around it, any help is appreciated :)
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Tim
