Cool, Ill try that out later! I was wondering about that, but obviously I
wasn't able to default dims::Array{Int64,1} = 0.
How do I make sure it returns a definite type? I know that it should always
be an Array{UInt64,1}. I added the code of mdhcounters to the gist.
Thanks for the input!
On Saturday, 12 March 2016 15:15:51 UTC+1, Dan wrote:
>
> Yep, `peCounters`, `paCounters` and `dims` are not type-stable. They are
> one type by their default values and then assigned another. Perhaps rename
> the default parameters, and copy them to `peCounters`, `paCounters` and
> `dims` only if they are set to something other than `0`.
>
> Also, `mdhcounters` might not return a definite type (need to check that
> function).
> Fixing these should make the loop efficient.
>
> On Saturday, March 12, 2016 at 4:05:39 PM UTC+2, Tim Loderhose wrote:
>>
>> Here's the actual code:
>> https://gist.github.com/timlod/0f607e311d0464fd6c63
>>
>> I am running the code from the REPL, may that be a problem? (As I read in
>> the REPL everything is global). In the file nothing is global.
>> Also, the counters are UInt16s, but that shouldnt matter I guess.
>>
>> Thanks for the help so far!
>>
>> On Saturday, 12 March 2016 14:22:38 UTC+1, Dan wrote:
>>>
>>> It's better to have code which actually runs in the post. In any case,
>>> the allocations at the `for` lines is suspicious - the for should basically
>>> only allocate a counter. Are there any global variables? Is `counter1` or
>>> `counter2` or `dims` global? Globals are always a good source of confusion
>>> to the type-inference engine.
>>>
>>> On Saturday, March 12, 2016 at 2:28:51 PM UTC+2, Tim Loderhose wrote:
>>>>
>>>> The code is in a function. I changed the names a bit to make it more
>>>> understandable. The actual function is longer and has different variable
>>>> names.
>>>>
>>>> On Saturday, 12 March 2016 13:01:28 UTC+1, tshort wrote:
>>>>>
>>>>> Is that code in a function? (It should be.) Also, one of your variable
>>>>> names changed to `counter1s`. Suspect a type instability.
>>>>> On Mar 12, 2016 4:12 AM, "Tim Loderhose" <[email protected]> wrote:
>>>>>
>>>>>> I tried around with that a bit, but then it gets much worse: From ~1s
>>>>>> to ~6s, allocation as shown:
>>>>>>
>>>>>> 153710487 mat = Array{Complex64}(dims...)
>>>>>> 4722450 file = Mmap.mmap(filename, Array{Complex64,2},
>>>>>> (dims[2],length(counter1)))
>>>>>> 9568 for i = 1:dims[2]
>>>>>> 4000 for j = 1:length(counter1)
>>>>>> 1690462534 mat[counter1s[j],i,counter2[j]] = file[i,j]
>>>>>> - end
>>>>>>
>>>>>> I swapped the for loops around here, but that didn't matter. I can
>>>>>> gain a little bit by indexing i into the first dimension of mat, but it
>>>>>> still lags far behind.
>>>>>> Any other ideas?
>>>>>>
>>>>>> On Saturday, 12 March 2016 03:15:33 UTC+1, Greg Plowman wrote:
>>>>>>>
>>>>>>> I think array slices (on right hand side of assignment) create new
>>>>>>> arrays, hence the allocation.
>>>>>>> Try writing an explicit loop instead, something like:
>>>>>>>
>>>>>>> for j = 1:length(counter1)
>>>>>>> for i = 1:size(file,1)
>>>>>>> mat[counter1[j],i,counter2[j]] = file[i,j]
>>>>>>> end
>>>>>>> end
>>>>>>>
>>>>>>>
>>>>>>> On Saturday, March 12, 2016 at 12:25:00 PM UTC+11, Tim Loderhose
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I have a question regarding some allocation in my code I would like
>>>>>>>> to get rid of.
>>>>>>>> I am memory mapping a file (which could be very large) which is
>>>>>>>> part of a complex 3D matrix, and then put its contents into the
>>>>>>>> preallocated matrix along the second dimension. I need the counters
>>>>>>>> because
>>>>>>>> the contents of file are only a subset of the full matrix.
>>>>>>>>
>>>>>>>> Here's a profiled snippet, where the file which is loaded has
>>>>>>>> 120619520 bytes.
>>>>>>>>
>>>>>>>> 153705063 mat = Array{Complex64}(dims...)
>>>>>>>> 4721282 file = Mmap.mmap(filename, Array{Complex64,2},
>>>>>>>> (dims[2],length(counter1)))
>>>>>>>> 16 for i = 1:length(counter1)
>>>>>>>> 148179531 mat[counter1[i],:,counter2[i]] = file[:,i]
>>>>>>>> - end
>>>>>>>>
>>>>>>>> Why does the code allocate so much memory inside the for-loop (even
>>>>>>>> more bytes than the contents of file)?
>>>>>>>> It seems like this is a trivial matter, right now I just can't get
>>>>>>>> my head around it, any help is appreciated :)
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Tim
>>>>>>>>
>>>>>>>
