Yep, `peCounters`, `paCounters` and `dims` are not type-stable. They are one type by their default values and then assigned another. Perhaps rename the default parameters, and copy them to `peCounters`, `paCounters` and `dims` only if they are set to something other than `0`.
Also, `mdhcounters` might not return a definite type (need to check that function). Fixing these should make the loop efficient. On Saturday, March 12, 2016 at 4:05:39 PM UTC+2, Tim Loderhose wrote: > > Here's the actual code: > https://gist.github.com/timlod/0f607e311d0464fd6c63 > > I am running the code from the REPL, may that be a problem? (As I read in > the REPL everything is global). In the file nothing is global. > Also, the counters are UInt16s, but that shouldnt matter I guess. > > Thanks for the help so far! > > On Saturday, 12 March 2016 14:22:38 UTC+1, Dan wrote: >> >> It's better to have code which actually runs in the post. In any case, >> the allocations at the `for` lines is suspicious - the for should basically >> only allocate a counter. Are there any global variables? Is `counter1` or >> `counter2` or `dims` global? Globals are always a good source of confusion >> to the type-inference engine. >> >> On Saturday, March 12, 2016 at 2:28:51 PM UTC+2, Tim Loderhose wrote: >>> >>> The code is in a function. I changed the names a bit to make it more >>> understandable. The actual function is longer and has different variable >>> names. >>> >>> On Saturday, 12 March 2016 13:01:28 UTC+1, tshort wrote: >>>> >>>> Is that code in a function? (It should be.) Also, one of your variable >>>> names changed to `counter1s`. Suspect a type instability. >>>> On Mar 12, 2016 4:12 AM, "Tim Loderhose" <[email protected]> wrote: >>>> >>>>> I tried around with that a bit, but then it gets much worse: From ~1s >>>>> to ~6s, allocation as shown: >>>>> >>>>> 153710487 mat = Array{Complex64}(dims...) >>>>> 4722450 file = Mmap.mmap(filename, Array{Complex64,2}, >>>>> (dims[2],length(counter1))) >>>>> 9568 for i = 1:dims[2] >>>>> 4000 for j = 1:length(counter1) >>>>> 1690462534 mat[counter1s[j],i,counter2[j]] = file[i,j] >>>>> - end >>>>> >>>>> I swapped the for loops around here, but that didn't matter. I can >>>>> gain a little bit by indexing i into the first dimension of mat, but it >>>>> still lags far behind. >>>>> Any other ideas? >>>>> >>>>> On Saturday, 12 March 2016 03:15:33 UTC+1, Greg Plowman wrote: >>>>>> >>>>>> I think array slices (on right hand side of assignment) create new >>>>>> arrays, hence the allocation. >>>>>> Try writing an explicit loop instead, something like: >>>>>> >>>>>> for j = 1:length(counter1) >>>>>> for i = 1:size(file,1) >>>>>> mat[counter1[j],i,counter2[j]] = file[i,j] >>>>>> end >>>>>> end >>>>>> >>>>>> >>>>>> On Saturday, March 12, 2016 at 12:25:00 PM UTC+11, Tim Loderhose >>>>>> wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> I have a question regarding some allocation in my code I would like >>>>>>> to get rid of. >>>>>>> I am memory mapping a file (which could be very large) which is part >>>>>>> of a complex 3D matrix, and then put its contents into the preallocated >>>>>>> matrix along the second dimension. I need the counters because the >>>>>>> contents >>>>>>> of file are only a subset of the full matrix. >>>>>>> >>>>>>> Here's a profiled snippet, where the file which is loaded has >>>>>>> 120619520 bytes. >>>>>>> >>>>>>> 153705063 mat = Array{Complex64}(dims...) >>>>>>> 4721282 file = Mmap.mmap(filename, Array{Complex64,2}, >>>>>>> (dims[2],length(counter1))) >>>>>>> 16 for i = 1:length(counter1) >>>>>>> 148179531 mat[counter1[i],:,counter2[i]] = file[:,i] >>>>>>> - end >>>>>>> >>>>>>> Why does the code allocate so much memory inside the for-loop (even >>>>>>> more bytes than the contents of file)? >>>>>>> It seems like this is a trivial matter, right now I just can't get >>>>>>> my head around it, any help is appreciated :) >>>>>>> >>>>>>> Thanks, >>>>>>> Tim >>>>>>> >>>>>>
