Here's the actual code: https://gist.github.com/timlod/0f607e311d0464fd6c63
I am running the code from the REPL, may that be a problem? (As I read in the REPL everything is global). In the file nothing is global. Also, the counters are UInt16s, but that shouldnt matter I guess. Thanks for the help so far! On Saturday, 12 March 2016 14:22:38 UTC+1, Dan wrote: > > It's better to have code which actually runs in the post. In any case, the > allocations at the `for` lines is suspicious - the for should basically > only allocate a counter. Are there any global variables? Is `counter1` or > `counter2` or `dims` global? Globals are always a good source of confusion > to the type-inference engine. > > On Saturday, March 12, 2016 at 2:28:51 PM UTC+2, Tim Loderhose wrote: >> >> The code is in a function. I changed the names a bit to make it more >> understandable. The actual function is longer and has different variable >> names. >> >> On Saturday, 12 March 2016 13:01:28 UTC+1, tshort wrote: >>> >>> Is that code in a function? (It should be.) Also, one of your variable >>> names changed to `counter1s`. Suspect a type instability. >>> On Mar 12, 2016 4:12 AM, "Tim Loderhose" <[email protected]> wrote: >>> >>>> I tried around with that a bit, but then it gets much worse: From ~1s >>>> to ~6s, allocation as shown: >>>> >>>> 153710487 mat = Array{Complex64}(dims...) >>>> 4722450 file = Mmap.mmap(filename, Array{Complex64,2}, >>>> (dims[2],length(counter1))) >>>> 9568 for i = 1:dims[2] >>>> 4000 for j = 1:length(counter1) >>>> 1690462534 mat[counter1s[j],i,counter2[j]] = file[i,j] >>>> - end >>>> >>>> I swapped the for loops around here, but that didn't matter. I can gain >>>> a little bit by indexing i into the first dimension of mat, but it still >>>> lags far behind. >>>> Any other ideas? >>>> >>>> On Saturday, 12 March 2016 03:15:33 UTC+1, Greg Plowman wrote: >>>>> >>>>> I think array slices (on right hand side of assignment) create new >>>>> arrays, hence the allocation. >>>>> Try writing an explicit loop instead, something like: >>>>> >>>>> for j = 1:length(counter1) >>>>> for i = 1:size(file,1) >>>>> mat[counter1[j],i,counter2[j]] = file[i,j] >>>>> end >>>>> end >>>>> >>>>> >>>>> On Saturday, March 12, 2016 at 12:25:00 PM UTC+11, Tim Loderhose wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> I have a question regarding some allocation in my code I would like >>>>>> to get rid of. >>>>>> I am memory mapping a file (which could be very large) which is part >>>>>> of a complex 3D matrix, and then put its contents into the preallocated >>>>>> matrix along the second dimension. I need the counters because the >>>>>> contents >>>>>> of file are only a subset of the full matrix. >>>>>> >>>>>> Here's a profiled snippet, where the file which is loaded has >>>>>> 120619520 bytes. >>>>>> >>>>>> 153705063 mat = Array{Complex64}(dims...) >>>>>> 4721282 file = Mmap.mmap(filename, Array{Complex64,2}, >>>>>> (dims[2],length(counter1))) >>>>>> 16 for i = 1:length(counter1) >>>>>> 148179531 mat[counter1[i],:,counter2[i]] = file[:,i] >>>>>> - end >>>>>> >>>>>> Why does the code allocate so much memory inside the for-loop (even >>>>>> more bytes than the contents of file)? >>>>>> It seems like this is a trivial matter, right now I just can't get my >>>>>> head around it, any help is appreciated :) >>>>>> >>>>>> Thanks, >>>>>> Tim >>>>>> >>>>>
