I implemented the suggestions (see updated gist: https://gist.github.com/timlod/0f607e311d0464fd6c63). The allocation in the for loops disappeared and the time required halved to ~3s. The mmap access though still allocated a lot. And the vectorised access is still much faster and allocates less (although I want to get rid of the allocation altogether). Any other ideas?
On Saturday, 12 March 2016 15:15:51 UTC+1, Dan wrote: > > Yep, `peCounters`, `paCounters` and `dims` are not type-stable. They are > one type by their default values and then assigned another. Perhaps rename > the default parameters, and copy them to `peCounters`, `paCounters` and > `dims` only if they are set to something other than `0`. > > Also, `mdhcounters` might not return a definite type (need to check that > function). > Fixing these should make the loop efficient. > > On Saturday, March 12, 2016 at 4:05:39 PM UTC+2, Tim Loderhose wrote: >> >> Here's the actual code: >> https://gist.github.com/timlod/0f607e311d0464fd6c63 >> <https://www.google.com/url?q=https%3A%2F%2Fgist.github.com%2Ftimlod%2F0f607e311d0464fd6c63&sa=D&sntz=1&usg=AFQjCNHB4_Itk64Xevbo5RACmmrF0lsgBA> >> >> I am running the code from the REPL, may that be a problem? (As I read in >> the REPL everything is global). In the file nothing is global. >> Also, the counters are UInt16s, but that shouldnt matter I guess. >> >> Thanks for the help so far! >> >> On Saturday, 12 March 2016 14:22:38 UTC+1, Dan wrote: >>> >>> It's better to have code which actually runs in the post. In any case, >>> the allocations at the `for` lines is suspicious - the for should basically >>> only allocate a counter. Are there any global variables? Is `counter1` or >>> `counter2` or `dims` global? Globals are always a good source of confusion >>> to the type-inference engine. >>> >>> On Saturday, March 12, 2016 at 2:28:51 PM UTC+2, Tim Loderhose wrote: >>>> >>>> The code is in a function. I changed the names a bit to make it more >>>> understandable. The actual function is longer and has different variable >>>> names. >>>> >>>> On Saturday, 12 March 2016 13:01:28 UTC+1, tshort wrote: >>>>> >>>>> Is that code in a function? (It should be.) Also, one of your variable >>>>> names changed to `counter1s`. Suspect a type instability. >>>>> On Mar 12, 2016 4:12 AM, "Tim Loderhose" <[email protected]> wrote: >>>>> >>>>>> I tried around with that a bit, but then it gets much worse: From ~1s >>>>>> to ~6s, allocation as shown: >>>>>> >>>>>> 153710487 mat = Array{Complex64}(dims...) >>>>>> 4722450 file = Mmap.mmap(filename, Array{Complex64,2}, >>>>>> (dims[2],length(counter1))) >>>>>> 9568 for i = 1:dims[2] >>>>>> 4000 for j = 1:length(counter1) >>>>>> 1690462534 mat[counter1s[j],i,counter2[j]] = file[i,j] >>>>>> - end >>>>>> >>>>>> I swapped the for loops around here, but that didn't matter. I can >>>>>> gain a little bit by indexing i into the first dimension of mat, but it >>>>>> still lags far behind. >>>>>> Any other ideas? >>>>>> >>>>>> On Saturday, 12 March 2016 03:15:33 UTC+1, Greg Plowman wrote: >>>>>>> >>>>>>> I think array slices (on right hand side of assignment) create new >>>>>>> arrays, hence the allocation. >>>>>>> Try writing an explicit loop instead, something like: >>>>>>> >>>>>>> for j = 1:length(counter1) >>>>>>> for i = 1:size(file,1) >>>>>>> mat[counter1[j],i,counter2[j]] = file[i,j] >>>>>>> end >>>>>>> end >>>>>>> >>>>>>> >>>>>>> On Saturday, March 12, 2016 at 12:25:00 PM UTC+11, Tim Loderhose >>>>>>> wrote: >>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> I have a question regarding some allocation in my code I would like >>>>>>>> to get rid of. >>>>>>>> I am memory mapping a file (which could be very large) which is >>>>>>>> part of a complex 3D matrix, and then put its contents into the >>>>>>>> preallocated matrix along the second dimension. I need the counters >>>>>>>> because >>>>>>>> the contents of file are only a subset of the full matrix. >>>>>>>> >>>>>>>> Here's a profiled snippet, where the file which is loaded has >>>>>>>> 120619520 bytes. >>>>>>>> >>>>>>>> 153705063 mat = Array{Complex64}(dims...) >>>>>>>> 4721282 file = Mmap.mmap(filename, Array{Complex64,2}, >>>>>>>> (dims[2],length(counter1))) >>>>>>>> 16 for i = 1:length(counter1) >>>>>>>> 148179531 mat[counter1[i],:,counter2[i]] = file[:,i] >>>>>>>> - end >>>>>>>> >>>>>>>> Why does the code allocate so much memory inside the for-loop (even >>>>>>>> more bytes than the contents of file)? >>>>>>>> It seems like this is a trivial matter, right now I just can't get >>>>>>>> my head around it, any help is appreciated :) >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Tim >>>>>>>> >>>>>>>
