Where does your variable `dims` come from? As pointed out above, global variables can hurt type inference. On Mar 13, 2016 8:29 AM, "Tim Loderhose" <[email protected]> wrote:
> I implemented the suggestions (see updated gist: > https://gist.github.com/timlod/0f607e311d0464fd6c63). > The allocation in the for loops disappeared and the time required halved > to ~3s. The mmap access though still allocated a lot. > And the vectorised access is still much faster and allocates less > (although I want to get rid of the allocation altogether). > Any other ideas? > > On Saturday, 12 March 2016 15:15:51 UTC+1, Dan wrote: >> >> Yep, `peCounters`, `paCounters` and `dims` are not type-stable. They are >> one type by their default values and then assigned another. Perhaps rename >> the default parameters, and copy them to `peCounters`, `paCounters` and >> `dims` only if they are set to something other than `0`. >> >> Also, `mdhcounters` might not return a definite type (need to check that >> function). >> Fixing these should make the loop efficient. >> >> On Saturday, March 12, 2016 at 4:05:39 PM UTC+2, Tim Loderhose wrote: >>> >>> Here's the actual code: >>> https://gist.github.com/timlod/0f607e311d0464fd6c63 >>> <https://www.google.com/url?q=https%3A%2F%2Fgist.github.com%2Ftimlod%2F0f607e311d0464fd6c63&sa=D&sntz=1&usg=AFQjCNHB4_Itk64Xevbo5RACmmrF0lsgBA> >>> >>> I am running the code from the REPL, may that be a problem? (As I read >>> in the REPL everything is global). In the file nothing is global. >>> Also, the counters are UInt16s, but that shouldnt matter I guess. >>> >>> Thanks for the help so far! >>> >>> On Saturday, 12 March 2016 14:22:38 UTC+1, Dan wrote: >>>> >>>> It's better to have code which actually runs in the post. In any case, >>>> the allocations at the `for` lines is suspicious - the for should basically >>>> only allocate a counter. Are there any global variables? Is `counter1` or >>>> `counter2` or `dims` global? Globals are always a good source of confusion >>>> to the type-inference engine. >>>> >>>> On Saturday, March 12, 2016 at 2:28:51 PM UTC+2, Tim Loderhose wrote: >>>>> >>>>> The code is in a function. I changed the names a bit to make it more >>>>> understandable. The actual function is longer and has different variable >>>>> names. >>>>> >>>>> On Saturday, 12 March 2016 13:01:28 UTC+1, tshort wrote: >>>>>> >>>>>> Is that code in a function? (It should be.) Also, one of your >>>>>> variable names changed to `counter1s`. Suspect a type instability. >>>>>> On Mar 12, 2016 4:12 AM, "Tim Loderhose" <[email protected]> wrote: >>>>>> >>>>>>> I tried around with that a bit, but then it gets much worse: From >>>>>>> ~1s to ~6s, allocation as shown: >>>>>>> >>>>>>> 153710487 mat = Array{Complex64}(dims...) >>>>>>> 4722450 file = Mmap.mmap(filename, Array{Complex64,2}, >>>>>>> (dims[2],length(counter1))) >>>>>>> 9568 for i = 1:dims[2] >>>>>>> 4000 for j = 1:length(counter1) >>>>>>> 1690462534 mat[counter1s[j],i,counter2[j]] = file[i,j] >>>>>>> - end >>>>>>> >>>>>>> I swapped the for loops around here, but that didn't matter. I can >>>>>>> gain a little bit by indexing i into the first dimension of mat, but it >>>>>>> still lags far behind. >>>>>>> Any other ideas? >>>>>>> >>>>>>> On Saturday, 12 March 2016 03:15:33 UTC+1, Greg Plowman wrote: >>>>>>>> >>>>>>>> I think array slices (on right hand side of assignment) create new >>>>>>>> arrays, hence the allocation. >>>>>>>> Try writing an explicit loop instead, something like: >>>>>>>> >>>>>>>> for j = 1:length(counter1) >>>>>>>> for i = 1:size(file,1) >>>>>>>> mat[counter1[j],i,counter2[j]] = file[i,j] >>>>>>>> end >>>>>>>> end >>>>>>>> >>>>>>>> >>>>>>>> On Saturday, March 12, 2016 at 12:25:00 PM UTC+11, Tim Loderhose >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> I have a question regarding some allocation in my code I would >>>>>>>>> like to get rid of. >>>>>>>>> I am memory mapping a file (which could be very large) which is >>>>>>>>> part of a complex 3D matrix, and then put its contents into the >>>>>>>>> preallocated matrix along the second dimension. I need the counters >>>>>>>>> because >>>>>>>>> the contents of file are only a subset of the full matrix. >>>>>>>>> >>>>>>>>> Here's a profiled snippet, where the file which is loaded has >>>>>>>>> 120619520 bytes. >>>>>>>>> >>>>>>>>> 153705063 mat = Array{Complex64}(dims...) >>>>>>>>> 4721282 file = Mmap.mmap(filename, Array{Complex64,2}, >>>>>>>>> (dims[2],length(counter1))) >>>>>>>>> 16 for i = 1:length(counter1) >>>>>>>>> 148179531 mat[counter1[i],:,counter2[i]] = file[:,i] >>>>>>>>> - end >>>>>>>>> >>>>>>>>> Why does the code allocate so much memory inside the for-loop >>>>>>>>> (even more bytes than the contents of file)? >>>>>>>>> It seems like this is a trivial matter, right now I just can't get >>>>>>>>> my head around it, any help is appreciated :) >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Tim >>>>>>>>> >>>>>>>>
