I *think* it may work better if you replace the last 3 lines in your loop by:
a=all_TSFC[0] if len(all_TSFC) > 1: N.maximum(a, TSFC, out=a) Not 100% sure that would work though, as I'm not entirely confident I understand your code. -=- Olivier 2011/12/6 questions anon <questions.a...@gmail.com> > Something fancier I think, > I am able to compare the result with my previous method so I can easily > see I am doing something wrong. > see code below: > > > all_TSFC=[] > for (path, dirs, files) in os.walk(MainFolder): > for dir in dirs: > print dir > path=path+'/' > for ncfile in files: > if ncfile[-3:]=='.nc': > print "dealing with ncfiles:", ncfile > ncfile=os.path.join(path,ncfile) > ncfile=Dataset(ncfile, 'r+', 'NETCDF4') > TSFC=ncfile.variables['T_SFC'][:] > fillvalue=ncfile.variables['T_SFC']._FillValue > TSFC=MA.masked_values(TSFC, fillvalue) > ncfile.close() > all_TSFC.append(TSFC) > a=TSFC[0] > for b in TSFC[1:]: > N.maximum(a,b,out=a) > > big_array=N.ma.concatenate(all_TSFC) > Max=big_array.max(axis=0) > print "max is", Max,"a is", a > > > On Wed, Dec 7, 2011 at 2:34 PM, Olivier Delalleau <sh...@keba.be> wrote: > >> Is 'a' a regular numpy array or something fancier? >> >> >> -=- Olivier >> >> 2011/12/6 questions anon <questions.a...@gmail.com> >> >>> thanks again my only problem though is that the out=a in the loop does >>> not seem to replace my a= outside the loop so my final a is whatever I >>> started with for a. >>> Not sure what I am doing wrong whether it is something with the loop or >>> with the command. >>> >>> On Wed, Dec 7, 2011 at 1:44 PM, <josef.p...@gmail.com> wrote: >>> >>>> On Tue, Dec 6, 2011 at 9:36 PM, Olivier Delalleau <sh...@keba.be> >>>> wrote: >>>> > The "out=a" keyword will ensure your first array will keep being >>>> updated. So >>>> > you can do something like: >>>> > >>>> > a = my_list_of_arrays[0] >>>> > for b in my_list_of_arrays[1:]: >>>> > numpy.maximum(a, b, out=a) >>>> >>>> I didn't think of the out argument which makes it more efficient, but >>>> in my example I used Python's reduce which takes an iterable and not >>>> one huge array. >>>> >>>> Josef >>>> >>>> >>>> > >>>> > -=- Olivier >>>> > >>>> > 2011/12/6 questions anon <questions.a...@gmail.com> >>>> >> >>>> >> thanks for all of your help, that does look appropriate but I am not >>>> sure >>>> >> how to loop it over thousands of files. >>>> >> I need to keep the first array to compare with but replace any >>>> greater >>>> >> values as I loop through each array comparing back to the same >>>> array. does >>>> >> that make sense? >>>> >> >>>> >> >>>> >> On Wed, Dec 7, 2011 at 1:12 PM, Olivier Delalleau <sh...@keba.be> >>>> wrote: >>>> >>> >>>> >>> Thanks, I didn't know you could specify the out array :) >>>> >>> >>>> >>> (to the OP: my initial suggestion, although probably not very >>>> efficient, >>>> >>> seems to work with 2D arrays too, so I have no idea why it didn't >>>> work for >>>> >>> you -- but Nathaniel's one seems to be the ideal one anyway). >>>> >>> >>>> >>> -=- Olivier >>>> >>> >>>> >>> >>>> >>> 2011/12/6 Nathaniel Smith <n...@pobox.com> >>>> >>>> >>>> >>>> I think you want >>>> >>>> np.maximum(a, b, out=a) >>>> >>>> >>>> >>>> - Nathaniel >>>> >>>> >>>> >>>> On Dec 6, 2011 9:04 PM, "questions anon" <questions.a...@gmail.com >>>> > >>>> >>>> wrote: >>>> >>>>> >>>> >>>>> thanks for responding Josef but that is not really what I am >>>> looking >>>> >>>>> for, I have a multidimensional array and if the next array has >>>> any values >>>> >>>>> greater than what is in my first array I want to replace them. >>>> The data are >>>> >>>>> contained in netcdf files. >>>> >>>>> I can achieve what I want if I combine all of my arrays using >>>> numpy >>>> >>>>> concatenate and then using the command numpy.max(myarray, axis=0) >>>> but >>>> >>>>> because I have so many arrays I end up with a memory error so I >>>> need to find >>>> >>>>> a way to get the maximum while looping. >>>> >>>>> >>>> >>>>> >>>> >>>>> >>>> >>>>> On Wed, Dec 7, 2011 at 12:36 PM, <josef.p...@gmail.com> wrote: >>>> >>>>>> >>>> >>>>>> On Tue, Dec 6, 2011 at 7:55 PM, Olivier Delalleau <sh...@keba.be >>>> > >>>> >>>>>> wrote: >>>> >>>>>> > It may not be the most efficient way to do this, but you can >>>> do: >>>> >>>>>> > mask = b > a >>>> >>>>>> > a[mask] = b[mask] >>>> >>>>>> > >>>> >>>>>> > -=- Olivier >>>> >>>>>> > >>>> >>>>>> > 2011/12/6 questions anon <questions.a...@gmail.com> >>>> >>>>>> >> >>>> >>>>>> >> I would like to produce an array with the maximum values out >>>> of >>>> >>>>>> >> many >>>> >>>>>> >> (10000s) of arrays. >>>> >>>>>> >> I need to loop through many multidimentional arrays and if a >>>> value >>>> >>>>>> >> is >>>> >>>>>> >> larger (in the same place as the previous array) then I would >>>> like >>>> >>>>>> >> that >>>> >>>>>> >> value to replace it. >>>> >>>>>> >> >>>> >>>>>> >> e.g. >>>> >>>>>> >> a=[1,1,2,2 >>>> >>>>>> >> 11,2,2 >>>> >>>>>> >> 1,1,2,2] >>>> >>>>>> >> b=[1,1,3,2 >>>> >>>>>> >> 2,1,0,0 >>>> >>>>>> >> 1,1,2,0] >>>> >>>>>> >> >>>> >>>>>> >> where b>a replace with value in b, so the new a should be : >>>> >>>>>> >> >>>> >>>>>> >> a=[1,1,3,2] >>>> >>>>>> >> 2,1,2,2 >>>> >>>>>> >> 1,1,2,2] >>>> >>>>>> >> >>>> >>>>>> >> and then keep looping through many arrays and replace whenever >>>> >>>>>> >> value is >>>> >>>>>> >> larger. >>>> >>>>>> >> >>>> >>>>>> >> I have tried numpy.putmask but that results in >>>> >>>>>> >> TypeError: putmask() argument 1 must be numpy.ndarray, not >>>> list >>>> >>>>>> >> Any other ideas? Thanks >>>> >>>>>> >>>> >>>>>> if I understand correctly it's a minimum.reduce >>>> >>>>>> >>>> >>>>>> numpy >>>> >>>>>> >>>> >>>>>> >>> a = np.concatenate((np.arange(5)[::-1], >>>> >>>>>> >>> np.arange(5)))*np.ones((4,3,1)) >>>> >>>>>> >>> np.minimum.reduce(a, axis=2) >>>> >>>>>> array([[ 0., 0., 0.], >>>> >>>>>> [ 0., 0., 0.], >>>> >>>>>> [ 0., 0., 0.], >>>> >>>>>> [ 0., 0., 0.]]) >>>> >>>>>> >>> a.T.shape >>>> >>>>>> (10, 3, 4) >>>> >>>>>> >>>> >>>>>> python with iterable >>>> >>>>>> >>>> >>>>>> >>> reduce(np.maximum, a.T) >>>> >>>>>> array([[ 4., 4., 4., 4.], >>>> >>>>>> [ 4., 4., 4., 4.], >>>> >>>>>> [ 4., 4., 4., 4.]]) >>>> >>>>>> >>> reduce(np.minimum, a.T) >>>> >>>>>> array([[ 0., 0., 0., 0.], >>>> >>>>>> [ 0., 0., 0., 0.], >>>> >>>>>> [ 0., 0., 0., 0.]]) >>>> >>>>>> >>>> >>>>>> Josef >>>> >>>>>> >>>> >>>>>> >> >>>> >>>>>> >> _______________________________________________ >>>> >>>>>> >> NumPy-Discussion mailing list >>>> >>>>>> >> NumPy-Discussion@scipy.org >>>> >>>>>> >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>> >>>>>> >> >>>> >>>>>> > >>>> >>>>>> > >>>> >>>>>> > _______________________________________________ >>>> >>>>>> > NumPy-Discussion mailing list >>>> >>>>>> > NumPy-Discussion@scipy.org >>>> >>>>>> > http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>> >>>>>> > >>>> >>>>>> _______________________________________________ >>>> >>>>>> NumPy-Discussion mailing list >>>> >>>>>> NumPy-Discussion@scipy.org >>>> >>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>> >>>>> >>>> >>>>> >>>> >>>>> >>>> >>>>> _______________________________________________ >>>> >>>>> NumPy-Discussion mailing list >>>> >>>>> NumPy-Discussion@scipy.org >>>> >>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>> >>>>> >>>> >>>> >>>> >>>> _______________________________________________ >>>> >>>> NumPy-Discussion mailing list >>>> >>>> NumPy-Discussion@scipy.org >>>> >>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>> >>>> >>>> >>> >>>> >>> >>>> >>> _______________________________________________ >>>> >>> NumPy-Discussion mailing list >>>> >>> NumPy-Discussion@scipy.org >>>> >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>> >>> >>>> >> >>>> >> >>>> >> _______________________________________________ >>>> >> NumPy-Discussion mailing list >>>> >> NumPy-Discussion@scipy.org >>>> >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>> >> >>>> > >>>> > >>>> > _______________________________________________ >>>> > NumPy-Discussion mailing list >>>> > NumPy-Discussion@scipy.org >>>> > http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>> > >>>> _______________________________________________ >>>> NumPy-Discussion mailing list >>>> NumPy-Discussion@scipy.org >>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>> >>> >>> >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion@scipy.org >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >>> >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion@scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > >
_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion