Re: [Numpy-discussion] Rounding float to integer while minizing the difference between the two arrays?
Sorry, there is one error in this part of code, it should be: def convert_integer(x,threshold=0): This fucntion converts the float number x to integer according to the threshold. if abs(x-0) 1e-5: return 0 else: pdec,pint = math.modf(x) if pdec threshold: return int(math.ceil(pint)+1) else: return int(math.ceil(pint)) On Wed, Jul 16, 2014 at 3:18 PM, Chao YUE chaoyue...@gmail.com wrote: Dear all, I have two arrays with both float type, let's say X and Y. I want to round the X to integers (intX) according to some decimal threshold, at the same time I want to limit the following difference as small: diff = np.sum(X*Y) - np.sum(intX*Y) I don't have to necessarily minimize the diff variable (If with this demand the computation time is too long). But I would like to limit the diff to, let's say ten percent within np.sum(X*Y). I have tried to write some functions, but I don't know where to start the opitimization. def convert_integer(x,threshold=0): This fucntion converts the float number x to integer according to the threshold. if abs(x-0) 1e5: return 0 else: pdec,pint = math.modf(x) if pdec threshold: return int(math.ceil(pint)+1) else: return int(math.ceil(pint)) def convert_arr(arr,threshold=0): out = arr.copy() for i,num in enumerate(arr): out[i] = convert_integer(num,threshold=threshold) return out In [147]: convert_arr(np.array([0.14,1.14,0.12]),0.13) Out[147]: array([1, 2, 0]) Now my problem is, how can I minimize or limit the following? diff = np.sum(X*Y) - np.sum(convert_arr(X,threshold=?)*Y) Because it's the first time I encounter such kind of question, so please give me some clue to start :p Thanks a lot in advance. Best, Chao -- please visit: http://www.globalcarbonatlas.org/ *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 -- please visit: http://www.globalcarbonatlas.org/ *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Rounding float to integer while minizing the difference between the two arrays?
Dear all, A bit sorry, this is not difficult. scipy.optimize.minimize_scalar seems to solve my problem. Thanks anyway, for this great tool. Cheers, Chao On Wed, Jul 16, 2014 at 3:18 PM, Chao YUE chaoyue...@gmail.com wrote: Dear all, I have two arrays with both float type, let's say X and Y. I want to round the X to integers (intX) according to some decimal threshold, at the same time I want to limit the following difference as small: diff = np.sum(X*Y) - np.sum(intX*Y) I don't have to necessarily minimize the diff variable (If with this demand the computation time is too long). But I would like to limit the diff to, let's say ten percent within np.sum(X*Y). I have tried to write some functions, but I don't know where to start the opitimization. def convert_integer(x,threshold=0): This fucntion converts the float number x to integer according to the threshold. if abs(x-0) 1e5: return 0 else: pdec,pint = math.modf(x) if pdec threshold: return int(math.ceil(pint)+1) else: return int(math.ceil(pint)) def convert_arr(arr,threshold=0): out = arr.copy() for i,num in enumerate(arr): out[i] = convert_integer(num,threshold=threshold) return out In [147]: convert_arr(np.array([0.14,1.14,0.12]),0.13) Out[147]: array([1, 2, 0]) Now my problem is, how can I minimize or limit the following? diff = np.sum(X*Y) - np.sum(convert_arr(X,threshold=?)*Y) Because it's the first time I encounter such kind of question, so please give me some clue to start :p Thanks a lot in advance. Best, Chao -- please visit: http://www.globalcarbonatlas.org/ *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 -- please visit: http://www.globalcarbonatlas.org/ *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Rounding float to integer while minizing the difference between the two arrays?
Dear all, I have two arrays with both float type, let's say X and Y. I want to round the X to integers (intX) according to some decimal threshold, at the same time I want to limit the following difference as small: diff = np.sum(X*Y) - np.sum(intX*Y) I don't have to necessarily minimize the diff variable (If with this demand the computation time is too long). But I would like to limit the diff to, let's say ten percent within np.sum(X*Y). I have tried to write some functions, but I don't know where to start the opitimization. def convert_integer(x,threshold=0): This fucntion converts the float number x to integer according to the threshold. if abs(x-0) 1e5: return 0 else: pdec,pint = math.modf(x) if pdec threshold: return int(math.ceil(pint)+1) else: return int(math.ceil(pint)) def convert_arr(arr,threshold=0): out = arr.copy() for i,num in enumerate(arr): out[i] = convert_integer(num,threshold=threshold) return out In [147]: convert_arr(np.array([0.14,1.14,0.12]),0.13) Out[147]: array([1, 2, 0]) Now my problem is, how can I minimize or limit the following? diff = np.sum(X*Y) - np.sum(convert_arr(X,threshold=?)*Y) Because it's the first time I encounter such kind of question, so please give me some clue to start :p Thanks a lot in advance. Best, Chao -- please visit: http://www.globalcarbonatlas.org/ *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] percentile function for masked array?
Dear all, Thank you for this information. I will return to this issue later and probably make patch (as temporary solution) for this. Because I never tried before, so it may take me some time. For the other overall masked array constructing issues, it might be left further for more discussion. Best, Chao On Mon, Jun 2, 2014 at 6:50 PM, Alexander Belopolsky ndar...@mac.com wrote: On Mon, Jun 2, 2014 at 12:25 PM, Charles R Harris charlesr.har...@gmail.com wrote: I think the masked array code is also due a cleanup/rationalization. Any comments you have along that line are welcome. Here are a few thoughts: 1. Please avoid another major rewrite. 2. Stop pretending that instances of ma.MaskedArray and ndarray have is a relationship. Use of inheritance should become an implementation detail and any method that is not explicitly overridden should raise an exception. 3. Add a mechanism to keep numpy and numpy.ma APIs in sync. At a minimum - add a test comparing public functions and methods and for pure python functions compare signatures. 4. Consider deprecating the ma.masked scalar. 5. Support duck-typing in MaskedArray constructors. If supplied data object has mask attribute it should be used as mask. This will allow interoperability with alternative missing values implementations. (ndarray may itself grow mask attribute one day which will be equivalent to isnan. Bit views, anyone?) ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -- please visit: http://www.globalcarbonatlas.org/ *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] percentile function for masked array?
Dear all, It seems that there is not a percentile function for masked array in numpy or scipy? I checked numpy.percentile and scipy.percentile, it seems not support only nonmasked array? And there is no percentile function in scipy.stats.mstats, so I have to use np.percentile(arr.compressed()) I guess. Thanks for any comments. Best, Chao -- please visit: http://www.globalcarbonatlas.org/ *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] simple way to denote unchanged dimension in reshape?
Dear all, I have a simple question. Is there a way to denote the unchanged dimension in the reshape function? like suppose I have an array named arr having three dims with the first dimension length as 48, I want to reshape the first dim into 12*4, but keeping all the other dimension length unchanged. like when we slice the array, we can use: arr[10:40, ... ], ...' represents all remaining dimesions. however when doing reshape, we must use: arr.reshape(12,-1,arr.shape(1),arr.shape(2)) Is there something allowing more flexible reshape, like: arr.reshape(12,-1,...)? thanks a lot in advance, best, Chao -- please visit: http://www.globalcarbonatlas.org/ *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] simple way to denote unchanged dimension in reshape?
Oh, I didn't think it out. thanks. Chao On Thu, May 29, 2014 at 11:59 AM, Dave Hirschfeld dave.hirschf...@gmail.com wrote: Chao YUE chaoyuejoy at gmail.com writes: Dear all, I have a simple question. Is there a way to denote the unchanged dimension in the reshape function? like suppose I have an array named arr having three dims with the first dimension length as 48, I want to reshape the first dim into 12*4, but keeping all the other dimension length unchanged. like when we slice the array, we can use: arr[10:40, ... ], ...' represents all remaining dimesions. however when doing reshape, we must use: arr.reshape(12,-1,arr.shape(1),arr.shape(2)) Is there something allowing more flexible reshape, like: arr.reshape(12,-1,...)? thanks a lot in advance,best, Chao For the example given the below code works: In [1]: x = randn(48,5,4,3,2) In [2]: x.reshape(12,-1,*x.shape[1:]).shape Out[2]: (12L, 4L, 5L, 4L, 3L, 2L) HTH, Dave ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -- please visit: http://www.globalcarbonatlas.org/ *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] np.ma.argmax not respecting the mask?
Dear all, I am using 1.7.1 version of numpy and np.ma.argmax is not repecting the mask? I expect for all data that are masked, it should also return a mask, but this is not the case. In [96]: d3 Out[96]: masked_array(data = [[-- -- -- -- 4] [5 -- 7 8 9]], mask = [[ True True True True False] [False True False False False]], fill_value = 6) In [97]: np.ma.argmax(d3,axis=0) Out[97]: array([1, 0, 1, 1, 1]) In [98]: np.__version__ Out[98]: '1.7.1' Can I file a bug report on this? thanks, Chao -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] np.ma.argmax not respecting the mask?
Sorry I didn't the docs very carefully. there is no doc for np.ma.argmax for indeed there is for np.ma.argmin so it's an expected behavior rather than a bug. Let some heavy users to say their ideas. Practicaly, the returned value of 0 will be always confused with the values which are not masked but do have the minimum or maximum values at the 0 position over the specified axis. One way to walk around is: data_mask = np.ma.mean(axis=0).mask np.ma.masked_array(np.ma.argmax(data,axis=0), mask=data_mask) Chao On Tue, Jul 9, 2013 at 4:26 PM, Pierre Gerard-Marchant pgmdevl...@gmail.com wrote: On Jul 9, 2013, at 16:08 , Sebastian Berg sebast...@sipsolutions.net wrote: On Tue, 2013-07-09 at 15:14 +0200, Stéfan van der Walt wrote: On Tue, Jul 9, 2013 at 2:55 PM, Chao YUE chaoyue...@gmail.com wrote: I am using 1.7.1 version of numpy and np.ma.argmax is not repecting the mask? In [96]: d3 Out[96]: masked_array(data = [[-- -- -- -- 4] [5 -- 7 8 9]], mask = [[ True True True True False] [False True False False False]], fill_value = 6) In [97]: np.ma.argmax(d3,axis=0) Out[97]: array([1, 0, 1, 1, 1]) This is the result I would expect. If both values are masked, the fill value is used, so there is always an argmin value. To be honest, I would expect the exact opposite. If there is no value, there is no minimum argument - either its an error, or it signals invalid in some other way. On masked arrays I would expect it to be masked to signal this. The doc is quite clear: masked values are replaced by `fill_value` when determining the argmax/argmin. Attaching a mask a posteriori is always doable, but making the output of np.ma.argstuff a MaskedArray may be a nuisance at this point (any input from heavy users?). ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] np.ma.argmax not respecting the mask?
Thanks Pierre, good to know there are so many tricks available. Chao On Tue, Jul 9, 2013 at 4:55 PM, Pierre Gerard-Marchant pgmdevl...@gmail.com wrote: On Jul 9, 2013, at 16:38 , Chao YUE chaoyue...@gmail.com wrote: Sorry I didn't the docs very carefully. there is no doc for np.ma.argmax for indeed there is for np.ma.argmin Yeah, the doc of the function asks you to go check the doc of the method… Not the best. so it's an expected behavior rather than a bug. Let some heavy users to say their ideas. Practicaly, the returned value of 0 will be always confused with the values which are not masked but do have the minimum or maximum values at the 0 position over the specified axis. Well, it's just an index: if you take the corresponding value from the input array, it'll be masked... One way to walk around is: data_mask = np.ma.mean(axis=0).mask np.ma.masked_array(np.ma.argmax(data,axis=0), mask=data_mask) I find easier to use `mask=x.mask.prod(axis)` to get the combined mask along the desired axis (you could also use a `reduce(np.logical_and, x.mask)` for axis=0, but it's less convenient I think). ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] easy way to collapse the last two dimension of an array?
Dear all, I have an array with 4 dim: In [24]: dd.shape Out[24]: (12, 13, 120, 170) I would like to collapse the last two dim for applying np.sum(axis=-1) In [25]: dd.reshape(12,13,-1).shape Out[25]: (12, 13, 20400) is there a more general way to do this? something like In [21]: dd.reshape(*dd.shape[0:2],-1).shape File ipython console, line 1 SyntaxError: only named arguments may follow *expression (ipython console, line 1) thanks a lot for any hints, cheers, Chao -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] easy way to collapse the last two dimension of an array?
Thanks to the two of you!! Chao On Mon, Jun 3, 2013 at 11:53 AM, Sebastian Berg sebast...@sipsolutions.netwrote: On Mon, 2013-06-03 at 10:44 +0200, Chao YUE wrote: Dear all, I have an array with 4 dim: In [24]: dd.shape Out[24]: (12, 13, 120, 170) I would like to collapse the last two dim for applying np.sum(axis=-1) If you use Numpy = 1.7. you can also just use dd.sum(axis=(-1,-2)) - Sebastian In [25]: dd.reshape(12,13,-1).shape Out[25]: (12, 13, 20400) is there a more general way to do this? something like In [21]: dd.reshape(*dd.shape[0:2],-1).shape File ipython console, line 1 SyntaxError: only named arguments may follow *expression (ipython console, line 1) thanks a lot for any hints, cheers, Chao -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] howto reduce along arbitrary axis
Hi Neal, I forward you this mail which I think might be of help to your question. Chao -- Forwarded message -- From: Chao YUE chaoyue...@gmail.com Date: Sat, Mar 16, 2013 at 5:40 PM Subject: indexing of arbitrary axis and arbitrary slice? To: Discussion of Numerical Python numpy-discussion@scipy.org Dear all, Is there some way to index the numpy array by specifying arbitrary axis and arbitrary slice, while not knowing the actual shape of the data? For example, I have a 3-dim data, data.shape = (3,4,5) Is there a way to retrieve data[:,0,:] by using something like np.retrieve_data(data,axis=2,slice=0), by this way you don't have to know the actual shape of the array. for for 4-dim data, np.retrieve_data(data,axis=2,slice=0) will actually be data[:,0,:,:] thanks in advance, Chao -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] howto reduce along arbitrary axis
Oh sorry, my fault... here is the answer by Nathaniel Smith: def retrieve_data(a, ax, idx): full_idx = [slice(None)] * a.ndim full_idx[ax] = idx return a[tuple(full_idx)] Or for the specific case where you do know the axis in advance, you just don't know how many trailing axes there are, use a[:, :, 0, ...] and the ... will expand to represent the appropriate number of :'s. probably you can sue something simlaer. Chao On Tue, Mar 26, 2013 at 3:33 PM, Neal Becker ndbeck...@gmail.com wrote: Thank you, but do you also have an answer to this question? I only see the question. On Tue, Mar 26, 2013 at 10:23 AM, Chao YUE chaoyue...@gmail.com wrote: Hi Neal, I forward you this mail which I think might be of help to your question. Chao -- Forwarded message -- From: Chao YUE chaoyue...@gmail.com Date: Sat, Mar 16, 2013 at 5:40 PM Subject: indexing of arbitrary axis and arbitrary slice? To: Discussion of Numerical Python numpy-discussion@scipy.org Dear all, Is there some way to index the numpy array by specifying arbitrary axis and arbitrary slice, while not knowing the actual shape of the data? For example, I have a 3-dim data, data.shape = (3,4,5) Is there a way to retrieve data[:,0,:] by using something like np.retrieve_data(data,axis=2,slice=0), by this way you don't have to know the actual shape of the array. for for 4-dim data, np.retrieve_data(data,axis=2,slice=0) will actually be data[:,0,:,:] thanks in advance, Chao -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] indexing of arbitrary axis and arbitrary slice?
Hi Nathaniel, thanks for your reply, it works fine and suffice for my purpose. cheers, Chao On Sat, Mar 16, 2013 at 5:49 PM, Nathaniel Smith n...@pobox.com wrote: On 16 Mar 2013 16:41, Chao YUE chaoyue...@gmail.com wrote: Dear all, Is there some way to index the numpy array by specifying arbitrary axis and arbitrary slice, while not knowing the actual shape of the data? For example, I have a 3-dim data, data.shape = (3,4,5) Is there a way to retrieve data[:,0,:] by using something like np.retrieve_data(data,axis=2,slice=0), by this way you don't have to know the actual shape of the array. for for 4-dim data, np.retrieve_data(data,axis=2,slice=0) will actually be data[:,0,:,:] I don't know of anything quite like that, but it's easy to fake it: def retrieve_data(a, ax, idx): full_idx = [slice(None)] * a.ndim full_idx[ax] = idx return a[tuple(full_idx)] Or for the specific case where you do know the axis in advance, you just don't know how many trailing axes there are, use a[:, :, 0, ...] and the ... will expand to represent the appropriate number of :'s. -n ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] indexing of arbitrary axis and arbitrary slice?
Dear all, Is there some way to index the numpy array by specifying arbitrary axis and arbitrary slice, while not knowing the actual shape of the data? For example, I have a 3-dim data, data.shape = (3,4,5) Is there a way to retrieve data[:,0,:] by using something like np.retrieve_data(data,axis=2,slice=0), by this way you don't have to know the actual shape of the array. for for 4-dim data, np.retrieve_data(data,axis=2,slice=0) will actually be data[:,0,:,:] thanks in advance, Chao -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] return index of maximum value in an array easily?
Dear all, Are we going to consider returning the index of maximum value in an array easily without calling np.argmax and np.unravel_index consecutively? I saw few posts in mailing archive and stackover flow on this, when I tried to return the index of maximum value of 2d array. It seems that I am not the first to be confused by this. http://stackoverflow.com/questions/11377028/getting-index-of-numpy-ndarray http://old.nabble.com/maximum-value-and-corresponding-index-td24834930.html http://stackoverflow.com/questions/5469286/how-to-get-the-index-of-a-maximum-element-in-a-numpy-array http://stackoverflow.com/questions/4150542/determine-index-of-highest-value-in-pythons-numpy cheers, Chao -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] numpy.testing.asserts and masked array
Thanks. I tried again, it works. On Thu, Dec 27, 2012 at 10:35 PM, Ralf Gommers ralf.gomm...@gmail.comwrote: from numpy.ma import testutils -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] numpy.testing.asserts and masked array
Dear all, I found here http://mail.scipy.org/pipermail/numpy-discussion/2009-January/039681.html that to use* numpy.ma.testutils.assert_almost_equal* for masked array assertion, but I cannot find the np.ma.testutils module? Am I getting somewhere wrong? my numpy version is 1.6.2 thanks! Chao -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] an easy way to know if a functions works or not for masked array?
Dear numpy users, I think since long I am confused by if a function works as expected or not for masked array. like np.reshape works for masked array, but not np.sum (I mean, I expect np.sum to drop the masked elements when do summing, of course we have np.ma.sum). So I always use fuctions preceding by np.ma to make sure there is nothing going woring if I expect there will be masked array participating in the calculation in the data process chain. When I handle masked array, Before I use a function, normally I check if there is an equivalent np.ma function and if there is, I use np.mafunction; Then I check how the documentation says about np.func and np.ma.func and see there is anything mentioned explicitly on the handling of masked array. Howevery, In most cases I will try with simple arrays to test both np.ma.func and np.func before I use some function. but this is sometimes time consuming. Does anyone have similar situation? thanks! Chao -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] np.seterr doesn't work for masked array?
Thanks. You mean actually when numpy handle masked array, it will first treat all the base data, and then apply the mask after the treatment. and normally the base data of maksed elements will very likely to intrigure these errors, and you will see a lot errory warning or print in the process, and will make it impossible to see the error information your want to see for those elements that are not masked but still can intriguer the error you would like to see or check. I didn't realize this. It's a good to overwrite the error setting. thanks for your explanation. Chao On Fri, Dec 14, 2012 at 3:15 PM, Robert Kern robert.k...@gmail.com wrote: On Fri, Dec 14, 2012 at 1:57 PM, Chao YUE chaoyue...@gmail.com wrote: Dear all, I tried to capture the zero divide error when I divide a masked array by another. It seems that np.seterr is not working for masked array? when I do np.divide on two masked array, it directly put the zero divides part as being masked. The np.seterr works if the two arrays for dividing are not masked arrays. could anyone explain? thanks!! numpy.ma uses np.seterr(divide='ignore', invalid='ignore') for most of its operations, so it is overriding your settings. This is usually desirable since many of the masked values will be trip these errors spuriously even though they will be masked out in the result. -- Robert Kern ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] pandas dataframe memory layout
I don't know. Maybe you can ask here: http://stackoverflow.com/questions/tagged/pandas chao On Sat, Dec 8, 2012 at 12:20 PM, Daniel Wu staywith...@gmail.com wrote: For numpy array, we can choose to use either C style or Forran stype. For dataframe in Pandas, is it possible to choose memory layout as in numpy array? ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] the difference between + and np.add?
Dear all, if I have two ndarray arr1 and arr2 (with the same shape), is there some difference when I do: arr = arr1 + arr2 and arr = np.add(arr1, arr2), and then if I have more than 2 arrays: arr1, arr2, arr3, arr4, arr5, then I cannot use np.add anymore as it only recieves 2 arguments. then what's the best practice to add these arrays? should I do arr = arr1 + arr2 + arr3 + arr4 + arr5 or I do arr = np.sum(np.array([arr1, arr2, arr3, arr4, arr5]), axis=0)? because I just noticed recently that there are functions like np.add, np.divide, np.substract... before I am using all like directly arr1/arr2, rather than np.divide(arr1,arr2). best regards, Chao -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] the difference between + and np.add?
Thanks for the explanations. Yes, what I am thinking is basically the same
but I didn't test the time.
I never try numexpr, but it would be nice to try it.
Chao
On Thu, Nov 22, 2012 at 3:20 PM, Francesc Alted franc...@continuum.iowrote:
On 11/22/12 1:41 PM, Chao YUE wrote:
Dear all,
if I have two ndarray arr1 and arr2 (with the same shape), is there
some difference when I do:
arr = arr1 + arr2
and
arr = np.add(arr1, arr2),
and then if I have more than 2 arrays: arr1, arr2, arr3, arr4, arr5,
then I cannot use np.add anymore as it only recieves 2 arguments.
then what's the best practice to add these arrays? should I do
arr = arr1 + arr2 + arr3 + arr4 + arr5
or I do
arr = np.sum(np.array([arr1, arr2, arr3, arr4, arr5]), axis=0)?
because I just noticed recently that there are functions like np.add,
np.divide, np.substract... before I am using all like directly
arr1/arr2, rather than np.divide(arr1,arr2).
As Nathaniel said, there is not a difference in terms of *what* is
computed. However, the methods that you suggested actually differ on
*how* they are computed, and that has dramatic effects on the time
used. For example:
In []: arr1, arr2, arr3, arr4, arr5 = [np.arange(1e7) for x in range(5)]
In []: %time arr1 + arr2 + arr3 + arr4 + arr5
CPU times: user 0.05 s, sys: 0.10 s, total: 0.14 s
Wall time: 0.15 s
Out[]:
array([ 0.e+00, 5.e+00, 1.e+01, ...,
4.9850e+07, 4.9900e+07, 4.9950e+07])
In []: %time np.sum(np.array([arr1, arr2, arr3, arr4, arr5]), axis=0)
CPU times: user 2.98 s, sys: 0.15 s, total: 3.13 s
Wall time: 3.14 s
Out[]:
array([ 0.e+00, 5.e+00, 1.e+01, ...,
4.9850e+07, 4.9900e+07, 4.9950e+07])
The difference is how memory is used. In the first case, the additional
memory was just a temporary with the size of the operands, while for the
second case a big temporary has to be created, so the difference in is
speed is pretty large.
There are also ways to minimize the size of temporaries, and numexpr is
one of the simplests:
In []: import numexpr as ne
In []: %time ne.evaluate('arr1 + arr2 + arr3 + arr4 + arr5')
CPU times: user 0.04 s, sys: 0.04 s, total: 0.08 s
Wall time: 0.04 s
Out[]:
array([ 0.e+00, 5.e+00, 1.e+01, ...,
4.9850e+07, 4.9900e+07, 4.9950e+07])
Again, the computations are the same, but how you manage memory is
critical.
--
Francesc Alted
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
--
***
Chao YUE
Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL)
UMR 1572 CEA-CNRS-UVSQ
Batiment 712 - Pe 119
91191 GIF Sur YVETTE Cedex
Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] the fast way to loop over ndarray elements?
Dear Thouis, I take some time to check, before I tried with cython, I tried the np.interp first, and very luckily, it's exeactly what I need. And with the old written by me, it spend 20 seconds, now it's 0.2 seconds! Thanks a lot to all you guys. Chao On Mon, Nov 19, 2012 at 3:08 PM, Thouis (Ray) Jones tho...@gmail.comwrote: On Sat, Nov 17, 2012 at 8:28 AM, Chao YUE chaoyue...@gmail.com wrote: Dear all, I need to make a linear contrast of the 2D numpy array data from an interval to another, the approach is: I have another two list: base target, then I check for each ndarray element data[i,j], if base[m] = data[i,j] = base[m+1], then it will be linearly converted to be in the interval of (target[m], target[m+1]), using another function called lintrans. #The way I do is to loop each row and column of the 2D array, and finally loop the intervals constituted by base list: for row in range(data.shape[0]): for col in range(data.shape[1]): for i in range(len(base)-1): if data[row,col]=base[i] and data[row,col]=base[i+1]: data[row,col]=lintrans(data[row,col],(base[i],base[i+1]),(target[i],target[i+1])) break #use break to jump out of loop as the data have to be ONLY transferred ONCE. Now the profiling result shows that most of the time has been used in this loop over the array (plot_array_transg), and less time in calling the linear transformation fuction lintrans: ncalls tottime percallcumtimepercall filename:lineno(function) 180470.1100.000 0.1100.000 mathex.py:132(lintrans) 112.495 12.495 19.061 19.061 mathex.py:196(plot_array_transg) so is there anyway I can speed up this loop? Thanks for any suggestions!! best, Chao If lintrans() is a linear interpolation, could you use interp? http://docs.scipy.org/doc/numpy/reference/generated/numpy.interp.html ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] the fast way to loop over ndarray elements?
Dear all, I need to make a linear contrast of the 2D numpy array data from an interval to another, the approach is: I have another two list: base target, then I check for each ndarray element data[i,j], if base[m] = data[i,j] = base[m+1], then it will be linearly converted to be in the interval of (target[m], target[m+1]), using another function called lintrans. #The way I do is to loop each row and column of the 2D array, and finally loop the intervals constituted by base list: for row in range(data.shape[0]): for col in range(data.shape[1]): for i in range(len(base)-1): if data[row,col]=base[i] and data[row,col]=base[i+1]: data[row,col]=lintrans(data[row,col],(base[i],base[i+1]),(target[i],target[i+1])) break #use break to jump out of loop as the data have to be ONLY transferred ONCE. Now the profiling result shows that most of the time has been used in this loop over the array (plot_array_transg), and less time in calling the linear transformation fuction lintrans: ncalls tottime percallcumtimepercall filename:lineno(function) 180470.1100.000 0.1100.000 mathex.py:132(lintrans) 112.495 12.495 19.061 19.061 mathex.py:196(plot_array_transg) so is there anyway I can speed up this loop? Thanks for any suggestions!! best, Chao -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] the fast way to loop over ndarray elements?
Yes, both the base and target are ascending. Thanks! Chao On Sat, Nov 17, 2012 at 2:40 PM, Benjamin Root ben.r...@ou.edu wrote: On Saturday, November 17, 2012, Chao YUE wrote: Dear all, I need to make a linear contrast of the 2D numpy array data from an interval to another, the approach is: I have another two list: base target, then I check for each ndarray element data[i,j], if base[m] = data[i,j] = base[m+1], then it will be linearly converted to be in the interval of (target[m], target[m+1]), using another function called lintrans. #The way I do is to loop each row and column of the 2D array, and finally loop the intervals constituted by base list: for row in range(data.shape[0]): for col in range(data.shape[1]): for i in range(len(base)-1): if data[row,col]=base[i] and data[row,col]=base[i+1]: data[row,col]=lintrans(data[row,col],(base[i],base[i+1]),(target[i],target[i+1])) break #use break to jump out of loop as the data have to be ONLY transferred ONCE. Now the profiling result shows that most of the time has been used in this loop over the array (plot_array_transg), and less time in calling the linear transformation fuction lintrans: ncalls tottime percallcumtimepercall filename:lineno(function) 180470.1100.000 0.1100.000 mathex.py:132(lintrans) 112.495 12.495 19.061 19.061 mathex.py:196(plot_array_transg) so is there anyway I can speed up this loop? Thanks for any suggestions!! best, Chao If the values in base are ascending, you can use searchsorted() to find out where values from data can be placed into base while maintaining order. Don't know if it is faster, but it would certainly be easier to read. Cheers! Ben Root ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] easy way to change part of only unmasked elements value?
but I think I personally prefer the reverse. I would expect when I do a[3:6]=1 the mask state would not change. then I want to change the base, I would use a.base[3:6]=1 then the mask state would change also. By the way, I found b.data always be equal to b.base? cheers, Chao On Tue, Sep 11, 2012 at 5:24 PM, Chao YUE chaoyue...@gmail.com wrote: Dear Richard, this is what I want. Thanks! Chao On Tue, Sep 11, 2012 at 3:19 PM, Richard Hattersley rhatters...@gmail.com wrote: Hi Chao, If you don't mind modifying masked values, then if you write to the underlying ndarray it won't touch the mask: a = np.ma.masked_less(np.arange(10),5) a.base[3:6] = 1 a masked_array(data = [-- -- -- -- -- 1 6 7 8 9], mask = [ True True True True True False False False False False], fill_value = 99) Regards, Richard Hattersley On 10 September 2012 17:43, Chao YUE chaoyue...@gmail.com wrote: Dear all numpy users, what's the easy way if I just want to change part of the unmasked array elements into another new value? like an example below: in my real case, I would like to change a subgrid of a masked numpy array to another value, but this grid include both masked and unmasked data. If I do a simple array[index1:index2, index3:index4] = another_value, those data with original True mask will change into False. I am using numpy 1.6.2. Thanks for any ideas. In [91]: a = np.ma.masked_less(np.arange(10),5) In [92]: or_mask = a.mask.copy() In [93]: a Out[93]: masked_array(data = [-- -- -- -- -- 5 6 7 8 9], mask = [ True True True True True False False False False False], fill_value = 99) In [94]: a[3:6]=1 In [95]: a Out[95]: masked_array(data = [-- -- -- 1 1 1 6 7 8 9], mask = [ True True True False False False False False False False], fill_value = 99) In [96]: a = np.ma.masked_array(a,mask=or_mask) In [97]: a Out[97]: masked_array(data = [-- -- -- -- -- 1 6 7 8 9], mask = [ True True True True True False False False False False], fill_value = 99) Chao -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] easy way to change part of only unmasked elements value?
Dear Richard, this is what I want. Thanks! Chao On Tue, Sep 11, 2012 at 3:19 PM, Richard Hattersley rhatters...@gmail.comwrote: Hi Chao, If you don't mind modifying masked values, then if you write to the underlying ndarray it won't touch the mask: a = np.ma.masked_less(np.arange(10),5) a.base[3:6] = 1 a masked_array(data = [-- -- -- -- -- 1 6 7 8 9], mask = [ True True True True True False False False False False], fill_value = 99) Regards, Richard Hattersley On 10 September 2012 17:43, Chao YUE chaoyue...@gmail.com wrote: Dear all numpy users, what's the easy way if I just want to change part of the unmasked array elements into another new value? like an example below: in my real case, I would like to change a subgrid of a masked numpy array to another value, but this grid include both masked and unmasked data. If I do a simple array[index1:index2, index3:index4] = another_value, those data with original True mask will change into False. I am using numpy 1.6.2. Thanks for any ideas. In [91]: a = np.ma.masked_less(np.arange(10),5) In [92]: or_mask = a.mask.copy() In [93]: a Out[93]: masked_array(data = [-- -- -- -- -- 5 6 7 8 9], mask = [ True True True True True False False False False False], fill_value = 99) In [94]: a[3:6]=1 In [95]: a Out[95]: masked_array(data = [-- -- -- 1 1 1 6 7 8 9], mask = [ True True True False False False False False False False], fill_value = 99) In [96]: a = np.ma.masked_array(a,mask=or_mask) In [97]: a Out[97]: masked_array(data = [-- -- -- -- -- 1 6 7 8 9], mask = [ True True True True True False False False False False], fill_value = 99) Chao -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] easy way to change part of only unmasked elements value?
Dear all numpy users, what's the easy way if I just want to change part of the unmasked array elements into another new value? like an example below: in my real case, I would like to change a subgrid of a masked numpy array to another value, but this grid include both masked and unmasked data. If I do a simple array[index1:index2, index3:index4] = another_value, those data with original True mask will change into False. I am using numpy 1.6.2. Thanks for any ideas. In [91]: a = np.ma.masked_less(np.arange(10),5) In [92]: or_mask = a.mask.copy() In [93]: a Out[93]: masked_array(data = [-- -- -- -- -- 5 6 7 8 9], mask = [ True True True True True False False False False False], fill_value = 99) In [94]: a[3:6]=1 In [95]: a Out[95]: masked_array(data = [-- -- -- 1 1 1 6 7 8 9], mask = [ True True True False False False False False False False], fill_value = 99) In [96]: a = np.ma.masked_array(a,mask=or_mask) In [97]: a Out[97]: masked_array(data = [-- -- -- -- -- 1 6 7 8 9], mask = [ True True True True True False False False False False], fill_value = 99) Chao -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] encounter many warnings while it's installing scipy
I don't think so. Maybe you can just try some functions you're familiar to see if they work as expected. Chao On Sun, Sep 2, 2012 at 3:59 PM, 心如烛光 275438...@qq.com wrote: Hi,everybody. I encounter many warnings while it's installing scipy with the commend:pip install scipy Such as warning:XX variable is uninitialized before used [-Wuninitialized] What makes these warnings? And will they make something wrong with scipy?How can I fix it ?? Thank you so much for your help! ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] use slicing as argument values?
Thanks Daniele. I am writing a small plotting function that can receive the index range as argument value. like I have variables var1, var2, var3, var4, var5 which have exactly the same dimensions. def plot_eg(index_range): #here I need the function above which can use the index_range to retrieve data from variables plot(func(var1,index_range))) plot(func(var2,index_range)) plot(func(var3,index_range)) plot(func(var4,index_range)) plot(func(var5,index_range)) actually I can also put the [var1,var2,var3,var4,var5] as arguments in the plot_eg function so that I can pick any variables I want to plot as long as they have the same dimension. otherwise, I have to change the index_range for every variable. cheers, Chao 2012/7/13 Daniele Nicolodi dani...@grinta.net On 12/07/2012 23:32, Chao YUE wrote: Thanks all for the discussion. Actually I am trying to use something like numpy ndarray indexing in the function. Like when I call: func(a,'1:3,:,2:4'), it knows I want to retrieve a[1:3,:,2:4], and func(a,'1:3,:,4') for a[1:3,:,4] ect. I am very close now. I don't see the advantage of this approach over directly using the sliced array as an argument of your function, as in func(a[1:3,:,4]). Can you elaborate more why you are going through this route? Cheers, Daniele ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] use slicing as argument values?
Thanks Robert. This is exactly what I want. I have a feeling that there must be something in numpy that can do the job and I didn't know. Thanks again, Chao 2012/7/13 Robert Kern robert.k...@gmail.com On Thu, Jul 12, 2012 at 10:32 PM, Chao YUE chaoyue...@gmail.com wrote: Thanks all for the discussion. Actually I am trying to use something like numpy ndarray indexing in the function. Like when I call: func(a,'1:3,:,2:4'), it knows I want to retrieve a[1:3,:,2:4], and func(a,'1:3,:,4') for a[1:3,:,4] ect. I am very close now. [~] |1 from numpy import index_exp [~] |2 index_exp[1:3,:,2:4] (slice(1, 3, None), slice(None, None, None), slice(2, 4, None)) -- Robert Kern ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] use slicing as argument values?
Dear all, I want to create a function and I would like one of the arguments of the function to determine what slicing of numpy array I want to use. a simple example: a=np.arange(100).reshape(10,10) suppose I want to have a imaging function to show image of part of this data: def show_part_of_data(m,n): plt.imshow(a[m,n]) like I can give m=3:5, n=2:7, when I call function show_part_of_data(3:5,2:7), this means I try to do plt.imshow(a[3:5,2:7]). the above example doesn't work in reality. but it illustrates something similar that I desire, that is, I can specify what slicing of number array I want by giving values to function arguments. thanks a lot, Chao -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] use slicing as argument values?
Hi Ben, it helps a lot. I am nearly finishing a function in a way I think pythonic. Just one more question, I have: In [24]: b=np.arange(1,11) In [25]: b Out[25]: array([ 1, 2, 3, 4, 5, 6, 7, 8, 9, 10]) In [26]: b[slice(1)] Out[26]: array([1]) In [27]: b[slice(4)] Out[27]: array([1, 2, 3, 4]) In [28]: b[slice(None,4)] Out[28]: array([1, 2, 3, 4]) so slice(4) is actually slice(None,4), how can I exactly want retrieve a[4] using slice object? thanks again! Chao 2012/7/12 Benjamin Root ben.r...@ou.edu On Thu, Jul 12, 2012 at 3:38 PM, Chao YUE chaoyue...@gmail.com wrote: Dear all, I want to create a function and I would like one of the arguments of the function to determine what slicing of numpy array I want to use. a simple example: a=np.arange(100).reshape(10,10) suppose I want to have a imaging function to show image of part of this data: def show_part_of_data(m,n): plt.imshow(a[m,n]) like I can give m=3:5, n=2:7, when I call function show_part_of_data(3:5,2:7), this means I try to do plt.imshow(a[3:5,2:7]). the above example doesn't work in reality. but it illustrates something similar that I desire, that is, I can specify what slicing of number array I want by giving values to function arguments. thanks a lot, Chao What you want to do is create slice objects. a[3:5] is equivalent to sl = slice(3, 5) a[sl] and a[3:5, 5:14] is equivalent to sl = (slice(3, 5), slice(5, 14)) a[sl] Furthermore, notation such as ::-1 is equivalent to slice(None, None, -1) I hope this helps! Ben Root ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] use slicing as argument values?
Thanks all for the discussion. Actually I am trying to use something like
numpy ndarray indexing in the function. Like when I call:
func(a,'1:3,:,2:4'), it knows I want to retrieve a[1:3,:,2:4], and
func(a,'1:3,:,4') for a[1:3,:,4] ect.
I am very close now.
#so this function changes the string to list of slice objects.
def convert_string_to_slice(slice_string):
provide slice_string as '2:3,:', it will return [slice(2, 3, None),
slice(None, None, None)]
slice_list=[]
split_slice_string_list=slice_string.split(',')
for sub_slice_string in split_slice_string_list:
split_sub=sub_slice_string.split(':')
if len(split_sub)==1:
sub_slice=slice(int(split_sub[0]))
else:
if split_sub[0]=='':
sub1=None
else:
sub1=int(split_sub[0])
if split_sub[1]=='':
sub2=None
else:
sub2=int(split_sub[1])
sub_slice=slice(sub1,sub2)
slice_list.append(sub_slice)
return slice_list
In [119]: a=np.arange(3*4*5).reshape(3,4,5)
for this it works fine.
In [120]: convert_string_to_slice('1:3,:,2:4')
Out[120]: [slice(1, 3, None), slice(None, None, None), slice(2, 4, None)]
In [121]: a[slice(1, 3, None), slice(None, None, None), slice(2, 4,
None)]==a[1:3,:,2:4]
Out[121]:
array([[[ True, True],
[ True, True],
[ True, True],
[ True, True]],
[[ True, True],
[ True, True],
[ True, True],
[ True, True]]], dtype=bool)
And problems happens when I want to retrieve a single number along a given
dimension:
because it treats 1:3,:,4 as 1:3,:,:4, as shown below:
In [122]: convert_string_to_slice('1:3,:,4')
Out[122]: [slice(1, 3, None), slice(None, None, None), slice(None, 4, None)]
In [123]: a[1:3,:,4]
Out[123]:
array([[24, 29, 34, 39],
[44, 49, 54, 59]])
In [124]: a[slice(1, 3, None), slice(None, None, None), slice(None, 4,
None)]
Out[124]:
array([[[20, 21, 22, 23],
[25, 26, 27, 28],
[30, 31, 32, 33],
[35, 36, 37, 38]],
[[40, 41, 42, 43],
[45, 46, 47, 48],
[50, 51, 52, 53],
[55, 56, 57, 58]]])
Then I have a function:
#this function retrieves data from ndarray a by specifying slice_string:
def retrieve_data(a,slice_string):
slice_list=convert_string_to_slice(slice_string)
return a[*slice_list]
In the list line of the fuction retrieve_data I have problem, I get an
invalid syntax error.
return a[*slice_list]
^
SyntaxError: invalid syntax
I hope it's not too long, please comment as you like. Thanks a lot
Chao
2012/7/12 Benjamin Root ben.r...@ou.edu
On Thu, Jul 12, 2012 at 4:46 PM, Chao YUE chaoyue...@gmail.com wrote:
Hi Ben,
it helps a lot. I am nearly finishing a function in a way I think
pythonic.
Just one more question, I have:
In [24]: b=np.arange(1,11)
In [25]: b
Out[25]: array([ 1, 2, 3, 4, 5, 6, 7, 8, 9, 10])
In [26]: b[slice(1)]
Out[26]: array([1])
In [27]: b[slice(4)]
Out[27]: array([1, 2, 3, 4])
In [28]: b[slice(None,4)]
Out[28]: array([1, 2, 3, 4])
so slice(4) is actually slice(None,4), how can I exactly want retrieve
a[4] using slice object?
thanks again!
Chao
Tricky question. Note the difference between
a[4]
and
a[4:5]
The first returns a scalar, while the second returns an array. The first,
though, is not a slice, just an integer.
Also, note that the arguments for slice() behaves very similar to the
arguments for range() (with some exceptions/differences).
Cheers!
Ben Root
2012/7/12 Benjamin Root ben.r...@ou.edu
On Thu, Jul 12, 2012 at 3:38 PM, Chao YUE chaoyue...@gmail.com wrote:
Dear all,
I want to create a function and I would like one of the arguments of
the function to determine what slicing of numpy array I want to use.
a simple example:
a=np.arange(100).reshape(10,10)
suppose I want to have a imaging function to show image of part of this
data:
def show_part_of_data(m,n):
plt.imshow(a[m,n])
like I can give m=3:5, n=2:7, when I call function
show_part_of_data(3:5,2:7), this means I try to do plt.imshow(a[3:5,2:7]).
the above example doesn't work in reality. but it illustrates something
similar that I desire, that is, I can specify what slicing of
number array I want by giving values to function arguments.
thanks a lot,
Chao
What you want to do is create slice objects.
a[3:5]
is equivalent to
sl = slice(3, 5)
a[sl]
and
a[3:5, 5:14]
is equivalent to
sl = (slice(3, 5), slice(5, 14))
a[sl]
Furthermore, notation such as ::-1 is equivalent to slice(None, None,
-1)
I hope this helps!
Ben Root
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
--
***
Chao YUE
Re: [Numpy-discussion] indexes in an array where value is greater than 1?
for me, np.nonzero() and np.where() both work. It seems they have same function. chao 2012/5/27 Chris Withers ch...@simplistix.co.uk On 25/05/2012 16:21, Benjamin Root wrote: np.nonzero(arrrgh 1) Did you mean np.where(arrrgh 1)?for I didn't know you could use np.nonzero in the way your describe? Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] why Segmentation fault (core dumped)?
Dear Jeff, Thanks a lot for your reply. I think it might be related with the memory management on our sever. But anyway, as you suggested, I open an issue on netcdf4-python.google.co http://netcdf4-python.google.comm. you can find the data and script on ftp://ftp.cea.fr/incoming/y2k01/chaoyue/. The cal_cmi_big.py gave the core dumped error, in which I tried to operate with big files (~2G). The cal_cmi_small.py works fine, which handles data only ~25M (a subset of big files). I used a small function (which use NetCDF4) in the script to read and write nc data with NetCDF4. You can also find this function in ncfunc.py. I tested all the script and data before I upload on our ftp. thanks again for your help, cheers, Chao 2012/5/26 Jeff Whitaker jsw...@fastmail.fm On 5/26/12 5:51 AM, Chao YUE wrote: Dear all, Previously I am able to run a script on our server but now it gave me a Segmentation fault (core dumped) error. try I tried the script with same type of netcdf file but with much smaller file size and it works. So I think the error is related with memory stuff. I guess it's because our system administrator make some change somewhere and that cause my problem? the file size that cause the error to appear is 2.6G (in the script I read this file with NetCDF4 to numpy array and make some manipulation), the small one without error is only 48M. Chao: Without seeing your script, there's not much I can say. I suggest opening an issue at netcdf4-python.google.com, including your script as an attachment. You'll probably have to post the data file somewhere (dropbox perhaps?) so I can run the script that triggers the segfault. -Jeff -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] why Segmentation fault (core dumped)?
Just one more sentence. We are using version 0.9.7 on our server. when I
tried to use:
f=nc.Dataset('file.nc') to open the file (either big or small) within
ipython, it works fine. Then when I tried to do
%run cal_cmi_big.py, core dumped error. but %run cal_cmi_small.py works
fine. The cal_cmi_big.py did work several days ago, I still have the file
generated by this script in my directory.
cheers,
Chao
2012/5/27 Chao YUE chaoyue...@gmail.com
Dear Jeff,
Thanks a lot for your reply. I think it might be related with the memory
management on our sever. But anyway, as you suggested, I open an issue on
netcdf4-python.google.co http://netcdf4-python.google.comm. you can
find the data and script on ftp://ftp.cea.fr/incoming/y2k01/chaoyue/. The
cal_cmi_big.py gave the core dumped error, in which I tried to operate with
big files (~2G). The cal_cmi_small.py works fine, which handles data only
~25M (a subset of big files). I used a small function (which use NetCDF4)
in the script to read and write nc data with NetCDF4. You can also find
this function in ncfunc.py. I tested all the script and data before I
upload on our ftp.
thanks again for your help,
cheers,
Chao
2012/5/26 Jeff Whitaker jsw...@fastmail.fm
On 5/26/12 5:51 AM, Chao YUE wrote:
Dear all,
Previously I am able to run a script on our server but now it gave me a
Segmentation fault (core dumped) error.
try I tried the script with same type of netcdf file but with much
smaller file size and it works. So I think the error is related with memory
stuff.
I guess it's because our system administrator make some change somewhere
and that cause my problem?
the file size that cause the error to appear is 2.6G (in the script I
read this file with NetCDF4 to numpy array and make some manipulation),
the small one without error is only 48M.
Chao: Without seeing your script, there's not much I can say. I suggest
opening an issue at netcdf4-python.google.com, including your script as
an attachment. You'll probably have to post the data file somewhere
(dropbox perhaps?) so I can run the script that triggers the segfault.
-Jeff
--
***
Chao YUE
Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL)
UMR 1572 CEA-CNRS-UVSQ
Batiment 712 - Pe 119
91191 GIF Sur YVETTE Cedex
Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16
--
***
Chao YUE
Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL)
UMR 1572 CEA-CNRS-UVSQ
Batiment 712 - Pe 119
91191 GIF Sur YVETTE Cedex
Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] why Segmentation fault (core dumped)?
Dear all, Previously I am able to run a script on our server but now it gave me a Segmentation fault (core dumped) error. try I tried the script with same type of netcdf file but with much smaller file size and it works. So I think the error is related with memory stuff. I guess it's because our system administrator make some change somewhere and that cause my problem? the file size that cause the error to appear is 2.6G (in the script I read this file with NetCDF4 to numpy array and make some manipulation), the small one without error is only 48M. I tried to use limit command to list the following, then I change stacksize to unlimited and the problem still occurs. ychao@obelix2 - ...CRU_NEW - 12 limit cputime unlimited filesize unlimited datasize unlimited stacksize10240 kbytes coredumpsize 0 kbytes memoryuseunlimited vmemoryuse unlimited descriptors 1024 memorylocked 64 kbytes maxproc 1024 would anybody be able to give me a short explanation or direct me to some webpage which can help to understand the problem? thanks et cheers, Chao -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] command for retrieving unmasked data from a mask array?
Thanks Olivier. it works. chao 2012/5/23 Olivier Delalleau sh...@keba.be Should be dt3.compressed() -=- Olivier 2012/5/23 Chao YUE chaoyue...@gmail.com Dear all, is there a command for retrieving unmasked data from a mask array? excepting using dt3[~dt3.mask].flatten()? thanks, Chao -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] assign a float number to a member of integer array always return integer
Dear all, Just in case some one didn't know this. Assign a float number to an integer array element will always return integer. In [4]: a=np.arange(2,11,2) In [5]: a Out[5]: array([ 2, 4, 6, 8, 10]) In [6]: a[1]=4.5 In [7]: a Out[7]: array([ 2, 4, 6, 8, 10]) so I would always do this if I expected a transfer from integer to float? In [18]: b=a.astype(float) In [19]: b Out[19]: array([ 2., 4., 6., 8., 10.]) In [20]: b[1]=4.5 In [21]: b Out[21]: array([ 2. , 4.5, 6. , 8. , 10. ]) thanks et cheers, Chao -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] assign a float number to a member of integer array always return integer
Thanks Chris for informative post. cheers, Chao 2012/5/22 Chris Barker chris.bar...@noaa.gov On Tue, May 22, 2012 at 6:33 AM, Chao YUE chaoyue...@gmail.com wrote: Just in case some one didn't know this. Assign a float number to an integer array element will always return integer. right -- numpy arrays are typed -- that's one of the points of them -- you wouldn't want the entire array up-cast with a single assignment -- particularly since there are only python literals for a subset of the numpy types. so I would always do this if I expected a transfer from integer to float? In [18]: b=a.astype(float) yes -- but that's an odd way of thinking about -- what you want to do is think about what type you need your array to be before you create it, then create it the way you need it: In [87]: np.arange(5, dtype=np.float) Out[87]: array([ 0., 1., 2., 3., 4.]) or better: In [91]: np.linspace(0,5,6) Out[91]: array([ 0., 1., 2., 3., 4., 5.]) note that most (all?) numpy array constructors take a dtype argument. -Chris In [19]: b Out[19]: array([ 2., 4., 6., 8., 10.]) In [20]: b[1]=4.5 In [21]: b Out[21]: array([ 2. , 4.5, 6. , 8. , 10. ]) thanks et cheers, Chao -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/ORR(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] why not zerodivision error?
Dear all, could anybody give one sentence about this? why in the loop I didn't get zerodivision error by when I explicitly do this, I get a zerodivision error? thanks. In [7]: for i in np.arange(-10,10): print 1./i ...: -0.1 -0. -0.125 -0.142857142857 -0.1667 -0.2 -0.25 -0. -0.5 -1.0 inf 1.0 0.5 0. 0.25 0.2 0.1667 0.142857142857 0.125 0. In [8]: 1/0. --- ZeroDivisionError Traceback (most recent call last) /mnt/f/data/DROUGTH/ipython-input-8-7e0bf5b37da6 in module() 1 1/0. ZeroDivisionError: float division by zero In [9]: 1./0. --- ZeroDivisionError Traceback (most recent call last) /mnt/f/data/DROUGTH/ipython-input-9-3543596c47ff in module() 1 1./0. ZeroDivisionError: float division by zero In [10]: 1./0 --- ZeroDivisionError Traceback (most recent call last) /mnt/f/data/DROUGTH/ipython-input-10-523760448f92 in module() 1 1./0 ZeroDivisionError: float division by zero Chao -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] why not zerodivision error?
thanks for this information. Chao 2012/5/20 eat e.antero.ta...@gmail.com Hi, On Sun, May 20, 2012 at 10:21 AM, Chao YUE chaoyue...@gmail.com wrote: Dear all, could anybody give one sentence about this? why in the loop I didn't get zerodivision error by when I explicitly do this, I get a zerodivision error? thanks. In [7]: for i in np.arange(-10,10): print 1./i ...: -0.1 -0. -0.125 -0.142857142857 -0.1667 -0.2 -0.25 -0. -0.5 -1.0 inf 1.0 0.5 0. 0.25 0.2 0.1667 0.142857142857 0.125 0. In [8]: 1/0. --- ZeroDivisionError Traceback (most recent call last) /mnt/f/data/DROUGTH/ipython-input-8-7e0bf5b37da6 in module() 1 1/0. ZeroDivisionError: float division by zero In [9]: 1./0. --- ZeroDivisionError Traceback (most recent call last) /mnt/f/data/DROUGTH/ipython-input-9-3543596c47ff in module() 1 1./0. ZeroDivisionError: float division by zero In [10]: 1./0 --- ZeroDivisionError Traceback (most recent call last) /mnt/f/data/DROUGTH/ipython-input-10-523760448f92 in module() 1 1./0 ZeroDivisionError: float division by zero You may like to read more on here http://docs.scipy.org/doc/numpy/reference/generated/numpy.seterr.html#numpy.seterr So, for your specific example: In []: a= arange(-10, 10) In []: 1./ a Out[]: array([-0.1 , -0., -0.125 , -0.14285714, -0.1667, -0.2 , -0.25 , -0., -0.5 , -1., inf, 1., 0.5 , 0., 0.25 , 0.2 , 0.1667, 0.14285714, 0.125 , 0.]) In []: seterr(divide= 'raise') In []: 1./ a Traceback (most recent call last): File ipython console, line 1, in module FloatingPointError: divide by zero encountered in divide My 2 cents, -eat Chao -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] python import question
I forgot whether I installed numpy 1.5.1 by esay_install or manually. But anyway, I had the same issue with you that I cannot use pip uninstall numpy to remove 1.5.1. chao 2012/5/19 Tim Cera t...@cerazone.net On Fri, May 18, 2012 at 5:49 PM, Chao YUE chaoyue...@gmail.com wrote: Previously I have installed numpy 1.5.1. and then I used pip install --upgrade numpy to install numpy 1.6.1 Why was the old 1.5.1 installation in /usr/lib/pymodules/python2.7? I have in the past used 'pip uninstall package' a couple of times in a row in order to remove old versions of packages, then run 'pip install package'. Old packages and pip seem to be a problem if you have used 'easy_install' to install the older packages. Kindest regards, Tim ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] python import question
Dear all, This is only a small python import question. I think I'm right but just want some confirmation. Previously I have installed numpy 1.5.1. and then I used pip install --upgrade numpy to install numpy 1.6.1 But when I try to import numpy as np within ipython shell, I still get the version 1.5.1 then I checked my sys.path: In [21]: sys.path Out[21]: ['', '/usr/local/bin', '/usr/local/lib/python2.7/dist-packages/pupynere-1.0.15-py2.7.egg', '/usr/lib/pymodules/python2.7', '/usr/local/lib/python2.7/dist-packages/scikits.statsmodels-0.3.1-py2.7.egg', '/usr/local/lib/python2.7/dist-packages/Shapely-1.2.13-py2.7-linux-i686.egg', '/usr/local/lib/python2.7/dist-packages/pandas-0.7.3-py2.7-linux-i686.egg', '/home/chaoyue/python/python_lib', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-linux2', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/usr/local/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages/PIL', '/usr/lib/pymodules/python2.7/gtk-2.0', '/usr/lib/python2.7/dist-packages/gst-0.10', '/usr/lib/python2.7/dist-packages/gtk-2.0', '/usr/lib/pymodules/python2.7/ubuntuone-client', '/usr/lib/pymodules/python2.7/ubuntuone-control-panel', '/usr/lib/pymodules/python2.7/ubuntuone-storage-protocol', '/usr/lib/python2.7/dist-packages/wx-2.8-gtk2-unicode', '/usr/local/lib/python2.7/dist-packages/IPython/extensions'] Actually I found I have numpy 1.5.1 in /usr/lib/pymodules/python2.7 and numpy 1.6.1 in /usr/local/lib/python2.7/dist-packages/numpy/ but because the first path is before the second one in sys.path, so ipython imports only the first one and ignore the second one. Then I delete the directory of /usr/lib/pymodules/python2.7/numpy and redo the import, I get the version 1.6.1 This means that import will try to find the first occurrence of the module and will ignore the ones with same name in later occurrences? cheers, Chao -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] read line mixed with string and number?
Dear all, I have files which contain lines like this: 30516F5 Sep1985 1-Day RainTrace 0.23.2 Trace 0.0 0.00.00.00.20.0 Trace 29.20.00.00.0 0.01.8 30516F5 Sep1985 1-Day SnowTrace 0.00.00.0 14.8 10.1 Trace 0.00.00.00.00.00.00.00.0 Trace Trace 0.0 30516F5 Sep1985 1-Day Pcpn. Trace 0.23.2 Trace 18.9 9.8 Trace 0.00.20.0 Trace 29.20.00.00.0 Trace 1.80.0 30516F5 May1986 Max. Temp. Misg Misg Misg Misg Misg Misg 9.08.08.00.06.01.01.0 -3.03. 30516F5 May1986 Min. Temp. Misg Misg Misg Misg Misg Misg Misg -1.0 -2.0 -6.0 -5.0 -5.0 -3.0 -7.0 -6.0 -5.0 -3.0 different columns were separated by blank spaces. with the first column as sitename, second as month name, then year, then variable name and data. I want to read them line by line into a list, and then connect all the numerical data within one year into a list, and then combining different year data into one masked ndarray, in this process, I check the flags (Trace, Misg, etc.) and replace them as unique values (or missing values). and then begin to analyse the data. each file contains only one site, it can be big or small depending on the number of years. I don't know what's the good way to do this job. what I am thinking is to read one file line by line, and then divide this line by blank space, and replace special flag. but during this process, I need to do type conversion. any suggestion would be appreciated. Chao -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] change masked array member values with conditional selection
Dear all numpy users, I am using numpy 1.6.1 I find that if you want to change some member values in a masked array according to some conditional selection. suppose a is a masked array, you want to change all value below zero to zero. you must always use a[np.nonzero(a0)]=0 rather than a[a0]=0. the latter will lose all masked elements. an example: In [24]: a=np.arange(10.) In [25]: a=np.ma.masked_array(a,mask=a3) In [28]: a[a5]=2. In [29]: a Out[29]: masked_array(data = [2.0 2.0 2.0 2.0 2.0 5.0 6.0 7.0 8.0 9.0], mask = [False False False False False False False False False False], fill_value = 1e+20) In [30]: b=np.arange(10.) In [31]: b=np.ma.masked_array(b,mask=b3) In [34]: b[np.nonzero(b5)]=2. In [35]: b Out[35]: masked_array(data = [-- -- -- 2.0 2.0 5.0 6.0 7.0 8.0 9.0], mask = [ True True True False False False False False False False], fill_value = 1e+20) cheers, Chao -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] change masked array member values with conditional selection
Thanks Pierre. It's a good idea to always use a[(a5).filled(False)] = 2 I don't understand very well the underlying structure but it's good to know some. Chao 2012/4/12 Pierre GM pgmdevl...@gmail.com Ciao Chao, That known quirk deserves to be better documented, I agree. There's a simple explanation for this behavior: Because `a` is a masked array, `(a 5)` is also a masked array with dtype=np.bool, and whose mask is the same as `a`'s. In your example, that's: masked_array(data = [-- -- -- True True False False False False False], mask = [ True True True False False False False False False False], fill_value = True) Now, what should we do with the masked entries ? Should we consider them as False? As True? That's up to you, actually... Because it's never a good idea to use masked arrays as condition (as you just experienced), I advise you to be explicit. In your case, that'd be a[(a5).filled(False)] = 2 If you go in the source code of numpy.ma.core, in the __getitem__/__setitem__ methods, you'll find a little warning that I commented (because numpy.ma is already slow enough that I didn't want to make it even slower)... On 4/12/12, Chao YUE chaoyue...@gmail.com wrote: Dear all numpy users, I am using numpy 1.6.1 I find that if you want to change some member values in a masked array according to some conditional selection. suppose a is a masked array, you want to change all value below zero to zero. you must always use a[np.nonzero(a0)]=0 rather than a[a0]=0. the latter will lose all masked elements. an example: In [24]: a=np.arange(10.) In [25]: a=np.ma.masked_array(a,mask=a3) In [28]: a[a5]=2. In [29]: a Out[29]: masked_array(data = [2.0 2.0 2.0 2.0 2.0 5.0 6.0 7.0 8.0 9.0], mask = [False False False False False False False False False False], fill_value = 1e+20) In [30]: b=np.arange(10.) In [31]: b=np.ma.masked_array(b,mask=b3) In [34]: b[np.nonzero(b5)]=2. In [35]: b Out[35]: masked_array(data = [-- -- -- 2.0 2.0 5.0 6.0 7.0 8.0 9.0], mask = [ True True True False False False False False False False], fill_value = 1e+20) cheers, Chao -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] small bug in ndarray.flatten()?
Dear all, Is there a small bug in following? In [2]: b Out[2]: array([[ 0, 1, 2, 3, 4, 5], [ 6, 7, 8, 9, 10, 11], [12, 13, 14, 15, 16, 17], [18, 19, 20, 21, 22, 23]]) In [3]: b.flatten(order='C') --- TypeError Traceback (most recent call last) /mnt/f/ipython console in module() TypeError: flatten() takes no keyword arguments order='F' gave tha same. cheers, chao -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] small bug in ndarray.flatten()?
nice to know this. can also use b.transpose().flatten() to circumvent it.
thanks,
Chao
2012/4/5 Pierre Haessig pierre.haes...@crans.org
Hi,
Le 05/04/2012 15:00, Olivier Delalleau a écrit :
Ok, it looks weird indeed. I was using numpy 1.6.1 myself, not sure if
it's a bug that's been fixed in 1.6.
Try without the keyword argument (b.flatten('C')), see if at least
that works.
I can reproduce Chao's bug with my numpy 1.5.
As you've just suggested, it runs fine when there is no keyword.
In [1]: a= np.arange(10)
In [2]: a.flatten('C') # Works well
Out[2]: array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
In [3]: a.flatten(order='C') # Chao's bug
---
TypeError Traceback (most recent call last)
/home/pierre/ipython-input-3-33cfe1eff84b in module()
1 a.flatten(order='C')
TypeError: flatten() takes no keyword arguments
In [4]: a.flatten? # indeed says there is a keyword order='C'
In [5]: np.__version__
Out[5]: '1.5.1'
Now if the bug is fixed in 1.6, there's nothing more to do than just
wait for the update !
(Debian testing in my case)
Best,
Pierre
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
--
***
Chao YUE
Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL)
UMR 1572 CEA-CNRS-UVSQ
Batiment 712 - Pe 119
91191 GIF Sur YVETTE Cedex
Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] how to check type of array?
Dear all,
how can I check type of array in if condition expression?
In [75]: type(a)
Out[75]: type 'numpy.ndarray'
In [76]: a.dtype
Out[76]: dtype('int32')
a.dtype=='int32'?
thanks!
Chao
--
***
Chao YUE
Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL)
UMR 1572 CEA-CNRS-UVSQ
Batiment 712 - Pe 119
91191 GIF Sur YVETTE Cedex
Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] how to check type of array?
thanks to all. more than what I need. cheers, chao 2012/3/29 Derek Homeier de...@astro.physik.uni-goettingen.de On 29 Mar 2012, at 14:49, Robert Kern wrote: all work. For a more general check (e.g. if it is any type of integer), you can do np.issubclass_(a.dtype.type, np.integer) I don't recommend using that. Use np.issubdtype(a.dtype, np.integer) instead. Sorry, you're right, this works the same way - I had the impression from the documentation that tests like np.issubdtype(np.int16, np.integer) would not work, but they do. Cheers, Derek ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Trying to read 500M txt file using numpy.genfromtxt within ipython shell
Dear all, I received a file from others which contains ~30 million lines and in size of ~500M. I try read it with numpy.genfromtxt in ipython interactive mode. Then ipython crashed. The data contains lat,lon,var1,year, the year ranges from 1001 to 2006. Finally I want to write the data to netcdf for separate years and feed them into the model. I guess I need a better way to do this? anyone would be any idea is highly appreciated. lon,lat,year,area_burned -180.0,65.0,1001,0 -180.0,65.0,1002,0 -180.0,65.0,1003,0 -180.0,65.0,1004,0 -180.0,65.0,1005,0 -180.0,65.0,1006,0 -180.0,65.0,1007,0 thanks and cheers, Chao -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Trying to read 500M txt file using numpy.genfromtxt within ipython shell
I would be in agree. thanks! I use gawk to separate the file into many files by year, then it would be easier to handle. anyway, it's not a good practice to produce such huge line txt files Chao 2012/3/20 David Froger david.fro...@gmail.com Hi, I think writing a Python script that convert your txt file to one netcdf file, reading the txt file one line at a time, and then use the netcdf file normally would be a good solution! Best, David Excerpts from Chao YUE's message of mar. mars 20 13:33:56 +0100 2012: Dear all, I received a file from others which contains ~30 million lines and in size of ~500M. I try read it with numpy.genfromtxt in ipython interactive mode. Then ipython crashed. The data contains lat,lon,var1,year, the year ranges from 1001 to 2006. Finally I want to write the data to netcdf for separate years and feed them into the model. I guess I need a better way to do this? anyone would be any idea is highly appreciated. lon,lat,year,area_burned -180.0,65.0,1001,0 -180.0,65.0,1002,0 -180.0,65.0,1003,0 -180.0,65.0,1004,0 -180.0,65.0,1005,0 -180.0,65.0,1006,0 -180.0,65.0,1007,0 thanks and cheers, Chao ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] remove redundant dimension in a ndarray?
Dear all, Do we have a function in numpy that can automatically shrink a ndarray with redundant dimension? like I have a ndarray with shape of (13,1,1,160,1), now I have written a small function to change the array to dimension of (13,160) [reduce the extra dimension with length as 1]. but I just would like to know maybe there is already something which can do this there ? cheers, Chao -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] copy mask from existing masked array?
Dear all, I have a matrix with dimension of (360,720) but with all global data. I have another land-sea mask matrix with only 2 unique values in it (land=1, sea=-1). So I can easily create transform the second array to a masked array. the problem is, how can I quickly transform the first one to a masked array using the same mask as the land-sea mask array? I hope my question is clear. If not, here is an example: In [93]: a=np.arange(10).reshape(2,5) In [95]: a=np.ma.masked_equal(a,2 In [96]: a=np.ma.masked_equal(a,8) In [97]: a Out[97]: masked_array(data = [[0 1 -- 3 4] [5 6 7 -- 9]], mask = [[False False True False False] [False False False True False]], fill_value = 8) In [100]: b=np.random.normal(0,2,size=(2,5)) I want to convert b to a masked array using exactly the same mask as a. thanks to all, cheers, Chao -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] python geospatial package?
Hi all, Is anyone using some python geospatial package that can do jobs like intersection, etc. the job is like you automatically extract a region on a global map etc. thanks and cheers, Chao -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] change the mask state of one element in a masked array
Dear all, I built a new empty masked array: In [91]: a=np.ma.empty((2,5)) In [92]: a Out[92]: masked_array(data = [[ 1.20569155e-312 3.34730819e-316 1.13580079e-316 1.11459945e-316 9.69610549e-317] [ 6.94900258e-310 8.48292532e-317 6.94900258e-310 9.76397825e-317 6.94900258e-310]], mask = False, fill_value = 1e+20) as you see, the mask for all the elements are false. so how can I set for some elements to masked elements (mask state as true)? let's say, I want a[0,0] to be masked. thanks cheers, Chao -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Addressing arrays
I am afraid you have to write index inquire function by yourself. I did like this. chao 2012/1/30 Ted To rainexpec...@theo.to Hi, Is there some straightforward way to access an array by values across a subset of its dimensions? For example, if I have a three dimensional array a=(x,y,z), can I look at the values of z given particular values for x and y? Thanks, Ted ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Addressing arrays
he is not asking for slicing. he is asking for how to index array by element value but not element index. 2012/1/30 Zachary Pincus zachary.pin...@yale.edu a[x,y,:] Read the slicing part of the tutorial: http://www.scipy.org/Tentative_NumPy_Tutorial (section 1.6) And the documentation: http://docs.scipy.org/doc/numpy/reference/arrays.indexing.html On Jan 30, 2012, at 10:25 AM, Ted To wrote: Hi, Is there some straightforward way to access an array by values across a subset of its dimensions? For example, if I have a three dimensional array a=(x,y,z), can I look at the values of z given particular values for x and y? Thanks, Ted ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] how to cite 1Xn array as nX1 array?
Dear all, suppose I have a ndarray a: In [66]: a Out[66]: array([0, 1, 2, 3, 4]) how can use it as 5X1 array without doing a=a.reshape(5,1)? thanks Chao -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] how to cite 1Xn array as nX1 array?
Thanks all. chao 2012/1/27 Tony Yu tsy...@gmail.com On Fri, Jan 27, 2012 at 9:28 AM, Paul Anton Letnes paul.anton.let...@gmail.com wrote: On 27. jan. 2012, at 14:52, Chao YUE wrote: Dear all, suppose I have a ndarray a: In [66]: a Out[66]: array([0, 1, 2, 3, 4]) how can use it as 5X1 array without doing a=a.reshape(5,1)? Several ways, this is one, although not much simpler. In [6]: a Out[6]: array([0, 1, 2, 3, 4]) In [7]: a.shape = 5, 1 In [8]: a Out[8]: array([[0], [1], [2], [3], [4]]) Paul I'm assuming your issue with that call to reshape is that you need to know the dimensions beforehand. An alternative is to call: a.reshape(-1, 1) The -1 allows numpy to infer the length based on the given sizes. Another alternative is: a[:, np.newaxis] -Tony ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] NumPy / SciPy related tutorials at PyCon 2012
Does anybody know if there is similar chance for training in Paris? (or other places of France)/ the price is nice, just because it's in US thanks, Chao 2012/1/18 Olivier Grisel olivier.gri...@ensta.org Hi all, Just a quick email to advertise this year's PyCon tutorials as they are very focused on HPC data analytics. In particular the numpy / scipy ecosystem is well covered, see: https://us.pycon.org/2012/schedule/tutorials/ Here is a selection of tutorials with an abstracts that mention numpy or a related project (scipy, ipython, matplotlib...): - Bayesian statistics made (as) simple (as possible) - Allen Downey https://us.pycon.org/2012/schedule/presentation/10/ - IPython in-depth: high-productivity interactive and parallel python - Fernando Pérez , Brian E. Granger , Min Ragan-Kelley https://us.pycon.org/2012/schedule/presentation/121/ - Faster Python Programs through Optimization - Mike Müller https://us.pycon.org/2012/schedule/presentation/245/ - Graph Analysis from the Ground Up - Van Lindberg https://us.pycon.org/2012/schedule/presentation/228/ - Data analysis in Python with pandas - Wes McKinney https://us.pycon.org/2012/schedule/presentation/427/ - Social Network Analysis with Python - Maksim Tsvetovat https://us.pycon.org/2012/schedule/presentation/15/ - High Performance Python I - Ian Ozsvald https://us.pycon.org/2012/schedule/presentation/174/ - Plotting with matplotlib - Mike Müller https://us.pycon.org/2012/schedule/presentation/238/ - Introduction to Interactive Predictive Analytics in Python with scikit-learn - Olivier Grisel https://us.pycon.org/2012/schedule/presentation/195/ - High Performance Python II - Travis Oliphant https://us.pycon.org/2012/schedule/presentation/343/ Also the main conference has also very interesting talks: https://us.pycon.org/2012/schedule/ The early birds rate for the PyCOn ends on Jan 25. See you in PyCon in March, -- Olivier http://twitter.com/ogrisel - http://github.com/ogrisel ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] [JOB] Extracting subset of dataset using latitude and longitude
Hi, I don't know if numpy has ready tool for this. I also have this use in my study. So I write a simple code for my personal use. It might no be great. I hope others can also respond as this is very basic function in earth data analysis. 3 import numpy as np lat=np.arange(89.75,-90,-0.5) lon=np.arange(-179.75,180,0.5) lon0,lat0=np.meshgrid(lon,lat) #crate the grid from demonstration def Get_GridValue(data,(vlat1,vlat2),(vlon1,vlon2)): index_lat=np.nonzero((lat[:]=vlat1)(lat[:]=vlat2))[0] index_lon=np.nonzero((lon[:]=vlon1)(lon[:]=vlon2))[0] target=data[...,index_lat[0]:index_lat[-1]+1,index_lon[0]:index_lon[-1]+1] return target Get_GridValue(lat0,(40,45),(-30,-25)) Get_GridValue(lon0,(40,45),(-30,-25)) Chao 2012/1/13 Jeremy Lounds jlou...@dynamiteinc.com Hello, I am looking for some help extracting a subset of data from a large dataset. The data is being read from a wgrib2 (World Meterological Organization standard gridded data) using the pygrib library. The data values, latitudes and longitudes are in separate lists (arrays?), and I would like a regional subset. The budget is not very large, but I am hoping that this is pretty simple job. I am just way too green at Python / numpy to know how to proceed, or even what to search for on Google. If interested, please e-mail jlou...@dynamiteinc.com Thank you! Jeremy Lounds DynamiteInc.com 1-877-762-7723, ext 711 Fax: 877-202-3014 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] strange nan in np.ma.average
Dear all numpy users, I have 10 90X720 arrays. let's say they are in a list 'a' with each element a 90X720 numpy masked array. then I create a new empty ndarray: data data=np.empty([10,90,720]) ##first I store all the 10 ndarray in a 10X90X720 array: for i,d in enumerate(a): data[i]=a data.shape=(10, 90, 720) then I use data_av=np.ma.average(data, axis=0) to get the average. The strange thing is, I don't have any 'nan' in all the 10 90X720 array, but I have nan value in the final data_av. how does this come? In [26]: np.nonzero(np.isnan(data_av)) Out[26]: (array([ 0, 0, 2, 2, 3, 5, 5, 6, 6, 6, 6, 7, 8, 8, 8, 9, 10, 10, 10, 11, 11, 12, 13, 13, 14, 17, 17, 19, 22, 22, 44, 63, 64, 64, 67, 68, 71, 72, 73, 76, 77, 77, 78, 79, 80, 80, 81, 82, 82, 84, 85, 85, 86, 86, 87, 87, 88, 89, 89, 89]), array([159, 541, 497, 548, 90, 97, 170, 244, 267, 587, 590, 150, 126, 168, 477, 240, 271, 277, 588, 99, 179, 528, 52, 256, 230, 109, 190, 617, 377, 389, 707, 539, 193, 361, 262, 465, 100, 232, 206, 90, 87, 93, 522, 229, 200, 482, 325, 195, 239, 228, 159, 194, thanks, Chao -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] strange nan in np.ma.average
the problem is here, data=np.empty([10,90,720]) you should always use np.ma.empty if you want to construct a masked empty array. Chao 2012/1/3 Chao YUE chaoyue...@gmail.com Dear all numpy users, I have 10 90X720 arrays. let's say they are in a list 'a' with each element a 90X720 numpy masked array. then I create a new empty ndarray: data data=np.empty([10,90,720]) ##first I store all the 10 ndarray in a 10X90X720 array: for i,d in enumerate(a): data[i]=a data.shape=(10, 90, 720) then I use data_av=np.ma.average(data, axis=0) to get the average. The strange thing is, I don't have any 'nan' in all the 10 90X720 array, but I have nan value in the final data_av. how does this come? In [26]: np.nonzero(np.isnan(data_av)) Out[26]: (array([ 0, 0, 2, 2, 3, 5, 5, 6, 6, 6, 6, 7, 8, 8, 8, 9, 10, 10, 10, 11, 11, 12, 13, 13, 14, 17, 17, 19, 22, 22, 44, 63, 64, 64, 67, 68, 71, 72, 73, 76, 77, 77, 78, 79, 80, 80, 81, 82, 82, 84, 85, 85, 86, 86, 87, 87, 88, 89, 89, 89]), array([159, 541, 497, 548, 90, 97, 170, 244, 267, 587, 590, 150, 126, 168, 477, 240, 271, 277, 588, 99, 179, 528, 52, 256, 230, 109, 190, 617, 377, 389, 707, 539, 193, 361, 262, 465, 100, 232, 206, 90, 87, 93, 522, 229, 200, 482, 325, 195, 239, 228, 159, 194, thanks, Chao -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] output different columns of ndarray in different formats
Dear all,
Just a small question, how can I output different columns of ndarray in
different formats,
the manual says, A single format (%10.5f), a sequence of formats, or a
multi-format string
but I use
np.savetxt('new.csv',data,fmt=['%i4','%f6.3'])
or
np.savetxt('new.csv',data,fmt=('%i4','%f6.3'))
give strange results.
In [33]: data.shape
Out[33]: (6506, 2)
I want the first column integer and second column float.
cheers,
Chao
--
***
Chao YUE
Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL)
UMR 1572 CEA-CNRS-UVSQ
Batiment 712 - Pe 119
91191 GIF Sur YVETTE Cedex
Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] output different columns of ndarray in different formats
O Yes You're right... It's fine now.
Merry Christmas to all!
Chao
2011/12/22 Aronne Merrelli aronne.merre...@gmail.com
On Thu, Dec 22, 2011 at 10:27 AM, Chao YUE chaoyue...@gmail.com wrote:
Dear all,
Just a small question, how can I output different columns of ndarray in
different formats,
the manual says, A single format (%10.5f), a sequence of formats, or a
multi-format string
but I use
np.savetxt('new.csv',data,fmt=['%i4','%f6.3'])
or
np.savetxt('new.csv',data,fmt=('%i4','%f6.3'))
give strange results.
I think you've flipped the format codes; try:
fmt = ('%4i', '%6.3f')
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
--
***
Chao YUE
Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL)
UMR 1572 CEA-CNRS-UVSQ
Batiment 712 - Pe 119
91191 GIF Sur YVETTE Cedex
Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] what statistical module to use for python?
Hi all, I just want to broadly ask what statistical package are you guys using? I mean routine statistical function like linear regression, GLM, ANOVA... etc. I know there is SciKits packages like statsmodels, but are there more general and complete ones? thanks to all, Chao -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] what statistical module to use for python?
thanks, I should do it but I forgot chao 2011/12/1 josef.p...@gmail.com On Wed, Nov 30, 2011 at 1:16 PM, Chao YUE chaoyue...@gmail.com wrote: Hi all, I just want to broadly ask what statistical package are you guys using? I mean routine statistical function like linear regression, GLM, ANOVA... etc. I know there is SciKits packages like statsmodels, but are there more general and complete ones? thanks to all, I forwarded it to the scipy-user mailing list since that is more suitable. Josef Chao -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] how to use the name of a ndarray as a string
Thanks to all people for this very nice discussions. the solutions are more that what I want!! and help me to clarify some concepts, and really begin to use class as a beginner :) I would say either Olivier or denis's solution can solve my problem completely. cheers, Chao 2011/11/18 denis denis-bz...@t-online.de as Olivier and Chris say, use a dict from the start. To make this a bit easier / more readable, use this tiny class Dotdict: class Dotdict( dict ): d = dotdict(), d.key same as d[key] # aka Bag, Bunch # cPickle.dumps( d, -1 ): PicklingError in 2.6.4 def __init__(self, **kwargs): dict.__init__(self, kwargs) self.__dict__ = self d = Dotdict() d.a = np.arange(10) d.b = np.ones((3,4)) d.b[1,2] += 1 print d.b One can also save all the d.* arrays like this: def savedotdict( d, prefix=d., save=np.savetxt, **kwargs ): save all d.* to files for name, val in d.items(): out = prefix + name if isinstance( val, basestring ): with open( out, w ) as f: f.write( val.rstrip() ) f.write( \n ) else: print saving %s to %s % (val.shape, out) save( out, val, **kwargs ) d.info = 2011-11-18 nov kilroy # From() savedotdict( d, d., fmt=%.3g ) cheers -- denis (If you use this, could you post it to the numpy-discussion group please ? It rejects me, grr.) On Nov 10, 11:17 am, Chao YUE chaoyue...@gmail.com wrote: Hi, Does anyone know how I can quickly use the name of a ndarray as a string? -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] how to use the name of a ndarray as a string
Hi, Does anyone know how I can quickly use the name of a ndarray as a string? for example, I have In [54]: index=np.arange(100) then I want to use the name 'index' as a key in a new dictionary: d3=dict() d3['index']=index I can do it like the way above, but I have many ndarray variables that need to be included in the dictionary. is there something like: d3[index.name()]=index while index.name() would equal the string 'index'? I hope my question is clear. thanks to all. Chao -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] index the last several members of a ndarray
Dear all, if I have In [395]: a Out[395]: array([[0, 1, 2, 3, 4], [5, 6, 7, 8, 9]]) In [396]: a[...,-1] Out[396]: array([4, 9]) In [397]: a[...,-4:-1] Out[397]: array([[1, 2, 3], [6, 7, 8]]) In [398]: a[...,-4:0] Out[398]: array([], shape=(2, 0), dtype=int64) how can I pick up something like: array([[1, 2, 3, 4], [6, 7, 8, 9]]) I want to index the final 4 rows. I cannot figure out how to do this? Thanks for any help, Chao -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] index the last several members of a ndarray
Thanks Jean I just want the last several numbers by indexing from the end. In [400]: b=np.arange(20).reshape(2,10) In [401]: b Out[401]: array([[ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9], [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]]) I want something like b[...,...(index from the end by using negative number)] to get: array([[ 6, 7, 8, 9], [16, 17, 18, 19]]) but it's strange that if you use b[...,-1], you get: In [402]: b[...,-1] Out[402]: array([ 9, 19]) if use b[...,-4:-1], you get: Out[403]: array([[ 6, 7, 8], [16, 17, 18]]) but you cannot use b[...,-4:-1] to get array([[ 6, 7, 8, 9], [16, 17, 18, 19]]) because In [403]: b[...,-4:-1] Out[403]: array([[ 6, 7, 8], [16, 17, 18]]) I don't know I am more clear this time Chao 2011/10/18 Jean-Luc Menut jeanluc.me...@free.fr how can I pick up something like: array([[1, 2, 3, 4], [6, 7, 8, 9]]) I'm not sure to understand, should not a[:,1:] be sufficient ? Did I miss something in your message ? -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] index the last several members of a ndarray
you are right Eric, In [405]: b[...,-4:] Out[405]: array([[ 6, 7, 8, 9], [16, 17, 18, 19]]) cheers, Chao 2011/10/18 Chao YUE chaoyue...@gmail.com Thanks Jean I just want the last several numbers by indexing from the end. In [400]: b=np.arange(20).reshape(2,10) In [401]: b Out[401]: array([[ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9], [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]]) I want something like b[...,...(index from the end by using negative number)] to get: array([[ 6, 7, 8, 9], [16, 17, 18, 19]]) but it's strange that if you use b[...,-1], you get: In [402]: b[...,-1] Out[402]: array([ 9, 19]) if use b[...,-4:-1], you get: Out[403]: array([[ 6, 7, 8], [16, 17, 18]]) but you cannot use b[...,-4:-1] to get array([[ 6, 7, 8, 9], [16, 17, 18, 19]]) because In [403]: b[...,-4:-1] Out[403]: array([[ 6, 7, 8], [16, 17, 18]]) I don't know I am more clear this time Chao 2011/10/18 Jean-Luc Menut jeanluc.me...@free.fr how can I pick up something like: array([[1, 2, 3, 4], [6, 7, 8, 9]]) I'm not sure to understand, should not a[:,1:] be sufficient ? Did I miss something in your message ? -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] np.ma.mean is not working?
Dear all, previoulsy I think np.ma.mean() will automatically filter the masked (missing) value but it's not? In [489]: a=np.arange(20.).reshape(2,10) In [490]: a=np.ma.masked_array(a,(a==2)|(a==5)|(a==11)|(a==18),fill_value=np.nan) In [491]: a Out[491]: masked_array(data = [[0.0 1.0 -- 3.0 4.0 -- 6.0 7.0 8.0 9.0] [10.0 -- 12.0 13.0 14.0 15.0 16.0 17.0 -- 19.0]], mask = [[False False True False False True False False False False] [False True False False False False False False True False]], fill_value = nan) In [492]: a.mean(0) Out[492]: masked_array(data = [5.0 1.0 12.0 8.0 9.0 15.0 11.0 12.0 8.0 14.0], mask = [False False False False False False False False False False], fill_value = 1e+20) In [494]: np.ma.mean(a,0) Out[494]: masked_array(data = [5.0 1.0 12.0 8.0 9.0 15.0 11.0 12.0 8.0 14.0], mask = [False False False False False False False False False False], fill_value = 1e+20) In [495]: np.ma.mean(a,0)==a.mean(0) Out[495]: masked_array(data = [ True True True True True True True True True True], mask = False, fill_value = True) only use a.filled().mean(0) can I get the result I want: In [496]: a.filled().mean(0) Out[496]: array([ 5., NaN, NaN, 8., 9., NaN, 11., 12., NaN, 14.]) I am doing this because I tried to have a small fuction from the web to do moving average for data: import numpy as np def rolling_window(a, window): if window 1: raise ValueError, `window` must be at least 1. if window a.shape[-1]: raise ValueError, `window` is too long. shape = a.shape[:-1] + (a.shape[-1] - window + 1, window) strides = a.strides + (a.strides[-1],) return np.lib.stride_tricks.as_strided(a, shape=shape, strides=strides) def move_ave(a,window): temp=rolling_window(a,window) pre=int(window)/2 post=int(window)-pre-1 return np.concatenate((a[...,0:pre],np.mean(temp,-1),a[...,-post:]),axis=-1) In [489]: a=np.arange(20.).reshape(2,10) In [499]: move_ave(a,4) Out[499]: masked_array(data = [[ 0.1.1.5 2.5 3.5 4.5 5.5 6.5 7.5 9. ] [ 10. 11. 11.5 12.5 13.5 14.5 15.5 16.5 17.5 19. ]], mask = False, fill_value = 1e+20) thanks, Chao -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] anyway to check a ndarray is a mased array or not?
Thanks Olivier. but I don't know how can I check it in the code (not in an interactive mode)? I would like: if ndarray a is a mased array: code 1 else code 2 thanks again, Chao 2011/10/18 Olivier Delalleau sh...@keba.be You could simply check if it has a 'mask' attribute. You can also check if it is an instance of numpy.ma.core.MaskedArray. -=- Olivier Le 18 octobre 2011 08:49, Chao YUE chaoyue...@gmail.com a écrit : Just one more question, how can I check a ndarray is a masked array or not? Chao -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] anyway to check a ndarray is a mased array or not?
really cool, thanks. Chao 2011/10/18 Olivier Delalleau sh...@keba.be if hasattr(a, 'mask'): # or if isinstance(a, numpy.ma.core.MaskedArray.) code 1 else code 2 -=- Olivier 2011/10/18 Chao YUE chaoyue...@gmail.com Thanks Olivier. but I don't know how can I check it in the code (not in an interactive mode)? I would like: if ndarray a is a mased array: code 1 else code 2 thanks again, Chao 2011/10/18 Olivier Delalleau sh...@keba.be You could simply check if it has a 'mask' attribute. You can also check if it is an instance of numpy.ma.core.MaskedArray. -=- Olivier Le 18 octobre 2011 08:49, Chao YUE chaoyue...@gmail.com a écrit : Just one more question, how can I check a ndarray is a masked array or not? Chao -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] np.ma.mean is not working?
thanks. Olivier. I see. Chao 2011/10/18 Olivier Delalleau sh...@keba.be As far as I can tell ma.mean() is working as expected here: it computes the mean only over non-masked values. If you want to get rid of any mean that was computed over a series containing masked value you can do: b = a.mean(0) b.mask[a.mask.any(0)] = True Then b will be: masked_array(data = [5.0 -- -- 8.0 9.0 -- 11.0 12.0 -- 14.0], mask = [False True True False False True False False True False], fill_value = 1e+20) -=- Olivier 2011/10/18 Chao YUE chaoyue...@gmail.com Dear all, previoulsy I think np.ma.mean() will automatically filter the masked (missing) value but it's not? In [489]: a=np.arange(20.).reshape(2,10) In [490]: a=np.ma.masked_array(a,(a==2)|(a==5)|(a==11)|(a==18),fill_value=np.nan) In [491]: a Out[491]: masked_array(data = [[0.0 1.0 -- 3.0 4.0 -- 6.0 7.0 8.0 9.0] [10.0 -- 12.0 13.0 14.0 15.0 16.0 17.0 -- 19.0]], mask = [[False False True False False True False False False False] [False True False False False False False False True False]], fill_value = nan) In [492]: a.mean(0) Out[492]: masked_array(data = [5.0 1.0 12.0 8.0 9.0 15.0 11.0 12.0 8.0 14.0], mask = [False False False False False False False False False False], fill_value = 1e+20) In [494]: np.ma.mean(a,0) Out[494]: masked_array(data = [5.0 1.0 12.0 8.0 9.0 15.0 11.0 12.0 8.0 14.0], mask = [False False False False False False False False False False], fill_value = 1e+20) In [495]: np.ma.mean(a,0)==a.mean(0) Out[495]: masked_array(data = [ True True True True True True True True True True], mask = False, fill_value = True) only use a.filled().mean(0) can I get the result I want: In [496]: a.filled().mean(0) Out[496]: array([ 5., NaN, NaN, 8., 9., NaN, 11., 12., NaN, 14.]) I am doing this because I tried to have a small fuction from the web to do moving average for data: import numpy as np def rolling_window(a, window): if window 1: raise ValueError, `window` must be at least 1. if window a.shape[-1]: raise ValueError, `window` is too long. shape = a.shape[:-1] + (a.shape[-1] - window + 1, window) strides = a.strides + (a.strides[-1],) return np.lib.stride_tricks.as_strided(a, shape=shape, strides=strides) def move_ave(a,window): temp=rolling_window(a,window) pre=int(window)/2 post=int(window)-pre-1 return np.concatenate((a[...,0:pre],np.mean(temp,-1),a[...,-post:]),axis=-1) In [489]: a=np.arange(20.).reshape(2,10) In [499]: move_ave(a,4) Out[499]: masked_array(data = [[ 0.1.1.5 2.5 3.5 4.5 5.5 6.5 7.5 9. ] [ 10. 11. 11.5 12.5 13.5 14.5 15.5 16.5 17.5 19. ]], mask = False, fill_value = 1e+20) thanks, Chao -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] index the last several members of a ndarray
thanks Scott. very good explanation. cheers, Chao 2011/10/18 Scott Sinclair scott.sinclair...@gmail.com On 18 October 2011 13:56, Chao YUE chaoyue...@gmail.com wrote: but it's strange that if you use b[...,-1], you get: In [402]: b[...,-1] Out[402]: array([ 9, 19]) if use b[...,-4:-1], you get: Out[403]: array([[ 6, 7, 8], [16, 17, 18]]) That's because you're mixing two different indexing constructs. In the first case, you're using direct indexing, so you get the values in b at the index you specify. In the second example you're using slicing syntax, where you get the values in b at the range of indices starting with -4 and ending *one before* -1 i.e. the values at b[..., -2]. Here's a simpler example: In [1]: a = range(5) In [2]: a Out[2]: [0, 1, 2, 3, 4] In [3]: a[0] Out[3]: 0 In [4]: a[2] Out[4]: 2 In [5]: a[0:2] Out[5]: [0, 1] In [6]: a[-3] Out[6]: 2 In [7]: a[-1] Out[7]: 4 In [8]: a[-3:-1] Out[8]: [2, 3] Cheers, Scott ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] np.ma.mean is not working?
Thanks Bruce. 2011/10/18 Bruce Southey bsout...@gmail.com ** On 10/18/2011 09:12 AM, Chao YUE wrote: thanks. Olivier. I see. Chao 2011/10/18 Olivier Delalleau sh...@keba.be As far as I can tell ma.mean() is working as expected here: it computes the mean only over non-masked values. If you want to get rid of any mean that was computed over a series containing masked value you can do: b = a.mean(0) b.mask[a.mask.any(0)] = True Then b will be: masked_array(data = [5.0 -- -- 8.0 9.0 -- 11.0 12.0 -- 14.0], mask = [False True True False False True False False True False], fill_value = 1e+20) -=- Olivier 2011/10/18 Chao YUE chaoyue...@gmail.com Dear all, previoulsy I think np.ma.mean() will automatically filter the masked (missing) value but it's not? In [489]: a=np.arange(20.).reshape(2,10) In [490]: a=np.ma.masked_array(a,(a==2)|(a==5)|(a==11)|(a==18),fill_value=np.nan) In [491]: a Out[491]: masked_array(data = [[0.0 1.0 -- 3.0 4.0 -- 6.0 7.0 8.0 9.0] [10.0 -- 12.0 13.0 14.0 15.0 16.0 17.0 -- 19.0]], mask = [[False False True False False True False False False False] [False True False False False False False False True False]], fill_value = nan) In [492]: a.mean(0) Out[492]: masked_array(data = [5.0 1.0 12.0 8.0 9.0 15.0 11.0 12.0 8.0 14.0], mask = [False False False False False False False False False False], fill_value = 1e+20) In [494]: np.ma.mean(a,0) Out[494]: masked_array(data = [5.0 1.0 12.0 8.0 9.0 15.0 11.0 12.0 8.0 14.0], mask = [False False False False False False False False False False], fill_value = 1e+20) In [495]: np.ma.mean(a,0)==a.mean(0) Out[495]: masked_array(data = [ True True True True True True True True True True], mask = False, fill_value = True) only use a.filled().mean(0) can I get the result I want: In [496]: a.filled().mean(0) Out[496]: array([ 5., NaN, NaN, 8., 9., NaN, 11., 12., NaN, 14.]) I am doing this because I tried to have a small fuction from the web to do moving average for data: import numpy as np def rolling_window(a, window): if window 1: raise ValueError, `window` must be at least 1. if window a.shape[-1]: raise ValueError, `window` is too long. shape = a.shape[:-1] + (a.shape[-1] - window + 1, window) strides = a.strides + (a.strides[-1],) return np.lib.stride_tricks.as_strided(a, shape=shape, strides=strides) def move_ave(a,window): temp=rolling_window(a,window) pre=int(window)/2 post=int(window)-pre-1 return np.concatenate((a[...,0:pre],np.mean(temp,-1),a[...,-post:]),axis=-1) In [489]: a=np.arange(20.).reshape(2,10) In [499]: move_ave(a,4) Out[499]: masked_array(data = [[ 0.1.1.5 2.5 3.5 4.5 5.5 6.5 7.5 9. ] [ 10. 11. 11.5 12.5 13.5 14.5 15.5 16.5 17.5 19. ]], mask = False, fill_value = 1e+20) thanks, Chao -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ___ NumPy-Discussion mailing listNumPy-Discussion@scipy.orghttp://mail.scipy.org/mailman/listinfo/numpy-discussion Looked at pandas for your rolling window functionality: http://pandas.sourceforge.net *Time series*-specific functionality: date range generation and frequency conversion, moving window statistics, moving window linear regressions, date shifting and lagging, etc. Bruce ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE
Re: [Numpy-discussion] ndarray with double comparison
Thanks. quite useful!! Chao 2011/10/15 Neil neilcrigh...@gmail.com Marc Shivers marc.shivers at gmail.com writes: you could use bitwise comparison with paretheses: In [8]: (a4)(a8)Out[8]: array([False, False, False, False, False, True, True, True, False, False, False], dtype=bool) For cases like this I find it very useful to define a function between() - e.g. https://bitbucket.org/nhmc/pyserpens/src/4e2cc9b656ae/utilities.py#cl-88 Then you can use between(a, 4, 8) instead of (4 a) (a 8), which I find less readable and more difficult to type. Neil ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] the axis parameter in the np.ma.concatenate is not working?
Thanks Josef, you're right. Could you explain me what's the difference between In [4]: a=np.arange(10) In [5]: a.shape Out[5]: (10,) and In [6]: a=np.arange(10).reshape(10,1) In [7]: a.shape Out[7]: (10, 1) (10) means the first a is only a one-dimensional ndarray, but the (10,1) means the second a is a two-dimensional ndarray? another question, if I have In [70]: f Out[70]: masked_array(data = [[0 0] [1 1] [2 2] [3 3] [-- 4] [5 5] [6 6] [7 --] [8 8] [9 9]], mask = [[False False] [False False] [False False] [False False] [ True False] [False False] [False False] [False True] [False False] [False False]], fill_value = 99) but when I do In [71]: f.data Out[71]: array([[0, 0], [1, 1], [2, 2], [3, 3], [4, 4], [5, 5], [6, 6], [7, 7], [8, 8], [9, 9]]) it still shows the original value, so what 's the usage of fill_value in masked array? can I set a fill_value as np.nan? Thanks, Chao 2011/10/13 josef.p...@gmail.com On Thu, Oct 13, 2011 at 1:17 PM, Chao YUE chaoyue...@gmail.com wrote: Dear all, I use numpy version 1.5.1 which is installed by default when I do sudo apt-get install numpy on ubuntu 11.04. but it seems that for np.ma.concatenate(arrays, axis), the axis parameter is not working? In [460]: a=np.arange(10) In [461]: a=np.ma.masked_array(a,a3) In [462]: a Out[462]: masked_array(data = [-- -- -- 3 4 5 6 7 8 9], mask = [ True True True False False False False False False False], fill_value = 99) In [463]: b=np.arange(10) In [464]: b=np.ma.masked_array(a,b7) In [465]: b Out[465]: masked_array(data = [-- -- -- 3 4 5 6 7 -- --], mask = [ True True True False False False False False True True], fill_value = 99) In [466]: c=np.ma.concatenate((a,b),axis=0) In [467]: c Out[467]: masked_array(data = [-- -- -- 3 4 5 6 7 8 9 -- -- -- 3 4 5 6 7 -- --], mask = [ True True True False False False False False False False True True True False False False False False True True], fill_value = 99) In [468]: c.shape Out[468]: (20,) In [469]: c=np.ma.concatenate((a,b),axis=1) maybe you want numpy.ma.column_stack for concatenate you need to add extra axis first something like c=np.ma.concatenate((a[:,None], b[:,None]),axis=1) (not tested) Josef In [470]: c.shape Out[470]: (20,) cheers, Chao -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] how to list all the values in a ndarray without repeat (like filter in excel)
Dear all, if I have a ndarray like array([1,2,3,2,3,1,1,1,2,2,,2,2,3]) containing some values that are flag for data quality. how can list all the values in this array, like doing a descriptive statistics. I guess I should use Scipy statistics ? Thanks for any ideas. Chao -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] how to list all the values in a ndarray without repeat (like filter in excel)
Yes, np.unique() is exactly what I want. thanks. chao 2011/10/13 Benjamin Root ben.r...@ou.edu On Thursday, October 13, 2011, Chao YUE chaoyue...@gmail.com wrote: Dear all, if I have a ndarray like array([1,2,3,2,3,1,1,1,2,2,,2,2,3]) containing some values that are flag for data quality. how can list all the values in this array, like doing a descriptive statistics. I guess I should use Scipy statistics ? Thanks for any ideas. Chao Would np.unique() do the job? Ben Root ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] ndarray with double comparison
Dear all, sorry for this stupid question but I cannot find it in numpy tutorial or google. suppose I have a=np.arange(11). In [32]: a 8 Out[32]: array([ True, True, True, True, True, True, True, True, False, False, False], dtype=bool) In [34]: a 4 Out[34]: array([False, False, False, False, False, True, True, True, True, True, True], dtype=bool) how can I have boolean index like 4 a 8 np.where(a4 and a8);or plainly input a4 and a8 doesn't work. thanks, Chao -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] ndarray with double comparison
Thanks. I starts to use python do some real data processing and has bunch of questions. Chao 2011/10/13 Benjamin Root ben.r...@ou.edu On Thu, Oct 13, 2011 at 11:13 AM, Chao YUE chaoyue...@gmail.com wrote: Dear all, sorry for this stupid question but I cannot find it in numpy tutorial or google. suppose I have a=np.arange(11). In [32]: a 8 Out[32]: array([ True, True, True, True, True, True, True, True, False, False, False], dtype=bool) In [34]: a 4 Out[34]: array([False, False, False, False, False, True, True, True, True, True, True], dtype=bool) how can I have boolean index like 4 a 8 np.where(a4 and a8);or plainly input a4 and a8 doesn't work. thanks, Chao Unfortunately, you can't use and, or, not keywords with boolean arrays because numpy can't overload them. Instead, use the bitwise operators: '', '|', and '~'. Be careful, though, because of operator precedence is different for bitwise operators than the boolean keywords. I am in the habit of always wrapping my boolean expressions in parentheses, just in case. (a 4) (a 8) is what you want. Note that a 4 a 8 would be evaluated in a different order -- 4 a would be first. I hope that helps! Ben Root ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] the axis parameter in the np.ma.concatenate is not working?
Dear all, I use numpy version 1.5.1 which is installed by default when I do sudo apt-get install numpy on ubuntu 11.04. but it seems that for np.ma.concatenate(arrays, axis), the axis parameter is not working? In [460]: a=np.arange(10) In [461]: a=np.ma.masked_array(a,a3) In [462]: a Out[462]: masked_array(data = [-- -- -- 3 4 5 6 7 8 9], mask = [ True True True False False False False False False False], fill_value = 99) In [463]: b=np.arange(10) In [464]: b=np.ma.masked_array(a,b7) In [465]: b Out[465]: masked_array(data = [-- -- -- 3 4 5 6 7 -- --], mask = [ True True True False False False False False True True], fill_value = 99) In [466]: c=np.ma.concatenate((a,b),axis=0) In [467]: c Out[467]: masked_array(data = [-- -- -- 3 4 5 6 7 8 9 -- -- -- 3 4 5 6 7 -- --], mask = [ True True True False False False False False False False True True True False False False False False True True], fill_value = 99) In [468]: c.shape Out[468]: (20,) In [469]: c=np.ma.concatenate((a,b),axis=1) In [470]: c.shape Out[470]: (20,) cheers, Chao -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] what python module to modify NetCDF data?
Thanks for all for very useful discussions. I am just on the beginning of modeling and for now I only use netCDF4 data. I think it might be a good idea to try netcdf4-pythonhttp://code.google.com/p/netcdf4-python/first. I think whitaker will continue to develop this package. But it gave me a good idea what python package people are using for netCDF data. Chao 2011/10/9 Konrad Hinsen konrad.hin...@fastmail.net On 8 oct. 11, at 17:57, Vicente Sole wrote: I have never seen myself a NetCDF file but if your NetCDF file is using HDF5 as format (possible since NetCDF 4 if I am not mistaken), you should be able to use h5py or PyTables to access and or modify it. I haven't tried this, but I don't think it's a good plan. PyTables can read arbitrary HDF5 files, but writes only a subset. The HDF5-based netCDF 4 format is also a subset of valid HDF5 files. It is unlikely that those two subsets are compatible, meaning that sooner or later PyTables will produce a file that netCDF 4 will not accept. h5py would be a better choice for an HDF5-based approach. However, there are two reasons for using a specific netCDF interface rather than HDF5: 1) There are still plenty of non-HDF5 netCDF files around, and netCDF4 continues to support them. 2) The netCDF interface is simpler than the HDF5 interface, and therefore easier to use, even if that difference is much less important in Python than in C or Fortran. A full netCDF interface has been available for many years as part of the ScientificPython package: http://dirac.cnrs-orleans.fr/ScientificPython/ScientificPythonManual/ Recent versions fully support netCDF4, including the HDF5-based formats. There are other Python interfaces to the netCDF libraries, but I haven't used them. Konrad. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] what python module to modify NetCDF data?
Dear all, I want to change some variable values in a series of NetCDF file. Did anybody else did this before using python? Now I use pupynere for reading data from NetCDF files and making plots. but the document of pupynere for writing data to NetCDF file is quite simple and I still feel difficult to do this with pupynere. the NetCDF file I want to change is a global data (0.5X0.5d resolution, 360X720grid with 12 time steps) and have approx. 10 variables. I just want to change some points for a specific variable for all 12 time steps. I know it's possible use NCO ncap2 utility to do the job. but now I have some problem in using ncap2 within a shell script. I guess there is some easy way to use some python module to do the job? like mainly altering the data that need to change while let the others remaining intact? Any idea will be greatly appreciated. I will all a good weekend, Chao -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] how to count Nan occurrence in a ndarray?
Dear all, I have an ndarray with dimension of 4X62500. is there anyway I can count the number of missing value (NaN)? because I want to know how many observations are missing? Thanks for any idea, Chao -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] how to count Nan occurrence in a ndarray?
The data is read from a .mat file. 2011/10/7 Chao YUE chaoyue...@gmail.com Dear all, I have an ndarray with dimension of 4X62500. is there anyway I can count the number of missing value (NaN)? because I want to know how many observations are missing? Thanks for any idea, Chao -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] how to count Nan occurrence in a ndarray?
Thanks Olivier. Chao 2011/10/7 Olivier Delalleau sh...@keba.be You can use numpy.isnan(array).sum() -=- Olivier 2011/10/7 Chao YUE chaoyue...@gmail.com The data is read from a .mat file. 2011/10/7 Chao YUE chaoyue...@gmail.com Dear all, I have an ndarray with dimension of 4X62500. is there anyway I can count the number of missing value (NaN)? because I want to know how many observations are missing? Thanks for any idea, Chao -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] import
Yes, you're right. The problem is, when you use the first one, you may cause a 'name pollution' to the current namespace. read this: http://bytebaker.com/2008/07/30/python-namespaces/ cheers, Chao 2011/7/19 Alex Ter-Sarkissov ater1...@gmail.com this is probably silly question, I've seen in this in one of the tutorials: from tkinter import * import tkinter.messagebox given that * implies importing the whole module, why would anyone bother with importing a specific command on top of it? ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 77 30; Fax:01.69.08.77.16 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] data type specification when using numpy.genfromtxt
Thanks very much!! you are right. It's becuase the extra semicolon in the
head row. I have no problems anymore.
I thank you for your time.
cheeers,
Chao
2011/6/28 Derek Homeier de...@astro.physik.uni-goettingen.de
Hi Chao,
by mistake did not reply to the list last time...
On 27.06.2011, at 10:30PM, Chao YUE wrote:
Hi Derek!
I tried with the lastest version of python(x,y) package with numpy
version of 1.6.0. I gave the data to you with reduced columns (10 column)
and rows.
b=np.genfromtxt('99Burn2003all_new.csv',delimiter=';',names=True,usecols=tuple(range(10)),dtype=['S10']
+ [ float for n in range(9)]) works.
if you change usecols=tuple(range(10)) to usecols=range(10), it still
works.
b=np.genfromtxt('99Burn2003all_new.csv',delimiter=';',names=True,dtype=None)
works.
but
b=np.genfromtxt('99Burn2003all_new.csv',delimiter=';',names=True,dtype=['S10']
+ [ float for n in range(9)]) didn't work.
I use Python(x,y)-2.6.6.1 with numpy version as 1.6.0, I use windows
32-bit system.
Please don't spend too much time on this if it's not a potential problem.
OK, dtype=None works on 1.6.0, that's the important bit.
From your example file it seems the dtype list does work not without
specifying usecols, because your header contains and excess semicolon in the
field Air temperature (High; HMP45C), thus genfromtxt expects more data
columns than actually exist. If you replace the semicolon you should be set
(or, if I may suggest, write another header line with catchier field names
so you don't have to work with array fields like b['Water vapor density by
LiCor 7500'] ;-).
Otherwise both options work for me with python2.6+numpy-1.5.1 as well as
1.6.0/1.6.1rc1.
I am curious though why your python interpreter gave this error message:
ValueErrorTraceback (most recent call
last)
D:\data\LaThuile_ancillary\Jim_Randerson_data\ipython console in
module()
C:\Python26\lib\site-packages\numpy\lib\npyio.pyc in genfromtxt(fname,
dtype, co
mments, delimiter, skiprows, skip_header, skip_footer, converters,
missing, miss
ing_values, filling_values, usecols, names, excludelist, deletechars,
replace_sp
ace, autostrip, case_sensitive, defaultfmt, unpack, usemask, loose,
invalid_rais
e)
1449 # Raise an exception ?
1450 if invalid_raise:
- 1451 raise ValueError(errmsg)
1452 # Issue a warning ?
1453 else:
ValueError
since ipython2.6 on my Mac reported this:
...
1450 if invalid_raise:
- 1451 raise ValueError(errmsg)
1452 # Issue a warning ?
1453 else:
ValueError: Some errors were detected !
Line #3 (got 10 columns instead of 11)
Line #4 (got 10 columns instead of 11)
etc
which of course provided the right lead to the problem - was the actual
errmsg really missing, or did you cut the message too soon?
the final thing is, when I try to do this (I want to try the
missing_values in numpy 1.6.0), it gives error:
In [33]: import StringIO as StringIO
In [34]: data = 1, 2, 3\n4, 5, 6
In [35]: np.genfromtxt(StringIO(data),
delimiter=,,dtype=int,int,int,missing_values=2)
---
TypeError Traceback (most recent call
last)
D:\data\LaThuile_ancillary\Jim_Randerson_data\ipython console in
module()
TypeError: 'module' object is not callable
You want to use from StringIO import StringIO (or write
StringIO.StringIO(data).
But again, this will not work the way you expect it to with int/float
numbers set as missing_values, and reading to regular arrays. I've tested
this on 1.6.1 and the current development branch as well, and the
missing_values are only considered for masked arrays. This is not likely to
change soon, and may actually be intentional, so to process those numbers on
read-in, your best option would be to define a custom set of
converters=conv as shown in my last mail.
Cheers,
Derek
2011/6/27 Derek Homeier de...@astro.physik.uni-goettingen.de
Hi Chao,
this seems to have become quite a number of different issues!
But let's make sure I understand what's going on...
Thanks very much for your quick reply. I make a short summary of what
I've tried. Actually the ['S10'] + [ float for n in range(48) ] only works
when you explicitly specify the columns to be read, and genfromtxt cannot
automatically determine the type if you don't specify the type
In [164]:
b=np.genfromtxt('99Burn2003all.csv',delimiter=';',names=True,usecols=tuple(range(49)),dtype=['S10']
+ [ float for n in range(48)])
...
But if I use the following, it gives error:
In [171]:
b=np.genfromtxt('99Burn2003all.csv',delimiter=';',names=True,dtype=['S
10
Re: [Numpy-discussion] data type specification when using numpy.genfromtxt
, Derek -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 77 30; Fax:01.69.08.77.16 TIMESTAMP;CO2 flux;Net radiation;Sensible heat flux;Latent heat flux;u*;Water vapor density by LiCor 7500;CO2 concentration;Air temperature (High; HMP45C);AGC #TIMESTAMP;umol m-2 s-1;W/m^2;W/m^2;W/m^2;m/s;g/(m^3);ppm;C;unitless 01/01/2003 00:00;-999;-1.028;-999;-999;-999;-999;-999;-25.3684;-999 01/01/2003 00:30;-999;-0.409;-999;-999;-999;-999;-999;-25.3233;-999 01/01/2003 01:00;-999;0.167;-999;-999;-999;-999;-999;-25.3589;-999 01/01/2003 01:30;-999;-0.188;-999;-999;-999;-999;-999;-25.4535;-999 01/01/2003 02:00;-999;-0.017;-999;-999;-999;-999;-999;-25.5717;-999 01/01/2003 02:30;-999;0.911;-999;-999;-999;-999;-999;-25.6927;-999 01/01/2003 03:00;-999;1.675;-999;-999;-999;-999;-999;-26.261;-999 01/01/2003 03:30;-999;1.856;-999;-999;-999;-999;-999;-26.6863;-999 01/01/2003 04:00;-999;3.377;-999;-999;-999;-999;-999;-26.7538;-999 01/01/2003 04:30;-999;4.012;-999;-999;-999;-999;-999;-27.3459;-999 01/01/2003 05:00;-999;2.985;-999;-999;-999;-999;-999;-27.8764;-999 01/01/2003 05:30;-999;3.396;-999;-999;-999;-999;-999;-28.7779;-999 01/01/2003 06:00;-999;4.641;-999;-999;-999;-999;-999;-29.2211;-999 01/01/2003 06:30;-999;5.066;-999;-999;-999;-999;-999;-29.5942;-999 01/01/2003 07:00;-999;5.225;-999;-999;-999;-999;-999;-30.3728;-999 01/01/2003 07:30;-999;5.272;-999;-999;-999;-999;-999;-31.0314;-999 01/01/2003 08:00;-999;4.8;-999;-999;-999;-999;-999;-31.5624;-999 01/01/2003 08:30;-999;4.71;-999;-999;-999;-999;-999;-32.0388;-999 01/01/2003 09:00;-999;4.345;-999;-999;-999;-999;-999;-31.1014;-999 01/01/2003 09:30;-999;4.681;-999;-999;-999;-999;-999;-31.6566;-999 01/01/2003 10:00;-999;4.567;-999;-999;-999;-999;-999;-31.9319;-999 01/01/2003 10:30;-999;4.491;-999;-999;-999;-999;-999;-32.2679;-999 01/01/2003 11:00;-999;4.277;-999;-999;-999;-999;-999;-32.457;-999 01/01/2003 11:30;-999;4.054;-999;-999;-999;-999;-999;-32.4047;-999 01/01/2003 12:00;-999;2.173;-999;-999;-999;-999;-999;-32.6372;-999 01/01/2003 12:30;-999;1.132;-999;-999;-999;-999;-999;-32.7332;-999 01/01/2003 13:00;-999;-1.394;-999;-999;-999;-999;-999;-32.59;-999 01/01/2003 13:30;-999;-2.617;-999;-999;-999;-999;-999;-31.564;-999 01/01/2003 14:00;-999;-2.794;-999;-999;-999;-999;-999;-31.2353;-999 01/01/2003 14:30;-999;-8.017;-999;-999;-999;-999;-999;-31.5474;-999 01/01/2003 15:00;-999;-2.115;-999;-999;-999;-999;-999;-31.765;-999 01/01/2003 15:30;-999;-0.143;-999;-999;-999;-999;-999;-31.3201;-999 01/01/2003 16:00;-999;1.267;-999;-999;-999;-999;-999;-31.5929;-999 01/01/2003 16:30;-999;-0.282;-999;-999;-999;-999;-999;-31.4198;-999 01/01/2003 17:00;-999;-0.281;-999;-999;-999;-999;-999;-31.8246;-999 01/01/2003 17:30;-999;-1.001;-999;-999;-999;-999;-999;-31.0432;-999 01/01/2003 18:00;-999;0.054;-999;-999;-999;-999;-999;-30.972;-999 01/01/2003 18:30;-999;1.008;-999;-999;-999;-999;-999;-30.6776;-999 01/01/2003 19:00;-999;-2.189;-999;-999;-999;-999;-999;-30.5332;-999 01/01/2003 19:30;-999;-3.992;-999;-999;-999;-999;-999;-29.8003;-999 01/01/2003 20:00;-999;-2.537;-999;-999;-999;-999;-999;-28.5913;-999 01/01/2003 20:30;-999;-2.093;-999;-999;-999;-999;-999;-27.8852;-999 01/01/2003 21:00;-999;0.324;-999;-999;-999;-999;-999;-27.4831;-999 01/01/2003 21:30;-999;1.97;-999;-999;-999;-999;-999;-27.5662;-999 01/01/2003 22:00;-999;2.527;-999;-999;-999;-999;-999;-27.8688;-999 01/01/2003 22:30;-999;2.553;-999;-999;-999;-999;-999;-28.0078;-999 01/01/2003 23:00;-999;2.719;-999;-999;-999;-999;-999;-28.858;-999 01/01/2003 23:30;-999;3.2;-999;-999;-999;-999;-999;-29.1183;-999 02/01/2003 00:00;-999;3.176;-999;-999;-999;-999;-999;-29.6443;-999 02/01/2003 00:30;-999;1.682;-999;-999;-999;-999;-999;-30.6337;-999 02/01/2003 01:00;-999;1.569;-999;-999;-999;-999;-999;-30.0727;-999 02/01/2003 01:30;-999;1.247;-999;-999;-999;-999;-999;-29.8676;-999 02/01/2003 02:00;-999;2.679;-999;-999;-999;-999;-999;-29.4964;-999 02/01/2003 02:30;-999;2.339;-999;-999;-999;-999;-999;-30.039;-999 02/01/2003 03:00;-999;2.85;-999;-999;-999;-999;-999;-30.0223;-999 02/01/2003 03:30;-999;1.844;-999;-999;-999;-999;-999;-30.979;-999 02/01/2003 04:00;-999;1.383;-999;-999;-999;-999;-999;-30.9313;-999 02/01/2003 04:30;-999;0.157;-999;-999;-999;-999;-999;-30.8023;-999 02/01/2003 05:00;-999;0.23;-999;-999;-999;-999;-999;-29.6911;-999 02/01/2003 05:30;-999;3.226;-999;-999;-999;-999;-999;-30.1023;-999 02/01/2003 06:00;-999;1.02;-999;-999;-999;-999;-999;-29.7733;-999 02/01/2003 06:30;-999;1.998;-999;-999;-999;-999;-999;-29.7671;-999 02/01/2003 07:00;-999;1.737;-999;-999;-999;-999;-999;-29.1955;-999 02/01/2003 07:30;-999;0.688;-999;-999;-999;-999;-999;-28.6603;-999 02/01/2003 08:00;-999;2.919;-999;-999;-999;-999;-999
[Numpy-discussion] data type specification when using numpy.genfromtxt
Dear all numpy users,
I want to read a csv file with many (49) columns, the first column is string
and remaning can be float.
how can I avoid type in like
data=numpy.genfromtxt('data.csv',delimiter=';',names=True, dtype=(S10,
float, float, ..))
Can I just specify the type of first cloumn is tring and the remaing float?
how can I do that?
Thanks a lot,
Chao
--
***
Chao YUE
Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL)
UMR 1572 CEA-CNRS-UVSQ
Batiment 712 - Pe 119
91191 GIF Sur YVETTE Cedex
Tel: (33) 01 69 08 77 30; Fax:01.69.08.77.16
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] data type specification when using numpy.genfromtxt
the '-999.0' in the data is missing
value, but I cannot display it as 'nan' by specifying the missing_values as
'-999.0':
but either I set the missing_values as -999.0 or using a dictionary, it
neither work...
*In [178]:
b=np.genfromtxt('99Burn2003all.csv',delimiter=';',names=True,usecols=(0,1,2),dtype=|S18,float,float,missing_values=-999.0)
In [179]: b
Out[179]:
array([('01/01/2003 00:00', -999.0, -1.028),
('01/01/2003 00:30', -999.0, -0.40897),
('01/01/2003 01:00', -999.0, 0.16701), ...,
('31/12/2003 22:30', -999.0, -999.0),
('31/12/2003 23:00', -999.0, -999.0),
('31/12/2003 23:30', -999.0, -999.0)],
dtype=[('TIMESTAMP', '|S18'), ('CO2_flux', 'f8'), ('Net_radiation',
'f8'
)])
In [180]:
b=np.genfromtxt('99Burn2003all.csv',delimiter=';',names=True,usecols=(
0,1,2),dtype=|S18,float,float,missing_values={1:'-999.0'})
In [181]:
In [182]: b
Out[182]:
array([('01/01/2003 00:00', -999.0, -1.028),
('01/01/2003 00:30', -999.0, -0.40897),
('01/01/2003 01:00', -999.0, 0.16701), ...,
('31/12/2003 22:30', -999.0, -999.0),
('31/12/2003 23:00', -999.0, -999.0),
('31/12/2003 23:30', -999.0, -999.0)],
dtype=[('TIMESTAMP', '|S18'), ('CO2_flux', 'f8'), ('Net_radiation',
'f8'
)])*
the value of is actually -999.0
*In [183]: b['CO2_flux'][1]==-999.0
Out[183]: True
*Even this doesn't work (suppose 2 is our missing_value),*
In [184]: data = 1, 2, 3\n4, 5, 6
In [185]: np.genfromtxt(StringIO(data),
delimiter=,,dtype=int,int,int,missin
g_values=2)
Out[185]:
array([(1, 2, 3), (4, 5, 6)],
dtype=[('f0', 'i4'), ('f1', 'i4'), ('f2', 'i4')])
In [186]: np.genfromtxt(StringIO(data),
delimiter=,,dtype=int,int,int,names=
a,b,c,missing_values={'b':2},filling_values=nan)
Out[186]:
array([(1, 2, 3), (4, 5, 6)],
dtype=[('a', 'i4'), ('b', 'i4'), ('c', 'i4')])*
Can you give me some suggestion? Thanks in advance~~*
*
Chao
*
2011/6/26 Derek Homeier de...@astro.physik.uni-goettingen.de
On 26.06.2011, at 8:48PM, Chao YUE wrote:
I want to read a csv file with many (49) columns, the first column is
string and remaning can be float.
how can I avoid type in like
data=numpy.genfromtxt('data.csv',delimiter=';',names=True, dtype=(S10,
float, float, ..))
Can I just specify the type of first cloumn is tring and the remaing
float? how can I do that?
Simply use 'dtype=None' to let genfromtxt automatically determine the type
(it is perhaps a bit confusing that this is not the default - maybe it
should be repeated in the docstring for clarity that the default is for
dtype is 'float'...).
Also, a shorter way of typing the dtype above (e.g. in case some columns
would be auto-detected as int) would be
['S10'] + [ float for n in range(48) ]
HTH,
Derek
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
--
***
Chao YUE
Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL)
UMR 1572 CEA-CNRS-UVSQ
Batiment 712 - Pe 119
91191 GIF Sur YVETTE Cedex
Tel: (33) 01 69 08 77 30; Fax:01.69.08.77.16
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
