[Numpy-discussion] Getting masked array boundary indices
Hi, I have a numpy array which is masked ( bathymetry), as seen below [ True] [ True] [ True] [ True]], fill_value = -.0) In [10]: depth[:,1130:1131] I need to find the indices where land(mask) is there along the boundaries and where water(value) is there along the boundaries, the above listing is along eastern boundary. Please help if there is a way to get starting and ending index of mask. I tried np.where but it gives another array as there are several mask points are there, I need to use some thing like if neighbouring points are True and False then index =i, but I am not getting the pythonic way to get this done. with best regards, Sudheer *** Sudheer Joseph Indian National Centre for Ocean Information Services Ministry of Earth Sciences, Govt. of India POST BOX NO: 21, IDA Jeedeemetla P.O. Via Pragathi Nagar,Kukatpally, Hyderabad; Pin:5000 55 Tel:+91-40-23886047(O),Fax:+91-40-23895011(O), Tel:+91-40-23044600(R),Tel:+91-40-9440832534(Mobile) E-mail:sjo.in...@gmail.com;sudheer.jos...@yahoo.com Web- http://oppamthadathil.tripod.com *** ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Getting masked array boundary indices
On Wed, Nov 27, 2013 at 7:01 AM, Sudheer Joseph sudheer.jos...@yahoo.com wrote: Hi, I have a numpy array which is masked ( bathymetry), as seen below [ True] [ True] [ True] [ True]], fill_value = -.0) In [10]: depth[:,1130:1131] I need to find the indices where land(mask) is there along the boundaries and where water(value) is there along the boundaries, the above listing is along eastern boundary. Please help if there is a way to get starting and ending index of mask. I tried np.where but it gives another array as there are several mask points are there, I need to use some thing like if neighbouring points are True and False then index =i, but I am not getting the pythonic way to get this done. if I understand correctly np.nonzero(np.diff(depth.mask))[0] Josef with best regards, Sudheer *** Sudheer Joseph Indian National Centre for Ocean Information Services Ministry of Earth Sciences, Govt. of India POST BOX NO: 21, IDA Jeedeemetla P.O. Via Pragathi Nagar,Kukatpally, Hyderabad; Pin:5000 55 Tel:+91-40-23886047(O),Fax:+91-40-23895011(O), Tel:+91-40-23044600(R),Tel:+91-40-9440832534(Mobile) E-mail:sjo.in...@gmail.com;sudheer.jos...@yahoo.com Web- http://oppamthadathil.tripod.com *** ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] PyArray_BASE equivalent in python
First, sorry for not responding to your other replies, there was a jam
in Thunderbird and I did not receive your answers.
The bits() seem to stay alive after deleting the image:
from PySide import QtGui
image = QtGui.QImage('/home/peter/code/pyTools/sandbox/images/faceDemo.jpg')
a = image.bits()
del image
b = str(a)
Works. But it might still be a Garbage collector thing and it might
break later. I do not know enough about Python and its buffers to know
what to expect here.
As a solution I have done something similar as it was proposed earlier,
just that I derived from ndarray and kept the QImage reference it it:
from PySide import QtGui as _qt
import numpy as _np
class MemoryTie(np.ndarray):
def __new__(cls, image):
# Retrieving parameters for _np.ndarray()
dims = [image.height(), image.width()]
strides = [[] for i in range(2)]
strides[0] = image.bytesPerLine()
bits = image.bits()
if image.format() == _qt.QImage.Format_Indexed8:
dtype = _np.uint8
strides[1] = 1
elif image.format() == _qt.QImage.Format_RGB32 \
or image.format() == _qt.QImage.Format_ARGB32 \
or image.format() == _qt.QImage.Format_ARGB32_Premultiplied:
dtype = _np.uint32
strides[1] = 4
elif image.format() == _qt.QImage.Format_Invalid:
raise ValueError(qimageview got invalid QImage)
else:
raise ValueError(qimageview can only handle 8- or 32-bit
QImages)
# creation of ndarray
obj = _np.ndarray(dims, _np.uint32, bits, 0, strides,
'C').view(cls)
obj._image = image
return obj
Thanks all for your help,
P
On 11/26/2013 10:58 PM, Nathaniel Smith wrote:
On Tue, Nov 26, 2013 at 2:55 PM, Peter Rennert p.renn...@cs.ucl.ac.uk wrote:
Btw, I just wanted to file a bug at PySide, but it might be alright at
their end, because I can do this:
from PySide import QtGui
image = QtGui.QImage('/home/peter/code/pyTools/sandbox/images/faceDemo.jpg')
a = image.bits()
del image
a
#read-write buffer ptr 0x7f5fe0034010, size 1478400 at 0x3c1a6b0
That just means that the buffer still has a pointer to the QImage's
old memory. It doesn't mean that following that pointer won't crash.
Try str(a) or something that actually touches the buffer contents...
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] PyArray_BASE equivalent in python
On Wed, Nov 27, 2013 at 2:16 PM, Peter Rennert p.renn...@cs.ucl.ac.uk
wrote:
As a solution I have done something similar as it was proposed earlier,
just that I derived from ndarray and kept the QImage reference it it:
from PySide import QtGui as _qt
import numpy as _np
class MemoryTie(np.ndarray):
def __new__(cls, image):
# Retrieving parameters for _np.ndarray()
dims = [image.height(), image.width()]
strides = [[] for i in range(2)]
strides[0] = image.bytesPerLine()
bits = image.bits()
if image.format() == _qt.QImage.Format_Indexed8:
dtype = _np.uint8
strides[1] = 1
elif image.format() == _qt.QImage.Format_RGB32 \
or image.format() == _qt.QImage.Format_ARGB32 \
or image.format() == _qt.QImage.Format_ARGB32_Premultiplied:
dtype = _np.uint32
strides[1] = 4
elif image.format() == _qt.QImage.Format_Invalid:
raise ValueError(qimageview got invalid QImage)
else:
raise ValueError(qimageview can only handle 8- or 32-bit
QImages)
# creation of ndarray
obj = _np.ndarray(dims, _np.uint32, bits, 0, strides,
'C').view(cls)
obj._image = image
return obj
Don't do that. Use my recipe. You don't really want your RGBA tuples
interpreted as uint32s, do you?
from PySide import QtGui
import numpy as np
class QImageArray(object):
def __init__(self, qimage):
shape = (qimage.height(), qimage.width(), -1)
# Generate an ndarray from the image bits and steal its
# __array_interface__ information.
arr = np.frombuffer(qimage.bits(), dtype=np.uint8).reshape(shape)
self.__array_interface__ = arr.__array_interface__
# Keep the QImage alive.
self.qimage = qimage
def qimage2ndarray(qimage):
return np.asarray(QImageArray(qimage))
qimage =
QtGui.QImage('/Users/rkern/git/scipy/scipy/misc/tests/data/icon.png')
arr = qimage2ndarray(qimage)
del qimage
print arr
--
Robert Kern
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Getting masked array boundary indices
Thank you, Though it did not get what I expected it is a strong clue, let me explore it, With. Best regards Sudheer___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] xkcd on git commit messages.
Here http://xkcd.com/. Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] xkcd on git commit messages.
On Wed, Nov 27, 2013 at 11:56 AM, Charles R Harris charlesr.har...@gmail.com wrote: Here. And his later commit messages went straight to https://twitter.com/gitlost ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Silencing NumPy output
Hi,
After more investigation, I found that there already exist a way to
suppress those message on posix system. So I reused it in the PR. That
way, it was faster, but prevent change in that area. So there is less
change of breaking other syste:
https://github.com/numpy/numpy/pull/4081
But it remove the stdout when we run this command:
numpy.distutils.system_info.get_info(blas_opt)
But during compilation, we still have the info about what is found:
atlas_blas_threads_info:
Setting PTATLAS=ATLAS
Setting PTATLAS=ATLAS
customize Gnu95FCompiler
Found executable /usr/bin/gfortran
customize Gnu95FCompiler
customize Gnu95FCompiler using config
compiling '_configtest.c':
/* This file is generated from numpy/distutils/system_info.py */
void ATL_buildinfo(void);
int main(void) {
ATL_buildinfo();
return 0;
}
C compiler: gcc -pthread -fno-strict-aliasing -DNDEBUG -g -O2 -fPIC
compile options: '-c'
gcc: _configtest.c
gcc -pthread _configtest.o -L/usr/lib64/atlas -lptf77blas -lptcblas
-latlas -o _configtest
success!
removing: _configtest.c _configtest.o _configtest
Setting PTATLAS=ATLAS
FOUND:
libraries = ['ptf77blas', 'ptcblas', 'atlas']
library_dirs = ['/usr/lib64/atlas']
language = c
define_macros = [('ATLAS_INFO', '\\3.8.3\\')]
include_dirs = ['/usr/include']
FOUND:
libraries = ['ptf77blas', 'ptcblas', 'atlas']
library_dirs = ['/usr/lib64/atlas']
language = c
define_macros = [('ATLAS_INFO', '\\3.8.3\\')]
include_dirs = ['/usr/include']
non-existing path in 'numpy/lib': 'benchmarks'
lapack_opt_info:
lapack_mkl_info:
mkl_info:
libraries mkl,vml,guide not found in
['/opt/lisa/os_v2/common/Canopy_64bit/User/lib', '/usr/local/lib64',
'/usr/local/lib', '/usr/lib64', '/usr/lib']
NOT AVAILABLE
NOT AVAILABLE
atlas_threads_info:
Setting PTATLAS=ATLAS
libraries ptf77blas,ptcblas,atlas not found in
/opt/lisa/os_v2/common/Canopy_64bit/User/lib
libraries lapack_atlas not found in
/opt/lisa/os_v2/common/Canopy_64bit/User/lib
libraries ptf77blas,ptcblas,atlas not found in /usr/local/lib64
libraries lapack_atlas not found in /usr/local/lib64
libraries ptf77blas,ptcblas,atlas not found in /usr/local/lib
libraries lapack_atlas not found in /usr/local/lib
libraries lapack_atlas not found in /usr/lib64/atlas
numpy.distutils.system_info.atlas_threads_info
Setting PTATLAS=ATLAS
Setting PTATLAS=ATLAS
FOUND:
libraries = ['lapack', 'ptf77blas', 'ptcblas', 'atlas']
library_dirs = ['/usr/lib64/atlas']
language = f77
define_macros = [('ATLAS_INFO', '\\3.8.3\\')]
include_dirs = ['/usr/include']
FOUND:
libraries = ['lapack', 'ptf77blas', 'ptcblas', 'atlas']
library_dirs = ['/usr/lib64/atlas']
language = f77
define_macros = [('ATLAS_INFO', '\\3.8.3\\')]
include_dirs = ['/usr/include']
Frédéric
On Fri, Nov 22, 2013 at 4:26 PM, Frédéric Bastien no...@nouiz.org wrote:
I didn't forgot this, but I got side tracked. Here is the Theano code
I would like to try to use to replace os.system:
https://github.com/Theano/Theano/blob/master/theano/misc/windows.py
But I won't be able to try this before next week.
Fred
On Fri, Nov 15, 2013 at 5:49 PM, David Cournapeau courn...@gmail.com wrote:
On Fri, Nov 15, 2013 at 7:41 PM, Robert Kern robert.k...@gmail.com wrote:
On Fri, Nov 15, 2013 at 7:28 PM, David Cournapeau courn...@gmail.com
wrote:
On Fri, Nov 15, 2013 at 6:21 PM, Charles R Harris
charlesr.har...@gmail.com wrote:
Sure, give it a shot. Looks like subprocess.Popen was intended to
replace os.system in any case.
Except that output is not 'real time' with straight Popen, and doing so
reliably on every platform (cough - windows - cough) is not completely
trivial. You also have to handle buffered output, etc... That code is very
fragile, so this would be quite a lot of testing to change, and I am not
sure it worths it.
It doesn't have to be real time. Just use .communicate() and print out
the stdout and stderr to their appropriate streams after the subprocess
finishes.
Indeed, it does not have to be, but that's useful for debugging compilation
issues (not so much for numpy itself, but for some packages which have files
that takes a very long time to build, like scipy.sparsetools or bottleneck).
That's a minor point compared to the potential issues when building on
windows, though.
David
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] numpy datetime64 NaT string conversion bug patch
If you convert an array of strings to datetime64s and 'NaT' (or one of
its variants) appears in the string, all subsequent values are
rendered as NaT:
(this is in 1.7.1 but the problem is present in current dev version as well)
import numpy as np
a = np.array(['2010', 'nat', '2030'])
a.astype(np.datetime64)
array(['2010', 'NaT', 'NaT'], dtype='datetime64[Y]')
The fix is to re-initalize 'dt' inside the loop in
_strided_to_strided_string_to_datetime
(patch attached)
Correct behavior (with patch):
import numpy as np
a=np.array(['2010', 'nat', '2020'])
a.astype(np.datetime64)
array(['2010', 'NaT', '2020'], dtype='datetime64[Y]')
diff --git a/numpy/core/src/multiarray/dtype_transfer.c b/numpy/core/src/multiarray/dtype_transfer.c
index f758139..3bd362c 100644
--- a/numpy/core/src/multiarray/dtype_transfer.c
+++ b/numpy/core/src/multiarray/dtype_transfer.c
@@ -884,12 +884,13 @@ _strided_to_strided_string_to_datetime(char *dst, npy_intp dst_stride,
NpyAuxData *data)
{
_strided_datetime_cast_data *d = (_strided_datetime_cast_data *)data;
-npy_int64 dt = ~NPY_DATETIME_NAT;
npy_datetimestruct dts;
char *tmp_buffer = d-tmp_buffer;
char *tmp;
while (N 0) {
+npy_int64 dt = ~NPY_DATETIME_NAT;
+
/* Replicating strnlen with memchr, because Mac OS X lacks it */
tmp = memchr(src, '\0', src_itemsize);
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] (no subject)
Hi, Thanks both - very helpful, Matthew On 11/22/13, Robert Kern robert.k...@gmail.com wrote: On Fri, Nov 22, 2013 at 9:23 PM, Matthew Brett matthew.br...@gmail.com wrote: Hi, I'm sorry if I missed something obvious - but is there a vectorized way to look for None in an array? In [3]: a = np.array([1, 1]) In [4]: a == object() Out[4]: array([False, False], dtype=bool) In [6]: a == None Out[6]: False [~] |1 x = np.array([1, None, 2], dtype=object) [~] |2 np.equal(x, None) array([False, True, False], dtype=bool) -- Robert Kern ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] sum of array for masked area
Hi All,
I just posted this on the SciPy forum but realised it might be more
appropriate here?
I have a separate text file for daily rainfall data that covers the whole
country. I would like to calculate the monthly mean, min, max and the mean
of the sum for one state.
The mean, max and min are just the mean, max and min for all data in that
month however the sum data needs to work out the total for the month across
the array and then sum that value.
I use gdal tools to mask out the rest of the country and I use numpy tools
for the summary stats.
I can get the max, min and mean for the state, but the mean of the sum
keeps giving me a result for the whole country rather than just the state,
even though I am performing the analysis on the state only data. I am not
sure if this is a masking issue or a numpy calculation issue. The mask
works fine for the other summary statistics.
Any feedback will be greatly appreciated!
import numpy as np
import matplotlib.pyplot as plt
from numpy import ma as MA
from mpl_toolkits.basemap import Basemap
from datetime import datetime as dt
from datetime import timedelta
import os
from StringIO import StringIO
from osgeo import gdal, gdalnumeric, ogr, osr
import glob
import matplotlib.dates as mdates
import sys
shapefile=r/Users/state.shp
## Create masked array from shapefile
xmin,ymin,xmax,ymax=[111.975,-
9.975, 156.275,-44.525]
ncols,nrows=[886, 691] #Your rows/cols
maskvalue = 1
xres=(xmax-xmin)/float(ncols)
yres=(ymax-ymin)/float(nrows)
geotransform=(xmin,xres,0,ymax,0, -yres)
0
src_ds = ogr.Open(shapefile)
src_lyr=src_ds.GetLayer()
dst_ds = gdal.GetDriverByName('MEM').Create('',ncols, nrows, 1
,gdal.GDT_Byte)
dst_rb = dst_ds.GetRasterBand(1)
dst_rb.Fill(0) #initialise raster with zeros
dst_rb.SetNoDataValue(0)
dst_ds.SetGeoTransform(geotransform)
err = gdal.RasterizeLayer(dst_ds, [maskvalue], src_lyr)
dst_ds.FlushCache()
mask_arr=dst_ds.GetRasterBand(1).ReadAsArray()
np.set_printoptions(threshold='nan')
mask_arr[mask_arr == 255] = 1
newmask=MA.masked_equal(mask_arr,0)
### calculate monthly summary stats for state Only
rainmax=[]
rainmin=[]
rainmean=[]
rainsum=[]
yearmonthlist=[]
yearmonth_int=[]
errors=[]
OutputFolder=r/outputfolder
GLOBTEMPLATE =
r/daily-rainfall/combined/rainfall-{year}/r{year}{month:02}??.txt
def accumulate_month(year, month):
files = glob.glob(GLOBTEMPLATE.format(year=year, month=month))
monthlyrain=[]
for ifile in files:
try:
f=np.genfromtxt(ifile,skip_header=6)
except:
print ERROR with file:, ifile
errors.append(ifile)
f=np.flipud(f)
stateonly_f=np.ma.masked_array(f, mask=newmask.mask) # this masks
data to state
print stateonly_f:, stateonly_f.max(), stateonly_f.mean(),
stateonly_f.sum()
monthlyrain.append(stateonly_f)
yearmonth=dt(year,month,1)
yearmonthlist.append(yearmonth)
yearmonthint=str(year)+str(month)
d=dt.strptime(yearmonthint, '%Y%m')
print d
date_string=dt.strftime(d,'%Y%m')
yearmonthint=int(date_string)
yearmonth_int.append(yearmonthint)
r_sum=np.sum(monthlyrain, axis=0)
r_mean_of_sum=MA.mean(r_sum)
r_max, r_mean, r_min=MA.max(monthlyrain), MA.mean(monthlyrain),
MA.min(monthlyrain)
rainmax.append(r_max)
rainmean.append(r_mean)
rainmin.append(r_min)
rainsum.append(r_mean_of_sum)
print state only:, yearmonthint,r_max, r_mean, r_min, r_mean_of_sum
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
