Re: [Numpy-discussion] [Numpy-svn] r8457 - trunk

[Stéfan van der Walt]

On 6 June 2010 22:49, David  wrote:
> On 06/07/2010 02:36 PM, Stéfan van der Walt wrote:
>> I guess this changeset is up for discussion, but I'd be very glad if
>> we could track the .gitignore.
>
> I don't think we should. It is easy to set it up by yourself, and it may
> hide things that some people may want to see - different people may want
> to hide different things.

That doesn't seem to stop us from using svn-ignore?

S.
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] [Numpy-svn] r8457 - trunk

[David]

On 06/07/2010 02:36 PM, Stéfan van der Walt wrote:
> I guess this changeset is up for discussion, but I'd be very glad if
> we could track the .gitignore.

I don't think we should. It is easy to set it up by yourself, and it may 
hide things that some people may want to see - different people may want 
to hide different things.

cheers,

David
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] [Numpy-svn] r8457 - trunk

[Stéfan van der Walt]

I guess this changeset is up for discussion, but I'd be very glad if
we could track the .gitignore.  This makes life a lot easier for
everybody using git-svn.

Cheers
Stéfan

On 6 June 2010 20:43,   wrote:
> Author: charris
> Date: 2010-06-06 22:43:07 -0500 (Sun, 06 Jun 2010)
> New Revision: 8457
>
> Removed:
>   trunk/.gitignore
> Log:
> Don't track .gitignore file.
>
> Deleted: trunk/.gitignore
> ===
> --- trunk/.gitignore    2010-06-05 15:02:50 UTC (rev 8456)
> +++ trunk/.gitignore    2010-06-07 03:43:07 UTC (rev 8457)
> @@ -1,13 +0,0 @@
> -*.pyc
> -*.so
> -*~
> -\#*
> -__*
> -
> -build
> -/dist
> -/numpy.egg-inf
> -/numpy/version.py
> -
> -/tools/win32build/misc/msvcrt90/msvcr90.def
> -/doc/source/reference/generated
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Bug in nanmin called with unsigned integers

[Charles R Harris]

On Wed, May 26, 2010 at 7:59 AM, Tony S Yu  wrote:

>
> On May 25, 2010, at 10:57 PM, Charles R Harris wrote:
>
>
>
> On Tue, May 25, 2010 at 8:21 PM, Tony S Yu  wrote:
>
>> I got bit again by this bug with unsigned 
>> integers.
>> (My original changes got overwritten when I updated from svn and,
>> unfortunately, merged conflicts without actually looking over the changes.)
>>
>> In any case, I thought it'd be a good time to bump the issue (with 
>> patch
>> ).
>>
>> Cheers,
>> -Tony
>>
>> PS: Just for context, this issue comes up when displaying images with
>> Chaco (which converts images to unsigned integer arrays and calls nanmin).
>>
>>
> Fixed in r8445. Please add some tests.
>
>
>
> I'm not totally sure what's appropriate to test, so I just added a simple
> test to the comments for the 
> ticket
> .
>
> On a side note, I noticed that all the nan-ops degenerate to their
> non-nan-ops counterparts (i.e. nanmin --> min) when called with integer
> dtypes. Below is a diff where that's made a little more obvious by returning
> early for integer dtypes.
>
> Cheers,
> -Tony
>
>
> Index: numpy/lib/function_base.py
> ===
> --- numpy/lib/function_base.py (revision 8445)
> +++ numpy/lib/function_base.py (working copy)
> @@ -1295,15 +1295,15 @@
>
>  """
>  y = array(a, subok=True)
> -mask = isnan(a)
>
>  # We only need to take care of NaN's in floating point arrays
> -if not np.issubdtype(y.dtype, np.integer):
> -# y[mask] = fill
> -# We can't use fancy indexing here as it'll mess w/ MaskedArrays
> -# Instead, let's fill the array directly...
> -np.putmask(y, mask, fill)
> -
> +if np.issubdtype(y.dtype, np.integer):
> +return op(y, axis=axis)
> +mask = isnan(a)
> +# y[mask] = fill
> +# We can't use fancy indexing here as it'll mess w/ MaskedArrays
> +# Instead, let's fill the array directly...
> +np.putmask(y, mask, fill)
>  res = op(y, axis=axis)
>  mask_all_along_axis = mask.all(axis=axis)
>
>
>
Applied. Thanks.
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Technicalities of the SVN -> GIT transition

[David]

On 06/05/2010 11:43 PM, Pauli Virtanen wrote:
> Fri, 04 Jun 2010 15:28:52 -0700, Matthew Brett wrote:
 I think it should be opt-in.  How would opt-out work?  Would someone
 create new accounts for all the contributors and then give them
 access?
>>>
>>> Just to be clear, this has nothing to do with accounts on github, or
>>> any registered thing. This is *only* about username/email as recognized
>>> by git itself (as recorded in the commit objects).
>>
>> Actually - isn't it better if people do give you their github username /
>> email combo - assuming that's the easiest combo to work with later?
>
> I think the Github user name is not really needed here, as what goes into
> the history is the Git ID: name + email address.

Indeed. IOW, the output of

git config user.name
git config user.email

if you already use git is all that I need,

David
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] "Dynamic convolution" in Numpy

[David]

On 06/07/2010 12:08 AM, Anne Archibald wrote:

>
> I think the kicker is here: what is the right way to interpolate
> between filters?

There is no right way that I know of, it really depends on what you are 
doing. The big issue here is that a filter which changes is obviously 
not time independent anymore. For IIR, this has the unfortunate 
consequence that even if your filter is stable at every interpolation 
point, transitioning from one to the other can still blow up.

For filters used in music processing, it is common to have filters 
changing really fast, for example in synthesizers (up to a few hundred 
times / sec) - and interesting effects are obtained for filters with 
very high and concentrated resonance. The solution is usually a mix of 
upsampling to avoid big transitions and using filter representations 
which are more stable (state-based representation instead of direct 
filters coefficients, for example).

I don't think it is directly applicable to your problem, but the series 
of papers by Dattorro ten years ago is a goldmine:

"Effect Design Part 1|2|3, Jon Dattorro, J. Audio Eng. Soc., Vol 45, No. 
9, 1997 September"

> As far as convolution, as David says, take a look at existing
> algorithms and maybe even music software - there's a trade-off between
> the n^2 computation of a brute-force FIR filter and the delay
> introduced by an FFT approach, but on-the-fly convolution is a
> well-studied problem.

It may be pretty cool to have an implementation of scipy, now that I 
think of it :) One issue is that there are several patents on those 
techniques, but I was told there are ways around it in term of 
implementation. Not sure how to proceed here for scipy,

David
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] memory usage question

[Eric Firing]

On 06/06/2010 02:17 PM, Tom Kuiper wrote:
> Greetings all.
>
> I have a feeling that, coming at this with a background in FORTRAN and
> C, I'm missing some subtlety, possibly of an OO nature.   Basically, I'm
> looping over very large data arrays and memory usage just keeps growing
> even though I re-use the arrays.  Below is a stripped down version of
> what I'm doing.  You'll recognize it as gulping a great quantity of data
> (1 million complex samples), Fourier transforming these by 1000 sample
> blocks into spectra, co-adding the spectra, and doing this 255 times,
> for a grand 1000 point total spectrum.  At iteration 108 of the outer
> loop, I get a memory error.  By then, according to 'top', ipython (or
> python) is using around 85% of 3.5 GB of memory.
>
>  P = zeros(fft_size)
>nsecs = 255
>fft_size = 1000
>for i in range(nsecs):
>  header,data = get_raw_record(fd_in)
>  num_bytes = len(data)
>  label, reclen, recver, softver, spcid, vsrid, schanid,
> bits_per_sample, \
>  ksamps_per_sec, sdplr, prdx_dss_id, prdx_sc_id, prdx_pass_num, \
>  prdx_uplink_band,prdx_downlink_band, trk_mode, uplink_dss_id,
> ddc_lo, \
>  rf_to_if_lo, data_error, year, doy, sec, data_time_offset, frov,
> fro, \
>  frr, sfro,rf_freq, schan_accum_phase, (scpp0,scpp1,scpp2,scpp3), \
>  schan_label = header
>  # ksamp_per_sec = 1e3, number of complex samples in 'data' = 1e6
>  num_32bit_words = len(data)*8/BITS_PER_32BIT_WORD
>  cmplx_samp_per_word = (BITS_PER_32BIT_WORD/(2*bits_per_sample))
>  cmplx_samples =
> unpack_vdr_data(num_32bit_words,cmplx_samp_per_word,data)
>  del(data) # This makes no difference
>  for j in range(0,ksamps_per_sec*1000/fft_size):
>index = int(j*fft_size)
>S = fft(cmplx_samples[index:index+fft_size])
>P += S*conjugate(S)
>  del(cmplx_samples) # This makes no difference
>if (i % 20) == 0:
>  gc.collect(0) # This makes no difference
>P /= nsecs
>sample_period = 1./ksamps_per_sec # kHz
>f = fftfreq(fft_size, d=sample_period)
>
> What am I missing?

I don't know, but I would suggest that you strip the example down 
further: instead of reading data from a file, use numpy.random.randn to 
generate fake data as needed.  In other words, use only numpy 
functions--no readers, no unpackers.  Put this minimal script into a 
file and run it from the command line, not in ipython.  (Have you 
verified that you get the same result running a standalone script from 
the command line as running from ipython?)  Put a memory-monitoring step 
inside, maybe at each outer loop iteration.  You can use the 
matplotlib.cbook.report_memory function or similar:

def report_memory(i=0):  # argument may go away
 'return the memory consumed by process'
 from subprocess import Popen, PIPE
 pid = os.getpid()
 if sys.platform=='sunos5':
 a2 = Popen('ps -p %d -o osz' % pid, shell=True,
 stdout=PIPE).stdout.readlines()
 mem = int(a2[-1].strip())
 elif sys.platform.startswith('linux'):
 a2 = Popen('ps -p %d -o rss,sz' % pid, shell=True,
 stdout=PIPE).stdout.readlines()
 mem = int(a2[1].split()[1])
 elif sys.platform.startswith('darwin'):
 a2 = Popen('ps -p %d -o rss,vsz' % pid, shell=True,
 stdout=PIPE).stdout.readlines()
 mem = int(a2[1].split()[0])

 return mem

I'm suspecting the problem may be in your data reader and/or unpacker, 
not in the application of numpy functions.  Also, ipython can confuse 
the issue by keeping references to objects.  In any case, with a simpler 
test script and regular memory monitoring, it should be easier for you 
to track down the problem.

Eric


>
> Best regards
>
> Tom
>
> p.s.  Many of you will see this twice, for which I apologize.
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] memory usage question

[Tom Kuiper]

Greetings all.

I have a feeling that, coming at this with a background in FORTRAN and 
C, I'm missing some subtlety, possibly of an OO nature.   Basically, I'm 
looping over very large data arrays and memory usage just keeps growing 
even though I re-use the arrays.  Below is a stripped down version of 
what I'm doing.  You'll recognize it as gulping a great quantity of data 
(1 million complex samples), Fourier transforming these by 1000 sample 
blocks into spectra, co-adding the spectra, and doing this 255 times, 
for a grand 1000 point total spectrum.  At iteration 108 of the outer 
loop, I get a memory error.  By then, according to 'top', ipython (or 
python) is using around 85% of 3.5 GB of memory.

P = zeros(fft_size)
  nsecs = 255
  fft_size = 1000
  for i in range(nsecs):
header,data = get_raw_record(fd_in)
num_bytes = len(data)
label, reclen, recver, softver, spcid, vsrid, schanid, 
bits_per_sample, \
ksamps_per_sec, sdplr, prdx_dss_id, prdx_sc_id, prdx_pass_num, \
prdx_uplink_band,prdx_downlink_band, trk_mode, uplink_dss_id, 
ddc_lo, \
rf_to_if_lo, data_error, year, doy, sec, data_time_offset, frov, 
fro, \
frr, sfro,rf_freq, schan_accum_phase, (scpp0,scpp1,scpp2,scpp3), \
schan_label = header
# ksamp_per_sec = 1e3, number of complex samples in 'data' = 1e6
num_32bit_words = len(data)*8/BITS_PER_32BIT_WORD
cmplx_samp_per_word = (BITS_PER_32BIT_WORD/(2*bits_per_sample))
cmplx_samples = 
unpack_vdr_data(num_32bit_words,cmplx_samp_per_word,data)
del(data) # This makes no difference
for j in range(0,ksamps_per_sec*1000/fft_size):
  index = int(j*fft_size)
  S = fft(cmplx_samples[index:index+fft_size])
  P += S*conjugate(S)
del(cmplx_samples) # This makes no difference
  if (i % 20) == 0:
gc.collect(0) # This makes no difference
  P /= nsecs
  sample_period = 1./ksamps_per_sec # kHz
  f = fftfreq(fft_size, d=sample_period)

What am I missing?

Best regards

Tom

p.s.  Many of you will see this twice, for which I apologize.
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] How to resize numpy.memmap?

[Pearu Peterson]

Hi again,
To answer to the second part of my question, here follows an example
demonstrating how to "resize" a memmap:

>>> fp = numpy.memmap('test.dat', shape=(10,), mode='w+')
>>> fp._mmap.resize(11)
>>> cp = numpy.ndarray.__new__(numpy.memmap, (fp._mmap.size(),), 
>>> dtype=fp.dtype, buffer=fp._mmap, offset=0, order='C')
>>> cp[-1] = 99
>>> cp[1] = 33
>>> cp
memmap([ 0, 33,  0,  0,  0,  0,  0,  0,  0,  0, 99], dtype=uint8)
>>> fp
memmap([ 0, 33,  0,  0,  0,  0,  0,  0,  0,  0], dtype=uint8)
>>> del fp, cp
>>> fp = numpy.memmap('test.dat', mode='r')
>>> fp
memmap([ 0, 33,  0,  0,  0,  0,  0,  0,  0,  0, 99], dtype=uint8)

Would there be any interest in turning the above code to numpy.memmap
method, say, to resized(newshape)? For example, for resolving the original
problem, one could have

fp = numpy.memmap('test.dat', shape=(10,), mode='w+')
fp = fp.resized(11)

Regards,
Pearu

On Sun, Jun 6, 2010 at 10:19 PM, Pearu Peterson
 wrote:
> Hi,
>
> I am creating a rather large file (typically 100MBi-1GBi) with numpy.memmap
> but in some cases the initial estimate to the file size is just few
> bytes too small.
> So, I was trying to resize the memmap with a failure as demonstrated
> with the following
> example:
>
 fp = numpy.memmap('test.dat', shape=(10,), mode='w+')
 fp.resize(11, refcheck=False)
> ...
> ValueError: cannot resize this array:  it does not own its data
>
> My question is, is there a way to "fix" this or may be there exist some other
> technique to resize memmap. I have tried resizing memmap's _mmap attribute
> directly:
>
 fp._mmap.resize(11)
 fp._mmap.size()
>    11
>
> but the size of memmap instance remains unchanged:
 fp.size
>    10
>
> Thanks,
> Pearu
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] How to resize numpy.memmap?

[Pearu Peterson]

Hi,

I am creating a rather large file (typically 100MBi-1GBi) with numpy.memmap
but in some cases the initial estimate to the file size is just few
bytes too small.
So, I was trying to resize the memmap with a failure as demonstrated
with the following
example:

>>> fp = numpy.memmap('test.dat', shape=(10,), mode='w+')
>>> fp.resize(11, refcheck=False)
...
ValueError: cannot resize this array:  it does not own its data

My question is, is there a way to "fix" this or may be there exist some other
technique to resize memmap. I have tried resizing memmap's _mmap attribute
directly:

>>> fp._mmap.resize(11)
>>> fp._mmap.size()
11

but the size of memmap instance remains unchanged:
>>> fp.size
10

Thanks,
Pearu
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] "Dynamic convolution" in Numpy

[Anne Archibald]

On 6 June 2010 04:44, David Cournapeau  wrote:
> On Thu, Jun 3, 2010 at 7:49 PM, arthur de conihout
>  wrote:
>
>> I don't know if i made myself very clear.
>> if anyone has suggestions or has already operated a dynamic filtering i
>> would be well interested.
>
> Does fade-in/fade-out actually works ? I would have thought that it
> had killed the properties of your filters ?
>
> There are two issues:
>  - how to do convolution fast
>  - how to go from one filter to the other

I think the kicker is here: what is the right way to interpolate
between filters?

If you have, or can generate, a lot of filters, then at least you can
evaluate the quality of the interpolation.

The right way to understand what kind of interpolation to is to have
some idea of the physics you're dealing with. In this case, as I
understand it, you're dealing with the auditory effects of the head,
seen from different angles. I would say that the ear senses something
integrated power in logarithmically-spaced frequency bins over roughly
twentieth-of-a-second time intervals. So the relevant effects you care
about are amplitude absorption and time delays, if they are as long as
a twentieth of a second. Doing simple linear interpolation,
unfortunately, will probably get you in trouble - imagine, for
example, that two impulse responses have the same amplitude at 440 Hz
but different phases. A linear interpolation will change the amplitude
(for example if they're 180 degrees apart it'll pass through zero).
You might do better with interpolating in polar coordinates, though
you might have phase wrapping issues. A really thorough approach might
be to take a granular-synthesis approach to the impulse responses,
breaking them up into orthogonal time-domain channels within which the
response is defined by an amplitude and a time delay, which you'd
interpolate in the natural way. I'd try polar interpolation on the
FFTs of the amplitudes first, though (since in fact it's the same
thing with the minimal possible frequency channels).

I suspect that some interpolation, however unrealistic (even linear
interpolation) is necessary or listeners may perceive sounds
"snapping" from place to place in the aural field.

> The main issue with changing filters is that your system is not LTI
> anymore. If your filters have finite impulse answer, I guess it should
> not be too much of an issue. To do convolution quickly, you need to
> use FFT, which is a bit tricky if you want to do things in real-time,
> as you need to partition the impulse response. Using "partitioned
> impulse answer" as keywords should give you plenty of references on
> how to do it,

As far as convolution, as David says, take a look at existing
algorithms and maybe even music software - there's a trade-off between
the n^2 computation of a brute-force FIR filter and the delay
introduced by an FFT approach, but on-the-fly convolution is a
well-studied problem.

Anne

> David
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] "Dynamic convolution" in Numpy

[David Cournapeau]

On Thu, Jun 3, 2010 at 7:49 PM, arthur de conihout
 wrote:

> I don't know if i made myself very clear.
> if anyone has suggestions or has already operated a dynamic filtering i
> would be well interested.

Does fade-in/fade-out actually works ? I would have thought that it
had killed the properties of your filters ?

There are two issues:
 - how to do convolution fast
 - how to go from one filter to the other

The main issue with changing filters is that your system is not LTI
anymore. If your filters have finite impulse answer, I guess it should
not be too much of an issue. To do convolution quickly, you need to
use FFT, which is a bit tricky if you want to do things in real-time,
as you need to partition the impulse response. Using "partitioned
impulse answer" as keywords should give you plenty of references on
how to do it,

David
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion