Re: [Numpy-discussion] Problem with correlate
On Tue, Jun 2, 2009 at 10:56 PM, Ryan May rma...@gmail.com wrote: On Tue, Jun 2, 2009 at 5:59 AM, David Cournapeau da...@ar.media.kyoto-u.ac.jp wrote: Robin wrote: On Tue, Jun 2, 2009 at 11:36 AM, David Cournapeau courn...@gmail.com wrote: Done in r7031 - correlate/PyArray_Correlate should be unchanged, and acorrelate/PyArray_Acorrelate implement the conventional definitions, I don't know if it's been discussed before but while people are thinking about/changing correlate I thought I'd like to request as a user a matlab style xcorr function (basically with the functionality of the matlab version). I don't know if this is a deliberate emission, but it is often one of the first things my colleagues try when I get them using Python, and as far as I know there isn't really a good answer. There is xcorr in pylab, but it isn't vectorised like xcorr from matlab... There is one in the talkbox scikit: http://github.com/cournape/talkbox/blob/202135a9d848931ebd036b97302f1e10d7488c63/scikits/talkbox/tools/correlations.py It uses the fft, and bonus point, the file is independent of the rest of toolbox. There is another version which uses direct implementation (this is faster if you need only a few lags, and it takes less memory too). I'd be +1 on including something like this (provided it expanded to include complex-valued data). I think it's a real need, since everyone seems to keep rolling their own. I had to write my own just so that I can calculate a few lags in a vectorized fashion. The code in talkbox is not good enough for scipy. I made an attempt for scipy.signal here: http://github.com/cournape/scipy3/blob/b004d17d824f1c03921d9663207ee40adadc5762/scipy/signal/correlations.py It is reasonably fast when only a few lags are needed, both double and complex double are supported, and it works on arbitrary axis and lags. Other precisions should be easy to add, but I think I need to extend the numpy code generators to support cython sources to avoid code duplication. Does that fill your need ? cheers, David ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Problem with correlate
On Thu, Jun 4, 2009 at 5:14 AM, David Cournapeau courn...@gmail.com wrote: On Tue, Jun 2, 2009 at 10:56 PM, Ryan May rma...@gmail.com wrote: On Tue, Jun 2, 2009 at 5:59 AM, David Cournapeau da...@ar.media.kyoto-u.ac.jp wrote: Robin wrote: On Tue, Jun 2, 2009 at 11:36 AM, David Cournapeau courn...@gmail.com wrote: Done in r7031 - correlate/PyArray_Correlate should be unchanged, and acorrelate/PyArray_Acorrelate implement the conventional definitions, I don't know if it's been discussed before but while people are thinking about/changing correlate I thought I'd like to request as a user a matlab style xcorr function (basically with the functionality of the matlab version). I don't know if this is a deliberate emission, but it is often one of the first things my colleagues try when I get them using Python, and as far as I know there isn't really a good answer. There is xcorr in pylab, but it isn't vectorised like xcorr from matlab... There is one in the talkbox scikit: http://github.com/cournape/talkbox/blob/202135a9d848931ebd036b97302f1e10d7488c63/scikits/talkbox/tools/correlations.py It uses the fft, and bonus point, the file is independent of the rest of toolbox. There is another version which uses direct implementation (this is faster if you need only a few lags, and it takes less memory too). I'd be +1 on including something like this (provided it expanded to include complex-valued data). I think it's a real need, since everyone seems to keep rolling their own. I had to write my own just so that I can calculate a few lags in a vectorized fashion. The code in talkbox is not good enough for scipy. I made an attempt for scipy.signal here: http://github.com/cournape/scipy3/blob/b004d17d824f1c03921d9663207ee40adadc5762/scipy/signal/correlations.py It is reasonably fast when only a few lags are needed, both double and complex double are supported, and it works on arbitrary axis and lags. Other precisions should be easy to add, but I think I need to extend the numpy code generators to support cython sources to avoid code duplication. Does that fill your need ? It would fill mine. Would it make sense to make y default to x, so that you can use xcorr to do the autocorrelation as: xcorr(x) ? Ryan -- Ryan May Graduate Research Assistant School of Meteorology University of Oklahoma ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Problem with correlate
I wonder if xcorrelate would be a better name than acorrelate? I think it would. ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Problem with correlate
On Tue, Jun 2, 2009 at 12:37 PM, David Cournapeau da...@ar.media.kyoto-u.ac.jp wrote: Robert Kern wrote: This does not solve the C function problem (PyArray_Correlate). The easy solution would be to keep the current C version, deal with the problem in python for acorrelate for the time being, and replace the old C function with the 'correct' one once we remove the deprecated correlate ? No, you do the same thing at the C level. Done in r7031 - correlate/PyArray_Correlate should be unchanged, and acorrelate/PyArray_Acorrelate implement the conventional definitions, David ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Problem with correlate
On Tue, Jun 2, 2009 at 11:36 AM, David Cournapeau courn...@gmail.com wrote: Done in r7031 - correlate/PyArray_Correlate should be unchanged, and acorrelate/PyArray_Acorrelate implement the conventional definitions, I don't know if it's been discussed before but while people are thinking about/changing correlate I thought I'd like to request as a user a matlab style xcorr function (basically with the functionality of the matlab version). I don't know if this is a deliberate emission, but it is often one of the first things my colleagues try when I get them using Python, and as far as I know there isn't really a good answer. There is xcorr in pylab, but it isn't vectorised like xcorr from matlab... Cheers Robin ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Problem with correlate
Robin wrote: On Tue, Jun 2, 2009 at 11:36 AM, David Cournapeau courn...@gmail.com wrote: Done in r7031 - correlate/PyArray_Correlate should be unchanged, and acorrelate/PyArray_Acorrelate implement the conventional definitions, I don't know if it's been discussed before but while people are thinking about/changing correlate I thought I'd like to request as a user a matlab style xcorr function (basically with the functionality of the matlab version). I don't know if this is a deliberate emission, but it is often one of the first things my colleagues try when I get them using Python, and as far as I know there isn't really a good answer. There is xcorr in pylab, but it isn't vectorised like xcorr from matlab... There is one in the talkbox scikit: http://github.com/cournape/talkbox/blob/202135a9d848931ebd036b97302f1e10d7488c63/scikits/talkbox/tools/correlations.py It uses the fft, and bonus point, the file is independent of the rest of toolbox. There is another version which uses direct implementation (this is faster if you need only a few lags, and it takes less memory too). David ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Problem with correlate
On Tue, Jun 2, 2009 at 5:59 AM, David Cournapeau da...@ar.media.kyoto-u.ac.jp wrote: Robin wrote: On Tue, Jun 2, 2009 at 11:36 AM, David Cournapeau courn...@gmail.com wrote: Done in r7031 - correlate/PyArray_Correlate should be unchanged, and acorrelate/PyArray_Acorrelate implement the conventional definitions, I don't know if it's been discussed before but while people are thinking about/changing correlate I thought I'd like to request as a user a matlab style xcorr function (basically with the functionality of the matlab version). I don't know if this is a deliberate emission, but it is often one of the first things my colleagues try when I get them using Python, and as far as I know there isn't really a good answer. There is xcorr in pylab, but it isn't vectorised like xcorr from matlab... There is one in the talkbox scikit: http://github.com/cournape/talkbox/blob/202135a9d848931ebd036b97302f1e10d7488c63/scikits/talkbox/tools/correlations.py It uses the fft, and bonus point, the file is independent of the rest of toolbox. There is another version which uses direct implementation (this is faster if you need only a few lags, and it takes less memory too). I'd be +1 on including something like this (provided it expanded to include complex-valued data). I think it's a real need, since everyone seems to keep rolling their own. I had to write my own just so that I can calculate a few lags in a vectorized fashion. Ryan -- Ryan May Graduate Research Assistant School of Meteorology University of Oklahoma ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Problem with correlate
I also think that the conjugate should be taken. I spent the last few weeks using correlate to experiment with signal processing and I got strange results until I realised that I had to manually take the conjugate. It would also be good if the function did it since applying the conjugate to the wrong sequence yields the complex conjugate of the correlation. Who would want to use the correlation without the conjugate, if someone is only using real values it won't affect them, if they are using complex values they probably want to conjugate. One function that does depend on correlate though is convolution! Changes made to correlate will affect it! but I have understand that a new function acorrelate is being created instead of changing correlate? Otherwise I've never used matlab but it does seem like xcorr has some good features. The modes 'same' and 'valid' were initially quite confusing especially as the default is 'valid', meaning that autocorrelations lead to a single value by default! I also think that having weighting options would be good. I now understand the complexities of the various weightings that can be applied to the correlation i.e. biased vs unbiased but I think that having correlate include these options might prompt users to investigate which one they really needed. Correlate seemed so simple when I first used it but it took me ages to realise that these are choices to be made. regards Rob ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Problem with correlate
Charles R Harris wrote: I also think that having weighting options would be good. I now understand the complexities of the various weightings that can be applied to the correlation i.e. biased vs unbiased but I think that having correlate include these options might prompt users to investigate which one they really needed. Correlate seemed so simple when I first used it but it took me ages to realise that these are choices to be made. I wonder if xcorrelate would be a better name than acorrelate? It may be confusing, because what acorrelate currently does is not what xcorr does under matlab. Under matlab, xcorr computes the 1d cross correlation column-wise (would be axis wise under numpy). If I get the time today, I will try to implement a first version using generalized ufunc - that would be a good occasion to learn generalized ufunc. If that works well, maybe there is no need for the current acorrelate at all (but we should still keep PyArray_Acorrelate, I think). cheers, David ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Problem with correlate
On Mon, Jun 1, 2009 at 00:05, David Cournapeau da...@ar.media.kyoto-u.ac.jp wrote: I think we should just fix it to use conjugate - I will do this in the branch, and I will integrate it in the trunk later unless someone stands up vehemently against the change. I opened up a ticket to track this, though, It breaks everyone's code that works around the current behavior. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Problem with correlate
On Mon, Jun 1, 2009 at 11:48 AM, Charles R Harris charlesr.har...@gmail.com wrote: On Mon, Jun 1, 2009 at 10:35 AM, Robert Kern robert.k...@gmail.comwrote: On Mon, Jun 1, 2009 at 00:05, David Cournapeau da...@ar.media.kyoto-u.ac.jp wrote: I think we should just fix it to use conjugate - I will do this in the branch, and I will integrate it in the trunk later unless someone stands up vehemently against the change. I opened up a ticket to track this, though, It breaks everyone's code that works around the current behavior. Maybe we need a new function. But what to call it? How about introducing acorrelate and deprecating the old version? Chuck ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Problem with correlate
On Mon, Jun 1, 2009 at 13:44, Charles R Harris charlesr.har...@gmail.com wrote: On Mon, Jun 1, 2009 at 11:48 AM, Charles R Harris charlesr.har...@gmail.com wrote: On Mon, Jun 1, 2009 at 10:35 AM, Robert Kern robert.k...@gmail.com wrote: On Mon, Jun 1, 2009 at 00:05, David Cournapeau da...@ar.media.kyoto-u.ac.jp wrote: I think we should just fix it to use conjugate - I will do this in the branch, and I will integrate it in the trunk later unless someone stands up vehemently against the change. I opened up a ticket to track this, though, It breaks everyone's code that works around the current behavior. Maybe we need a new function. But what to call it? How about introducing acorrelate and deprecating the old version? Sure. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Problem with correlate
Charles R Harris wrote: On Mon, Jun 1, 2009 at 11:48 AM, Charles R Harris charlesr.har...@gmail.com mailto:charlesr.har...@gmail.com wrote: On Mon, Jun 1, 2009 at 10:35 AM, Robert Kern robert.k...@gmail.com mailto:robert.k...@gmail.com wrote: On Mon, Jun 1, 2009 at 00:05, David Cournapeau da...@ar.media.kyoto-u.ac.jp mailto:da...@ar.media.kyoto-u.ac.jp wrote: I think we should just fix it to use conjugate - I will do this in the branch, and I will integrate it in the trunk later unless someone stands up vehemently against the change. I opened up a ticket to track this, though, It breaks everyone's code that works around the current behavior. Maybe we need a new function. But what to call it? How about introducing acorrelate and deprecating the old version? This does not solve the C function problem (PyArray_Correlate). The easy solution would be to keep the current C version, deal with the problem in python for acorrelate for the time being, and replace the old C function with the 'correct' one once we remove the deprecated correlate ? David ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Problem with correlate
On Mon, Jun 1, 2009 at 22:33, David Cournapeau da...@ar.media.kyoto-u.ac.jp wrote: Charles R Harris wrote: On Mon, Jun 1, 2009 at 11:48 AM, Charles R Harris charlesr.har...@gmail.com mailto:charlesr.har...@gmail.com wrote: On Mon, Jun 1, 2009 at 10:35 AM, Robert Kern robert.k...@gmail.com mailto:robert.k...@gmail.com wrote: On Mon, Jun 1, 2009 at 00:05, David Cournapeau da...@ar.media.kyoto-u.ac.jp mailto:da...@ar.media.kyoto-u.ac.jp wrote: I think we should just fix it to use conjugate - I will do this in the branch, and I will integrate it in the trunk later unless someone stands up vehemently against the change. I opened up a ticket to track this, though, It breaks everyone's code that works around the current behavior. Maybe we need a new function. But what to call it? How about introducing acorrelate and deprecating the old version? This does not solve the C function problem (PyArray_Correlate). The easy solution would be to keep the current C version, deal with the problem in python for acorrelate for the time being, and replace the old C function with the 'correct' one once we remove the deprecated correlate ? No, you do the same thing at the C level. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Problem with correlate
Hi,
After my previous email, I have opened a ticket #1117 (correlate not order
dependent)
I have found that the correlate function is defined in multiarraymodule.c and
that inputs are being swapped using the following code
n1 = ap1-dimensions[0];
n2 = ap2-dimensions[0];
if (n1 n2) {
ret = ap1;
ap1 = ap2;
ap2 = ret;
ret = NULL;
i = n1;
n1 = n2;
n2 = i;
}
I do not know the code well enough to see whether this could just be removed (I
don't know c either).
Maybe the algorithmn requires the inputs to be length ordered? I will try to
work it out.
Regards
Rob
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Problem with correlate
Charles R Harris wrote:
On Sun, May 31, 2009 at 11:54 AM, rob steed rjst...@talk21.com
mailto:rjst...@talk21.com wrote:
Hi,
After my previous email, I have opened a ticket #1117 (correlate
not order dependent)
I have found that the correlate function is defined in
multiarraymodule.c and
that inputs are being swapped using the following code
n1 = ap1-dimensions[0];
n2 = ap2-dimensions[0];
if (n1 n2) {
ret = ap1;
ap1 = ap2;
ap2 = ret;
ret = NULL;
i = n1;
n1 = n2;
n2 = i;
}
I do not know the code well enough to see whether this could just
be removed (I don't know c either).
Maybe the algorithmn requires the inputs to be length ordered? I
will try to work it out.
If the correlation algorithm doesn't use an fft and is done
explicitly, then the maximum overlap for any shift is the length of
the shortest input. Swapping the arrays makes that logic easier to
implement, but it isn't necessary.
But this logic is also wrong if the swapping is not taken into account -
as the OP mentioned, correlate(a, b) is not equal to correlate(b, a) in
the general case. The output is reversed in the second case compared to
the first case.
cheers,
David
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Problem with correlate
On Sun, May 31, 2009 at 7:18 PM, David Cournapeau
da...@ar.media.kyoto-u.ac.jp wrote:
Charles R Harris wrote:
On Sun, May 31, 2009 at 11:54 AM, rob steed rjst...@talk21.com
mailto:rjst...@talk21.com wrote:
Hi,
After my previous email, I have opened a ticket #1117 (correlate
not order dependent)
I have found that the correlate function is defined in
multiarraymodule.c and
that inputs are being swapped using the following code
n1 = ap1-dimensions[0];
n2 = ap2-dimensions[0];
if (n1 n2) {
ret = ap1;
ap1 = ap2;
ap2 = ret;
ret = NULL;
i = n1;
n1 = n2;
n2 = i;
}
I do not know the code well enough to see whether this could just
be removed (I don't know c either).
Maybe the algorithmn requires the inputs to be length ordered? I
will try to work it out.
If the correlation algorithm doesn't use an fft and is done
explicitly, then the maximum overlap for any shift is the length of
the shortest input. Swapping the arrays makes that logic easier to
implement, but it isn't necessary.
But this logic is also wrong if the swapping is not taken into account -
as the OP mentioned, correlate(a, b) is not equal to correlate(b, a) in
the general case. The output is reversed in the second case compared to
the first case.
I didn't say it was *correctly* implemented ;)
Chuck
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Problem with correlate
Charles R Harris wrote:
On Sun, May 31, 2009 at 7:18 PM, David Cournapeau
da...@ar.media.kyoto-u.ac.jp mailto:da...@ar.media.kyoto-u.ac.jp
wrote:
Charles R Harris wrote:
On Sun, May 31, 2009 at 11:54 AM, rob steed rjst...@talk21.com
mailto:rjst...@talk21.com
mailto:rjst...@talk21.com mailto:rjst...@talk21.com wrote:
Hi,
After my previous email, I have opened a ticket #1117 (correlate
not order dependent)
I have found that the correlate function is defined in
multiarraymodule.c and
that inputs are being swapped using the following code
n1 = ap1-dimensions[0];
n2 = ap2-dimensions[0];
if (n1 n2) {
ret = ap1;
ap1 = ap2;
ap2 = ret;
ret = NULL;
i = n1;
n1 = n2;
n2 = i;
}
I do not know the code well enough to see whether this could
just
be removed (I don't know c either).
Maybe the algorithmn requires the inputs to be length ordered? I
will try to work it out.
If the correlation algorithm doesn't use an fft and is done
explicitly, then the maximum overlap for any shift is the length of
the shortest input. Swapping the arrays makes that logic easier to
implement, but it isn't necessary.
But this logic is also wrong if the swapping is not taken into
account -
as the OP mentioned, correlate(a, b) is not equal to correlate(b,
a) in
the general case. The output is reversed in the second case
compared to
the first case.
I didn't say it was *correctly* implemented ;)
:) So I gave it a shot
http://github.com/cournape/numpy/commits/fix_correlate
(It took me a while to realize that PyArray_ISFLEXIBLE returns false for
array object. Is this expected ? The documentation concerning copyswap
says that it is necessary for flexible arrays, but I think it is
necessary for object arrays as well).
It still bothers me that correlate does not conjugate the second
argument for complex arrays...
cheers,
David
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Problem with correlate
On Sun, May 31, 2009 at 9:08 PM, David Cournapeau
da...@ar.media.kyoto-u.ac.jp wrote:
Charles R Harris wrote:
On Sun, May 31, 2009 at 7:18 PM, David Cournapeau
da...@ar.media.kyoto-u.ac.jp mailto:da...@ar.media.kyoto-u.ac.jp
wrote:
Charles R Harris wrote:
On Sun, May 31, 2009 at 11:54 AM, rob steed rjst...@talk21.com
mailto:rjst...@talk21.com
mailto:rjst...@talk21.com mailto:rjst...@talk21.com wrote:
Hi,
After my previous email, I have opened a ticket #1117
(correlate
not order dependent)
I have found that the correlate function is defined in
multiarraymodule.c and
that inputs are being swapped using the following code
n1 = ap1-dimensions[0];
n2 = ap2-dimensions[0];
if (n1 n2) {
ret = ap1;
ap1 = ap2;
ap2 = ret;
ret = NULL;
i = n1;
n1 = n2;
n2 = i;
}
I do not know the code well enough to see whether this could
just
be removed (I don't know c either).
Maybe the algorithmn requires the inputs to be length ordered?
I
will try to work it out.
If the correlation algorithm doesn't use an fft and is done
explicitly, then the maximum overlap for any shift is the length of
the shortest input. Swapping the arrays makes that logic easier to
implement, but it isn't necessary.
But this logic is also wrong if the swapping is not taken into
account -
as the OP mentioned, correlate(a, b) is not equal to correlate(b,
a) in
the general case. The output is reversed in the second case
compared to
the first case.
I didn't say it was *correctly* implemented ;)
:) So I gave it a shot
http://github.com/cournape/numpy/commits/fix_correlate
(It took me a while to realize that PyArray_ISFLEXIBLE returns false for
array object. Is this expected ? The documentation concerning copyswap
says that it is necessary for flexible arrays, but I think it is
necessary for object arrays as well).
Don't know. PyArray_ISFLEXIBLE looks like a macro...
#define PyArray_ISFLEXIBLE(obj) PyTypeNum_ISFLEXIBLE(PyArray_TYPE(obj))
#define PyTypeNum_ISFLEXIBLE(type) (((type) =NPY_STRING) \
((type) =NPY_VOID))
And the typecodes are '?bhilqpBHILQPfdgFDGSUVO'. So 'SUV' are flexible and O
is not. I'm not clear on how correlate should apply to any of 'SUV' but it
might be worth having it work for objects.
It still bothers me that correlate does not conjugate the second
argument for complex arrays...
It bothers me also... Chuck
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Problem with correlate
Charles R Harris wrote:
On Sun, May 31, 2009 at 9:08 PM, David Cournapeau
da...@ar.media.kyoto-u.ac.jp mailto:da...@ar.media.kyoto-u.ac.jp
wrote:
Charles R Harris wrote:
On Sun, May 31, 2009 at 7:18 PM, David Cournapeau
da...@ar.media.kyoto-u.ac.jp
mailto:da...@ar.media.kyoto-u.ac.jp
mailto:da...@ar.media.kyoto-u.ac.jp
mailto:da...@ar.media.kyoto-u.ac.jp
wrote:
Charles R Harris wrote:
On Sun, May 31, 2009 at 11:54 AM, rob steed
rjst...@talk21.com mailto:rjst...@talk21.com
mailto:rjst...@talk21.com mailto:rjst...@talk21.com
mailto:rjst...@talk21.com mailto:rjst...@talk21.com
mailto:rjst...@talk21.com mailto:rjst...@talk21.com wrote:
Hi,
After my previous email, I have opened a ticket #1117
(correlate
not order dependent)
I have found that the correlate function is defined in
multiarraymodule.c and
that inputs are being swapped using the following code
n1 = ap1-dimensions[0];
n2 = ap2-dimensions[0];
if (n1 n2) {
ret = ap1;
ap1 = ap2;
ap2 = ret;
ret = NULL;
i = n1;
n1 = n2;
n2 = i;
}
I do not know the code well enough to see whether this
could
just
be removed (I don't know c either).
Maybe the algorithmn requires the inputs to be length
ordered? I
will try to work it out.
If the correlation algorithm doesn't use an fft and is done
explicitly, then the maximum overlap for any shift is the
length of
the shortest input. Swapping the arrays makes that logic
easier to
implement, but it isn't necessary.
But this logic is also wrong if the swapping is not taken into
account -
as the OP mentioned, correlate(a, b) is not equal to
correlate(b,
a) in
the general case. The output is reversed in the second case
compared to
the first case.
I didn't say it was *correctly* implemented ;)
:) So I gave it a shot
http://github.com/cournape/numpy/commits/fix_correlate
(It took me a while to realize that PyArray_ISFLEXIBLE returns
false for
array object. Is this expected ? The documentation concerning copyswap
says that it is necessary for flexible arrays, but I think it is
necessary for object arrays as well).
Don't know. PyArray_ISFLEXIBLE looks like a macro...
#define PyArray_ISFLEXIBLE(obj) PyTypeNum_ISFLEXIBLE(PyArray_TYPE(obj))
#define PyTypeNum_ISFLEXIBLE(type) (((type) =NPY_STRING) \
((type) =NPY_VOID))
And the typecodes are '?bhilqpBHILQPfdgFDGSUVO'. So 'SUV' are flexible
and O is not.
I re-read the copyswap documentation, and realized I did not read it
correctly. Now, I am not sure when to use copyswap vs memcpy (memcpy
should be much faster on basic types, as memcpy should be inlined
generally, whereas I doubt copyswap can).
I'm not clear on how correlate should apply to any of 'SUV' but it
might be worth having it work for objects.
It already does (I added a couple of unit tests in the branch, since
there were no test for correlate, and one is for Decimal object arrays).
It still bothers me that correlate does not conjugate the second
argument for complex arrays...
It bothers me also...
I think we should just fix it to use conjugate - I will do this in the
branch, and I will integrate it in the trunk later unless someone stands
up vehemently against the change. I opened up a ticket to track this,
though,
cheers,
David
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Problem with correlate
Hi all, I have been using numpy.correlate and was finding something weird. I now think that there might be a bug. Correlations should be order dependent eg. correlate(x,y) != correlate(y,x) in general (whereas convolutions are symmetric) import numpy as N x = N.array([1,0,0]) y = N.array([0,0,1]) N.correlate(x,y,'full') array([1, 0, 0, 0, 0]) N.correlate(y,x,'full') array([0, 0, 0, 0, 1]) This works fine. However, if the arrays have different lengths, we get a problem. y2=N.array([0,0,0,1]) N.correlate(x,y2,'full') array([0, 0, 0, 0, 0, 1]) N.correlate(y2,x,'full') array([0, 0, 0, 0, 0, 1]) I believe that somewhere in the code, the arrays are re-ordered by their length. Initially I thought that this was because correlate was deriving from convolution but looking at numpy.core, I can see that in fact convolution derives from correlate. After that, it becomes C code which I haven't managed to look at yet. Am I correct, is this a bug? regards Rob Steed ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Problem with correlate
2009/5/18 rob steed rjst...@talk21.com: This works fine. However, if the arrays have different lengths, we get a problem. y2=N.array([0,0,0,1]) N.correlate(x,y2,'full') This looks like a bug to me. In [54]: N.correlate([1, 0, 0, 0], [0, 0, 0, 1],'full') Out[54]: array([1, 0, 0, 0, 0, 0, 0]) In [55]: N.correlate([1, 0, 0, 0, 0], [0, 0, 0, 1],'full') Out[55]: array([1, 0, 0, 0, 0, 0, 0, 0]) In [56]: N.correlate([1, 0, 0, 0, 0], [0, 0, 0, 0, 1],'full') Out[56]: array([1, 0, 0, 0, 0, 0, 0, 0, 0]) In [57]: N.correlate([1, 0, 0, 0, 0], [0, 0, 0, 0, 0, 1],'full') Out[57]: array([0, 0, 0, 0, 0, 0, 0, 0, 0, 1]) Regards Stéfan ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Problem with correlate
2009/5/18 Stéfan van der Walt ste...@sun.ac.za: 2009/5/18 rob steed rjst...@talk21.com: This works fine. However, if the arrays have different lengths, we get a problem. y2=N.array([0,0,0,1]) N.correlate(x,y2,'full') This looks like a bug to me. In [54]: N.correlate([1, 0, 0, 0], [0, 0, 0, 1],'full') Out[54]: array([1, 0, 0, 0, 0, 0, 0]) In [55]: N.correlate([1, 0, 0, 0, 0], [0, 0, 0, 1],'full') Out[55]: array([1, 0, 0, 0, 0, 0, 0, 0]) In [56]: N.correlate([1, 0, 0, 0, 0], [0, 0, 0, 0, 1],'full') Out[56]: array([1, 0, 0, 0, 0, 0, 0, 0, 0]) In [57]: N.correlate([1, 0, 0, 0, 0], [0, 0, 0, 0, 0, 1],'full') Out[57]: array([0, 0, 0, 0, 0, 0, 0, 0, 0, 1]) comparing with scipy: signal.correlate behaves the same flipping way as np.correlate, ndimage.correlate keeps the orientation. np.correlate([1, 2, 0, 0, 0], [0, 0, 1, 0, 0,0],'same') array([0, 0, 0, 2, 1, 0]) np.correlate([1, 2, 0, 0, 0], [0, 0, 1, 0, 0],'same') array([1, 2, 0, 0, 0]) np.correlate([1, 2, 0, 0, 0], [0, 0, 1, 0, 0, 0],'full') array([0, 0, 0, 0, 0, 2, 1, 0, 0, 0]) np.correlate([1, 2, 0, 0, 0], [0, 0, 1, 0, 0],'full') array([0, 0, 1, 2, 0, 0, 0, 0, 0]) signal.correlate([1, 2, 0, 0, 0], [0, 0, 1, 0, 0, 0]) array([0, 0, 0, 0, 0, 2, 1, 0, 0, 0]) signal.correlate([1, 2, 0, 0, 0], [0, 0, 1, 0, 0]) array([0, 0, 1, 2, 0, 0, 0, 0, 0]) ndimage.filters.correlate([1, 2, 0, 0, 0], [0, 0, 1, 0, 0, 0],mode='constant') array([0, 1, 2, 0, 0]) ndimage.filters.correlate([1, 2, 0, 0, 0], [0, 0, 1, 0, 0],mode='constant') array([1, 2, 0, 0, 0]) Josef ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Problem with correlate?
Hi Stefan, yes, indeed, that's what I thought. This result is odd. Has correlate been changed since version 1.0.4, or should I submit this as a bug? Best regards, Hanno Stéfan van der Walt [EMAIL PROTECTED] said: --=_Part_25307_10322093.1219268954678 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Content-Disposition: inline 2008/8/20 Hanno Klemm [EMAIL PROTECTED]: In [29]: x =3D array([0.,0.,1, 0, 0]) In [35]: y1 =3D array([1,0,0,0,0]) In [36]: correlate(x,y1,mode=3D'full') Out[36]: array([ 0., 0., 0., 0., 0., 0., 1., 0., 0.]) That doesn't look right. Under r5661: In [60]: np.convolve([0, 0, 1, 0, 0], [1, 0, 0, 0, 0], mode=3D'f') Out[60]: array([0, 0, 1, 0, 0, 0, 0, 0, 0]) Regards St=E9fan --=_Part_25307_10322093.1219268954678 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Content-Disposition: inline div dir=3Dltr2008/8/20 Hanno Klemm lt;a href=3Dmailto:[EMAIL PROTECTED] z.ch[EMAIL PROTECTED]/agt;:brgt; In [29]: x =3D array([0.,0.,1, 0,= 0])brgt; In [35]: y1 =3D array([1,0,0,0,0])brgt;brgt; In [36]: co= rrelate(x,y1,mode=3D#39;full#39;)br gt; Out[36]: array([ 0., nbsp;0., nbsp;0., nbsp;0., nbsp;0., nbsp;0.,= nbsp;1., nbsp;0., nbsp;0.])brbrThat doesn#39;t look right.nbsp; U= nder r5661:brbrIn [60]: np.convolve([0, 0, 1, 0, 0], [1, 0, 0, 0, 0], m= ode=3D#39;f#39;)brOut[60]: array([0, 0, 1, 0, 0, 0, 0, 0, 0])br brRegardsbrSt=E9fanbrbr/div --=_Part_25307_10322093.1219268954678-- -- Hanno Klemm [EMAIL PROTECTED] ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Problem with correlate?
Hi Hanno 2008/8/22 Hanno Klemm [EMAIL PROTECTED]: yes, indeed, that's what I thought. This result is odd. Has correlate been changed since version 1.0.4, or should I submit this as a bug? Is there any way that you could try out the latest release on your machine and see if it solves your problem? We probably won't be releasing bug-fixes on 1.0.4, but if it exists in 1.1 we'll still address it. I'm not aware of any changes, but I may simply have missed it. Regards Stéfan ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Problem with correlate?
Hi Stefan, I checked it with numpy version 1.1.1 just now and the result is the same: x = N.array([0.,0,1,0,0]) y1 = N.array([1.,0,0,0,0]) N.correlate(x,y1, mode='full') array([ 0., 0., 0., 0., 0., 0., 1., 0., 0.]) y2 = N.array([1.,0,0,0,0,0,0]) N.correlate(x,y2, mode='full') array([ 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0.]) N.__version__ '1.1.1' Best regards, Hanno Stéfan van der Walt [EMAIL PROTECTED] said: Hi Hanno 2008/8/22 Hanno Klemm [EMAIL PROTECTED]: yes, indeed, that's what I thought. This result is odd. Has correlate been changed since version 1.0.4, or should I submit this as a bug? Is there any way that you could try out the latest release on your machine and see if it solves your problem? We probably won't be releasing bug-fixes on 1.0.4, but if it exists in 1.1 we'll still address it. I'm not aware of any changes, but I may simply have missed it. Regards Stéfan ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion -- Hanno Klemm [EMAIL PROTECTED] ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Problem with correlate?
Hi All, after the discussion on numpy.correlate some time ago, regarding complex conjugation, etc. I today was pointed to yet another oddity, which I hope somebody could explain to me, as to why that's a feature, rather than a bug. I'm thoroughly confused by the following behaviour: In [29]: x = array([0.,0.,1, 0, 0]) In [30]: y = array([0, 0, 0, 1., 0, 0, 0]) In [31]: correlate(x,y) Out[31]: array([ 0., 1., 0.]) In [32]: correlate(x,y, mode='full') Out[32]: array([ 0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0.]) In [33]: y = array([0,0,1.,0,0]) In [34]: correlate(x,y,mode='full') Out[34]: array([ 0., 0., 0., 0., 1., 0., 0., 0., 0.]) So far, everything is allright and as expected. Now: In [35]: y1 = array([1,0,0,0,0]) In [36]: correlate(x,y1,mode='full') Out[36]: array([ 0., 0., 0., 0., 0., 0., 1., 0., 0.]) In [37]: y2 = array([1,0,0,0,0,0,0]) In [38]: correlate(x,y2,mode='full') Out[38]: array([ 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0.]) Could somebody please help me understanding, why the one switches places? Furthermore, this behaviour does not seem to be very consistent, if we switch the x and y argument: In [39]: correlate(y2,x,mode='full') Out[39]: array([ 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0.]) In [40]: correlate(y1,x,mode='full') Out[40]: array([ 0., 0., 1., 0., 0., 0., 0., 0., 0.]) The answer remains the same. Any help would be appreciated, the numpy version is 1.0.4. Best regards, Hanno -- Hanno Klemm [EMAIL PROTECTED] ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Problem with correlate?
2008/8/20 Hanno Klemm [EMAIL PROTECTED]: In [29]: x = array([0.,0.,1, 0, 0]) In [35]: y1 = array([1,0,0,0,0]) In [36]: correlate(x,y1,mode='full') Out[36]: array([ 0., 0., 0., 0., 0., 0., 1., 0., 0.]) That doesn't look right. Under r5661: In [60]: np.convolve([0, 0, 1, 0, 0], [1, 0, 0, 0, 0], mode='f') Out[60]: array([0, 0, 1, 0, 0, 0, 0, 0, 0]) Regards Stéfan ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
