Re: [Numpy-discussion] numpy 1.10.1 reduce operation on recarrays

2015-10-19 Thread josef.pktd 
On Fri, Oct 16, 2015 at 9:31 PM, Allan Haldane 
wrote:

> On 10/16/2015 09:17 PM, josef.p...@gmail.com wrote:
>
>>
>>
>> On Fri, Oct 16, 2015 at 8:56 PM, Allan Haldane > > wrote:
>>
>> On 10/16/2015 05:31 PM, josef.p...@gmail.com
>>  wrote:
>> >
>> >
>> > On Fri, Oct 16, 2015 at 2:21 PM, Charles R Harris
>> > 
>> > >> wrote:
>> >
>> >
>> >
>> > On Fri, Oct 16, 2015 at 12:20 PM, Charles R Harris
>> > 
>> > >> wrote:
>> >
>> >
>> >
>> > On Fri, Oct 16, 2015 at 11:58 AM, > 
>>  > >
>> >> wrote:
>>  >
>>  > was there a change with reduce operations with
>> recarrays in
>>  > 1.10 or 1.10.1?
>>  >
>>  > Travis shows a new test failure in the statsmodels
>> testsuite
>>  > with 1.10.1:
>>  >
>>  > ERROR: test suite for >  > 'statsmodels.base.tests.test_data.TestRecarrays'>
>>  >
>>  >   File
>>  >
>>
>> "/home/travis/miniconda/envs/statsmodels-test/lib/python2.7/site-packages/statsmodels-0.8.0-py2.7-linux-x86_64.egg/statsmodels/base/data.py",
>>  > line 131, in _handle_constant
>>  > const_idx = np.where(self.exog.ptp(axis=0) ==
>>  > 0)[0].squeeze()
>>  > TypeError: cannot perform reduce with flexible type
>>  >
>>  >
>>  > Sorry for asking so late.
>>  > (statsmodels is short on maintainers, and I'm
>> distracted)
>>  >
>>  >
>>  > statsmodels still has code to support recarrays and
>>  > structured dtypes from the time before pandas became
>>  > popular, but I don't think anyone is using them
>> together
>>  > with statsmodels anymore.
>>  >
>>  >
>>  > There were several commits dealing both recarrays and
>> ufuncs, so
>>  > this might well be a regression.
>>  >
>>  >
>>  > A bisection would be helpful. Also, open an issue.
>>  >
>>  >
>>  >
>>  > The reason for the test failure might be somewhere else hiding
>> behind
>>  > several layers of statsmodels, but only started to show up with
>> numpy 1.10.1
>>  >
>>  > I already have the reduce exception with my currently installed
>> numpy
>>  > '1.9.2rc1'
>>  >
>>   x = np.random.random(9*3).view([('const', 'f8'),('x_1', 'f8'),
>>  > ('x_2', 'f8')]).view(np.recarray)
>>  >
>>   np.ptp(x, axis=0)
>>  > Traceback (most recent call last):
>>  >   File "", line 1, in 
>>  >   File
>>  >
>>
>> "C:\programs\WinPython-64bit-3.4.3.1\python-3.4.3.amd64\lib\site-packages\numpy\core\fromnumeric.py",
>>  > line 2047, in ptp
>>  > return ptp(axis, out)
>>  > TypeError: cannot perform reduce with flexible type
>>  >
>>  >
>>  > Sounds like fun, and I don't even know how to automatically bisect.
>>  >
>>  > Josef
>>
>> That example isn't the problem (ptp should definitely fail on
>> structured
>> arrays), but I've tracked down what is - it has to do with views of
>> record arrays.
>>
>> The fix looks simple, I'll get it in for the next release.
>>
>>
>> Thanks,
>>
>> I realized that at that point in the statsmodels code we should have
>> only regular ndarrays, so the array conversion fails somewhere.
>>
>> AFAICS, the main helper function to convert is
>>
>> def struct_to_ndarray(arr):
>>  return arr.view((float, len(arr.dtype.names)))
>>
>> which doesn't look like it will handle other dtypes than float64. Nobody
>> ever complained, so maybe our test suite is the only user of this.
>>
>> What is now the recommended way of converting structured
>> dtypes/recarrays to ndarrays?
>>
>> Josef
>>
>
> Yes, that's the code I narrowed it down to as well. I think the code in
> statsmodels is fine, the problem is actually a  bug I must admit I
> introduced in changes to the way views of recarrays work.
>
> If you are curious, the bug is in this line:
>
> https://github.com/numpy/numpy/blob/master/numpy/core/records.py#L467
>
> This line was intended to fix the problem that accessing a nested record
> array field would lose the 'np.record' dtype. I only considered void
> structured arrays, and had forgotten about sub-arrays which statsmodels
> uses.
>
> I think the fix is to replace `issubclass(val.type,

[Numpy-discussion] numpy 1.10.1 reduce operation on recarrays

2015-10-16 Thread josef.pktd 
was there a change with reduce operations with recarrays in 1.10 or 1.10.1?

Travis shows a new test failure in the statsmodels testsuite with 1.10.1:

ERROR: test suite for 

  File
"/home/travis/miniconda/envs/statsmodels-test/lib/python2.7/site-packages/statsmodels-0.8.0-py2.7-linux-x86_64.egg/statsmodels/base/data.py",
line 131, in _handle_constant
const_idx = np.where(self.exog.ptp(axis=0) == 0)[0].squeeze()
TypeError: cannot perform reduce with flexible type


Sorry for asking so late.
(statsmodels is short on maintainers, and I'm distracted)


statsmodels still has code to support recarrays and structured dtypes from
the time before pandas became popular, but I don't think anyone is using
them together with statsmodels anymore.

Josef
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] numpy 1.10.1 reduce operation on recarrays

2015-10-16 Thread Charles R Harris 
On Fri, Oct 16, 2015 at 11:58 AM,  wrote:

> was there a change with reduce operations with recarrays in 1.10 or 1.10.1?
>
> Travis shows a new test failure in the statsmodels testsuite with 1.10.1:
>
> ERROR: test suite for  'statsmodels.base.tests.test_data.TestRecarrays'>
>
>   File
> "/home/travis/miniconda/envs/statsmodels-test/lib/python2.7/site-packages/statsmodels-0.8.0-py2.7-linux-x86_64.egg/statsmodels/base/data.py",
> line 131, in _handle_constant
> const_idx = np.where(self.exog.ptp(axis=0) == 0)[0].squeeze()
> TypeError: cannot perform reduce with flexible type
>
>
> Sorry for asking so late.
> (statsmodels is short on maintainers, and I'm distracted)
>
>
> statsmodels still has code to support recarrays and structured dtypes from
> the time before pandas became popular, but I don't think anyone is using
> them together with statsmodels anymore.
>
>
There were several commits dealing both recarrays and ufuncs, so this might
well be a regression.

Chuck
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] numpy 1.10.1 reduce operation on recarrays

2015-10-16 Thread josef.pktd 
On Fri, Oct 16, 2015 at 2:21 PM, Charles R Harris  wrote:

>
>
> On Fri, Oct 16, 2015 at 12:20 PM, Charles R Harris <
> charlesr.har...@gmail.com> wrote:
>
>>
>>
>> On Fri, Oct 16, 2015 at 11:58 AM,  wrote:
>>
>>> was there a change with reduce operations with recarrays in 1.10 or
>>> 1.10.1?
>>>
>>> Travis shows a new test failure in the statsmodels testsuite with 1.10.1:
>>>
>>> ERROR: test suite for >> 'statsmodels.base.tests.test_data.TestRecarrays'>
>>>
>>>   File
>>> "/home/travis/miniconda/envs/statsmodels-test/lib/python2.7/site-packages/statsmodels-0.8.0-py2.7-linux-x86_64.egg/statsmodels/base/data.py",
>>> line 131, in _handle_constant
>>> const_idx = np.where(self.exog.ptp(axis=0) == 0)[0].squeeze()
>>> TypeError: cannot perform reduce with flexible type
>>>
>>>
>>> Sorry for asking so late.
>>> (statsmodels is short on maintainers, and I'm distracted)
>>>
>>>
>>> statsmodels still has code to support recarrays and structured dtypes
>>> from the time before pandas became popular, but I don't think anyone is
>>> using them together with statsmodels anymore.
>>>
>>>
>> There were several commits dealing both recarrays and ufuncs, so this
>> might well be a regression.
>>
>>
> A bisection would be helpful. Also, open an issue.
>


The reason for the test failure might be somewhere else hiding behind
several layers of statsmodels, but only started to show up with numpy 1.10.1

I already have the reduce exception with my currently installed numpy
'1.9.2rc1'

>>> x = np.random.random(9*3).view([('const', 'f8'),('x_1', 'f8'), ('x_2',
'f8')]).view(np.recarray)

>>> np.ptp(x, axis=0)
Traceback (most recent call last):
  File "", line 1, in 
  File
"C:\programs\WinPython-64bit-3.4.3.1\python-3.4.3.amd64\lib\site-packages\numpy\core\fromnumeric.py",
line 2047, in ptp
return ptp(axis, out)
TypeError: cannot perform reduce with flexible type


Sounds like fun, and I don't even know how to automatically bisect.

Josef


>
> Chuck
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] numpy 1.10.1 reduce operation on recarrays

2015-10-16 Thread Allan Haldane 
On 10/16/2015 05:31 PM, josef.p...@gmail.com wrote:
> 
> 
> On Fri, Oct 16, 2015 at 2:21 PM, Charles R Harris
> > wrote:
> 
> 
> 
> On Fri, Oct 16, 2015 at 12:20 PM, Charles R Harris
> > wrote:
> 
> 
> 
> On Fri, Oct 16, 2015 at 11:58 AM,  > wrote:
> 
> was there a change with reduce operations with recarrays in
> 1.10 or 1.10.1?
> 
> Travis shows a new test failure in the statsmodels testsuite
> with 1.10.1:
> 
> ERROR: test suite for  'statsmodels.base.tests.test_data.TestRecarrays'>
> 
>   File
> 
> "/home/travis/miniconda/envs/statsmodels-test/lib/python2.7/site-packages/statsmodels-0.8.0-py2.7-linux-x86_64.egg/statsmodels/base/data.py",
> line 131, in _handle_constant
> const_idx = np.where(self.exog.ptp(axis=0) ==
> 0)[0].squeeze()
> TypeError: cannot perform reduce with flexible type
> 
> 
> Sorry for asking so late.
> (statsmodels is short on maintainers, and I'm distracted)
> 
> 
> statsmodels still has code to support recarrays and
> structured dtypes from the time before pandas became
> popular, but I don't think anyone is using them together
> with statsmodels anymore.
> 
> 
> There were several commits dealing both recarrays and ufuncs, so
> this might well be a regression.
> 
> 
> A bisection would be helpful. Also, open an issue.
> 
> 
> 
> The reason for the test failure might be somewhere else hiding behind
> several layers of statsmodels, but only started to show up with numpy 1.10.1
> 
> I already have the reduce exception with my currently installed numpy
> '1.9.2rc1'
> 
 x = np.random.random(9*3).view([('const', 'f8'),('x_1', 'f8'),
> ('x_2', 'f8')]).view(np.recarray)
> 
 np.ptp(x, axis=0)
> Traceback (most recent call last):
>   File "", line 1, in 
>   File
> "C:\programs\WinPython-64bit-3.4.3.1\python-3.4.3.amd64\lib\site-packages\numpy\core\fromnumeric.py",
> line 2047, in ptp
> return ptp(axis, out)
> TypeError: cannot perform reduce with flexible type
> 
> 
> Sounds like fun, and I don't even know how to automatically bisect.
> 
> Josef

That example isn't the problem (ptp should definitely fail on structured
arrays), but I've tracked down what is - it has to do with views of
record arrays.

The fix looks simple, I'll get it in for the next release.

Allan




___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] numpy 1.10.1 reduce operation on recarrays

2015-10-16 Thread josef.pktd 
On Fri, Oct 16, 2015 at 8:56 PM, Allan Haldane 
wrote:

> On 10/16/2015 05:31 PM, josef.p...@gmail.com wrote:
> >
> >
> > On Fri, Oct 16, 2015 at 2:21 PM, Charles R Harris
> > > wrote:
> >
> >
> >
> > On Fri, Oct 16, 2015 at 12:20 PM, Charles R Harris
> > >
> wrote:
> >
> >
> >
> > On Fri, Oct 16, 2015 at 11:58 AM,  > > wrote:
> >
> > was there a change with reduce operations with recarrays in
> > 1.10 or 1.10.1?
> >
> > Travis shows a new test failure in the statsmodels testsuite
> > with 1.10.1:
> >
> > ERROR: test suite for  > 'statsmodels.base.tests.test_data.TestRecarrays'>
> >
> >   File
> >
>  
> "/home/travis/miniconda/envs/statsmodels-test/lib/python2.7/site-packages/statsmodels-0.8.0-py2.7-linux-x86_64.egg/statsmodels/base/data.py",
> > line 131, in _handle_constant
> > const_idx = np.where(self.exog.ptp(axis=0) ==
> > 0)[0].squeeze()
> > TypeError: cannot perform reduce with flexible type
> >
> >
> > Sorry for asking so late.
> > (statsmodels is short on maintainers, and I'm distracted)
> >
> >
> > statsmodels still has code to support recarrays and
> > structured dtypes from the time before pandas became
> > popular, but I don't think anyone is using them together
> > with statsmodels anymore.
> >
> >
> > There were several commits dealing both recarrays and ufuncs, so
> > this might well be a regression.
> >
> >
> > A bisection would be helpful. Also, open an issue.
> >
> >
> >
> > The reason for the test failure might be somewhere else hiding behind
> > several layers of statsmodels, but only started to show up with numpy
> 1.10.1
> >
> > I already have the reduce exception with my currently installed numpy
> > '1.9.2rc1'
> >
>  x = np.random.random(9*3).view([('const', 'f8'),('x_1', 'f8'),
> > ('x_2', 'f8')]).view(np.recarray)
> >
>  np.ptp(x, axis=0)
> > Traceback (most recent call last):
> >   File "", line 1, in 
> >   File
> >
> "C:\programs\WinPython-64bit-3.4.3.1\python-3.4.3.amd64\lib\site-packages\numpy\core\fromnumeric.py",
> > line 2047, in ptp
> > return ptp(axis, out)
> > TypeError: cannot perform reduce with flexible type
> >
> >
> > Sounds like fun, and I don't even know how to automatically bisect.
> >
> > Josef
>
> That example isn't the problem (ptp should definitely fail on structured
> arrays), but I've tracked down what is - it has to do with views of
> record arrays.
>
> The fix looks simple, I'll get it in for the next release.
>

Thanks,

I realized that at that point in the statsmodels code we should have only
regular ndarrays, so the array conversion fails somewhere.

AFAICS, the main helper function to convert is

def struct_to_ndarray(arr):
return arr.view((float, len(arr.dtype.names)))

which doesn't look like it will handle other dtypes than float64. Nobody
ever complained, so maybe our test suite is the only user of this.

What is now the recommended way of converting structured dtypes/recarrays
to ndarrays?

Josef



>
> Allan
>
>
>
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] numpy 1.10.1 reduce operation on recarrays

2015-10-16 Thread Allan Haldane 

On 10/16/2015 09:17 PM, josef.p...@gmail.com wrote:



On Fri, Oct 16, 2015 at 8:56 PM, Allan Haldane > wrote:

On 10/16/2015 05:31 PM, josef.p...@gmail.com
 wrote:
>
>
> On Fri, Oct 16, 2015 at 2:21 PM, Charles R Harris
> 
>> wrote:
>
>
>
> On Fri, Oct 16, 2015 at 12:20 PM, Charles R Harris
> 
>> wrote:
>
>
>
> On Fri, Oct 16, 2015 at 11:58 AM, 
 > >> wrote:
 >
 > was there a change with reduce operations with
recarrays in
 > 1.10 or 1.10.1?
 >
 > Travis shows a new test failure in the statsmodels
testsuite
 > with 1.10.1:
 >
 > ERROR: test suite for  'statsmodels.base.tests.test_data.TestRecarrays'>
 >
 >   File
 >
  
"/home/travis/miniconda/envs/statsmodels-test/lib/python2.7/site-packages/statsmodels-0.8.0-py2.7-linux-x86_64.egg/statsmodels/base/data.py",
 > line 131, in _handle_constant
 > const_idx = np.where(self.exog.ptp(axis=0) ==
 > 0)[0].squeeze()
 > TypeError: cannot perform reduce with flexible type
 >
 >
 > Sorry for asking so late.
 > (statsmodels is short on maintainers, and I'm distracted)
 >
 >
 > statsmodels still has code to support recarrays and
 > structured dtypes from the time before pandas became
 > popular, but I don't think anyone is using them together
 > with statsmodels anymore.
 >
 >
 > There were several commits dealing both recarrays and
ufuncs, so
 > this might well be a regression.
 >
 >
 > A bisection would be helpful. Also, open an issue.
 >
 >
 >
 > The reason for the test failure might be somewhere else hiding behind
 > several layers of statsmodels, but only started to show up with
numpy 1.10.1
 >
 > I already have the reduce exception with my currently installed numpy
 > '1.9.2rc1'
 >
  x = np.random.random(9*3).view([('const', 'f8'),('x_1', 'f8'),
 > ('x_2', 'f8')]).view(np.recarray)
 >
  np.ptp(x, axis=0)
 > Traceback (most recent call last):
 >   File "", line 1, in 
 >   File
 >

"C:\programs\WinPython-64bit-3.4.3.1\python-3.4.3.amd64\lib\site-packages\numpy\core\fromnumeric.py",
 > line 2047, in ptp
 > return ptp(axis, out)
 > TypeError: cannot perform reduce with flexible type
 >
 >
 > Sounds like fun, and I don't even know how to automatically bisect.
 >
 > Josef

That example isn't the problem (ptp should definitely fail on structured
arrays), but I've tracked down what is - it has to do with views of
record arrays.

The fix looks simple, I'll get it in for the next release.


Thanks,

I realized that at that point in the statsmodels code we should have
only regular ndarrays, so the array conversion fails somewhere.

AFAICS, the main helper function to convert is

def struct_to_ndarray(arr):
 return arr.view((float, len(arr.dtype.names)))

which doesn't look like it will handle other dtypes than float64. Nobody
ever complained, so maybe our test suite is the only user of this.

What is now the recommended way of converting structured
dtypes/recarrays to ndarrays?

Josef


Yes, that's the code I narrowed it down to as well. I think the code in 
statsmodels is fine, the problem is actually a  bug I must admit I 
introduced in changes to the way views of recarrays work.


If you are curious, the bug is in this line:

https://github.com/numpy/numpy/blob/master/numpy/core/records.py#L467

This line was intended to fix the problem that accessing a nested record 
array field would lose the 'np.record' dtype. I only considered void 
structured arrays, and had forgotten about sub-arrays which statsmodels 
uses.


I think the fix is to replace `issubclass(val.type, nt.void)` with 
`val.names` or something similar. I'll take a closer look soon.


Allan

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] numpy 1.10.1 reduce operation on recarrays

2015-10-16 Thread Charles R Harris 
On Fri, Oct 16, 2015 at 12:20 PM, Charles R Harris <
charlesr.har...@gmail.com> wrote:

>
>
> On Fri, Oct 16, 2015 at 11:58 AM,  wrote:
>
>> was there a change with reduce operations with recarrays in 1.10 or
>> 1.10.1?
>>
>> Travis shows a new test failure in the statsmodels testsuite with 1.10.1:
>>
>> ERROR: test suite for > 'statsmodels.base.tests.test_data.TestRecarrays'>
>>
>>   File
>> "/home/travis/miniconda/envs/statsmodels-test/lib/python2.7/site-packages/statsmodels-0.8.0-py2.7-linux-x86_64.egg/statsmodels/base/data.py",
>> line 131, in _handle_constant
>> const_idx = np.where(self.exog.ptp(axis=0) == 0)[0].squeeze()
>> TypeError: cannot perform reduce with flexible type
>>
>>
>> Sorry for asking so late.
>> (statsmodels is short on maintainers, and I'm distracted)
>>
>>
>> statsmodels still has code to support recarrays and structured dtypes
>> from the time before pandas became popular, but I don't think anyone is
>> using them together with statsmodels anymore.
>>
>>
> There were several commits dealing both recarrays and ufuncs, so this
> might well be a regression.
>
>
A bisection would be helpful. Also, open an issue.

Chuck
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion