Re: [Numpy-discussion] ANN: Numpy 1.6.0 beta 2

[Christopher Barker]

On 4/5/11 10:33 PM, Matthew Brett wrote:
> Did you mean to send this just to me?  It seems like the whole is
> generally interesting and helpful, at least to me...

I did mean to send to the list -- I've done that now.

> Well, the current code doesn't split on \r in py3k, admittedly that
> must be a rare case.

I guess that's a key question here -- It really *should* split on /r, 
but maybe it's rare enough to be unimportant.

> The general point about whether to read binary or text is also in
> play.   I agree with you, reading text sounds like a better idea to
> me, but I don't know the performance hit.   Pauli had it as reading
> binary in the original conversion and was defending that in an earlier
> email...

The thing is -- we're talking about parsing text here, so we really are 
reading text files, NOT binary files.

So the question really is -- do we want py3's file reading code to 
handle encoding issues for us, or do we want to handle them ourselves. 
If we only want to support ansi encodings, then handling ourselves may 
well be easier and faster performing. If we go that way we need to 
handle line-endings, too. The easy way is to only support line endings 
with a '\n' in them -- that works out of the box. But it's not that hard 
to support 'r' also, depending on how you want to do it. Before 'U' 
existed, I did that all the time, something like:

some_text = file.read(some_size_buffer)
some_text.replace('\r\n', '\n')
some_text.replace('\r', '\n')
lines = some_text.split('\n')

(by the way, if you're going to support this, it's really nice to 
support mixed line-endings (like this approach does) -- there are a lot 
of editors that can make a mess of line endings.

If you can read the entire file into memory at once, this is almost 
trivial, if you can't -- there is a bit more bookeeping code to be written.

DARN -- I think I said my last note was the last on this topic!

-Chris






-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] ANN: Numpy 1.6.0 beta 2

[Christopher Barker]

Sorry to keep harping on this, but for history's sake, I was one of the 
folks that got 'U' introduced in the first place. I was dealing with a 
nightmare of unix, mac and dos test files, 'U' was a  godsend.


On 4/5/11 4:51 PM, Matthew Brett wrote:
> The difference between 'rt' and 'U' is (this is for my own benefit):
>
> For 'rt', a '\r' does not cause a line break - with 'U' - it does.

Perhaps semantics, but what 'U' does is actually change any of the line 
breaks to '\n' -- any line breaking happens after the fact. In Py2, the 
difference between 'U' and 't' is that 't' assumes that any file read 
uses the native line endings -- a bad idea, IMHO. Back in the day, Guido 
argued that text file line ending conversion was the job of file 
transfer tools. The reality, however, is that users don't always use 
file transfer tools correctly, nor even understand the implications of 
line endings.

All that being said,  mac-style files are pretty rare these days. 
(though I bet I've got a few still kicking around)

> Right - my argument is that the behavior implied by 'U' and 't' is
> conceptually separable.   'U' is for how to do line-breaks, and
> line-termination translations, 't' is for whether to decode the text
> or not.  In python 3.

but 't' and 'U' are the same in python 3 -- there is no distinction. It 
seems you are arguing that there could/should be a way to translate line 
termination without decoding the text, but ...

> In python 3 a binary file is a file which is not decoded, and returns
> bytes.  It still has a concept of a 'line', as defined by line
> terminators - you can iterate over one, or do .readlines().

I'll take your word for it that it does, but that's not really a binary 
file then, it's a file that you are assuming is encoded in an 
ascii-compatible way.

While I know that "practicality beats purity", we really should be 
opening the file as a text file (it is text, after all), and specifying 
utf-8 or latin-1 or something as the encoding.

However, IIUC, then the issue here is later on down the line, numpy uses 
regular old C code, which expects ascii strings. In that case, we could 
encode the text as ascii, into a bytes object.

That's a lot of overhead for line ending translation, so probably not 
worth it. But if nothing else, we should be clear in the docs that numpy 
text file reading code is expecting ascii-compatible data.

(and it would be nice to get the line-ending translation)

> Right - so obviously if you open a utf-16 file as binary, terrible
> things may happen - this was what Pauli was pointing out before.  His
> point was that utf-8 is the standard,

but it's not the standard -- it's a common use, but not a standard -- 
ideally numpy wouldn't enforce any particular encoding (though it could 
default to one, and utf-8 would be a good choice for that)

>> Once you really make the distiction between text and binary, the concept
>> of a "line terminator" doesn't really make sense anyway.
>
> Well - I was arguing that, given we can iterate over lines in binary
> files, then there must be the concept of what a line is, in a binary
> file, and that means that we need the concept of a line terminator.

maybe, but that concept is built on a assumption that your file is 
ascii-compatible (for \n anyway), and you know what they say about 
assumptions...

> I realize this is a discussion that would have to happen on the
> python-dev list...

I'm not sure -- I was thinking that python missed something here, but I 
don't think so anymore. In the unicode world, there is not choice but to 
be explicit about encodings, and if you do that, then python's "text or 
binary" distinction makes sense. .readline() for binary file doesn't, 
but so be it.

Honestly, I've never been sure in this discussion what code actually 
needs fixing, so I'm done now -- we've talked enough that the issues 
MUST have been covered by now!

-Chris

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] ANN: Numpy 1.6.0 beta 2

[Christoph Gohlke]

On 4/5/2011 7:44 PM, Mark Wiebe wrote:
> On Tue, Apr 5, 2011 at 7:21 PM, Christoph Gohlke  > wrote:
>
>
>
> On 4/5/2011 6:46 PM, Mark Wiebe wrote:
> >  On Tue, Apr 5, 2011 at 5:07 PM, Christoph Gohlke  
> >  >> wrote:
> >
> >
> >
> >  On 4/5/2011 4:05 PM, Mark Wiebe wrote:
> >  >  On Mon, Apr 4, 2011 at 9:10 PM, Christoph Gohlke
> mailto:cgoh...@uci.edu>
> >  >
> >  > 
>  >  >
> >  >
> >  > 
> >  >
> >  >  A few numpy tests fail on win-amd64:
> >  >
> >  > 
> >  >
> >  >
> >
> ==
> >  >  FAIL: test_iterator.test_iter_broadcasting_errors
> >  >
> >
> --
> >  >  Traceback (most recent call last):
> >  > File "X:\Python26-x64\lib\site-packages\nose\case.py", line
> >  187, in
> >  >  runTest
> >  >   self.test(*self.arg)
> >  > File
> >  >
> "X:\Python26-x64\lib\site-packages\numpy\core\tests\test_iterator.py",
> >  >  line 639, in test_iter_broadcasting_errors
> >  > 'Message "%s" doesn\'t contain operand shape (2,3)' % msg)
> >  > File
> >  "X:\Python26-x64\lib\site-packages\numpy\testing\utils.py", line
> >  >  34, in assert_
> >  >   raise AssertionError(msg)
> >  >  AssertionError: Message "non-broadcastable output operand with
> >  shape
> >  >  (%lld,%lld)
> >  >doesn't match the broadcast shape (%lld,%lld,%lld)" doesn't
> >  contain
> >  >  operand shape (2,3)
> >  >
> >  >
> >  >  I've pushed a fix for this to the 1.6.x branch, can you confirm
> >  that it
> >  >  works on win-amd64?
> >  >
> >  >  Thanks,
> >  >  Mark
> >  >
> >  > 
> >  >
> >
> >  Sorry, I forgot to mention that this test failed on 64-bit
> Python 2.6
> >  only. I now recognize it is due to a known issue with Python's
> >  PyErr_Format function .
> Unfortunately
> >  the fix will not be backported to Python 2.6. Maybe this test
> could be
> >  marked as known failure on win-amd64-py2.6?
> >
> >  Could you please revert your changes or set the format
> specifier to
> >  "%lld"? "%I64d" is not supported by PyErr_Format.
> >
> >
> >  I've committed an attempted workaround using the %zd formatter,
> can you
> >  check whether it works on 64-bit Windows Python 2.6 now?
> >
> >  Thanks,
> >  Mark
> >
>
> I have not tried but I do not expect it to work. According to
>  the "%zd" format is 'exactly
> equivalent to printf("%zd")'. The 'z' prefix character is C99 AFAIK and
> not supported by Visual Studio compilers
> .
>
>
> I think there's a good chance it will work, since the CPython code is
> re-interpreting the format codes itself. In the Python 2.6 code, it
> looks like it will become %Id on Windows, which would be correct.
>
> -Mark
>

Agreed, it'll work if NPY_INTP_FMT is only used with certain Python 
functions that use %Id instead of %zd on Windows. Then why not simply 
define NPY_INTP_FMT as %zd?

-#if (PY_VERSION_HEX > 0x0206)
-#define NPY_INTP_FMT "lld"
-#else
  #define NPY_INTP_FMT "zd"
-#endif

That should work unless 64 bit numpy is run on a LLP64 platform with 
Python < 2.5.

Christoph
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] ANN: Numpy 1.6.0 beta 2

[Mark Wiebe]

On Tue, Apr 5, 2011 at 7:21 PM, Christoph Gohlke  wrote:

>
>
> On 4/5/2011 6:46 PM, Mark Wiebe wrote:
> > On Tue, Apr 5, 2011 at 5:07 PM, Christoph Gohlke  > > wrote:
> >
> >
> >
> > On 4/5/2011 4:05 PM, Mark Wiebe wrote:
> > >  On Mon, Apr 4, 2011 at 9:10 PM, Christoph Gohlke  > 
> > >  >> wrote:
> > >
> > >
> > >  
> > >
> > >  A few numpy tests fail on win-amd64:
> > >
> > >  
> > >
> > >
> >
> ==
> > >  FAIL: test_iterator.test_iter_broadcasting_errors
> > >
> >
> --
> > >  Traceback (most recent call last):
> > > File "X:\Python26-x64\lib\site-packages\nose\case.py", line
> > 187, in
> > >  runTest
> > >   self.test(*self.arg)
> > > File
> > >
>  "X:\Python26-x64\lib\site-packages\numpy\core\tests\test_iterator.py",
> > >  line 639, in test_iter_broadcasting_errors
> > >  'Message "%s" doesn\'t contain operand shape (2,3)' % msg)
> > > File
> > "X:\Python26-x64\lib\site-packages\numpy\testing\utils.py", line
> > >  34, in assert_
> > >   raise AssertionError(msg)
> > >  AssertionError: Message "non-broadcastable output operand with
> > shape
> > >  (%lld,%lld)
> > >doesn't match the broadcast shape (%lld,%lld,%lld)" doesn't
> > contain
> > >  operand shape (2,3)
> > >
> > >
> > >  I've pushed a fix for this to the 1.6.x branch, can you confirm
> > that it
> > >  works on win-amd64?
> > >
> > >  Thanks,
> > >  Mark
> > >
> > >  
> > >
> >
> > Sorry, I forgot to mention that this test failed on 64-bit Python 2.6
> > only. I now recognize it is due to a known issue with Python's
> > PyErr_Format function .
> Unfortunately
> > the fix will not be backported to Python 2.6. Maybe this test could
> be
> > marked as known failure on win-amd64-py2.6?
> >
> > Could you please revert your changes or set the format specifier to
> > "%lld"? "%I64d" is not supported by PyErr_Format.
> >
> >
> > I've committed an attempted workaround using the %zd formatter, can you
> > check whether it works on 64-bit Windows Python 2.6 now?
> >
> > Thanks,
> > Mark
> >
>
> I have not tried but I do not expect it to work. According to
>  the "%zd" format is 'exactly
> equivalent to printf("%zd")'. The 'z' prefix character is C99 AFAIK and
> not supported by Visual Studio compilers
> .


I think there's a good chance it will work, since the CPython code is
re-interpreting the format codes itself. In the Python 2.6 code, it looks
like it will become %Id on Windows, which would be correct.

-Mark
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] ANN: Numpy 1.6.0 beta 2

[Christoph Gohlke]

On 4/5/2011 6:46 PM, Mark Wiebe wrote:
> On Tue, Apr 5, 2011 at 5:07 PM, Christoph Gohlke  > wrote:
>
>
>
> On 4/5/2011 4:05 PM, Mark Wiebe wrote:
> >  On Mon, Apr 4, 2011 at 9:10 PM, Christoph Gohlke  
> >  >> wrote:
> >
> >
> >  
> >
> >  A few numpy tests fail on win-amd64:
> >
> >  
> >
> >
> ==
> >  FAIL: test_iterator.test_iter_broadcasting_errors
> >
> --
> >  Traceback (most recent call last):
> > File "X:\Python26-x64\lib\site-packages\nose\case.py", line
> 187, in
> >  runTest
> >   self.test(*self.arg)
> > File
> >  "X:\Python26-x64\lib\site-packages\numpy\core\tests\test_iterator.py",
> >  line 639, in test_iter_broadcasting_errors
> >  'Message "%s" doesn\'t contain operand shape (2,3)' % msg)
> > File
> "X:\Python26-x64\lib\site-packages\numpy\testing\utils.py", line
> >  34, in assert_
> >   raise AssertionError(msg)
> >  AssertionError: Message "non-broadcastable output operand with
> shape
> >  (%lld,%lld)
> >doesn't match the broadcast shape (%lld,%lld,%lld)" doesn't
> contain
> >  operand shape (2,3)
> >
> >
> >  I've pushed a fix for this to the 1.6.x branch, can you confirm
> that it
> >  works on win-amd64?
> >
> >  Thanks,
> >  Mark
> >
> >  
> >
>
> Sorry, I forgot to mention that this test failed on 64-bit Python 2.6
> only. I now recognize it is due to a known issue with Python's
> PyErr_Format function . Unfortunately
> the fix will not be backported to Python 2.6. Maybe this test could be
> marked as known failure on win-amd64-py2.6?
>
> Could you please revert your changes or set the format specifier to
> "%lld"? "%I64d" is not supported by PyErr_Format.
>
>
> I've committed an attempted workaround using the %zd formatter, can you
> check whether it works on 64-bit Windows Python 2.6 now?
>
> Thanks,
> Mark
>

I have not tried but I do not expect it to work. According to 
 the "%zd" format is 'exactly 
equivalent to printf("%zd")'. The 'z' prefix character is C99 AFAIK and 
not supported by Visual Studio compilers 
.

Christoph
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] ANN: Numpy 1.6.0 beta 2

[Mark Wiebe]

On Tue, Apr 5, 2011 at 5:07 PM, Christoph Gohlke  wrote:

>
>
> On 4/5/2011 4:05 PM, Mark Wiebe wrote:
> > On Mon, Apr 4, 2011 at 9:10 PM, Christoph Gohlke  > > wrote:
> >
> >
> > 
> >
> > A few numpy tests fail on win-amd64:
> >
> > 
> >
> >
> ==
> > FAIL: test_iterator.test_iter_broadcasting_errors
> >
> --
> > Traceback (most recent call last):
> >File "X:\Python26-x64\lib\site-packages\nose\case.py", line 187,
> in
> > runTest
> >  self.test(*self.arg)
> >File
> >
> "X:\Python26-x64\lib\site-packages\numpy\core\tests\test_iterator.py",
> > line 639, in test_iter_broadcasting_errors
> > 'Message "%s" doesn\'t contain operand shape (2,3)' % msg)
> >File "X:\Python26-x64\lib\site-packages\numpy\testing\utils.py",
> line
> > 34, in assert_
> >  raise AssertionError(msg)
> > AssertionError: Message "non-broadcastable output operand with shape
> > (%lld,%lld)
> >   doesn't match the broadcast shape (%lld,%lld,%lld)" doesn't contain
> > operand shape (2,3)
> >
> >
> > I've pushed a fix for this to the 1.6.x branch, can you confirm that it
> > works on win-amd64?
> >
> > Thanks,
> > Mark
> >
> > 
> >
>
> Sorry, I forgot to mention that this test failed on 64-bit Python 2.6
> only. I now recognize it is due to a known issue with Python's
> PyErr_Format function . Unfortunately
> the fix will not be backported to Python 2.6. Maybe this test could be
> marked as known failure on win-amd64-py2.6?
>
> Could you please revert your changes or set the format specifier to
> "%lld"? "%I64d" is not supported by PyErr_Format.
>

I've committed an attempted workaround using the %zd formatter, can you
check whether it works on 64-bit Windows Python 2.6 now?

Thanks,
Mark


>
> Thanks,
>
> Christoph
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] ANN: Numpy 1.6.0 beta 2

[Mark Wiebe]

On Tue, Apr 5, 2011 at 5:07 PM, Christoph Gohlke  wrote:

>
>
> On 4/5/2011 4:05 PM, Mark Wiebe wrote:
> > On Mon, Apr 4, 2011 at 9:10 PM, Christoph Gohlke  > > wrote:
> >
> >
> > 
> >
> > A few numpy tests fail on win-amd64:
> >
> > 
> >
> >
> ==
> > FAIL: test_iterator.test_iter_broadcasting_errors
> >
> --
> > Traceback (most recent call last):
> >File "X:\Python26-x64\lib\site-packages\nose\case.py", line 187,
> in
> > runTest
> >  self.test(*self.arg)
> >File
> >
> "X:\Python26-x64\lib\site-packages\numpy\core\tests\test_iterator.py",
> > line 639, in test_iter_broadcasting_errors
> > 'Message "%s" doesn\'t contain operand shape (2,3)' % msg)
> >File "X:\Python26-x64\lib\site-packages\numpy\testing\utils.py",
> line
> > 34, in assert_
> >  raise AssertionError(msg)
> > AssertionError: Message "non-broadcastable output operand with shape
> > (%lld,%lld)
> >   doesn't match the broadcast shape (%lld,%lld,%lld)" doesn't contain
> > operand shape (2,3)
> >
> >
> > I've pushed a fix for this to the 1.6.x branch, can you confirm that it
> > works on win-amd64?
> >
> > Thanks,
> > Mark
> >
> > 
> >
>
> Sorry, I forgot to mention that this test failed on 64-bit Python 2.6
> only. I now recognize it is due to a known issue with Python's
> PyErr_Format function . Unfortunately
> the fix will not be backported to Python 2.6. Maybe this test could be
> marked as known failure on win-amd64-py2.6?
>
> Could you please revert your changes or set the format specifier to
> "%lld"? "%I64d" is not supported by PyErr_Format.
>

This means PyString_Format and PyOS_snprintf support different sets of
formatting characters, which begs the question of which set of functions
should the NPY_*_FMT macros be intended for? Currently it looks like just
the NPY_INTP_FMT is used with PyString_Format/PyErr_Format, and the rest are
used with PyOS_snprintf. Also, in other places npy_intp variables are cast
to long and "%ld" is used instead of NPY_INTP_FMT. I suppose I'll just
change NPY_INTP_FMT unconditionally to "%lld" when it's long long, since
that's what the PyString_Format 2.7 documentation says is the correct
portable formatter.

-Mark


>
> Thanks,
>
> Christoph
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] ANN: Numpy 1.6.0 beta 2

[Christoph Gohlke]

On 4/5/2011 4:05 PM, Mark Wiebe wrote:
> On Mon, Apr 4, 2011 at 9:10 PM, Christoph Gohlke  > wrote:
>
>
> 
>
> A few numpy tests fail on win-amd64:
>
> 
>
> ==
> FAIL: test_iterator.test_iter_broadcasting_errors
> --
> Traceback (most recent call last):
>File "X:\Python26-x64\lib\site-packages\nose\case.py", line 187, in
> runTest
>  self.test(*self.arg)
>File
> "X:\Python26-x64\lib\site-packages\numpy\core\tests\test_iterator.py",
> line 639, in test_iter_broadcasting_errors
> 'Message "%s" doesn\'t contain operand shape (2,3)' % msg)
>File "X:\Python26-x64\lib\site-packages\numpy\testing\utils.py", line
> 34, in assert_
>  raise AssertionError(msg)
> AssertionError: Message "non-broadcastable output operand with shape
> (%lld,%lld)
>   doesn't match the broadcast shape (%lld,%lld,%lld)" doesn't contain
> operand shape (2,3)
>
>
> I've pushed a fix for this to the 1.6.x branch, can you confirm that it
> works on win-amd64?
>
> Thanks,
> Mark
>
> 
>

Sorry, I forgot to mention that this test failed on 64-bit Python 2.6 
only. I now recognize it is due to a known issue with Python's 
PyErr_Format function . Unfortunately 
the fix will not be backported to Python 2.6. Maybe this test could be 
marked as known failure on win-amd64-py2.6?

Could you please revert your changes or set the format specifier to 
"%lld"? "%I64d" is not supported by PyErr_Format.

Thanks,

Christoph
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] ANN: Numpy 1.6.0 beta 2

[Matthew Brett]

Hi,

On Tue, Apr 5, 2011 at 4:12 PM, Christopher Barker
 wrote:
> On 4/5/11 3:36 PM, josef.p...@gmail.com wrote:
>>> I disagree that U makes no sense for binary file reading.
>
> I wasn't saying that it made no sense to have a "U" mode for binary file
> reading, what I meant is that by the python2 definition, it made no
> sense. In Python 2, the ONLY difference between binary and text mode is
> line-feed translation.

I think it's right to say that the difference between a text and a
binary file in python 2 is - none for unix, and '\r\n' -> '\n'
translation in windows.

The difference between 'rt' and 'U' is (this is for my own benefit):

For 'rt', a '\r' does not cause a line break - with 'U' - it does.
For 'rt' _not_ on Windows, '\r\n' stays the same - it is stripped to
'\n' with 'U'.

> As for Python 3:
>
>>> In python 3:
>>>
>>> 'b' means, "return byte objects"
>>> 't' means "return decoded strings"
>>>
>>> 'U' means two things:
>>>
>>> 1) When iterating by line, split lines at any of '\r', '\r\n', '\n'
>>> 2) When returning lines split this way, convert '\r' and '\r\n' to '\n'
>
> a) 'U' is default -- it's essentially the same as 't' (in PY3), so 't'
> means "return decoded and line-feed translated unicode objects"

Right - my argument is that the behavior implied by 'U' and 't' is
conceptually separable.   'U' is for how to do line-breaks, and
line-termination translations, 't' is for whether to decode the text
or not.  In python 3.

> b) I think the line-feed conversion is done regardless of if you are
> iterating by lines, i.e. with a full-on .read(). At least that's how it
> works in py2 -- not running py3 here to test.

Yes, that looks right.

>>> If you support returning lines from a binary file (which python 3
>>> does), then I think 'U' is a sensible thing to allow - as in this
>>> case.
>
> but what is a "binary file"?

In python 3 a binary file is a file which is not decoded, and returns
bytes.  It still has a concept of a 'line', as defined by line
terminators - you can iterate over one, or do .readlines().  In python
2, as you say, a binary file is essentially the same as a text file,
with the single exception of the windows \r\n -> \n translation.

> I THINK what you are proposing is that we'd want to be able to have both
> linefeed translation and no decoding done. But I think that's impossible
> -- aren't the linefeeds themselves encoded differently with different
> encodings?

Right - so obviously if you open a utf-16 file as binary, terrible
things may happen - this was what Pauli was pointing out before.  His
point was that utf-8 is the standard, and that we probably would not
hit many other encodings.I agree with you if you are saying that
it would be good to be able to deal with them if we can - presumably
by allowing 'rt' file objects, producing python 3 strings.

>> U looks appropriate in this case, better than the workarounds.
>> However, to me the python 3.2 docs seem to say that U only works for
>> text mode
>
> Agreed -- but I don't see the problem -- your files are either encoded
> in something that might treat newlines differently (UCS32, maybe?), in
> which case you'd want it decoded, or you are working with ascii or ansi
> or utf-8, in which case you can specify the encoding anyway.
>
> I don't understand why we'd want a binary blob for text parsing -- the
> parsing code is going to have to know something about the encoding to
> work -- it might as well get passed in to the file open call, and work
> with unicode. I suppose if we still want to assume ascii for parsing,
> then we could use 't' and then re-encode to ascii to work with it. Which
> I agree does seem heavy handed just for fixing newlines.
>
> Also, one problem I've often had with encodings is what happens if I
> think I have ascii, but really have a couple characters above 127 --
> then the default is to get an error in decoding. I'd like to be able to
> pass in a flag that either skips the un-decodable characters or replaces
> them with something, but it doesn't look like you can do that with the
> file open function in py3.
>
>> The line terminator is always b'\n' for binary files;
>
> Once you really make the distiction between text and binary, the concept
> of a "line terminator" doesn't really make sense anyway.

Well - I was arguing that, given we can iterate over lines in binary
files, then there must be the concept of what a line is, in a binary
file, and that means that we need the concept of a line terminator.

I realize this is a discussion that would have to happen on the
python-dev list...

See you,

Matthew
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] ANN: Numpy 1.6.0 beta 2

[Christopher Barker]

On 4/5/11 3:36 PM, josef.p...@gmail.com wrote:
>> I disagree that U makes no sense for binary file reading.

I wasn't saying that it made no sense to have a "U" mode for binary file 
reading, what I meant is that by the python2 definition, it made no 
sense. In Python 2, the ONLY difference between binary and text mode is 
line-feed translation.

As for Python 3:

>> In python 3:
>>
>> 'b' means, "return byte objects"
>> 't' means "return decoded strings"
>>
>> 'U' means two things:
>>
>> 1) When iterating by line, split lines at any of '\r', '\r\n', '\n'
>> 2) When returning lines split this way, convert '\r' and '\r\n' to '\n'

a) 'U' is default -- it's essentially the same as 't' (in PY3), so 't' 
means "return decoded and line-feed translated unicode objects"

b) I think the line-feed conversion is done regardless of if you are 
iterating by lines, i.e. with a full-on .read(). At least that's how it 
works in py2 -- not running py3 here to test.

>> If you support returning lines from a binary file (which python 3
>> does), then I think 'U' is a sensible thing to allow - as in this
>> case.

but what is a "binary file"?

I THINK what you are proposing is that we'd want to be able to have both 
linefeed translation and no decoding done. But I think that's impossible 
-- aren't the linefeeds themselves encoded differently with different 
encodings?

> U looks appropriate in this case, better than the workarounds.
> However, to me the python 3.2 docs seem to say that U only works for
> text mode

Agreed -- but I don't see the problem -- your files are either encoded 
in something that might treat newlines differently (UCS32, maybe?), in 
which case you'd want it decoded, or you are working with ascii or ansi 
or utf-8, in which case you can specify the encoding anyway.

I don't understand why we'd want a binary blob for text parsing -- the 
parsing code is going to have to know something about the encoding to 
work -- it might as well get passed in to the file open call, and work 
with unicode. I suppose if we still want to assume ascii for parsing, 
then we could use 't' and then re-encode to ascii to work with it. Which 
I agree does seem heavy handed just for fixing newlines.

Also, one problem I've often had with encodings is what happens if I 
think I have ascii, but really have a couple characters above 127 -- 
then the default is to get an error in decoding. I'd like to be able to 
pass in a flag that either skips the un-decodable characters or replaces 
them with something, but it doesn't look like you can do that with the 
file open function in py3.

> The line terminator is always b'\n' for binary files;

Once you really make the distiction between text and binary, the concept 
of a "line terminator" doesn't really make sense anyway.

In the ansi world, everyone should always have used 'U' for text. It 
probably would have been the default if it had been there from the 
beginning. People got away without it because:
  1) dos line feeds have a "\n" in them anyway
  2) most if the time it doesn't matter that there is an extra 
whitespace charater inther
  3) darn few of us ever had to deal with the mac "\r"

Now that we are in a unicode world (at least a little) there is simply 
no way around the fact that you can't reliably read a file without 
knowing how it is encoded.

My thought at this point is to say that the numpy text file reading 
stuff only works on 1byte, ansi encoding (nad maybe only ascii), and be 
done with it. utf-8 might be OK -- I don't know if there are any valid 
files in, say latin-1 that utf-8 will barf on -- you may not get the 
non-ascii symbols right, but that's better than barfing.

-Chris




-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] ANN: Numpy 1.6.0 beta 2

[Mark Wiebe]

On Mon, Apr 4, 2011 at 9:10 PM, Christoph Gohlke  wrote:

>
> 
>
> A few numpy tests fail on win-amd64:
>
> 
>
> ==
> FAIL: test_iterator.test_iter_broadcasting_errors
> --
> Traceback (most recent call last):
>File "X:\Python26-x64\lib\site-packages\nose\case.py", line 187, in
> runTest
> self.test(*self.arg)
>   File
> "X:\Python26-x64\lib\site-packages\numpy\core\tests\test_iterator.py",
> line 639, in test_iter_broadcasting_errors
> 'Message "%s" doesn\'t contain operand shape (2,3)' % msg)
>   File "X:\Python26-x64\lib\site-packages\numpy\testing\utils.py", line
> 34, in assert_
> raise AssertionError(msg)
> AssertionError: Message "non-broadcastable output operand with shape
> (%lld,%lld)
>  doesn't match the broadcast shape (%lld,%lld,%lld)" doesn't contain
> operand shape (2,3)
>

I've pushed a fix for this to the 1.6.x branch, can you confirm that it
works on win-amd64?

Thanks,
Mark


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] ANN: Numpy 1.6.0 beta 2

[josef . pktd]

On Tue, Apr 5, 2011 at 5:56 PM, Matthew Brett  wrote:
> Hi,
>
> On Tue, Apr 5, 2011 at 9:46 AM, Christopher Barker
>  wrote:
>> On 4/4/11 10:35 PM, Charles R Harris wrote:
>>>     IIUC, "Ub" is undefined -- "U" means universal newlines, which makes no
>>>     sense when used with "b" for binary. I looked at the code a ways back,
>>>     and I can't remember the resolution order, but there isn't any checking
>>>     for incompatible flags.
>
> I disagree that U makes no sense for binary file reading.
>
> In python 3:
>
> 'b' means, "return byte objects"
> 't' means "return decoded strings"
>
> 'U' means two things:
>
> 1) When iterating by line, split lines at any of '\r', '\r\n', '\n'
> 2) When returning lines split this way, convert '\r' and '\r\n' to '\n'
>
> If you support returning lines from a binary file (which python 3
> does), then I think 'U' is a sensible thing to allow - as in this
> case.

U looks appropriate in this case, better than the workarounds.
However, to me the python 3.2 docs seem to say that U only works for
text mode and readline only takes \n as line separator

readline(limit=-1)
Read and return one line from the stream. If limit is specified, at
most limit bytes will be read.

The line terminator is always b'\n' for binary files; for text files,
the newlines argument to open() can be used to select the line
terminator(s) recognized.

Josef


>
> Best,
>
> Matthew
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] ANN: Numpy 1.6.0 beta 2

[Matthew Brett]

Hi,

On Tue, Apr 5, 2011 at 9:46 AM, Christopher Barker
 wrote:
> On 4/4/11 10:35 PM, Charles R Harris wrote:
>>     IIUC, "Ub" is undefined -- "U" means universal newlines, which makes no
>>     sense when used with "b" for binary. I looked at the code a ways back,
>>     and I can't remember the resolution order, but there isn't any checking
>>     for incompatible flags.

I disagree that U makes no sense for binary file reading.

In python 3:

'b' means, "return byte objects"
't' means "return decoded strings"

'U' means two things:

1) When iterating by line, split lines at any of '\r', '\r\n', '\n'
2) When returning lines split this way, convert '\r' and '\r\n' to '\n'

If you support returning lines from a binary file (which python 3
does), then I think 'U' is a sensible thing to allow - as in this
case.

Best,

Matthew
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] ANN: Numpy 1.6.0 beta 2

[Matthew Brett]

Hi,

On Tue, Apr 5, 2011 at 10:56 AM, Charles R Harris
 wrote:
>
>
> On Tue, Apr 5, 2011 at 11:45 AM,  wrote:
>>
>> On Tue, Apr 5, 2011 at 1:20 PM, Charles R Harris
>>  wrote:
>> >
>> >
>> > On Tue, Apr 5, 2011 at 10:46 AM, Christopher Barker
>> > 
>> > wrote:
>> >>
>> >> On 4/4/11 10:35 PM, Charles R Harris wrote:
>> >> >     IIUC, "Ub" is undefined -- "U" means universal newlines, which
>> >> > makes
>> >> > no
>> >> >     sense when used with "b" for binary. I looked at the code a ways
>> >> > back,
>> >> >     and I can't remember the resolution order, but there isn't any
>> >> > checking
>> >> >     for incompatible flags.
>> >> >
>> >> >     I'd expect that genfromtxt, being txt, and line oriented, should
>> >> > use
>> >> >     'rU'. but if it wants the raw line endings (why would it?) then
>> >> > rb
>> >> >     should be fine.
>> >> >
>> >> >
>> >> > "U" has been kept around for backwards compatibility, the python
>> >> > documentation recommends that it not be used for new code.
>> >>
>> >> That is for  3.*  -- the 2.7.* docs say:
>> >>
>> >> """
>> >> In addition to the standard fopen() values mode may be 'U' or 'rU'.
>> >> Python is usually built with universal newline support; supplying 'U'
>> >> opens the file as a text file, but lines may be terminated by any of
>> >> the
>> >> following: the Unix end-of-line convention '\n', the Macintosh
>> >> convention '\r', or the Windows convention '\r\n'. All of these
>> >> external
>> >> representations are seen as '\n' by the Python program. If Python is
>> >> built without universal newline support a mode with 'U' is the same as
>> >> normal text mode. Note that file objects so opened also have an
>> >> attribute called newlines which has a value of None (if no newlines
>> >> have
>> >> yet been seen), '\n', '\r', '\r\n', or a tuple containing all the
>> >> newline types seen.
>> >>
>> >> Python enforces that the mode, after stripping 'U', begins with 'r',
>> >> 'w'
>> >> or 'a'.
>> >> ""
>> >>
>> >> which does, in fact indicate that 'Ub' is NOT allowed. We should be
>> >> using 'Ur', I think. Maybe the "python enforces" is what we saw the
>> >> error from -- it didn't used to enforce anything.
>> >>
>> >
>> > 'rbU' works and I put that in as a quick fix.
>> >>
>> >> On 4/5/11 7:12 AM, Charles R Harris wrote:
>> >>
>> >> > The 'Ub' mode doesn't work for '\r' on python 3. This may be a bug in
>> >> > python, as it works just fine on python 2.7.
>> >>
>> >> "Ub" never made any sense anywhere -- "U" means universal newline text
>> >> file. "b" means binary -- combining them makes no sense. On older
>> >> pythons, the behaviour of 'Ub' was undefined -- now, it looks like it
>> >> is
>> >> supposed to raise an error.
>> >>
>> >> does 'Ur' work with \r line endings on Python 3?
>> >
>> > Yes.
>> >
>> >>
>> >> According to my read of the docs, 'U' does nothing -- "universal"
>> >> newline support is supposed to be the default:
>> >>
>> >> """
>> >> On input, if newline is None, universal newlines mode is enabled. Lines
>> >> in the input can end in '\n', '\r', or '\r\n', and these are translated
>> >> into '\n' before being returned to the caller.
>> >> """
>> >>
>> >> > It may indeed be desirable
>> >> > to read the files as text, but that would require more work on both
>> >> > loadtxt and genfromtxt.
>> >>
>> >> Why can't we just open the file with mode 'Ur'? text is text, messing
>> >> with line endings shouldn't hurt anything, and it might help.
>> >>
>> >
>> > Well, text in the files then gets the numpy 'U' type instead of 'S', and
>> > there are places where byte streams are assumed for stripping and such.
>> > Which is to say that changing to text mode requires some work. Another
>> > possibility is to use a generator:
>> >
>> > def usetext(fname):
>> >     f = open(fname, 'rt')
>> >     for l in f:
>> >        yield asbytes(f.next())
>> >
>> > I think genfromtxt could use a refactoring and cleanup, but probably not
>> > for
>> > 1.6.
>>
>> I think it should also be possible to read "rb" and strip any \r, \r\n
>> in _iotools.py,
>> that's were the bytes are used, from my reading and the initial error
>> message.
>>
>
> Doesn't work for \r, you get the whole file at once instead of line by line.

Thanks for trying to sort out this ugliness.  I've added another pull request:

https://github.com/numpy/numpy/pull/71

 - tests for \n \r\n and \r files, raising skiptest for currently
failing 3.2 \r mode.

Matthew
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] ANN: Numpy 1.6.0 beta 2

[Charles R Harris]

On Tue, Apr 5, 2011 at 11:45 AM,  wrote:

> On Tue, Apr 5, 2011 at 1:20 PM, Charles R Harris
>  wrote:
> >
> >
> > On Tue, Apr 5, 2011 at 10:46 AM, Christopher Barker <
> chris.bar...@noaa.gov>
> > wrote:
> >>
> >> On 4/4/11 10:35 PM, Charles R Harris wrote:
> >> > IIUC, "Ub" is undefined -- "U" means universal newlines, which
> makes
> >> > no
> >> > sense when used with "b" for binary. I looked at the code a ways
> >> > back,
> >> > and I can't remember the resolution order, but there isn't any
> >> > checking
> >> > for incompatible flags.
> >> >
> >> > I'd expect that genfromtxt, being txt, and line oriented, should
> use
> >> > 'rU'. but if it wants the raw line endings (why would it?) then rb
> >> > should be fine.
> >> >
> >> >
> >> > "U" has been kept around for backwards compatibility, the python
> >> > documentation recommends that it not be used for new code.
> >>
> >> That is for  3.*  -- the 2.7.* docs say:
> >>
> >> """
> >> In addition to the standard fopen() values mode may be 'U' or 'rU'.
> >> Python is usually built with universal newline support; supplying 'U'
> >> opens the file as a text file, but lines may be terminated by any of the
> >> following: the Unix end-of-line convention '\n', the Macintosh
> >> convention '\r', or the Windows convention '\r\n'. All of these external
> >> representations are seen as '\n' by the Python program. If Python is
> >> built without universal newline support a mode with 'U' is the same as
> >> normal text mode. Note that file objects so opened also have an
> >> attribute called newlines which has a value of None (if no newlines have
> >> yet been seen), '\n', '\r', '\r\n', or a tuple containing all the
> >> newline types seen.
> >>
> >> Python enforces that the mode, after stripping 'U', begins with 'r', 'w'
> >> or 'a'.
> >> ""
> >>
> >> which does, in fact indicate that 'Ub' is NOT allowed. We should be
> >> using 'Ur', I think. Maybe the "python enforces" is what we saw the
> >> error from -- it didn't used to enforce anything.
> >>
> >
> > 'rbU' works and I put that in as a quick fix.
> >>
> >> On 4/5/11 7:12 AM, Charles R Harris wrote:
> >>
> >> > The 'Ub' mode doesn't work for '\r' on python 3. This may be a bug in
> >> > python, as it works just fine on python 2.7.
> >>
> >> "Ub" never made any sense anywhere -- "U" means universal newline text
> >> file. "b" means binary -- combining them makes no sense. On older
> >> pythons, the behaviour of 'Ub' was undefined -- now, it looks like it is
> >> supposed to raise an error.
> >>
> >> does 'Ur' work with \r line endings on Python 3?
> >
> > Yes.
> >
> >>
> >> According to my read of the docs, 'U' does nothing -- "universal"
> >> newline support is supposed to be the default:
> >>
> >> """
> >> On input, if newline is None, universal newlines mode is enabled. Lines
> >> in the input can end in '\n', '\r', or '\r\n', and these are translated
> >> into '\n' before being returned to the caller.
> >> """
> >>
> >> > It may indeed be desirable
> >> > to read the files as text, but that would require more work on both
> >> > loadtxt and genfromtxt.
> >>
> >> Why can't we just open the file with mode 'Ur'? text is text, messing
> >> with line endings shouldn't hurt anything, and it might help.
> >>
> >
> > Well, text in the files then gets the numpy 'U' type instead of 'S', and
> > there are places where byte streams are assumed for stripping and such.
> > Which is to say that changing to text mode requires some work. Another
> > possibility is to use a generator:
> >
> > def usetext(fname):
> > f = open(fname, 'rt')
> > for l in f:
> >yield asbytes(f.next())
> >
> > I think genfromtxt could use a refactoring and cleanup, but probably not
> for
> > 1.6.
>
> I think it should also be possible to read "rb" and strip any \r, \r\n
> in _iotools.py,
> that's were the bytes are used, from my reading and the initial error
> message.
>
>
Doesn't work for \r, you get the whole file at once instead of line by line.

Chuck
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] ANN: Numpy 1.6.0 beta 2

[josef . pktd]

On Tue, Apr 5, 2011 at 1:20 PM, Charles R Harris
 wrote:
>
>
> On Tue, Apr 5, 2011 at 10:46 AM, Christopher Barker 
> wrote:
>>
>> On 4/4/11 10:35 PM, Charles R Harris wrote:
>> >     IIUC, "Ub" is undefined -- "U" means universal newlines, which makes
>> > no
>> >     sense when used with "b" for binary. I looked at the code a ways
>> > back,
>> >     and I can't remember the resolution order, but there isn't any
>> > checking
>> >     for incompatible flags.
>> >
>> >     I'd expect that genfromtxt, being txt, and line oriented, should use
>> >     'rU'. but if it wants the raw line endings (why would it?) then rb
>> >     should be fine.
>> >
>> >
>> > "U" has been kept around for backwards compatibility, the python
>> > documentation recommends that it not be used for new code.
>>
>> That is for  3.*  -- the 2.7.* docs say:
>>
>> """
>> In addition to the standard fopen() values mode may be 'U' or 'rU'.
>> Python is usually built with universal newline support; supplying 'U'
>> opens the file as a text file, but lines may be terminated by any of the
>> following: the Unix end-of-line convention '\n', the Macintosh
>> convention '\r', or the Windows convention '\r\n'. All of these external
>> representations are seen as '\n' by the Python program. If Python is
>> built without universal newline support a mode with 'U' is the same as
>> normal text mode. Note that file objects so opened also have an
>> attribute called newlines which has a value of None (if no newlines have
>> yet been seen), '\n', '\r', '\r\n', or a tuple containing all the
>> newline types seen.
>>
>> Python enforces that the mode, after stripping 'U', begins with 'r', 'w'
>> or 'a'.
>> ""
>>
>> which does, in fact indicate that 'Ub' is NOT allowed. We should be
>> using 'Ur', I think. Maybe the "python enforces" is what we saw the
>> error from -- it didn't used to enforce anything.
>>
>
> 'rbU' works and I put that in as a quick fix.
>>
>> On 4/5/11 7:12 AM, Charles R Harris wrote:
>>
>> > The 'Ub' mode doesn't work for '\r' on python 3. This may be a bug in
>> > python, as it works just fine on python 2.7.
>>
>> "Ub" never made any sense anywhere -- "U" means universal newline text
>> file. "b" means binary -- combining them makes no sense. On older
>> pythons, the behaviour of 'Ub' was undefined -- now, it looks like it is
>> supposed to raise an error.
>>
>> does 'Ur' work with \r line endings on Python 3?
>
> Yes.
>
>>
>> According to my read of the docs, 'U' does nothing -- "universal"
>> newline support is supposed to be the default:
>>
>> """
>> On input, if newline is None, universal newlines mode is enabled. Lines
>> in the input can end in '\n', '\r', or '\r\n', and these are translated
>> into '\n' before being returned to the caller.
>> """
>>
>> > It may indeed be desirable
>> > to read the files as text, but that would require more work on both
>> > loadtxt and genfromtxt.
>>
>> Why can't we just open the file with mode 'Ur'? text is text, messing
>> with line endings shouldn't hurt anything, and it might help.
>>
>
> Well, text in the files then gets the numpy 'U' type instead of 'S', and
> there are places where byte streams are assumed for stripping and such.
> Which is to say that changing to text mode requires some work. Another
> possibility is to use a generator:
>
> def usetext(fname):
>     f = open(fname, 'rt')
>     for l in f:
>        yield asbytes(f.next())
>
> I think genfromtxt could use a refactoring and cleanup, but probably not for
> 1.6.

I think it should also be possible to read "rb" and strip any \r, \r\n
in _iotools.py,
that's were the bytes are used, from my reading and the initial error message.

Josef

>
> Chuck
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] ANN: Numpy 1.6.0 beta 2

[Charles R Harris]

On Tue, Apr 5, 2011 at 10:46 AM, Christopher Barker
wrote:

> On 4/4/11 10:35 PM, Charles R Harris wrote:
> > IIUC, "Ub" is undefined -- "U" means universal newlines, which makes
> no
> > sense when used with "b" for binary. I looked at the code a ways
> back,
> > and I can't remember the resolution order, but there isn't any
> checking
> > for incompatible flags.
> >
> > I'd expect that genfromtxt, being txt, and line oriented, should use
> > 'rU'. but if it wants the raw line endings (why would it?) then rb
> > should be fine.
> >
> >
> > "U" has been kept around for backwards compatibility, the python
> > documentation recommends that it not be used for new code.
>
> That is for  3.*  -- the 2.7.* docs say:
>
> """
> In addition to the standard fopen() values mode may be 'U' or 'rU'.
> Python is usually built with universal newline support; supplying 'U'
> opens the file as a text file, but lines may be terminated by any of the
> following: the Unix end-of-line convention '\n', the Macintosh
> convention '\r', or the Windows convention '\r\n'. All of these external
> representations are seen as '\n' by the Python program. If Python is
> built without universal newline support a mode with 'U' is the same as
> normal text mode. Note that file objects so opened also have an
> attribute called newlines which has a value of None (if no newlines have
> yet been seen), '\n', '\r', '\r\n', or a tuple containing all the
> newline types seen.
>
> Python enforces that the mode, after stripping 'U', begins with 'r', 'w'
> or 'a'.
> ""
>
> which does, in fact indicate that 'Ub' is NOT allowed. We should be
> using 'Ur', I think. Maybe the "python enforces" is what we saw the
> error from -- it didn't used to enforce anything.
>
>
'rbU' works and I put that in as a quick fix.

>
> On 4/5/11 7:12 AM, Charles R Harris wrote:
>
> > The 'Ub' mode doesn't work for '\r' on python 3. This may be a bug in
> > python, as it works just fine on python 2.7.
>
> "Ub" never made any sense anywhere -- "U" means universal newline text
> file. "b" means binary -- combining them makes no sense. On older
> pythons, the behaviour of 'Ub' was undefined -- now, it looks like it is
> supposed to raise an error.
>
> does 'Ur' work with \r line endings on Python 3?
>

Yes.


> According to my read of the docs, 'U' does nothing -- "universal"
> newline support is supposed to be the default:
>
> """
> On input, if newline is None, universal newlines mode is enabled. Lines
> in the input can end in '\n', '\r', or '\r\n', and these are translated
> into '\n' before being returned to the caller.
> """
>
> > It may indeed be desirable
> > to read the files as text, but that would require more work on both
> > loadtxt and genfromtxt.
>
> Why can't we just open the file with mode 'Ur'? text is text, messing
> with line endings shouldn't hurt anything, and it might help.
>
>
Well, text in the files then gets the numpy 'U' type instead of 'S', and
there are places where byte streams are assumed for stripping and such.
Which is to say that changing to text mode requires some work. Another
possibility is to use a generator:

def usetext(fname):
f = open(fname, 'rt')
for l in f:
   yield asbytes(f.next())

I think genfromtxt could use a refactoring and cleanup, but probably not for
1.6.

Chuck
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] ANN: Numpy 1.6.0 beta 2

[Christopher Barker]

On 4/4/11 10:35 PM, Charles R Harris wrote:
> IIUC, "Ub" is undefined -- "U" means universal newlines, which makes no
> sense when used with "b" for binary. I looked at the code a ways back,
> and I can't remember the resolution order, but there isn't any checking
> for incompatible flags.
>
> I'd expect that genfromtxt, being txt, and line oriented, should use
> 'rU'. but if it wants the raw line endings (why would it?) then rb
> should be fine.
>
>
> "U" has been kept around for backwards compatibility, the python
> documentation recommends that it not be used for new code.

That is for  3.*  -- the 2.7.* docs say:

"""
In addition to the standard fopen() values mode may be 'U' or 'rU'. 
Python is usually built with universal newline support; supplying 'U' 
opens the file as a text file, but lines may be terminated by any of the 
following: the Unix end-of-line convention '\n', the Macintosh 
convention '\r', or the Windows convention '\r\n'. All of these external 
representations are seen as '\n' by the Python program. If Python is 
built without universal newline support a mode with 'U' is the same as 
normal text mode. Note that file objects so opened also have an 
attribute called newlines which has a value of None (if no newlines have 
yet been seen), '\n', '\r', '\r\n', or a tuple containing all the 
newline types seen.

Python enforces that the mode, after stripping 'U', begins with 'r', 'w' 
or 'a'.
""

which does, in fact indicate that 'Ub' is NOT allowed. We should be 
using 'Ur', I think. Maybe the "python enforces" is what we saw the 
error from -- it didn't used to enforce anything.


On 4/5/11 7:12 AM, Charles R Harris wrote:

> The 'Ub' mode doesn't work for '\r' on python 3. This may be a bug in
> python, as it works just fine on python 2.7.

"Ub" never made any sense anywhere -- "U" means universal newline text 
file. "b" means binary -- combining them makes no sense. On older 
pythons, the behaviour of 'Ub' was undefined -- now, it looks like it is 
supposed to raise an error.

does 'Ur' work with \r line endings on Python 3?

According to my read of the docs, 'U' does nothing -- "universal" 
newline support is supposed to be the default:

"""
On input, if newline is None, universal newlines mode is enabled. Lines 
in the input can end in '\n', '\r', or '\r\n', and these are translated 
into '\n' before being returned to the caller.
"""

> It may indeed be desirable
> to read the files as text, but that would require more work on both
> loadtxt and genfromtxt.

Why can't we just open the file with mode 'Ur'? text is text, messing 
with line endings shouldn't hurt anything, and it might help.

If we stick with binary, then it comes down to:
- will having an extra \r with Windows files hurt anything? -- probably not.
- Are there many mac-style text files out there anymore? not many.

-Chris




-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] ANN: Numpy 1.6.0 beta 2

[George Nurser]

SUSE 11.3, python 2.7, gcc 4.3.4, gfortran from gcc 4.6.0, I get two failures on
commit 1439a8ddcb2eda20fa102aa44e846783f29c0af3 (head of 1.6.x
maintenance branch).

--George.


==
FAIL: Test basic arithmetic function errors
--
Traceback (most recent call last):
  File 
"/noc/users/agn/ext/SUSE_11/lib/python2.7/site-packages/numpy/testing/decorators.py",
line 215, in knownfailer
return f(*args, **kwargs)
  File 
"/noc/users/agn/ext/SUSE_11/lib/python2.7/site-packages/numpy/core/tests/test_numeric.py",
line 321, in test_floating_exceptions
lambda a,b:a/b, ft_tiny, ft_max)
  File 
"/noc/users/agn/ext/SUSE_11/lib/python2.7/site-packages/numpy/core/tests/test_numeric.py",
line 271, in assert_raises_fpe
"Type %s did not raise fpe error '%s'." % (ftype, fpeerr))
  File 
"/noc/users/agn/ext/SUSE_11/lib/python2.7/site-packages/numpy/testing/utils.py",
line 34, in assert_
raise AssertionError(msg)
AssertionError: Type  did not raise fpe error 'underflow'.

==
FAIL: test_kind.TestKind.test_all
--
Traceback (most recent call last):
  File "/noc/users/agn/ext/SUSE_11/lib/python2.7/site-packages/nose/case.py",
line 187, in runTest
self.test(*self.arg)
  File 
"/noc/users/agn/ext/SUSE_11/lib/python2.7/site-packages/numpy/f2py/tests/test_kind.py",
line 30, in test_all
'selectedrealkind(%s): expected %r but got %r' %  (i,
selected_real_kind(i), selectedrealkind(i)))
  File 
"/noc/users/agn/ext/SUSE_11/lib/python2.7/site-packages/numpy/testing/utils.py",
line 34, in assert_
raise AssertionError(msg)
AssertionError: selectedrealkind(19): expected -1 but got 16

--
Ran 3544 tests in 48.039s

FAILED (KNOWNFAIL=4, failures=2)



On 4 April 2011 21:04, Ralf Gommers  wrote:
> Hi,
>
> I am pleased to announce the availability of the second beta of NumPy
> 1.6.0. Due to the extensive changes in the Numpy core for this
> release, the beta testing phase will last at least one month. Please
> test this beta and report any problems on the Numpy mailing list.
>
> Many bug fixes went in since beta 1, among which:
> - fix installation of source release with python 3
> - f2py fixes for assumed shape arrays
> - several loadtxt bug fixes and enhancements
> - change default floating point error handling from "print" to "warn"
> - much more
>
> I quickly counted in the timeline, and in the last few weeks the
> number of open tickets has been decreased by over 100. Thanks to
> everyone who contributed to this spring cleaning!
>
> Sources and binaries can be found at
> http://sourceforge.net/projects/numpy/files/NumPy/1.6.0b2/
> For (preliminary) release notes see below.
>
> Enjoy,
> Ralf
>
>
>
> Note: NumPy 1.6.0 is not yet released.
>
>
> =
> NumPy 1.6.0 Release Notes
> =
>
> This release includes several new features as well as numerous bug fixes and
> improved documentation.  It is backward compatible with the 1.5.0 release, and
> supports Python 2.4 - 2.7 and 3.1 - 3.2.
>
>
> Highlights
> ==
>
> * Re-introduction of datetime dtype support to deal with dates in arrays.
>
> * A new 16-bit floating point type.
>
> * A new iterator, which improves performance of many functions.
>
>
> New features
> 
>
> New 16-bit floating point type
> --
>
> This release adds support for the IEEE 754-2008 binary16 format, available as
> the data type ``numpy.half``.  Within Python, the type behaves similarly to
> `float` or `double`, and C extensions can add support for it with the exposed
> half-float API.
>
>
> New iterator
> 
>
> A new iterator has been added, replacing the functionality of the
> existing iterator and multi-iterator with a single object and API.
> This iterator works well with general memory layouts different from
> C or Fortran contiguous, and handles both standard NumPy and
> customized broadcasting. The buffering, automatic data type
> conversion, and optional output parameters, offered by
> ufuncs but difficult to replicate elsewhere, are now exposed by this
> iterator.
>
>
> Legendre, Laguerre, Hermite, HermiteE polynomials in ``numpy.polynomial``
> -
>
> Extend the number of polynomials available in the polynomial package. In
> addition, a new ``window`` attribute has been added to the classes in
> order to specify the range the ``domain`` maps to. This is mostly useful
> for the Laguerre, Hermite, and HermiteE polynomials whose natural domains
> are infinite and provides a more intuitive way to get the correct mapping
> of values without playing unnatural tricks with the domain.
>
>
> Fortran as

Re: [Numpy-discussion] ANN: Numpy 1.6.0 beta 2

[Charles R Harris]

On Tue, Apr 5, 2011 at 8:12 AM, Charles R Harris
wrote:

>
>
> On Mon, Apr 4, 2011 at 11:22 PM, Chris Barker wrote:
>
>> On 4/4/11 9:03 PM, josef.p...@gmail.com wrote:
>> > On Mon, Apr 4, 2011 at 11:42 PM, Charles R Harris
>>
>> >>>File "/sw/lib/python2.4/site-packages/numpy/lib/_datasource.py",
>> >>> line 477, in open
>> >>>  return _file_openers[ext](found, mode=mode)
>> >>> IOError: invalid mode: Ub
>> >>>
>> >>
>> >> Guess that wasn't tested before ;) I thought that was strange when I
>> saw it.
>> >> The source of the problem is line 2035 in npyio.py. Additionally, Since
>> >> genloadtxt needs to have byte strings the 'rb" mode should probably be
>> used.
>> >> That works on linux, both for python 2 and python 3, but doing that
>> might
>> >> uncover genfromtxt problems on other platforms.
>>
>> > "rb" is fine on Windows with python 3.2, (that's what I tested
>> > initially for this bug)
>>
>> IIUC, "Ub" is undefined -- "U" means universal newlines, which makes no
>> sense when used with "b" for binary. I looked at the code a ways back,
>> and I can't remember the resolution order, but there isn't any checking
>> for incompatible flags.
>>
>> I'd expect that genfromtxt, being txt, and line oriented, should use
>> 'rU'. but if it wants the raw line endings (why would it?) then rb
>> should be fine.
>>
>> Note that if you only test with unix (\n) and dos (\r\n) line endings,
>> it may work with 'b', if it's splitting on '\n', but not if you try it
>> with Mac endings (\r). Of course with OS-X mac endings are getting
>> pretty uncommon.
>>
>>
> The 'Ub' mode doesn't work for '\r' on python 3. This may be a bug in
> python, as it works just fine on python 2.7. It may indeed be desirable to
> read the files as text, but that would require more work on both loadtxt and
> genfromtxt.
>
>
Curiously, 'rbU' and 'rU' do work on 2.4, but not 'Urb',  'Ub', 'Ur', or
'bU'. Due to Python 3 not hadling '\r' in binary mode, the shortest path
forward might be to open files as text and call asbytes on the lines as they
are read.

Chuck
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] ANN: Numpy 1.6.0 beta 2

[Charles R Harris]

On Mon, Apr 4, 2011 at 11:22 PM, Chris Barker  wrote:

> On 4/4/11 9:03 PM, josef.p...@gmail.com wrote:
> > On Mon, Apr 4, 2011 at 11:42 PM, Charles R Harris
>
> >>>File "/sw/lib/python2.4/site-packages/numpy/lib/_datasource.py",
> >>> line 477, in open
> >>>  return _file_openers[ext](found, mode=mode)
> >>> IOError: invalid mode: Ub
> >>>
> >>
> >> Guess that wasn't tested before ;) I thought that was strange when I saw
> it.
> >> The source of the problem is line 2035 in npyio.py. Additionally, Since
> >> genloadtxt needs to have byte strings the 'rb" mode should probably be
> used.
> >> That works on linux, both for python 2 and python 3, but doing that
> might
> >> uncover genfromtxt problems on other platforms.
>
> > "rb" is fine on Windows with python 3.2, (that's what I tested
> > initially for this bug)
>
> IIUC, "Ub" is undefined -- "U" means universal newlines, which makes no
> sense when used with "b" for binary. I looked at the code a ways back,
> and I can't remember the resolution order, but there isn't any checking
> for incompatible flags.
>
> I'd expect that genfromtxt, being txt, and line oriented, should use
> 'rU'. but if it wants the raw line endings (why would it?) then rb
> should be fine.
>
> Note that if you only test with unix (\n) and dos (\r\n) line endings,
> it may work with 'b', if it's splitting on '\n', but not if you try it
> with Mac endings (\r). Of course with OS-X mac endings are getting
> pretty uncommon.
>
>
The 'Ub' mode doesn't work for '\r' on python 3. This may be a bug in
python, as it works just fine on python 2.7. It may indeed be desirable to
read the files as text, but that would require more work on both loadtxt and
genfromtxt.

Chuck
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] ANN: Numpy 1.6.0 beta 2

[Alan G Isaac]

Superpack is OK on Windows Vista 64bit with Python 2.7 32bit.
Alan Isaac

Ran 2993 tests in 25.707s

OK (KNOWNFAIL=9, SKIP=5)
Running unit tests for numpy
NumPy version 1.6.0b2
NumPy is installed in c:\Python27\lib\site-packages\numpy
Python version 2.7 (r27:82525, Jul  4 2010, 09:01:59) [MSC v.1500 32 bit 
(Intel)]
nose version 0.11.0
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] ANN: Numpy 1.6.0 beta 2

[Charles R Harris]

On Mon, Apr 4, 2011 at 11:22 PM, Chris Barker  wrote:

> On 4/4/11 9:03 PM, josef.p...@gmail.com wrote:
> > On Mon, Apr 4, 2011 at 11:42 PM, Charles R Harris
>
> >>>File "/sw/lib/python2.4/site-packages/numpy/lib/_datasource.py",
> >>> line 477, in open
> >>>  return _file_openers[ext](found, mode=mode)
> >>> IOError: invalid mode: Ub
> >>>
> >>
> >> Guess that wasn't tested before ;) I thought that was strange when I saw
> it.
> >> The source of the problem is line 2035 in npyio.py. Additionally, Since
> >> genloadtxt needs to have byte strings the 'rb" mode should probably be
> used.
> >> That works on linux, both for python 2 and python 3, but doing that
> might
> >> uncover genfromtxt problems on other platforms.
>
> > "rb" is fine on Windows with python 3.2, (that's what I tested
> > initially for this bug)
>
> IIUC, "Ub" is undefined -- "U" means universal newlines, which makes no
> sense when used with "b" for binary. I looked at the code a ways back,
> and I can't remember the resolution order, but there isn't any checking
> for incompatible flags.
>
> I'd expect that genfromtxt, being txt, and line oriented, should use
> 'rU'. but if it wants the raw line endings (why would it?) then rb
> should be fine.
>
>
"U" has been kept around for backwards compatibility, the python
documentation recommends that it not be used for new code. Whatever 'new'
means in this case ;) If it is unneeded for python 2.4 I say drop it.

Note that if you only test with unix (\n) and dos (\r\n) line endings,
> it may work with 'b', if it's splitting on '\n', but not if you try it
> with Mac endings (\r). Of course with OS-X mac endings are getting
> pretty uncommon.
>


I suppose we could try the different line endings on a single platform and
see what happens. It would be nice to know just how portable text really is.

Chuck
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] ANN: Numpy 1.6.0 beta 2

[Chris Barker]

On 4/4/11 9:03 PM, josef.p...@gmail.com wrote:
> On Mon, Apr 4, 2011 at 11:42 PM, Charles R Harris

>>>File "/sw/lib/python2.4/site-packages/numpy/lib/_datasource.py",
>>> line 477, in open
>>>  return _file_openers[ext](found, mode=mode)
>>> IOError: invalid mode: Ub
>>>
>>
>> Guess that wasn't tested before ;) I thought that was strange when I saw it.
>> The source of the problem is line 2035 in npyio.py. Additionally, Since
>> genloadtxt needs to have byte strings the 'rb" mode should probably be used.
>> That works on linux, both for python 2 and python 3, but doing that might
>> uncover genfromtxt problems on other platforms.

> "rb" is fine on Windows with python 3.2, (that's what I tested
> initially for this bug)

IIUC, "Ub" is undefined -- "U" means universal newlines, which makes no 
sense when used with "b" for binary. I looked at the code a ways back, 
and I can't remember the resolution order, but there isn't any checking 
for incompatible flags.

I'd expect that genfromtxt, being txt, and line oriented, should use 
'rU'. but if it wants the raw line endings (why would it?) then rb 
should be fine.

Note that if you only test with unix (\n) and dos (\r\n) line endings, 
it may work with 'b', if it's splitting on '\n', but not if you try it 
with Mac endings (\r). Of course with OS-X mac endings are getting 
pretty uncommon.

-Chris


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] ANN: Numpy 1.6.0 beta 2

[josef . pktd]

On Tue, Apr 5, 2011 at 12:10 AM, Christoph Gohlke  wrote:
>
>
> On 4/4/2011 1:04 PM, Ralf Gommers wrote:
>> Hi,
>>
>> I am pleased to announce the availability of the second beta of NumPy
>> 1.6.0. Due to the extensive changes in the Numpy core for this
>> release, the beta testing phase will last at least one month. Please
>> test this beta and report any problems on the Numpy mailing list.
>>
>> Many bug fixes went in since beta 1, among which:
>> - fix installation of source release with python 3
>> - f2py fixes for assumed shape arrays
>> - several loadtxt bug fixes and enhancements
>> - change default floating point error handling from "print" to "warn"
>> - much more
>>
>> I quickly counted in the timeline, and in the last few weeks the
>> number of open tickets has been decreased by over 100. Thanks to
>> everyone who contributed to this spring cleaning!
>>
>> Sources and binaries can be found at
>> http://sourceforge.net/projects/numpy/files/NumPy/1.6.0b2/
>> For (preliminary) release notes see below.
>>
>> Enjoy,
>> Ralf
>>
>
> Thank you.
>
> numpy 1.6.0b2 builds well on Windows with msvc9/MKL.
>
> No license file is included in the installers.
>
> The installers for Python 3 include 2to3 backup files. This has already
> been fixed in scipy
> .
>
> All numpy tests pass on win32.
>
> A few numpy tests fail on win-amd64:
>
> ==
> ERROR: Ticket #99
> --
> Traceback (most recent call last):
>   File "X:\Python26-x64\lib\site-packages\numpy\testing\decorators.py",
> line 215, in knownfailer
>     return f(*args, **kwargs)
>   File
> "X:\Python26-x64\lib\site-packages\numpy\core\tests\test_regression.py",
> line 148, in test_intp
>     np.intp('0x' + 'f'*i_width,16)
> TypeError: function takes at most 1 argument (2 given)
>
> ==
> FAIL: test_iterator.test_iter_broadcasting_errors
> --
> Traceback (most recent call last):
>   File "X:\Python26-x64\lib\site-packages\nose\case.py", line 187, in
> runTest
>     self.test(*self.arg)
>   File
> "X:\Python26-x64\lib\site-packages\numpy\core\tests\test_iterator.py",
> line 639, in test_iter_broadcasting_errors
>     'Message "%s" doesn\'t contain operand shape (2,3)' % msg)
>   File "X:\Python26-x64\lib\site-packages\numpy\testing\utils.py", line
> 34, in assert_
>     raise AssertionError(msg)
> AssertionError: Message "non-broadcastable output operand with shape
> (%lld,%lld)
>  doesn't match the broadcast shape (%lld,%lld,%lld)" doesn't contain
> operand shape (2,3)
>
> ==
> FAIL: test_noncentral_f (test_random.TestRandomDist)
> --
> Traceback (most recent call last):
>   File
> "X:\Python26-x64\lib\site-packages\numpy\random\tests\test_random.py",
> line 297, in test_noncentral_f
>     np.testing.assert_array_almost_equal(actual, desired, decimal=14)
>   File "X:\Python26-x64\lib\site-packages\numpy\testing\utils.py", line
> 800, in assert_array_almost_equal
>     header=('Arrays are not almost equal to %d decimals' % decimal))
>   File "X:\Python26-x64\lib\site-packages\numpy\testing\utils.py", line
> 636, in assert_array_compare
>     raise AssertionError(msg)
> AssertionError:
> Arrays are not almost equal to 14 decimals
>
> (mismatch 100.0%)
>  x: array([[ 1.62003345,  1.7253997 ],
>        [ 0.96735921,  0.42933718],
>        [ 0.71714872,  6.24979552]])
>  y: array([[ 1.405981  ,  0.34207973],
>        [ 3.57715069,  7.92632663],
>        [ 0.43741599,  1.17742088]])
>
>
>
> One scipy 0.9.0 test fails (32 and 64 bit):
>
> ==
> FAIL: test_expon (test_morestats.TestAnderson)
> --
> Traceback (most recent call last):
>   File
> "X:\Python26-x64\lib\site-packages\scipy\stats\tests\test_morestats.py",
> line 72, in test_expon
>     assert_array_less(crit[:-1], A)
>   File "X:\Python26-x64\lib\site-packages\numpy\testing\utils.py", line
> 869, in assert_array_less
>     header='Arrays are not less-ordered')
>   File "X:\Python26-x64\lib\site-packages\numpy\testing\utils.py", line
> 613, in assert_array_compare
>     chk_same_position(x_id, y_id, hasval='inf')
>   File "X:\Python26-x64\lib\site-packages\numpy\testing\utils.py", line
> 588, in chk_same_position
>     raise AssertionError(msg)
> AssertionError:
> Arrays are not less-ordered
>
> x and y inf location mismatch:
>  x: array([ 0.911,  1.065,  1.325,  1.587])
>  y: array(inf)

I think this is from the change how inf are now handled in the
asserts, but I cannot check because I don't have 

Re: [Numpy-discussion] ANN: Numpy 1.6.0 beta 2

[Christoph Gohlke]

On 4/4/2011 1:04 PM, Ralf Gommers wrote:
> Hi,
>
> I am pleased to announce the availability of the second beta of NumPy
> 1.6.0. Due to the extensive changes in the Numpy core for this
> release, the beta testing phase will last at least one month. Please
> test this beta and report any problems on the Numpy mailing list.
>
> Many bug fixes went in since beta 1, among which:
> - fix installation of source release with python 3
> - f2py fixes for assumed shape arrays
> - several loadtxt bug fixes and enhancements
> - change default floating point error handling from "print" to "warn"
> - much more
>
> I quickly counted in the timeline, and in the last few weeks the
> number of open tickets has been decreased by over 100. Thanks to
> everyone who contributed to this spring cleaning!
>
> Sources and binaries can be found at
> http://sourceforge.net/projects/numpy/files/NumPy/1.6.0b2/
> For (preliminary) release notes see below.
>
> Enjoy,
> Ralf
>

Thank you.

numpy 1.6.0b2 builds well on Windows with msvc9/MKL.

No license file is included in the installers.

The installers for Python 3 include 2to3 backup files. This has already 
been fixed in scipy 
.

All numpy tests pass on win32.

A few numpy tests fail on win-amd64:

==
ERROR: Ticket #99
--
Traceback (most recent call last):
   File "X:\Python26-x64\lib\site-packages\numpy\testing\decorators.py", 
line 215, in knownfailer
 return f(*args, **kwargs)
   File 
"X:\Python26-x64\lib\site-packages\numpy\core\tests\test_regression.py", 
line 148, in test_intp
 np.intp('0x' + 'f'*i_width,16)
TypeError: function takes at most 1 argument (2 given)

==
FAIL: test_iterator.test_iter_broadcasting_errors
--
Traceback (most recent call last):
   File "X:\Python26-x64\lib\site-packages\nose\case.py", line 187, in 
runTest
 self.test(*self.arg)
   File 
"X:\Python26-x64\lib\site-packages\numpy\core\tests\test_iterator.py", 
line 639, in test_iter_broadcasting_errors
 'Message "%s" doesn\'t contain operand shape (2,3)' % msg)
   File "X:\Python26-x64\lib\site-packages\numpy\testing\utils.py", line 
34, in assert_
 raise AssertionError(msg)
AssertionError: Message "non-broadcastable output operand with shape 
(%lld,%lld)
  doesn't match the broadcast shape (%lld,%lld,%lld)" doesn't contain 
operand shape (2,3)

==
FAIL: test_noncentral_f (test_random.TestRandomDist)
--
Traceback (most recent call last):
   File 
"X:\Python26-x64\lib\site-packages\numpy\random\tests\test_random.py", 
line 297, in test_noncentral_f
 np.testing.assert_array_almost_equal(actual, desired, decimal=14)
   File "X:\Python26-x64\lib\site-packages\numpy\testing\utils.py", line 
800, in assert_array_almost_equal
 header=('Arrays are not almost equal to %d decimals' % decimal))
   File "X:\Python26-x64\lib\site-packages\numpy\testing\utils.py", line 
636, in assert_array_compare
 raise AssertionError(msg)
AssertionError:
Arrays are not almost equal to 14 decimals

(mismatch 100.0%)
  x: array([[ 1.62003345,  1.7253997 ],
[ 0.96735921,  0.42933718],
[ 0.71714872,  6.24979552]])
  y: array([[ 1.405981  ,  0.34207973],
[ 3.57715069,  7.92632663],
[ 0.43741599,  1.17742088]])



One scipy 0.9.0 test fails (32 and 64 bit):

==
FAIL: test_expon (test_morestats.TestAnderson)
--
Traceback (most recent call last):
   File 
"X:\Python26-x64\lib\site-packages\scipy\stats\tests\test_morestats.py", 
line 72, in test_expon
 assert_array_less(crit[:-1], A)
   File "X:\Python26-x64\lib\site-packages\numpy\testing\utils.py", line 
869, in assert_array_less
 header='Arrays are not less-ordered')
   File "X:\Python26-x64\lib\site-packages\numpy\testing\utils.py", line 
613, in assert_array_compare
 chk_same_position(x_id, y_id, hasval='inf')
   File "X:\Python26-x64\lib\site-packages\numpy\testing\utils.py", line 
588, in chk_same_position
 raise AssertionError(msg)
AssertionError:
Arrays are not less-ordered

x and y inf location mismatch:
  x: array([ 0.911,  1.065,  1.325,  1.587])
  y: array(inf)



Christoph
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] ANN: Numpy 1.6.0 beta 2

[josef . pktd]

On Tue, Apr 5, 2011 at 12:03 AM,   wrote:
> On Mon, Apr 4, 2011 at 11:42 PM, Charles R Harris
>  wrote:
>>
>>
>> On Mon, Apr 4, 2011 at 8:42 PM, Derek Homeier
>>  wrote:
>>>
>>> Hi all,
>>>
>>> On 4 Apr 2011, at 22:04, Ralf Gommers wrote:
>>>
>>> > I am pleased to announce the availability of the second beta of NumPy
>>> > 1.6.0. Due to the extensive changes in the Numpy core for this
>>> > release, the beta testing phase will last at least one month. Please
>>> > test this beta and report any problems on the Numpy mailing list.
>>> >
>>> the tests have a number of Python2.4-incompatibilities, one for a file
>>> opening mode and the rest for class declaration styles.
>>>
>>> Cheers,
>>>                                                Derek
>>>
>>> Running unit tests for numpy
>>> NumPy version 1.6.0b2
>>> NumPy is installed in /sw/lib/python2.4/site-packages/numpy
>>> Python version 2.4.4 (#1, Jan  5 2011, 03:05:41) [GCC 4.0.1 (Apple
>>> Inc. build 5493)]
>>> nose version 1.0.0
>>> .
>>> ==
>>> ERROR: Failure: SyntaxError (invalid syntax (test_multiarray.py, line
>>> 1023))
>>> ...
>>>   File "/sw/lib/python2.4/site-packages/numpy/core/tests/
>>> test_multiarray.py", line 1023
>>>     class TestPutmask():
>>>                       ^
>>> SyntaxError: invalid syntax
>>>
>>> ==
>>> ERROR: Failure: SyntaxError (invalid syntax (test_numeric.py, line
>>> 1068))
>>> ...
>>>   File "/sw/lib/python2.4/site-packages/numpy/core/tests/
>>> test_numeric.py", line 1068
>>>     class TestAllclose():
>>>                        ^
>>> SyntaxError: invalid syntax
>>>
>>> ==
>>> ERROR: Failure: SyntaxError (invalid syntax (test_scalarmath.py, line
>>> 84))
>>> ...
>>>   File "/sw/lib/python2.4/site-packages/numpy/core/tests/
>>> test_scalarmath.py", line 84
>>>     class TestRepr():
>>>                    ^
>>> SyntaxError: invalid syntax
>>>
>>> ==
>>> ERROR: Failure: SyntaxError (invalid syntax (test_twodim_base.py, line
>>> 280))
>>> ...
>>>   File "/sw/lib/python2.4/site-packages/numpy/lib/tests/
>>> test_twodim_base.py", line 280
>>>     class TestTriuIndices():
>>>                           ^
>>> SyntaxError: invalid syntax
>>>
>>> ==
>>> ERROR: Failure: SyntaxError (invalid syntax (test_linalg.py, line 243))
>>> ...
>>>   File "/sw/lib/python2.4/site-packages/numpy/linalg/tests/
>>> test_linalg.py", line 243
>>>     class TestMatrixPower():
>>>                           ^
>>> SyntaxError: invalid syntax
>>>
>>
>> Subclassing object should fix those, I think.
>>
>>>
>>> ==
>>> ERROR: test_gft_filename (test_io.TestFromTxt)
>>> --
>>> Traceback (most recent call last):
>>>   File "/sw/lib/python2.4/site-packages/numpy/lib/tests/test_io.py",
>>> line 1327, in test_gft_filename
>>>     assert_array_equal(np.genfromtxt(name), exp_res)
>>>   File "/sw/lib/python2.4/site-packages/numpy/lib/npyio.py", line
>>> 1235, in genfromtxt
>>>     fhd = iter(np.lib._datasource.open(fname, 'Ub'))
>>>   File "/sw/lib/python2.4/site-packages/numpy/lib/_datasource.py",
>>> line 145, in open
>>>     return ds.open(path, mode)
>>>   File "/sw/lib/python2.4/site-packages/numpy/lib/_datasource.py",
>>> line 477, in open
>>>     return _file_openers[ext](found, mode=mode)
>>> IOError: invalid mode: Ub
>>>
>>
>> Guess that wasn't tested before ;) I thought that was strange when I saw it.
>> The source of the problem is line 2035 in npyio.py. Additionally, Since
>> genloadtxt needs to have byte strings the 'rb" mode should probably be used.
>> That works on linux, both for python 2 and python 3, but doing that might
>> uncover genfromtxt problems on other platforms.
>
> "rb" is fine on Windows with python 3.2, (that's what I tested
> initially for this bug)

Sorry, I take this back. All our files have \n file endings not \r\n.
So I didn't test the latter case.

Josef

>
> Josef
>
>>
>> Chuck
>> ___
>> NumPy-Discussion mailing list
>> NumPy-Discussion@scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>>
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] ANN: Numpy 1.6.0 beta 2

[josef . pktd]

On Mon, Apr 4, 2011 at 11:42 PM, Charles R Harris
 wrote:
>
>
> On Mon, Apr 4, 2011 at 8:42 PM, Derek Homeier
>  wrote:
>>
>> Hi all,
>>
>> On 4 Apr 2011, at 22:04, Ralf Gommers wrote:
>>
>> > I am pleased to announce the availability of the second beta of NumPy
>> > 1.6.0. Due to the extensive changes in the Numpy core for this
>> > release, the beta testing phase will last at least one month. Please
>> > test this beta and report any problems on the Numpy mailing list.
>> >
>> the tests have a number of Python2.4-incompatibilities, one for a file
>> opening mode and the rest for class declaration styles.
>>
>> Cheers,
>>                                                Derek
>>
>> Running unit tests for numpy
>> NumPy version 1.6.0b2
>> NumPy is installed in /sw/lib/python2.4/site-packages/numpy
>> Python version 2.4.4 (#1, Jan  5 2011, 03:05:41) [GCC 4.0.1 (Apple
>> Inc. build 5493)]
>> nose version 1.0.0
>> .
>> ==
>> ERROR: Failure: SyntaxError (invalid syntax (test_multiarray.py, line
>> 1023))
>> ...
>>   File "/sw/lib/python2.4/site-packages/numpy/core/tests/
>> test_multiarray.py", line 1023
>>     class TestPutmask():
>>                       ^
>> SyntaxError: invalid syntax
>>
>> ==
>> ERROR: Failure: SyntaxError (invalid syntax (test_numeric.py, line
>> 1068))
>> ...
>>   File "/sw/lib/python2.4/site-packages/numpy/core/tests/
>> test_numeric.py", line 1068
>>     class TestAllclose():
>>                        ^
>> SyntaxError: invalid syntax
>>
>> ==
>> ERROR: Failure: SyntaxError (invalid syntax (test_scalarmath.py, line
>> 84))
>> ...
>>   File "/sw/lib/python2.4/site-packages/numpy/core/tests/
>> test_scalarmath.py", line 84
>>     class TestRepr():
>>                    ^
>> SyntaxError: invalid syntax
>>
>> ==
>> ERROR: Failure: SyntaxError (invalid syntax (test_twodim_base.py, line
>> 280))
>> ...
>>   File "/sw/lib/python2.4/site-packages/numpy/lib/tests/
>> test_twodim_base.py", line 280
>>     class TestTriuIndices():
>>                           ^
>> SyntaxError: invalid syntax
>>
>> ==
>> ERROR: Failure: SyntaxError (invalid syntax (test_linalg.py, line 243))
>> ...
>>   File "/sw/lib/python2.4/site-packages/numpy/linalg/tests/
>> test_linalg.py", line 243
>>     class TestMatrixPower():
>>                           ^
>> SyntaxError: invalid syntax
>>
>
> Subclassing object should fix those, I think.
>
>>
>> ==
>> ERROR: test_gft_filename (test_io.TestFromTxt)
>> --
>> Traceback (most recent call last):
>>   File "/sw/lib/python2.4/site-packages/numpy/lib/tests/test_io.py",
>> line 1327, in test_gft_filename
>>     assert_array_equal(np.genfromtxt(name), exp_res)
>>   File "/sw/lib/python2.4/site-packages/numpy/lib/npyio.py", line
>> 1235, in genfromtxt
>>     fhd = iter(np.lib._datasource.open(fname, 'Ub'))
>>   File "/sw/lib/python2.4/site-packages/numpy/lib/_datasource.py",
>> line 145, in open
>>     return ds.open(path, mode)
>>   File "/sw/lib/python2.4/site-packages/numpy/lib/_datasource.py",
>> line 477, in open
>>     return _file_openers[ext](found, mode=mode)
>> IOError: invalid mode: Ub
>>
>
> Guess that wasn't tested before ;) I thought that was strange when I saw it.
> The source of the problem is line 2035 in npyio.py. Additionally, Since
> genloadtxt needs to have byte strings the 'rb" mode should probably be used.
> That works on linux, both for python 2 and python 3, but doing that might
> uncover genfromtxt problems on other platforms.

"rb" is fine on Windows with python 3.2, (that's what I tested
initially for this bug)

Josef

>
> Chuck
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] ANN: Numpy 1.6.0 beta 2

[Charles R Harris]

On Mon, Apr 4, 2011 at 8:42 PM, Derek Homeier <
de...@astro.physik.uni-goettingen.de> wrote:

> Hi all,
>
> On 4 Apr 2011, at 22:04, Ralf Gommers wrote:
>
> > I am pleased to announce the availability of the second beta of NumPy
> > 1.6.0. Due to the extensive changes in the Numpy core for this
> > release, the beta testing phase will last at least one month. Please
> > test this beta and report any problems on the Numpy mailing list.
> >
> the tests have a number of Python2.4-incompatibilities, one for a file
> opening mode and the rest for class declaration styles.
>
> Cheers,
>Derek
>
> Running unit tests for numpy
> NumPy version 1.6.0b2
> NumPy is installed in /sw/lib/python2.4/site-packages/numpy
> Python version 2.4.4 (#1, Jan  5 2011, 03:05:41) [GCC 4.0.1 (Apple
> Inc. build 5493)]
> nose version 1.0.0
> .
> ==
> ERROR: Failure: SyntaxError (invalid syntax (test_multiarray.py, line
> 1023))
> ...
>   File "/sw/lib/python2.4/site-packages/numpy/core/tests/
> test_multiarray.py", line 1023
> class TestPutmask():
>   ^
> SyntaxError: invalid syntax
>
> ==
> ERROR: Failure: SyntaxError (invalid syntax (test_numeric.py, line
> 1068))
> ...
>   File "/sw/lib/python2.4/site-packages/numpy/core/tests/
> test_numeric.py", line 1068
> class TestAllclose():
>^
> SyntaxError: invalid syntax
>
> ==
> ERROR: Failure: SyntaxError (invalid syntax (test_scalarmath.py, line
> 84))
> ...
>   File "/sw/lib/python2.4/site-packages/numpy/core/tests/
> test_scalarmath.py", line 84
> class TestRepr():
>^
> SyntaxError: invalid syntax
>
> ==
> ERROR: Failure: SyntaxError (invalid syntax (test_twodim_base.py, line
> 280))
> ...
>   File "/sw/lib/python2.4/site-packages/numpy/lib/tests/
> test_twodim_base.py", line 280
> class TestTriuIndices():
>   ^
> SyntaxError: invalid syntax
>
> ==
> ERROR: Failure: SyntaxError (invalid syntax (test_linalg.py, line 243))
> ...
>   File "/sw/lib/python2.4/site-packages/numpy/linalg/tests/
> test_linalg.py", line 243
> class TestMatrixPower():
>   ^
> SyntaxError: invalid syntax
>
>
Subclassing object should fix those, I think.


> ==
> ERROR: test_gft_filename (test_io.TestFromTxt)
> --
> Traceback (most recent call last):
>   File "/sw/lib/python2.4/site-packages/numpy/lib/tests/test_io.py",
> line 1327, in test_gft_filename
> assert_array_equal(np.genfromtxt(name), exp_res)
>   File "/sw/lib/python2.4/site-packages/numpy/lib/npyio.py", line
> 1235, in genfromtxt
> fhd = iter(np.lib._datasource.open(fname, 'Ub'))
>   File "/sw/lib/python2.4/site-packages/numpy/lib/_datasource.py",
> line 145, in open
> return ds.open(path, mode)
>   File "/sw/lib/python2.4/site-packages/numpy/lib/_datasource.py",
> line 477, in open
> return _file_openers[ext](found, mode=mode)
> IOError: invalid mode: Ub
>
>
Guess that wasn't tested before ;) I thought that was strange when I saw it.
The source of the problem is line 2035 in npyio.py. Additionally, Since
genloadtxt needs to have byte strings the 'rb" mode should probably be used.
That works on linux, both for python 2 and python 3, but doing that might
uncover genfromtxt problems on other platforms.

Chuck
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] ANN: Numpy 1.6.0 beta 2

[Derek Homeier]

Hi all,

On 4 Apr 2011, at 22:04, Ralf Gommers wrote:

> I am pleased to announce the availability of the second beta of NumPy
> 1.6.0. Due to the extensive changes in the Numpy core for this
> release, the beta testing phase will last at least one month. Please
> test this beta and report any problems on the Numpy mailing list.
>
the tests have a number of Python2.4-incompatibilities, one for a file
opening mode and the rest for class declaration styles.

Cheers,
Derek

Running unit tests for numpy
NumPy version 1.6.0b2
NumPy is installed in /sw/lib/python2.4/site-packages/numpy
Python version 2.4.4 (#1, Jan  5 2011, 03:05:41) [GCC 4.0.1 (Apple  
Inc. build 5493)]
nose version 1.0.0
.
==
ERROR: Failure: SyntaxError (invalid syntax (test_multiarray.py, line  
1023))
...
   File "/sw/lib/python2.4/site-packages/numpy/core/tests/ 
test_multiarray.py", line 1023
 class TestPutmask():
   ^
SyntaxError: invalid syntax

==
ERROR: Failure: SyntaxError (invalid syntax (test_numeric.py, line  
1068))
...
   File "/sw/lib/python2.4/site-packages/numpy/core/tests/ 
test_numeric.py", line 1068
 class TestAllclose():
^
SyntaxError: invalid syntax

==
ERROR: Failure: SyntaxError (invalid syntax (test_scalarmath.py, line  
84))
...
   File "/sw/lib/python2.4/site-packages/numpy/core/tests/ 
test_scalarmath.py", line 84
 class TestRepr():
^
SyntaxError: invalid syntax

==
ERROR: Failure: SyntaxError (invalid syntax (test_twodim_base.py, line  
280))
...
   File "/sw/lib/python2.4/site-packages/numpy/lib/tests/ 
test_twodim_base.py", line 280
 class TestTriuIndices():
   ^
SyntaxError: invalid syntax

==
ERROR: Failure: SyntaxError (invalid syntax (test_linalg.py, line 243))
...
   File "/sw/lib/python2.4/site-packages/numpy/linalg/tests/ 
test_linalg.py", line 243
 class TestMatrixPower():
   ^
SyntaxError: invalid syntax

==
ERROR: test_gft_filename (test_io.TestFromTxt)
--
Traceback (most recent call last):
   File "/sw/lib/python2.4/site-packages/numpy/lib/tests/test_io.py",  
line 1327, in test_gft_filename
 assert_array_equal(np.genfromtxt(name), exp_res)
   File "/sw/lib/python2.4/site-packages/numpy/lib/npyio.py", line  
1235, in genfromtxt
 fhd = iter(np.lib._datasource.open(fname, 'Ub'))
   File "/sw/lib/python2.4/site-packages/numpy/lib/_datasource.py",  
line 145, in open
 return ds.open(path, mode)
   File "/sw/lib/python2.4/site-packages/numpy/lib/_datasource.py",  
line 477, in open
 return _file_openers[ext](found, mode=mode)
IOError: invalid mode: Ub

--
Ran 2648 tests in 28.573s

FAILED (KNOWNFAIL=3, SKIP=21, errors=6)

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion