Re: [Numpy-discussion] String & unicode arrays vs text loading in python 3

2016-09-14 Thread Lluís Vilanova
Stephan Hoyer writes: > On Tue, Sep 13, 2016 at 11:05 AM, Lluís Vilanova wrote: > Whenever we repr an array using 'S', we can instead show a unicode in py3. > That > keeps the binary representation, but will always show the expected result > to > users, and it's only a handful of

Re: [Numpy-discussion] String & unicode arrays vs text loading in python 3

2016-09-13 Thread Chris Barker
On Tue, Sep 13, 2016 at 11:05 AM, Lluís Vilanova wrote: > Great, that's the type of info I wanted to get before going forward. I > guess > there's code relying on the binary representation of 'S' to do mmap's or > access > the array's raw contents. Is that right? yes, there is a LOT of code, mo

Re: [Numpy-discussion] String & unicode arrays vs text loading in python 3

2016-09-13 Thread Stephan Hoyer
On Tue, Sep 13, 2016 at 11:05 AM, Lluís Vilanova wrote: > Whenever we repr an array using 'S', we can instead show a unicode in py3. > That > keeps the binary representation, but will always show the expected result > to > users, and it's only a handful of lines added to dump_data(). > > If neede

Re: [Numpy-discussion] String & unicode arrays vs text loading in python 3

2016-09-13 Thread Lluís Vilanova
Chris Barker writes: > We had a big long discussion about this on this list a while back (maybe 2 yrs > ago???) please search the archives to find it. Though I'm pretty sure that we > never did come to a conclusion. I think it stared with wanting better support > ofr unicode in loadtxt and the lik

Re: [Numpy-discussion] String & unicode arrays vs text loading in python 3

2016-09-13 Thread Chris Barker
We had a big long discussion about this on this list a while back (maybe 2 yrs ago???) please search the archives to find it. Though I'm pretty sure that we never did come to a conclusion. I think it stared with wanting better support ofr unicode in loadtxt and the like, and ended up delving into o

Re: [Numpy-discussion] String & unicode arrays vs text loading in python 3

2016-09-13 Thread Lluís Vilanova
Sebastian Berg writes: > On Di, 2016-09-13 at 15:02 +0200, Lluís Vilanova wrote: >> Hi! I'm giving a shot to issue #3184 [1], based on the observation >> that the >> string dtype ('S') under python 3 uses byte arrays instead of unicode >> (the only >> readable string type in python 3). >> >> This

Re: [Numpy-discussion] String & unicode arrays vs text loading in python 3

2016-09-13 Thread Lluís Vilanova
Sebastian Berg writes: > On Di, 2016-09-13 at 15:02 +0200, Lluís Vilanova wrote: >> Hi! I'm giving a shot to issue #3184 [1], based on the observation >> that the >> string dtype ('S') under python 3 uses byte arrays instead of unicode >> (the only >> readable string type in python 3). >> >> This

Re: [Numpy-discussion] String & unicode arrays vs text loading in python 3

2016-09-13 Thread Sebastian Berg
On Di, 2016-09-13 at 15:02 +0200, Lluís Vilanova wrote: > Hi! I'm giving a shot to issue #3184 [1], based on the observation > that the > string dtype ('S') under python 3 uses byte arrays instead of unicode > (the only > readable string type in python 3). > > This brings two major problems: > >

[Numpy-discussion] String & unicode arrays vs text loading in python 3

2016-09-13 Thread Lluís Vilanova
Hi! I'm giving a shot to issue #3184 [1], based on the observation that the string dtype ('S') under python 3 uses byte arrays instead of unicode (the only readable string type in python 3). This brings two major problems: * numpy code has to go through loops to open and read files as binary data