Re: [Python-Dev] OpenIndiana and Solaris support

[Brett Cannon]

On Mon, 26 Sep 2016 at 15:38 Guido van Rossum  wrote:

> Thanks for the reality check Trent! I think if enough people with core
> committer bits want to keep supporting Solaris / Illumos / OpenIndiana
> / other variants that's fine, but I don't think that just having some
> VMs to test on is enough -- we also need people who can fix problems
> if those buildbots start failing, and that requires pretty specialized
> knowledge. Plus of course we won't know if fixing it for OpenIndiana
> will also fix it for Solaris 11 Express or for other Illumos forks.
> (For Linux it's easier to assess these things because so many people
> in open source use Linux and its many forks.)
>

The official requirement to support a platform is a stable buildbot and a
core dev to keep the support up:
https://www.python.org/dev/peps/pep-0011/#supporting-platforms. Victor has
asked that the OpenIndiana buildbot be removed from the stable pool as it
consistently throws MemoryError which means its support is not improving.
If Trent is willing to maintain a buildbot in a Joyent VM that at least
takes care of that part, but it still requires Jesus to volunteer to keep
the support up if it's going to be supported for free. Otherwise Joyent
could consider contracting with one of the various core devs who happen to
be consultants to help maintain the support.

At minimum, though, a new buildbot could go into the unstable pool so
illumos devs can keep an eye on when things break to try and get
platform-independent changes upstreamed that happen to help illumos (e.g.
no #ifdef changes specific to illumos, but if something just needed to be
made more robust and it happens to help illumos that's typically fine).


>
> On Mon, Sep 26, 2016 at 3:32 PM, Trent Mick  wrote:
> > I work for Joyent (joyent.com) now, which employs a number of devs that
> work
> > on illumos (illumos.org). We also provide cloud infrastructure. Would it
> > help if we offered one or more instances (VMs) on which to run buildbot
> > slaves (and on which volunteers for bug fixing could hack)?  I know a
> lot of
> > people in the illumos community would be quite sad to have it dropped as
> a
> > core Python plat.
> >
> > Guido,
> > Yes you are correct that Oracle owns the Solaris brand.
> >
> > tl;dr history if you care:
> > - sunos -> Solaris
> > - Sun open sources Solaris, called OpenSolaris (2005)
> > - Oracle acquires Sun and closes Solaris (Aug 2010). Shortly after, the
> > community forks OpenSolaris and calls it illumos (Sep 2010)
> > - OpenIndiana is a distro of illumos (somewhat similar to how Ubuntu is a
> > distro of Linux). Other distros are SmartOS (the one Joyent works on),
> and
> > OmniOS.
> > - Oracle continues work on Solaris, releasing "Solaris 11 Express".
> >
> > I've no real numbers of usage of illumos vs Solaris 11 vs others.
> >
> > Cheers,
> > Trent
> >
> > p.s. I hear that Jesus is also in contact with some of the illumos-devs
> on
> > IRC (and perhaps email). I hope we can help there.
>
>
>
> --
> --Guido van Rossum (python.org/~guido)
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] TextIO seek and tell cookies

[Guido van Rossum]

Yeah, that should work. The implementation is something like a byte
offset to the start of a line plus a character count, plus some misc
flags. I found this implementation in the 2.6 code (the last version
where it was pure Python code):

def _pack_cookie(self, position, dec_flags=0,
   bytes_to_feed=0, need_eof=0, chars_to_skip=0):
# The meaning of a tell() cookie is: seek to position, set the
# decoder flags to dec_flags, read bytes_to_feed bytes, feed them
# into the decoder with need_eof as the EOF flag, then skip
# chars_to_skip characters of the decoded result.  For most simple
# decoders, tell() will often just give a byte offset in the file.
return (position | (dec_flags<<64) | (bytes_to_feed<<128) |
   (chars_to_skip<<192) | bool(need_eof)<<256)

def _unpack_cookie(self, bigint):
rest, position = divmod(bigint, 1<<64)
rest, dec_flags = divmod(rest, 1<<64)
rest, bytes_to_feed = divmod(rest, 1<<64)
need_eof, chars_to_skip = divmod(rest, 1<<64)
return position, dec_flags, bytes_to_feed, need_eof, chars_to_skip

On Mon, Sep 26, 2016 at 3:43 PM, Greg Ewing  wrote:
> Ben Leslie wrote:
>>
>> But the idea of transmitting these offsets outside of a running
>> process is not something that I had anticipated. It got me thinking:
>> is there a guarantee that these opaque values returned from tell() is
>> stable across different versions of Python?
>
>
> Are they even guaranteed to work on a different file
> object in the same process? I.e. if you read some stuff
> from a file, do tell() on it, then close it, open it
> again and seek() with that token, are you guaranteed to
> end up at the same place in the file?
>
> --
> Greg
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/guido%40python.org



-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] TextIO seek and tell cookies

[Greg Ewing]

Ben Leslie wrote:

But the idea of transmitting these offsets outside of a running
process is not something that I had anticipated. It got me thinking:
is there a guarantee that these opaque values returned from tell() is
stable across different versions of Python?


Are they even guaranteed to work on a different file
object in the same process? I.e. if you read some stuff
from a file, do tell() on it, then close it, open it
again and seek() with that token, are you guaranteed to
end up at the same place in the file?

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] OpenIndiana and Solaris support

[Guido van Rossum]

Thanks for the reality check Trent! I think if enough people with core
committer bits want to keep supporting Solaris / Illumos / OpenIndiana
/ other variants that's fine, but I don't think that just having some
VMs to test on is enough -- we also need people who can fix problems
if those buildbots start failing, and that requires pretty specialized
knowledge. Plus of course we won't know if fixing it for OpenIndiana
will also fix it for Solaris 11 Express or for other Illumos forks.
(For Linux it's easier to assess these things because so many people
in open source use Linux and its many forks.)

On Mon, Sep 26, 2016 at 3:32 PM, Trent Mick  wrote:
> I work for Joyent (joyent.com) now, which employs a number of devs that work
> on illumos (illumos.org). We also provide cloud infrastructure. Would it
> help if we offered one or more instances (VMs) on which to run buildbot
> slaves (and on which volunteers for bug fixing could hack)?  I know a lot of
> people in the illumos community would be quite sad to have it dropped as a
> core Python plat.
>
> Guido,
> Yes you are correct that Oracle owns the Solaris brand.
>
> tl;dr history if you care:
> - sunos -> Solaris
> - Sun open sources Solaris, called OpenSolaris (2005)
> - Oracle acquires Sun and closes Solaris (Aug 2010). Shortly after, the
> community forks OpenSolaris and calls it illumos (Sep 2010)
> - OpenIndiana is a distro of illumos (somewhat similar to how Ubuntu is a
> distro of Linux). Other distros are SmartOS (the one Joyent works on), and
> OmniOS.
> - Oracle continues work on Solaris, releasing "Solaris 11 Express".
>
> I've no real numbers of usage of illumos vs Solaris 11 vs others.
>
> Cheers,
> Trent
>
> p.s. I hear that Jesus is also in contact with some of the illumos-devs on
> IRC (and perhaps email). I hope we can help there.



-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] OpenIndiana and Solaris support

[Trent Mick]

I work for Joyent (joyent.com) now, which employs a number of devs that
work on illumos (illumos.org). We also provide cloud infrastructure. Would
it help if we offered one or more instances (VMs) on which to run buildbot
slaves (and on which volunteers for bug fixing could hack)?  I know a lot
of people in the illumos community would be quite sad to have it dropped as
a core Python plat.

Guido,
Yes you are correct that Oracle owns the Solaris brand.

tl;dr history if you care:
- sunos -> Solaris
- Sun open sources Solaris, called OpenSolaris (2005)
- Oracle acquires Sun and closes Solaris (Aug 2010). Shortly after, the
community forks OpenSolaris and calls it illumos (Sep 2010)
- OpenIndiana is a distro of illumos (somewhat similar to how Ubuntu is a
distro of Linux). Other distros are SmartOS (the one Joyent works on), and
OmniOS.
- Oracle continues work on Solaris, releasing "Solaris 11 Express".

I've no real numbers of usage of illumos vs Solaris 11 vs others.

Cheers,
Trent

p.s. I hear that Jesus is also in contact with some of the illumos-devs on
IRC (and perhaps email). I hope we can help there.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] 3.6.0 Beta Phase Development

[Guido van Rossum]

The issue tracker is your friend!

On Mon, Sep 26, 2016 at 5:25 AM, Michael Felt  wrote:
>
> On 13/09/2016 02:15, Ned Deily wrote:
>>
>> the challenge is to put the finishing touches on the features and
>> documentation, squash bugs, and test test test.  The next preview release
>> will be 3.6.0b2
>
>
> Found one typo in Modules/_io/_iomodule.h on line 156 - #endif^L rather than
> #endif (posted as an issue, but I suppose just a note here would have been
> enough)
>
> I have a longish list of messages to stderr from the compiler (IBM xlc) on
> AIX. Rather than spam everyone with those - would opening an issue be the
> way forward, or just sending the file to a person - rather than the list.
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/guido%40python.org



-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] 3.6.0 Beta Phase Development

[Michael Felt]

On 13/09/2016 02:15, Ned Deily wrote:

the challenge is to put the finishing touches on the features and 
documentation, squash bugs, and test test test.  The next preview release will 
be 3.6.0b2


Found one typo in Modules/_io/_iomodule.h on line 156 - #endif^L rather 
than #endif (posted as an issue, but I suppose just a note here would 
have been enough)


I have a longish list of messages to stderr from the compiler (IBM xlc) 
on AIX. Rather than spam everyone with those - would opening an issue be 
the way forward, or just sending the file to a person - rather than the 
list.

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] TextIO seek and tell cookies

[Random832]

On Mon, Sep 26, 2016, at 05:30, Ben Leslie wrote:
> I think the case of JSON or SQL database is even more important though.
> 
> tell/seek can return 129-bit integers (maybe even more? my maths might
> be off here).
> 
> The very large integers that can be returned by tell() will break
> serialization to JSON, and storing in a SQL database (at least for
> most database types).
> 
> What is the value of comparing these to plain integers? Unless you
> happen to know the magic encoding it isn't going to be very useful I
> think?

I assume the value is that in the circumstances in which all of the
flags and other bits are zero, they can be used as offsets in precisely
the way that you used them. It may also be possible that in some cases
where they are not zero, doing arithmetic with them is still "safe"
since the real offset is still in the low-order bits. I don't know if
those circumstances are predictable enough for it to be worthwhile.
Changing it would obviously break code that does this (unless, perhaps,
it were changed to be a class with arithmetic operators), the question
is whether such code "deserves" to be broken.

In my own tests, even a UTF-8-sig file with DOS line endings "worked".
Does anyone have information about what circumstances can reliably cause
tell() to return values that are *not* simple integers? Maybe it has
something to do with working with stateful encodings like iso-2022 or
UTF-7?

What was the situation that caused your problem?
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] TextIO seek and tell cookies

[Ben Leslie]

It was pointed out in private email that technically JSON can
represent very large integers even if ECMAScript itself can't.

But the idea of transmitting these offsets outside of a running
process is not something that I had anticipated. It got me thinking:
is there a guarantee that these opaque values returned from tell() is
stable across different versions of Python? My reading of opaque is
that it could be subject to change, but that possibly isn't the
intent.

It seems that since the sizeof(int) and sizeof(Py_off_t) could be
different in different builds of Python even off the same version,
then the opaque value returned is necessarily going to be different
between builds of even the same version of Python.

It seems like it would be prudent to discourage the sharing of these
opaque cookies (such as via a database or interchange formats) as
you'd have to be very sure that they would be interpreted correctly in
any receiving instance.

Cheers,

Ben

On 26 September 2016 at 02:30, Ben Leslie  wrote:
> I think the case of JSON or SQL database is even more important though.
>
> tell/seek can return 129-bit integers (maybe even more? my maths might
> be off here).
>
> The very large integers that can be returned by tell() will break
> serialization to JSON, and storing in a SQL database (at least for
> most database types).
>
> What is the value of comparing these to plain integers? Unless you
> happen to know the magic encoding it isn't going to be very useful I
> think?
>
> Cheers,
>
> Ben
>
> On 25 September 2016 at 21:18, Guido van Rossum  wrote:
>> Be careful though, comparing these to plain integers should probably
>> be allowed, and we also should make sure that things like
>> serialization via JSON or storing in an SQL database don't break. I
>> personally think it's one of those "learn not to touch the stove"
>> cases and there's limited value in making this API idiot proof.
>>
>> On Sun, Sep 25, 2016 at 9:05 PM, Nick Coghlan  wrote:
>>> On 26 September 2016 at 10:21, MRAB  wrote:
 On 2016-09-26 00:21, Ben Leslie wrote:
> Are there any downsides to this? I've made some progress developing a
> patch to change this functionality. Is it worth polishing and
> submitting?
>
 An alternative might be a subclass of int.
>>>
>>> It could make sense to use a subclass of int that emitted deprecation
>>> warnings for integer arithmetic, and then eventually disallowed it
>>> entirely.
>>>
>>> Cheers,
>>> Nick.
>>>
>>> --
>>> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
>>> ___
>>> Python-Dev mailing list
>>> Python-Dev@python.org
>>> https://mail.python.org/mailman/listinfo/python-dev
>>> Unsubscribe: 
>>> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>>
>>
>>
>> --
>> --Guido van Rossum (python.org/~guido)
>> ___
>> Python-Dev mailing list
>> Python-Dev@python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe: 
>> https://mail.python.org/mailman/options/python-dev/benno%40benno.id.au
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] TextIO seek and tell cookies

[Ben Leslie]

I think the case of JSON or SQL database is even more important though.

tell/seek can return 129-bit integers (maybe even more? my maths might
be off here).

The very large integers that can be returned by tell() will break
serialization to JSON, and storing in a SQL database (at least for
most database types).

What is the value of comparing these to plain integers? Unless you
happen to know the magic encoding it isn't going to be very useful I
think?

Cheers,

Ben

On 25 September 2016 at 21:18, Guido van Rossum  wrote:
> Be careful though, comparing these to plain integers should probably
> be allowed, and we also should make sure that things like
> serialization via JSON or storing in an SQL database don't break. I
> personally think it's one of those "learn not to touch the stove"
> cases and there's limited value in making this API idiot proof.
>
> On Sun, Sep 25, 2016 at 9:05 PM, Nick Coghlan  wrote:
>> On 26 September 2016 at 10:21, MRAB  wrote:
>>> On 2016-09-26 00:21, Ben Leslie wrote:
 Are there any downsides to this? I've made some progress developing a
 patch to change this functionality. Is it worth polishing and
 submitting?

>>> An alternative might be a subclass of int.
>>
>> It could make sense to use a subclass of int that emitted deprecation
>> warnings for integer arithmetic, and then eventually disallowed it
>> entirely.
>>
>> Cheers,
>> Nick.
>>
>> --
>> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
>> ___
>> Python-Dev mailing list
>> Python-Dev@python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe: 
>> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>
>
>
> --
> --Guido van Rossum (python.org/~guido)
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/benno%40benno.id.au
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] TextIO seek and tell cookies

[Ben Leslie]

On 25 September 2016 at 17:21, MRAB  wrote:
> On 2016-09-26 00:21, Ben Leslie wrote:
>>
>> Hi all,
>>
>> I recently shot myself in the foot by assuming that TextIO.tell
>> returned integers rather than opaque cookies. Specifically I was
>> adding an offset to the value returned by TextIO.tell. In retrospect
>> this doesn't make sense/
>>
>> Now, I don't want to drive change simply because I failed to read the
>> documentation carefully, but I think the current API is very easy to
>> misuse. Most of the time TextIO.tell returns a cookie that is actually
>> an integer and adding an offset to it and seek-ing works fine.
>>
>> The only indication you get that you are mis-using the API is that
>> sometimes tell returns a cookie that when you add an integer offset to
>> it will cause seek() to fail with an OverflowError.
>>
>> Would it be possible to change the API to return something more
>> opaque? E.g.: rather than converting the C cookie structure to a long,
>> could it instead be converted to  a bytes() object.
>>
>> (I.e.: Change textiowrapper_build_cookie to use
>> PyBytes_FromStringAndSize rather than _PyLong_FromByteArray and
>> equivalent for textiowrapper_parse_cookie).
>>
>> This would ensure the return value is never mis-used and is probably
>> also faster using bytes objects than converting to/from an integer.
>>
> why would it be faster? It's an integer internally.


It isn't an integer internally though, it is a cookie:

typedef struct {
   Py_off_t start_pos;
int dec_flags;
int bytes_to_feed;
int chars_to_skip;
char need_eof;
} cookie_type;

The memory view of this structure is then converted to a long. Surely
converting to a PyLong is more work than converting to bytes?
In any case, performance really isn't the motivation here.

Cheers,

Ben
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com