[Python-Dev] Re: PEP-376 and PEP-427 interpretation

[Daniel Holth]

I meant to exclude md5 and sha1, e.g. hash functions with known problems.
SHA224 would be a weird choice but it wouldn't personally offend me
otherwise. It would be fun to see how many wheel handlers support
non-sha256 hash functions.

On Mon, Mar 29, 2021 at 9:56 PM Theallredman via Python-Dev <
python-dev@python.org> wrote:

> Thank you.  I can't think of a compelling reason someone would want to
> choose SHA224 over SHA256 in the context of wheel generation.  It just that
> the PEPs are usually pretty explicit and SHA224 seemed to be implicitly
> excluded from RECORD files.  And I'm considering the details of making a
> pretty pedantic wheel generation PEP517 backend.
>
> Eldon
>
> ‐‐‐ Original Message ‐‐‐
> On Monday, March 29, 2021 2:16 PM, Paul Moore  wrote:
>
> > On Mon, 29 Mar 2021 at 17:40, Theallredman via Python-Dev
> > python-dev@python.org wrote:
> >
> > > So going back to my actual question SHA224 is disallowed in record
> files because it's bit length is less then 256?
> >
> > It doesn't look like it's ever been excluded. The only explicit
> > exclusions are MD5 and SHA1 as you point out. Do you have a particular
> > reason to want to use SHA224? Pretty much everyone is using SHA256, as
> > far as I know.
> >
> > Paul
>
>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/NB6MLDDDJHRTRMOEWDFG5IYGZCP65K6V/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/P4575PDTQCUYMYZVOSJPYGD3J5YKNCFY/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP: Modify the C API to hide implementation details

[Daniel Holth]

Sorry that this is a bit off-topic. cffi would be a user of any new C API.

I've tried to make sure ABI3 is supported in setuptools and wheel, with
varying success. Apparently virtualenvs and Windows have problems. I'm
excited about the possibility of a better C API and possibly ABI.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/XFHKIXSEEZMCC7HZ3N4GP6SBUBYACI6K/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP: Modify the C API to hide implementation details

[Daniel Holth]

It can be done exactly as passing a void* when registering a C callback,
and getting it passed back to your callback function.
https://cffi.readthedocs.io/en/latest/ref.html#ffi-new-handle-ffi-from-handle

https://bitbucket.org/dholth/kivyjoy/src/aaeab79b2891782209a1219cd65a4d9716cea669/kivyjoy/controller.py#lines-49
https://bitbucket.org/dholth/kivyjoy/src/aaeab79b2891782209a1219cd65a4d9716cea669/kivyjoy/__init__.py#lines-15
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/QDHQANZJAKPTTOJHXLGFSJWKLE5NA6ZU/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP: Modify the C API to hide implementation details

[Daniel Holth]

Was it regular cffi or cffi's embedding API, which is used a bit
differently than regular cffi, that "seems to only solve a fraction of the
problem"? Was just playing around with the embedding API and was impressed.

In Python:

@ffi.def_extern()
def uwsgi_pyexample_init():
print("init called")

return 0

In C (embedded in the same plugin):

CFFI_DLLEXPORT struct uwsgi_plugin pyexample_plugin = {
.init = uwsgi_pyexample_init
};

Seems to be happily importing and exporting APIs. Interpreter starts the
first time a @ffi.def_extern() function is called.

https://cffi.readthedocs.io/en/latest/embedding.html

https://github.com/unbit/uwsgi/blob/f6ad0c6dfe431d91ffe365bed3105ed052bef6e4/plugins/pyexample/pyexample_plugin.py
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/S5P524LXXNIXIB2MUATZNHYQE57MXCQ5/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: python3 -bb and hash collisions

[Daniel Holth]

On Tue, Sep 10, 2019 at 8:38 PM Matt Billenstein  wrote:

> On Tue, Sep 10, 2019 at 10:42:52AM -0400, Daniel Holth wrote:
> > I stopped using Python 3 after learning about str(bytes) by finding it
> in my
> > corrupted database. Ever since then I've been anxious about changing to
> the new
> > language, since it makes it so easy to convert from bytes to unicode by
> > accident without specifying a valid encoding. So I would like to see a
> future
> > where str(bytes) is effectively removed. I started working on a pull
> request
> > that adds an API to toggle str(bytes) at runtime with a thread local
> (instead
> > of requiring a command line argument), so you could do with
> no_str_bytes(): if
> > you were worried about the feature, but got a bit stuck in the internals.
>
> How is this different than all the str -> unicode bugs we had in python2?
> If
> you have special needs, you can always monkey-patch it in plain python
> code by
> overriding __builtins__.str with something that asserts the given arg is
> not
> bytes.
>
> m
>
> --
> Matt Billenstein
> m...@vazor.com
> http://www.vazor.com/
>

It's different. One hint is that there's already an option to disable the
feature. The old style of error will occasionally reveal itself with decode
errors but the new style error happens silently, you discover it somehow,
then enable the -bb option, track down the source of the error, and deal
with the fallout. The proposed change would allow `print(bytes)` for
(de)bugging by letting you toggle `python3 -bb` behavior at runtime instead
of only at the command line. Or you could debug more explicitly by
`print(bytes.decode('ebcdic'))` or `print(repr(bytes))`

I didn't realize you could override __builtins__.str. That's interesting.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/I5RQRUGLAKLZ6TLPSO2S5VRL5D4CQNGT/

[Python-Dev] Re: python3 -bb and hash collisions

[Daniel Holth]

On Sat, Jun 22, 2019 at 2:48 AM Serhiy Storchaka 
wrote:

> 22.06.19 01:08, Daniel Holth пише:
> > Thanks. I think I might like an option to disable str(bytes) without
> > disabling str != bytes. Unless the second operation would also corrupt
> > output.
> >
> > Came across this kind of set in the hyper http library which uses a set
> > to accept certain headers with either str or bytes keys.
>
> Does that library support Python 2?  If it is true than you have a
> problem, because u'abc' == b'abc' in Python 2 and u'abc' != b'abc' in
> Python 3.
>
> If it is Python 3 only, you can just ignore BytesWarning. It was added
> purely to help to catch subtle bugs in transition to Python 3. In
> future, after Python 2 be out of use, BytesWarning will become deprecated.
>

I stopped using Python 3 after learning about str(bytes) by finding it in
my corrupted database. Ever since then I've been anxious about changing to
the new language, since it makes it so easy to convert from bytes to
unicode by accident without specifying a valid encoding. So I would like to
see a future where str(bytes) is effectively removed. I started working on
a pull request that adds an API to toggle str(bytes) at runtime with a
thread local (instead of requiring a command line argument), so you could
do with no_str_bytes(): if you were worried about the feature, but got a
bit stuck in the internals.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/5HIHFIJODYSLFCEASM37K3V7XPAF56PK/

[Python-Dev] Re: python3 -bb and hash collisions

[Daniel Holth]

The answer bytes == str is just False. That doesn't put b'' in your
database by accident. It could be useful to separate the two kinds of
warnings.

On Fri, Jun 21, 2019, 18:57 Ivan Pozdeev via Python-Dev <
python-dev@python.org> wrote:

> On 22.06.2019 1:08, Daniel Holth wrote:
>
> Thanks. I think I might like an option to disable str(bytes) without
> disabling str != bytes. Unless the second operation would also corrupt
> output.
>
> You can't compare str to bytes without knowing the encoding the bytes are
> supposed to be in (see
> https://stackoverflow.com/questions/49991870/python-default-string-encoding
> for details).
>
> And if you do know the encoding, you can as well compare
> `str.encode(encoding) != bytes` / `str != bytes.decode(encoding)`.
>
>
> Came across this kind of set in the hyper http library which uses a set to
> accept certain headers with either str or bytes keys.
>
> On Tue, Jun 18, 2019, 13:05 Christian Heimes  wrote:
>
>> On 18/06/2019 18.32, Daniel Holth wrote:
>> > set([u"foo", b"foo]) will error because the two kinds of string have the
>> > same hash, and this causes a comparison. Is that correct?
>>
>> Yes, it will fail with -bb, because it turns comparison between str and
>> bytes into an error. This can also happen with other strings when
>> hash(u'somestring') & mask == hash(b'otherbytes') & mask. The mask of a
>> set starts with PySet_MINSIZE - 1 == 8 and increases over team.
>>
>> Christian
>>
>>
>> ___
>> Python-Dev mailing list -- python-dev@python.org
>> To unsubscribe send an email to python-dev-le...@python.org
>> https://mail.python.org/mailman3/lists/python-dev.python.org/
>> Message archived at
>> https://mail.python.org/archives/list/python-dev@python.org/message/ZIF2MRBWSMSCFP6E7PZOBI5KYP46QZPK/
>>
>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to 
> python-dev-leave@python.orghttps://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-dev@python.org/message/R6E7FAR36UO6XHQSIAVF4DIM7G23ADJP/
>
> --
> Regards,
> Ivan
>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/XAN44UH5X5PYNSHY5ONULXIJF4DLBXF6/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/36DKFLVTBABEZPDX7MYHP7H2TVDZTOHG/

[Python-Dev] Re: python3 -bb and hash collisions

[Daniel Holth]

Thanks. I think I might like an option to disable str(bytes) without
disabling str != bytes. Unless the second operation would also corrupt
output.

Came across this kind of set in the hyper http library which uses a set to
accept certain headers with either str or bytes keys.

On Tue, Jun 18, 2019, 13:05 Christian Heimes  wrote:

> On 18/06/2019 18.32, Daniel Holth wrote:
> > set([u"foo", b"foo]) will error because the two kinds of string have the
> > same hash, and this causes a comparison. Is that correct?
>
> Yes, it will fail with -bb, because it turns comparison between str and
> bytes into an error. This can also happen with other strings when
> hash(u'somestring') & mask == hash(b'otherbytes') & mask. The mask of a
> set starts with PySet_MINSIZE - 1 == 8 and increases over team.
>
> Christian
>
>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/ZIF2MRBWSMSCFP6E7PZOBI5KYP46QZPK/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/R6E7FAR36UO6XHQSIAVF4DIM7G23ADJP/

[Python-Dev] python3 -bb and hash collisions

[Daniel Holth]

set([u"foo", b"foo]) will error because the two kinds of string have the
same hash, and this causes a comparison. Is that correct?
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/MRKTBTY7ZTU3I6SCWRHB533O2QMYA4VR/

Re: [Python-Dev] Feature request: Change a Dependency Package Version During Package Initiation

[Daniel Holth]

This sounds exactly like what people used to do with eggs. You could have
multiple versions of a package on the path as eggs and then require a
version at runtime. The approach has problems. Ruby also abandoned a
strategy where random app code depends on package management code at
runtime.

One better strategy is to set up a python path in a wrapper script.

On Fri, May 17, 2019, 11:27 Brett Cannon  wrote:

> Thanks for the idea but there are currently no plans to support such a
> feature. If you would like to see it then you will need to write a PEP with
> a proof-of-concept to demonstrate how you would expect such a feature to
> work.
>
> On Fri., May 17, 2019, 07:55 Q via Python-Dev, 
> wrote:
>
>> A lot of the Python code we use in production are used directly as
>> imports in other python
>> distributions (such as the python comes with the finite element software
>> Abaqus and MSC Marc), many
>> packages (such as matplotlib, numpy) that may have varying versioned
>> dependencies.
>>
>> I was wondering if this could be expanded to allow a version to be set
>> within a package and have
>> that propagate to all modules in that package. For example in the root
>> init.py if I set
>> multiversion(tornado, 2.2.1) then all modules in that package will use
>> tornado 2.2.1 when I import
>> tornado.
>>
>> See a relevant issue on github:
>> https://github.com/mitsuhiko/multiversion/issues/1
>>
>> Thank you!
>> Qiang
>>
>> ___
>> Python-Dev mailing list
>> Python-Dev@python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/dholth%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Imports with underscores

[Daniel Holth]

Easily solved with the totally evil ninja mode pattern of module
initialization. It has yet to catch on.

def ninja():
global exported
import os
def exported():
# do something
ninja()
del ninja

On Mon, Jan 9, 2017 at 1:13 PM Sven R. Kunze  wrote:

> Interesting to see that others have the same problem.
>
> We also had this kind of "over-protective" behavior. As far as I know, our
> devs stopped doing it as it feels cumbersome.
>
>
> Another argument for this is: when using PyCharm, this IDE will suggest
> imports from those modules which aren't the original ones. So, you might
> import from a third-party module. Over time, however, people learn to pick
> the "right" module to import from.
>
> Cheers,
> Sven
>
>
> On 09.01.2017 12:42, Steve Holden wrote:
>
> One of my developers recently submitted a pull request incuding a number
> of lines like
>
> import os as _os
>
> When I asked him why he suggested a) this would improve encapsulation, and
> b) the practice was supported in the stdlib. Further investigation reveals
> that some modules (e.g. argparse, crypt, difflib, random) do use this
> technique, but it is far from universal.
>
> So I thought it would be useful to get input from current devs about the
> value of this practice, since to me it seems somewhat anti-pythonic. What
> advantages does it confer?
>
> regards
> Steve Holden
>
>
> ___
> Python-Dev mailing 
> listPython-Dev@python.orghttps://mail.python.org/mailman/listinfo/python-dev
>
>
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/srkunze%40mail.de
>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/dholth%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] File system path encoding on Windows

[Daniel Holth]

A nice recent blog post about paths encoding in a media player.
http://beets.io/blog/paths.html . It's not merely the porting that makes it
hard.

On Mon, Aug 29, 2016, 19:32 Victor Stinner  wrote:

> 2016-08-24 17:44 GMT+02:00 Steve Dower :
> > I know Nick and Victor like the idea of a -X flag (or a direct -utf8
> flag),
> > but I prefer more specific environment variables:
> >
> > - PYTHONWINDOWSLEGACYSTDIO (for the console changes)
> > - PYTHONWINDOWSLEGACYPATHENCODING (assuming getfilesystemencoding() is
> utf8)
> > - PYTHONWINDOWSLEGACYLOCALEENCODING (assuming getpreferredencoding() is
> > utf8)
>
> Hum, you propose the opposite behaviour of my proposed "-X utf8"
> switch :-) You propose to force UTF-8 by default, but add a new flag
> to revert to the old behaviour. Right?
>
> About (...)LEGACYSTDIO: I don't think that we need such option. Using
> Unicode is the obvious choice if stdout/stderr is the Windows console
> no? I'm not talking about the case where stdout/stderr is redirected
> (into a file or a pipe). Are you aware of any case where the "legacy
> stdio" is better than the "new" (win_unicode_console-like) stdio?
>
> (...)PATHENCODING and (...)LOCALEENCODING: please don't do that. I
> tried once in Python 3.2 on Linux, and it failed badly:
> https://mail.python.org/pipermail/python-dev/2010-October/104517.html
>
> I don't think that it's a good idea to add an option to use two
> different encodings inside the same process. It's too common to
> exchange data between the "path encoding world" and "locale encoding
> world".
>
> Hum, currently, is it possible to get two different encodings for
> "path encoding" and "locale encoding"? Or both encodings are the ANSI
> code page? I only recall that the console uses a different encoding,
> the OEM code page.
>
> Victor
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/dholth%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] File system path encoding on Windows

[Daniel Holth]

#1 sounds like a great idea. I suppose surrogatepass solves approximately
the same problem of Rust's WTF-8, which is a way to round-trip bad UCS-2?
https://simonsapin.github.io/wtf-8/

#2 sounds like it would leave several problems, since mbcs is not the same
as a normal text encoding, IIUC it depends on the active code page. So if
your active code page is Russian you might not be able to encode Japanese
characters into MBCS.

Solution #2a Modify Windows so utf-8 is a valid value for the current MBCS
code page.

On Fri, Aug 19, 2016 at 3:01 PM Steve Dower  wrote:

> Hi python-dev
>
> About a week ago I proposed on python-ideas making some changes to how
> Python deals with encodings on Windows, specifically in relation to how
> Python interacts with the operating system.
>
> Changes to the console were uncontroversial, and I have posted patches
> at http://bugs.python.org/issue1602 and
> http://bugs.python.org/issue17620 to enable the full range of Unicode
> input to be used at interactive stdin/stdout.
>
> However, changes to sys.getfilesystemencoding(), which determines how
> the os module (and most filesystem functions in general) interpret bytes
> parameters, were more heatedly discussed. I've summarised the discussion
> in this email
>
> I'll declare up front that my preferred change is to treat bytes as
> utf-8 in Python 3.6, and I've posted a patch to do that at
> http://bugs.python.org/issue27781. Hopefully I haven't been too biased
> in my presentation of the alternatives, but this is so you at least know
> which way I'm biased.
>
> I'm looking for some agreement on the answers to the questions I pose in
> the summary.
>
> There is much more detail about them presented after that, as there are
> a number of non-obvious issues at play here. I suspect this will
> eventually become a PEP, but it's presented here as a summary of a
> discussion and not a PEP.
>
> Cheers,
> Steve
>
> Summary
> ===
>
> Representing file system paths on Windows as bytes may result in data
> loss due to the way Windows encodes/decodes strings via its bytes API.
>
> We can mitigate this by only using Window's Unicode API and doing our
> own encoding and decoding (i.e. within posixmodule.c's path converter).
> Invalid characters could cause encoding exceptions rather than data loss.
>
> We can go further to fix this by declaring the encoding of bytes paths
> on Windows must be utf-8, which would also prevent encoding exceptions,
> as utf-8 can fully represent all paths on Windows (natively utf-16-le).
>
> Even though using bytes for paths on Windows has been deprecated for
> three releases, this is not widely known and it may be too soon to
> change the behaviour.
>
> Questions:
> * should we always use Window's Unicode APIs instead of switching
> between bytes/Unicode based on parameter type?
> * should we allow users to pass bytes and interpret them as utf-8 rather
> than letting Windows do the decoding?
> * should we do it in 3.6, 3.7 or 3.8?
>
> Background
> ==
>
> File system paths are almost universally represented as text in some
> encoding determined by the file system. In Python, we expose these paths
> via a number of interfaces, such as the os and io modules. Paths may be
> passed either direction across these interfaces, that is, from the
> filesystem to the application (for example, os.listdir()), or from the
> application to the filesystem (for example, os.unlink()).
>
> When paths are passed between the filesystem and the application, they
> are either passed through as a bytes blob or converted to/from str using
> sys.getfilesystemencoding(). The result of encoding a string with
> sys.getfilesystemencoding() is a blob of bytes in the native format for
> the default file system.
>
> On Windows, the native format for the filesystem is utf-16-le. The
> recommended platform APIs for accessing the filesystem all accept and
> return text encoded in this format. However, prior to Windows NT (and
> possibly further back), the native format was a configurable machine
> option and a separate set of APIs existed to accept this format. The
> option (the "active code page") and these APIs (the "*A functions")
> still exist in recent versions of Windows for backwards compatibility,
> though new functionality often only has a utf-16-le API (the "*W
> functions").
>
> In Python, we recommend using str as the default format because (with
> the surrogateescape handling on POSIX), it can correctly round-trip all
> characters used in paths. On Windows this is strongly recommended
> because the legacy OS support for bytes cannot round-trip all characters
> used in paths. Our support for bytes explicitly uses the *A functions
> and hence the encoding for the bytes is "whatever the active code page
> is". Since the active code page cannot represent all Unicode characters,
> the conversion of a path into bytes can lose information without warning
> (and we can't get a warning from the OS here - 

Re: [Python-Dev] PyPI index deprecation

[Daniel Holth]

You may know that there are approximately 3 pypi maintainers, all
overworked and one paid. It is amazing that it works at all. I don't know
anything about that particular decision though.

On Wed, Jul 13, 2016 at 1:21 PM Dmitry Trofimov <
dmitry.trofi...@jetbrains.com> wrote:

> Hi,
>
> as you probably already know, today the PyPI index page (
> https://pypi.python.org/pypi?%3Aaction=index) was deprecated and ceased
> to be.
>
> Among other things it affected PyCharm IDE that relied on that page to
> enable packaging related features from the IDE. As a result users of
> PyCharm can no longer install/update PyPI packages from PyCharm.
>
> Here is an issue about that in our tracker:
> https://youtrack.jetbrains.com/issue/PY-20081
>
> Given that there are several hundred thouthands of PyCharm users in the
> world -- all 3 editions: Professional, Community, and Educational are
> affected -- this can lead to a storm of a negative feedback, when people
> will start to face the denial of the service.
>
> The deprecation of the index was totally unexpected for us and we weren't
> prepared for that. Maybe we missed some announcement.
>
> We will be very happy if the functionality of the index is restored at
> least for some short
> period of time: please, give as a couple of weeks. That will allow us to
> implement a workaround and provide the fix for the several latest major
> versions of PyChram.
>
> Does anybody know who is responsible for that decision and  whom to
> connect about it? Please help.
>
> Best regards,
>
> Dmitry Trofimov
> PyCharm Team Lead
> JetBrainshttp://www.jetbrains.com
> The Drive To Develop
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/dholth%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Should PY_SSIZE_T_CLEAN break Py_LIMITED_API?

[Daniel Holth]

I was using Py_LIMITED_API under 3.5 and PY_SSIZE_T_CLEAN was set, this
causes some functions not in the limited api to be used and the resulting
extension segfaults in Linux. Is that right?

Thanks,

Daniel
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Making stdlib modules optional for distributions (Was: Breaking up the stdlib (Was: release cadence))

[Daniel Holth]

Yes, not too long ago I installed "every" Python package on Ubuntu, and
Python basically would not start. Perhaps some plugin system was trying to
import everything and caused a segfault in GTK. The "short sys.path" model
of everything installed is importable has its limits.

On Thu, Jul 7, 2016 at 9:24 AM Barry Warsaw  wrote:

> On Jul 07, 2016, at 08:12 AM, Eric V. Smith wrote:
>
> >One thing to keep in mind if we do this is how it interacts with the -S
> >command line option to not include site-packages in sys.path. I currently
> use
> >-S to basically mean "give my python as it was distributed, and don't
> include
> >anything that was subsequently added by adding other RPM's (or package
> >manager of your choice)". I realize that's a rough description, and
> possibly
> >an abuse of -S. If using -S were to start excluding parts of the stdlib,
> that
> >would be a problem for me.
>
> It's an important consideration, and leads to another discussion that's
> recurred over the years.  Operating systems often want an "isolated"
> Python,
> similar to what's given by -I, which cannot be altered by subsequent
> installs.  It's one of the things that lead to the Debian ecosystem using
> dist-packages for PyPI installed packages.  Without isolation, it's just
> too
> easy for some random PyPI thing to break your system, and yes, that has
> really
> happened in the past.
>
> So if we go down the path of moving more of the stdlib to site-packages, we
> also need to get serious about a system-specific isolated Python.
>
> Cheers,
> -Barry
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/dholth%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Why does base64 return bytes?

[Daniel Holth]

It would be a codec. base64_text in the codecs module. Probably 1 line
different than the existing codec. Very easy to use and maintain. Less
surprising and less error prone for everyone who thinks base64 should
convert between bytes to text. Sounds like an obvious win to me.

On Wed, Jun 15, 2016 at 11:08 AM Isaac Morland 
wrote:

> On Wed, 15 Jun 2016, Greg Ewing wrote:
>
> > Simon Cross wrote:
> >> If we only support one, I would prefer it to be bytes since (bytes ->
> >> bytes -> unicode) seems like less overhead and slightly conceptually
> >> clearer than (bytes -> unicode -> bytes),
> >
> > Whereas bytes -> unicode, followed if needed by unicode -> bytes,
> > seems conceptually clearer to me. IOW, base64 is conceptually a
> > bytes-to-text transformation, and the usual way to represent
> > text in Python 3 is unicode.
>
> And in CPython, do I understand correctly that the output text would be
> represented using one byte per character?  If so, would there be a way of
> encoding that into UTF-8 that re-used the raw memory that backs the
> Unicode object?  And, therefore, avoids almost all the inefficiency of
> going via Unicode?  If so, this would be a win - proper use of Unicode to
> represent a text string, combined with instantaneous conversion into a
> bytes object for the purpose of writing to the OS.
>
> Isaac Morland   CSCF Web Guru
> DC 2619, x36650 WWW Software Specialist
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/dholth%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Why does base64 return bytes?

[Daniel Holth]

In that case could we just add a base64_text() method somewhere? Who would
like to measure whether it would be a win?

On Wed, Jun 15, 2016 at 8:34 AM Steven D'Aprano  wrote:

> On Tue, Jun 14, 2016 at 09:40:51PM -0700, Guido van Rossum wrote:
> > I'm officially on vacation, but I was surprised that people now assume
> > RFCs, which specify internet protocols, would have a bearing on
> programming
> > languages. (With perhaps an exception for RFCs that specifically specify
> > how programming languages or their libraries should treat certain
> specific
> > issues -- but I found no evidence that this RFC is doing that.)
>
> Sorry to disturb your vacation!
>
> I hoped that there might have been a nice simple answer, like "the
> main use-case for Base64 is the email module, which needs bytes, and
> thus it was decided". Or even "because backwards compatibility".
>
> Thanks to everyone for their constructive comments, and expecially Mark
> for digging up the original discussion on the Python-3000 list. I'm
> satisfied that the choice made by Python is the right choice, and that
> it meets the spirit (if, arguably, not the letter) of the RFC.
>
>
> --
> Steve
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/dholth%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Why does base64 return bytes?

[Daniel Holth]

IMO this is more a philosophical problem than a programming problem. base64
has a dual-nature. It is both text and bytes. At least it should fit in a
1-byte-per-character efficient Python 3 unicode string also.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] New hash algorithms: SHA3, SHAKE, BLAKE2, truncated SHA512

[Daniel Holth]

OpenSSL sucks. Python would only have to bundle a reference implementation
of the new hash algorithm(s), and unlike TLS suites they tend to just work.

BLAKE2 is important, since it removes the last objection to replacing MD5 -
speed - that has made it hard for cryptography fans to convince MD5 users
to upgrade.

On Fri, May 27, 2016 at 7:13 AM M.-A. Lemburg  wrote:

> On 27.05.2016 13:03, Donald Stufft wrote:
> >
> >> On May 27, 2016, at 6:54 AM, M.-A. Lemburg  wrote:
> >>
> >> IMO, relying on OpenSSL is a better strategy than providing
> >> (and maintaining) our own compatibility versions. Until OpenSSL
> >> has them, people can use Björn's package:
> >
> > Even now, hashlib doesn’t rely on OpenSSL if I recall, I mean it will
> > use it if OpenSSL is available but otherwise it has internal
> implementations
> > too.
>
> I know, but still don't think that's a good idea. It makes
> sense in case you don't want to carry around OpenSSL all the
> time, but how often does that happen nowadays ?
>
> BTW: If I recall correctly, those hash implementations predate
> the deeper support for OpenSSL we now have in Python.
>
> --
> Marc-Andre Lemburg
> eGenix.com
>
> Professional Python Services directly from the Experts (#1, May 27 2016)
> >>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
> >>> Python Database Interfaces ...   http://products.egenix.com/
> >>> Plone/Zope Database Interfaces ...   http://zope.egenix.com/
> 
>
> ::: We implement business ideas - efficiently in both time and costs :::
>
>eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
> D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
>Registered at Amtsgericht Duesseldorf: HRB 46611
>http://www.egenix.com/company/contact/
>   http://www.malemburg.com/
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/dholth%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Adding a threadlocal to the Python interpreter

[Daniel Holth]

Thank you. I started my patch http://bugs.python.org/issue27134 to allow
Python code to set a flag causing the evil str(b'bytes') to raise an
exception. I wasn't sure exactly which module to put it in, so it's in
_string. Please let me know the best place to put the feature and what I
should polish to get it in.

Thanks,

Daniel Holth

On Thu, May 19, 2016 at 6:34 AM Christian Heimes <christ...@python.org>
wrote:

> On 2016-05-19 04:30, Nick Coghlan wrote:
> > On 18 May 2016 at 23:20, Daniel Holth <dho...@gmail.com> wrote:
> >> I would like to take another stab at adding a threadlocal "str(bytes)
> raises
> >> an exception" to the Python interpreter, but I had a very hard time
> >> understanding both how to add a threadlocal value to either the
> interpreter
> >> state or the threadlocal dict that is part of that state, and then how
> to
> >> access the same value from both Python and CPython code. The structs
> were
> >> there but it was just hard to understand. Can someone explain it to me?
> >
> > Christian covered the C aspects of the API, while the general purpose
> > Python aspects live in the threading module.
> >
> > However, the Python level thread-local API doesn't provide direct
> > access to the thread state dict. Instead, it provides access to
> > subdicts stored under per-object keys in that dict, keyed as
> > "thread.local.":
>
> In case you wonder about subdicts, they are required to provide multiple
> thread local objects. Each thread local instance has its own key in each
> thread state dict. The approach enables thread local to have independent
> storage objects.
>
> Christian
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Adding a threadlocal to the Python interpreter

[Daniel Holth]

I would like to take another stab at adding a threadlocal "str(bytes)
raises an exception" to the Python interpreter, but I had a very hard time
understanding both how to add a threadlocal value to either the interpreter
state or the threadlocal dict that is part of that state, and then how to
access the same value from both Python and CPython code. The structs were
there but it was just hard to understand. Can someone explain it to me?

Thanks,

Daniel Holth
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Type hints -- a mediocre programmer's reaction

[Daniel Holth]

On Thu, Apr 23, 2015 at 9:55 AM, Paul Sokolovsky pmis...@gmail.com wrote:
 Hello,

 On Thu, 23 Apr 2015 09:15:44 -0400
 Daniel Holth dho...@gmail.com wrote:

 []

  Also ask why no one used type specifier, they are possible since
  Python 3.0 ?
  Because it is the wrong way for Python.
 
  That's an example of how perceptions differ. In my list, everyone(*)
  uses them - MyPy, MicroPython, etc. Even more should use them (any
  JIT module, which are many), but sit in the bushes, waiting for a
  kick, like PEP484 provides.

 It's OK that type hints are only to assist the programmer.

 Yes, it's OK to have a situation where type hints assist only a
 programmer. It's not OK to think that type hints may be useful only for
 programmer, instead of bunch more purposes, several of which
 were already shown in the long previous discussion.

 PyPy's FAQ
 has an explanation of why type hints are not for performance.
 http://pypy.readthedocs.org/en/latest/faq.html#would-type-annotations-help-pypy-s-performance

 You probably intended to write why type hints are not for *PyPy's*
 performance. There're many other language implementations and modules
 for which it may be useful, please don't limit your imagination by a
 single case.

 And speaking of PyPy, it really should think how to improve its
 performance - not of generated programs, but of generation itself. If
 compilation of a trivial program on a pumpy hardware takes 5 minutes
 and gigabytes of RAM and diskspace, few people will use it for other
 purposes beyond curiosity. There's something very un-Pythonic in
 waiting 5 mins just to run 10-line script. Type hints can help here
 too ;-) (by not wasting resources propagating types thru the same old
 standard library for example).

Naturally, PyPy is very controversial.

Type annotations can help to compile Python into a subset of Python.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Type hints -- a mediocre programmer's reaction

[Daniel Holth]

On Thu, Apr 23, 2015 at 5:59 AM, Paul Sokolovsky pmis...@gmail.com wrote:
 Hello,

 On Thu, 23 Apr 2015 10:43:52 +0200
 Wolfgang Langner tds333+py...@gmail.com wrote:

 []

 Also ask why no one used type specifier, they are possible since
 Python 3.0 ?
 Because it is the wrong way for Python.

 That's an example of how perceptions differ. In my list, everyone(*)
 uses them - MyPy, MicroPython, etc. Even more should use them (any JIT
 module, which are many), but sit in the bushes, waiting for a kick, like
 PEP484 provides.

It's OK that type hints are only to assist the programmer. PyPy's FAQ
has an explanation of why type hints are not for performance.
http://pypy.readthedocs.org/en/latest/faq.html#would-type-annotations-help-pypy-s-performance
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Type hints -- a mediocre programmer's reaction

[Daniel Holth]

On Tue, Apr 21, 2015 at 1:10 PM, Guido van Rossum gu...@python.org wrote:
 On Tue, Apr 21, 2015 at 9:17 AM, R. David Murray rdmur...@bitdance.com
 wrote:

 Please be respectful rather than inflammatory.  If you read what I
 wrote, I did not say that I was going to stop contributing, I
 specifically talked about that gut reaction being both emotional and
 illogical.  That doesn't make the reaction any less real, and the fact
 that such reactions exist is a data point you should consider in
 conducting your PR campaign for this issue.  (I don't mean that last as
 a negative:  this issue *requires* an honest PR campaign.)


 Well, my own reactions at this point in the flame war are also quite
 emotional. :-(

 I have done my best in being honest in my PR campaign. But I feel like the
 opposition (not you, but definitely some others -- have you seen Twitter?)
 are spreading FUD based on an irrational conviction that this will destroy
 Python. It will not. It may not prove the solution to all Python's problems
 -- there's always 3.6. (Oh wait, Python 2.7 is perfect. I've heard that
 before -- Paul Everitt famously said the same of Python 1.5.2. Aren't you
 glad I didn't take him literally? :-P )

 --
 --Guido van Rossum (python.org/~guido)

I am looking forward to using type annotations. I am not very good at
remembering the operations that are available on the objects passed
around in my code, but I am very good at typing CTRL-space. To get
that I am happy to modify my code with weird docstrings or any other
notation. Good support for completion, aided by standard annotations,
eliminates a huge amount of cross-referencing while coding.

I'm also hopeful that static type checking, aided with annotations,
will help with unicode porting. Duck typing does not work very well
when you are trying to differentiate between bytes and str.

Also, Python 1.5.2 was pretty good :-)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] easy_install ?

[Daniel Holth]

That might mostly do what you want, since tox could install any
additional test requirements based on its configuration.

On Tue, Feb 24, 2015 at 4:21 PM, Alexander Belopolsky
alexander.belopol...@gmail.com wrote:

 On Tue, Feb 24, 2015 at 2:03 PM, Daniel Holth dho...@gmail.com wrote:

  Is there a recommended way to invoke pip from setup.py?  When I specify
  tests_require= and run python setup.py test, the requirements get
  installed using setuptools' easy_install function.

 The solution is to not do that. A substitute is to specify your test
 requirements in a [test] extra and install them with pip or to run
 tests with tox. This gives control of the installer back to the user
 instead of the setup.py author.


 Isn't this a chicken and egg problem?  I currently have

 tests_require=['tox'],

 and this is exactly what tox recommends:

 https://testrun.org/tox/latest/example/basic.html#integration-with-setuptools-distribute-test-commands


 Note that my CI box is a CentOS 6.5 with Python 2.6.6, setuptools 0.6.  This
 is still a very common server configuration.  What is the recommended way to
 bootstrap tox in such environment?

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] easy_install ?

[Daniel Holth]

On Tue, Feb 24, 2015 at 1:31 PM, Alexander Belopolsky
alexander.belopol...@gmail.com wrote:

 On Tue, Feb 24, 2015 at 11:44 AM, Paul Moore p.f.mo...@gmail.com wrote:

 And if pip won't work, it would be good to
 know why.


 Is there a recommended way to invoke pip from setup.py?  When I specify
 tests_require= and run python setup.py test, the requirements get
 installed using setuptools' easy_install function.

The solution is to not do that. A substitute is to specify your test
requirements in a [test] extra and install them with pip or to run
tests with tox. This gives control of the installer back to the user
instead of the setup.py author.

https://pythonhosted.org/setuptools/setuptools.html#declaring-extras-optional-features-with-their-
own-dependencies
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] easy_install ?

[Daniel Holth]

The other option might be to use http://bitbucket.org/dholth/setup-requires

It uses pip to install requirements into an isolated directory before
setup.py runs, with pip, doing exactly what you requested.
On Feb 24, 2015 5:44 PM, Nick Coghlan ncogh...@gmail.com wrote:


 On 25 Feb 2015 07:23, Alexander Belopolsky 
 alexander.belopol...@gmail.com wrote:
 
 
  On Tue, Feb 24, 2015 at 2:03 PM, Daniel Holth dho...@gmail.com wrote:
  
Is there a recommended way to invoke pip from setup.py?  When I
 specify
tests_require= and run python setup.py test, the requirements get
installed using setuptools' easy_install function.
  
   The solution is to not do that. A substitute is to specify your test
   requirements in a [test] extra and install them with pip or to run
   tests with tox. This gives control of the installer back to the user
   instead of the setup.py author.
 
 
  Isn't this a chicken and egg problem?  I currently have
 
  tests_require=['tox'],
 
  and this is exactly what tox recommends:
 
 
 https://testrun.org/tox/latest/example/basic.html#integration-with-setuptools-distribute-test-commands
 
 
  Note that my CI box is a CentOS 6.5 with Python 2.6.6, setuptools 0.6.
 This is still a very common server configuration.  What is the recommended
 way to bootstrap tox in such environment?

 If running in the system Python isn't absolutely essential, then the
 Python 2.7 collection from softwarecollections.org is the preferred way
 to get a newer Python 2 (including pip et al) on CentOS. You can also get
 access to Python 3 that way.

 Failing that, pip  virtualenv are also available from the EPEL 6 repos.

 Both of those approaches rely on the system package manager to do the
 bootstrapping of the Python specific tooling.

 If both softwarecollections.org and EPEL are considered unacceptable
 dependencies, then you're going to have to do your own bootstrapping for
 PyPI access on CentOS (which may include relying on easy_install to
 bootstrap pip and/or virtualenv)

 Regards,
 Nick.

 
 
  ___
  Python-Dev mailing list
  Python-Dev@python.org
  https://mail.python.org/mailman/listinfo/python-dev
  Unsubscribe:
 https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
 

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 441 - Improving Python ZIP Application Support

[Daniel Holth]

On Sun, Feb 15, 2015 at 1:45 PM, Serhiy Storchaka storch...@gmail.com wrote:
 On 15.02.15 18:21, Thomas Wouters wrote:

 which requires that extension modules are stored uncompressed (simple)
 and page-aligned (harder, as the zipfile.ZipFile class doesn't directly
 support page-aligning anything


 It is possible to add this feature to ZipFile. It can be useful because will
 allow to mmap uncompressed files in ZIP file.

One of the cool things about ZIP files is that they ignore any data
that is not referenced by the index (which is at the end of the file).
That is why we can put a #! at the beginning of the file, prefix .exe
self-extractors, or put extra data between files in the archive. You
could probably even overlap archive members if you were motivated. But
it would be best to let the uncompressed extensions be standard ZIP
members.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 441 - Improving Python ZIP Application Support

[Daniel Holth]

Go ahead, make my pep.

I will appreciate seeing it happen.
On Feb 15, 2015 8:47 AM, Nick Coghlan ncogh...@gmail.com wrote:

 On 15 February 2015 at 23:21, Paul Moore p.f.mo...@gmail.com wrote:
  On 15 February 2015 at 08:59, Nick Coghlan ncogh...@gmail.com wrote:
  The other option would to cut PEP 441 way back to *just* be about
  standardising and registering the file associations, and recommending
  the use of pip to obtain the build machinery with (whether pyzaa,
  pyzzer or Twitter's more comprehensive pex). It would be a short PEP,
  but potentially still worth it for the improved visibility of the
  decision when folks are trying to figure out what pyz and pyzw
  files are later.
 
  Ok, thinking about this a little more.
 
  Getting the extension support is the key thing on Windows - at the
  moment, people are faced with adding their own file associations or
  putting binary data in a .py file, neither of which is a nice choice.
  Tooling is important, though - sure, you can zip the data up and put a
  header on, but it's fiddly.
 
  Which brings us full circle. A simple module, executable as python -m
  zipapp (see below re name) which exports a single function, pack()
  that creates the archive. If we want to provide a script to wrap the
  module, like pyvenv.py does for venv, I've no objection to that -
  presumably it would go in Tools/Scripts? If people (like me) want to
  experiment with a more programmatic API for building pyz files, they
  can do so on PyPI, and if such a thing becomes sufficiently mature we
  can look then at proposing it for inclusion in the stdlib, as an
  extension to the zipapp module.
 
  Regarding naming, I'm happy to go with zipapp if it's your preference.
  Presumably the wrapper in Tools/Scripts would be pyzipapp.py?

 Or we just skip the wrapper and make python -m zipapp the
 recommended invocation style. Adding a wrapper later is fairly easy,
 but removing it would be difficult.

 
  So the usage would be something like
 
  python -m zipapp [options] dir_to_zip
 
  Options:
  -p interpreterThe interpreter to use on the shebang line
  (defaulting to /usr/bin/env python)
  -o archive_name The name of the output file (defaulting to the
  source directory name with a .pyz extension)
  If the argument has no extension, add '.pyz'
  -m module:function  The entry point to call (written to __main__.py)
  Using this is an error if there is a
  __main__.py, and mandatory if there isn't
 
  If you want anything more complex, it's easy enough to write your own
  script based on zipfile, or use one of the modules on PyPI.
 
  Does this sound reasonable? If it's OK, I'll go ahead and prepare an
  update to the PEP and an implementation. (Steve, looks like I may be
  learning how to maintain the wix files after all - wish me luck :-))
  If I hear no objections in the next couple of days, I'll assume
  everyone's OK with it and I'll prepare a PEP update and a patch.

 Sounds good to me.

 Daniel, do you mind if Paul becomes a co-author on PEP 441 and updates
 it as described? It seems a bit tidier than allocating a new PEP
 number and rejecting PEP 441, when the revised proposal is essentially
 just a simplified and renamed version of your original idea.

 Cheers,
 Nick.

 --
 Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Sysadmin tasks

[Daniel Holth]

On Wed, Oct 1, 2014 at 1:59 PM, Mark Shannon m...@hotpy.org wrote:
 Hi,

 http://speed.python.org/
 could do with some love.

+1
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Internal representation of strings and Micropython

[Daniel Holth]

On Thu, Jun 5, 2014 at 11:59 AM, Paul Moore p.f.mo...@gmail.com wrote:
 On 5 June 2014 14:15, Nick Coghlan ncogh...@gmail.com wrote:
 As I've said before in other contexts, find me Windows, Mac OS X and
 JVM developers, or educators and scientists that are as concerned by
 the text model changes as folks that are primarily focused on Linux
 system (including network) programming, and I'll be more willing to
 concede the point.

 There is once again a strong selection bias in this discussion, by its
 very nature. People who like the new model don't have anything to
 complain about, and so are not heard.

 Just to support Nick's point, I for one find the Python 3 text model a
 huge benefit, both in practical terms of making my programs more
 robust, and educationally, as I have a far better understanding of
 encodings and their issues than I ever did under Python 2. Whenever a
 discussion like this occurs, I find it hard not to resent the people
 arguing that the new model should be taken away from me and replaced
 with a form of the old error-prone (for me) approach - as if it was in
 my best interests.

 Internal details don't bother me - using UTF8 and having indexing be
 potentially O(N) is of little relevance. But make me work with a
 string type that *doesn't* abstract a string as a sequence of Unicode
 code points and I'll get very upset.

Once you get past whether str + bytes throws an exception which seems
to be the divide most people focus on, you can discover new things
like dance-encoded strings, bytes decoded using an incorrect encoding
intended to be transcoded into the correct encoding later, surrogates
that work perfectly until .encode(), str(bytes), APIs that disagree
with you about whether the result should be str or bytes, APIs that
return either string or bytes depending on their initializers and so
on. Unicode can still be complicated in Python 3 independent of any
judgement about whether it is worse, better, or different than Python
2.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Some notes about MicroPython from an observer

[Daniel Holth]

- micropython is designed to run on a machine with 192 kilobytes of
RAM and perhaps a megabyte of FLASH. The controller can execute
read-only code directly from FLASH. There is no dynamic linker in this
environment. (It also has a UNIX port).
- However it does include a full Python parser and REPL, so the board
can be programmed without a separate computer as opposed to, say,
having to upload bytecode compiled on a regular computer.
- It's definitely going to be a subset of Python. For example,
func.__name__ is not supported - to make it more micro?
- They have a C API. It is much different than the CPython C API.
- It mas more than one code emitter. A certain decorator causes a
function to be compiled to ARM Thumb code instead of bytecode.
- It even has an inline assembler than translates Python-syntax ARM
assembly (to re-use the same parser) into machine code.

Most information from
https://www.kickstarter.com/projects/214379695/micro-python-python-for-microcontrollers/posts
and http://micropython.org/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Internal representation of strings and Micropython

[Daniel Holth]

Can of worms, opened.
On Jun 4, 2014 7:20 AM, Chris Angelico ros...@gmail.com wrote:

 On Wed, Jun 4, 2014 at 9:12 PM, Paul Sokolovsky pmis...@gmail.com wrote:
  An alternative view is that the discussion on the tracker showed Python
  developers' mind-fixation on implementing something the way CPython does
  it. And I didn't yet go to that argument, but in the end, MicroPython
  does not try to rewrite CPython or compete with it. So, having few
  choices with pros and cons leading approximately to the tie among them,
  it's the least productive to make the same choice as CPython did.

 I'm not a CPython dev, nor a Python dev, and I don't think any of the
 big names of CPython or Python has showed up on that tracker as yet.
 But why is be different from CPython such a valuable choice? CPython
 works. It's had many hours of dev time put into it. Problems have been
 identified and avoided. Throwing that out means throwing away a
 freely-given shoulder to stand on, in an Isaac Newton way.

 http://www.joelonsoftware.com/articles/fog69.html

 ChrisA
 ___
 Python-Dev mailing list
 Python-Dev@python.org
 https://mail.python.org/mailman/listinfo/python-dev
 Unsubscribe:
 https://mail.python.org/mailman/options/python-dev/dholth%40gmail.com

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Internal representation of strings and Micropython

[Daniel Holth]

If we're voting I think representing Unicode internally in micropython
as utf-8 with O(N) indexing is a great idea, partly because I'm not
sure indexing into strings is a good idea - lots of Unicode code
points don't make sense by themselves; see also grapheme clusters. It
would probably work great.

On Wed, Jun 4, 2014 at 7:49 AM, Paul Sokolovsky pmis...@gmail.com wrote:
 Hello,

 On Wed, 4 Jun 2014 20:53:46 +1000
 Chris Angelico ros...@gmail.com wrote:

 On Wed, Jun 4, 2014 at 8:38 PM, Paul Sokolovsky pmis...@gmail.com
 wrote:
  And I'm saying that not to discourage Unicode addition to
  MicroPython, but to hint that force-force approach implemented by
  CPython3 and causing rage and split in the community is not
  appreciated.

 FWIW, it's Python 3 (the language) and not CPython 3.x (the
 implementation) that specifies Unicode strings in this way.

 Yeah, but it's CPython what dictates how language evolves (some people
 even think that it dictates how language should be implemented!), so all
 good parts belong to Python3, and all bad parts - to CPython3,
 right? ;-)

 I don't
 know why it has to cause a split in the community; this is the one way
 to make sure *everyone's* strings work perfectly, rather than having
 ASCII strings work fine and others start tripping over problems in
 various APIs.

 It did cause split in the community, that's the fact, that's why
 Python2 and Python3 are at the respective positions. Anyway, I'm not
 interested in participating in that split, I did not yet uttered my
 opinion on that publicly enough, so I seized a chance to drop some
 witty remarks, but I don't want to start yet another Unicode flame.



 So, let's please be back to Unicode storage representation in
 MicroPython. So, https://github.com/micropython/micropython/issues/657
 discussed technical aspects, in a recent mail on this list I expressed
 my opinion why following CPython way is not productive (for development
 satisfaction and evolution of Python community, to be explicit).

 Final argument I would have is that you certainly can implement Unicode
 support the PEP393 way - it would be enormous help and would be gladly
 accepted. The question, how useful it will be for MicroPython. It
 certainly will be useful to report passing of testsuites. But will it
 be *really* used?

 For microcontroller board, it might be too heavy (put simple, with it,
 people will be able to do less (== heap running out sooner)), than
 without it, so one may expect it to be disabled by default. Then POSIX
 port is there surely not to let people replace python command
 with micropython and run Django, but to let people develop and debug
 their apps with more comfort than on embedded board. So, it should
 behave close to MCU version, and would follow with MCU choice
 re: Unicode.

 That's actually the reason why I keep up this discussion - not for the
 sake of argument or to bash Python3's Unicode choices. With recent
 MicroPython announcement, we surely looked for more people to
 contribute to its development. But then we (or at least I can speak for
 myself), would like to make sure that these contribution are actually
 the most useful ones (for both MicroPython, and Python community in
 general, which gets more choices, rather than just getting N% smaller
 CPython rewrite).

 So, you're not sure how O(N) string indexing will work? But MicroPython
 offers a great opportunity to try! And it's something new and exciting,
 which surely will be useful (== will save people memory), not just
 something old and boring ;-).



 ChrisA


 --
 Best regards,
  Paul  mailto:pmis...@gmail.com
 ___
 Python-Dev mailing list
 Python-Dev@python.org
 https://mail.python.org/mailman/listinfo/python-dev
 Unsubscribe: 
 https://mail.python.org/mailman/options/python-dev/dholth%40gmail.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Internal representation of strings and Micropython

[Daniel Holth]

MicroPython is going to be significantly incompatible with Python
anyway. But you should be able to run your mp code on regular Python.

On Wed, Jun 4, 2014 at 9:39 AM, Serhiy Storchaka storch...@gmail.com wrote:
 04.06.14 04:17, Steven D'Aprano написав(ла):

 Would either of these trade-offs be acceptable while still claiming
 Python 3.4 compatibility?

 My own feeling is that O(1) string indexing operations are a quality of
 implementation issue, not a deal breaker to call it a Python. I can't
 see any requirement in the docs that str[n] must take O(1) time, but
 perhaps I have missed something.


 I think than breaking O(1) expectation for indexing makes the implementation
 significant incompatible with Python. Virtually all string operations in
 Python operates with indices.

 O(1) indexing operations can be kept with minimal memory requirements if
 implement Unicode internally as modified UTF-8 plus optional array of
 offsets for every, say, 32th character (which even can be compressed to an
 array of 16-bit or 32-bit integers).


 ___
 Python-Dev mailing list
 Python-Dev@python.org
 https://mail.python.org/mailman/listinfo/python-dev
 Unsubscribe:
 https://mail.python.org/mailman/options/python-dev/dholth%40gmail.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Internal representation of strings and Micropython

[Daniel Holth]

On Wed, Jun 4, 2014 at 10:12 AM, Steven D'Aprano st...@pearwood.info wrote:
 On Wed, Jun 04, 2014 at 01:14:04PM +, Steve Dower wrote:
 I'm agree with Daniel. Directly indexing into text suggests an
 attempted optimization that is likely to be incorrect for a set of
 strings.

 I'm afraid I don't understand this argument. The language semantics says
 that a string is an array of code points. Every index relates to a
 single code point, no code point extends over two or more indexes.
 There's a 1:1 relationship between code points and indexes. How is
 direct indexing likely to be incorrect?

Useful is probably a better word. When you get into the complicated
languages and you want to know how wide something is, and you might
have y with two dots on it as one code point or two and left-to-right
and right-to-left indicators and who knows what else... then looking
at individual code points only works sometimes. I get the slicing
idea.

I like the idea that encoding to utf-8 would be the fastest thing you
can do with a string. You could consider doing regexps in that domain,
and other implementation specific optimizations in exactly the same
way that any Python implementation has them.

None of this would make it harder to move a servo.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] pep8 reasoning

[Daniel Holth]

Fortunately, Unicode provides us with the COMBINING LOW LINE
character, combining the horizontal space-savings of camelCase with
the underscore-indicates-separation properties of _. And it's a valid
Python identifier.

convertx̲mlt̲oj̲son



On Thu, Apr 24, 2014 at 12:25 PM, Chris Angelico ros...@gmail.com wrote:
 On Thu, Apr 24, 2014 at 11:59 PM, Barry Warsaw ba...@python.org wrote:
 I will say this: the original preference for underscore_names in PEP 8 was
 spurred by user studies some of our early non-native English speaking users
 conducted many years ago.  We learned that it was more difficult for many of
 them to parse mixedCase names than underscore_names.  I'm afraid I probably 
 no
 longer have references to those studies, but the difference was pronounced,
 IIRC, and I think it's easy to see why.  Underscores can be scanned by the 
 eye
 as spaces, while I'd hypothesize that the brain has to do more work to read
 mixedCase names.

 Underscores also play much more nicely with initialisms. How would you
 spell a function named Add HTTP Header?

 add_HTTP_header
 add_http_header

 addHTTPHeader
 addHttpHeader

 Four options to choose from. The first two clearly separate the
 initialism from the other words; take your pick whether you want it
 uppercased or not, because it's separated either way. In mixedCase,
 the first one merges the H of Header in with HTTP; with something less
 well known, that can be a nasty readability problem. The second one is
 probably more readable, but looks weird. Or, here's another one:
 converting one thing into another, where both are named by their
 initials:

 convert_XML_to_JSON
 convert_xml_to_json

 convertXMLToJSON
 convertXmlToJson

 Same four options. Which is the more readable?

 ChrisA
 ___
 Python-Dev mailing list
 Python-Dev@python.org
 https://mail.python.org/mailman/listinfo/python-dev
 Unsubscribe: 
 https://mail.python.org/mailman/options/python-dev/dholth%40gmail.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Software integrators vs end users (was Re: Language Summit notes)

[Daniel Holth]

On Fri, Apr 18, 2014 at 12:55 PM, Paul Moore p.f.mo...@gmail.com wrote:
 On 18 April 2014 16:58, Nick Coghlan ncogh...@gmail.com wrote:
 As part of thrashing out the respective distribution ecosystem roles
 of pip and conda (still a work in progress), we're at least converging
 on the notion that there are actually now *two* main ways of consuming
 Python: as a software integrator (the way most of us have
 traditionally consumed it, and the way that dominates most project
 documentation outside the scientific Python community) and as an end
 user (the way Linux system administrators have long consumed it, and
 the way scientists, financial analysts and folks just learning Python
 are likely best off consuming it).

 Making these different personas explicit is a process that has barely
 begun (this email is mostly based on some conversations I had in
 person at PyCon and via email during the sprints), but here's the gist
 (based on listing examples):

 Interesting perspective. However, I'm not convinced it's complete.
 Specifically, there's one group of people who I encounter relatively
 often, who don't seem to me to fit well into either category you're
 proposing. That is, (Windows in my experience, but maybe Linux as
 well) users who want to write simple scripts and for whom batch
 files or similar are not sufficient. Such people typically don't have
 the sort of single application area focus that your end user
 category seems to imply, but on the other hand they don't really fit
 the software integrators role in the sense of necessarily being
 comfortable setting up their own development environment.

 I worry that your classification risks ignoring that group (maybe
 because Unix users are well served with other alternatives than Python
 for this type of task, or because on Unix use the system Python is
 the right answer).

 Your list of end user targeted distributions seem to be limited to:

   - Linux distribution vendors
   - Vendors focused on the essentially scientific community (in the
 broadest sense)
   - Embedded Python

 That's very far from being complete coverage of all the people *I'd*
 like to be able to recommend Python to. Specifically, unless we're not
 interested in generic Windows users, I think we need to offer *some*
 form of equivalent of the OS-packaged Python on Linux for Windows
 users. That's what the python.org builds, plus pip, wheels and PyPI,
 give for Windows users now. Hmm, if we assume that supporting that
 remains a priority, is what you're really saying that we *don't* try
 to extend that to work for Linux/OSX, as doing so competes with the OS
 vendors - but rather we see python.org binaries and binary
 infrastructure like wheels as being focused on the Windows user
 experience?

 (I wish I'd been at PyCon, this would have been a very interesting
 discussion to have face to face. Email isn't ideal for this...)

One more group that I find interesting is application users. These
people should not need to notice that Python is in use at all, in
contrast to the build a virtualenv / install / pass through fire and
death / use workflow that is sometimes promoted. They are well served
by good tools that make single-file zip distributions or py2exe etc.
from a collection of wheels or sdists. Application users are using
Python because a best-in-class program is written in Python and not
because a Python program integrates better with other Python
libraries.


Conda is interesting because it is a system package manager, except it
installs everything into what they sometimes describe as C-level
virtualenvs rather than /. They've come up with a design that works
well with Python programs but isn't particularly Python-specific at
all.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

[Daniel Holth]

IIRC it is no longer the case that ZIP imports (involving only one
file for a lot of modules) are much faster than regular FS imports?

On Tue, Apr 15, 2014 at 10:34 AM, Eric Snow ericsnowcurren...@gmail.com wrote:
 On Tue, Apr 15, 2014 at 1:45 AM, Chris Angelico ros...@gmail.com wrote:
 Specific use-case that I can see: Mercurial. In a git vs hg shoot-out,
 git will usually win on performance, and hg is using Py2; migrating hg
 to Py3 will (if I understand the above figures correctly) widen that
 gap, so any improvement done to startup performance will give a very
 real advantage.

 Perhaps not so much a very real advantage as less of a
 distraction. It's still significantly slower than 2.7.  :)

 -eric
 ___
 Python-Dev mailing list
 Python-Dev@python.org
 https://mail.python.org/mailman/listinfo/python-dev
 Unsubscribe: 
 https://mail.python.org/mailman/options/python-dev/dholth%40gmail.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Mercurial sluggishness (was: this is what happens if you freeze all the modules required for startup)

[Daniel Holth]

I find Python's startup time to be very sluggish. I wish it was less
than 50 milliseconds (0.05 seconds) including running hg, which is the
common threshold for instant. On my machine 'python -c ' takes
about 0.05 seconds but 'python3 -c ' takes 0.125 seconds. I will be
very happy to see any speedup.

On Tue, Apr 15, 2014 at 11:34 AM, Skip Montanaro s...@pobox.com wrote:
 Eric wrote:

 Perhaps not so much a very real advantage as less of a
 distraction.  It's still significantly slower than 2.7.  :)

 I'm still confused. I peeked in /usr/bin/hg. The only system modules
 it imports directly are os and system (maybe I'm using an ancient
 version?). After that, it imports its own lazy import module. This
 suggests to me that Mercurial's import slowness is mostly in its own
 code (I counted 104 Python modules and six shared objects in its
 mercurial package, which isn't going to be affected (much?) by
 freezing the Python standard modules.

 I'm not trying to be difficult here. I thought that way back in the
 day a huge amount of work was done to remove needless filesystem
 activity, and zip packaging has been around for quite awhile.

 As an experiment, I ran hg pull as

 /usr/bin/python -v /usr/bin/hg pull

 in my cpython repo then looked at the -v output. Summarizing the
 output I saw the following:

 30 imports (0 dlopens)

 Python banner printed

 86 imports (18 dlopens)

 adding changesets message

 5 imports (2 dlopens)

 adding manifests message

 1 import (0 dlopens)

 adding file changes message

 7 imports (3 dlopens)

 added ... changesets message

 4 imports (0 dlopens)

 run 'hg update' message

 (I missed a searching message in there somewhere.)

 That's a total of 133 imports, 23 of which were satisfied by loading
 an extension module. The imports break down as follows:

 51 imports (4 dlopens) from the mercurial package
 5 imports from the hgext package
 7 imports from the encodings package

 Everything else is imported from the top level, and at a glance
 appears to be all Python stdlib stuff.  The key period of execution
 looks to be between the printing of the Python banner and the printing
 of the adding changesets message. I see 46 imports (2 dlopens) from
 the mercurial or hgext packages. That leaves 40 stdlib imports, of
 which 16 were satisfied by dlopen.

 As a final check, I saved all the stdlib import statements from the -v
 output (77 statements) to a file and timed its execution:

 % time /usr/bin/python ~/tmp/pyimp.py

 real0m0.162s
 user0m0.034s
 sys 0m0.010s

 It doesn't take much time to import every stdlib module that Mercurial
 needs.  I don't know how much slower all this import machinery is in
 3.x (and can't test at work, as we don't have a copy laying about). It
 would probably have to be 3x or more slower for it to have much
 visible impact on even simple hg commands.  I find it hard to believe
 that freezing the stdlib is going to lower the barrier enough for the
 Mercurial folks, if, in fact, import slowness is their main reason for
 not moving to 3.x.

 Skip
 ___
 Python-Dev mailing list
 Python-Dev@python.org
 https://mail.python.org/mailman/listinfo/python-dev
 Unsubscribe: 
 https://mail.python.org/mailman/options/python-dev/dholth%40gmail.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Mercurial sluggishness (was: this is what happens if you freeze all the modules required for startup)

[Daniel Holth]

In case you were wondering, I'm using Ubuntu's 2.7.5+ and 3.3.2+.

My feeling has long been that the speed of getting at the --help
option or any initial user feedback from Mercurial or git is a big
driver in perceived speed as opposed to how long the entire operation
might take. But for me any initial speed improvements from git are
fully offset by the feeling of irritation afterwards. /troll

For me Python's startup time (warm) takes about 1/4 of the hg startup
time in the worst case. I expect to both notice and appreciate any
speedups and encourage all optimizers to optimize.

On Tue, Apr 15, 2014 at 12:47 PM, Antoine Pitrou solip...@pitrou.net wrote:
 Le 15/04/2014 17:42, Daniel Holth a écrit :

 I find Python's startup time to be very sluggish. I wish it was less
 than 50 milliseconds (0.05 seconds) including running hg, which is the
 common threshold for instant. On my machine 'python -c ' takes
 about 0.05 seconds but 'python3 -c ' takes 0.125 seconds.


 Please quote exact versions. Different versions of Python 3 will have
 different startup characteristics.

 Regards

 Antoine.



 ___
 Python-Dev mailing list
 Python-Dev@python.org
 https://mail.python.org/mailman/listinfo/python-dev
 Unsubscribe:
 https://mail.python.org/mailman/options/python-dev/dholth%40gmail.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Mercurial sluggishness (was: this is what happens if you freeze all the modules required for startup)

[Daniel Holth]

On Tue, Apr 15, 2014 at 1:29 PM, Antoine Pitrou solip...@pitrou.net wrote:
 Le 15/04/2014 19:09, Daniel Holth a écrit :

 In case you were wondering, I'm using Ubuntu's 2.7.5+ and 3.3.2+.

 My feeling has long been that the speed of getting at the --help
 option or any initial user feedback from Mercurial or git is a big
 driver in perceived speed as opposed to how long the entire operation
 might take. But for me any initial speed improvements from git are
 fully offset by the feeling of irritation afterwards. /troll

 For me Python's startup time (warm) takes about 1/4 of the hg startup
 time in the worst case. I expect to both notice and appreciate any
 speedups and encourage all optimizers to optimize.


 Well, if we optimize 11% out of that 1/4, I don't expect you to notice the
 speedup at all ;-)


 Regards

 Antoine.

No one expects the Spanish Inquisition.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 465: A dedicated infix operator for matrix multiplication

[Daniel Holth]

On Tue, Apr 8, 2014 at 12:08 AM, Nick Coghlan ncogh...@gmail.com wrote:

 On 7 Apr 2014 21:58, MRAB pyt...@mrabarnett.plus.com wrote:

 On 2014-04-08 02:45, Guido van Rossum wrote:

 So what? Aren't we allowed to have fun? :-)

 Next thing you know, he'll be threatening people with The Comfy Chair!

 You may want to take a look at the packaging metadata 2.0 spec ;)

 I was also going to add a +1 for the actual topic of this thread, but
 Guido's acceptance of the PEP rendered that point rather moot :)

@ [1]
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 3

[Daniel Holth]

I feel not including %s is nuts. Should I write .replace('%b', '%s')?
All I desperately need are APIs that provide enough unicode / str type
safety that I get an exception when mixing them accidentally... in my
own code, dynamic typing is usually a bug. As has been endlessly
discussed, %s for bytes is a bit like exposing sprintf()...

On Thu, Mar 27, 2014 at 2:41 PM, Guido van Rossum gu...@python.org wrote:
 Much better, but I'm still not happy with including %s at all. Otherwise
 it's accept-worthy. (How's that for pressure. :-)


 On Thu, Mar 27, 2014 at 11:04 AM, Ethan Furman et...@stoneleaf.us wrote:

 On 03/27/2014 10:55 AM, Ethan Furman wrote:

 On 03/27/2014 10:29 AM, Guido van Rossum wrote:


 I also don't understand why we can't use %b instead of %s. AFAIK %b
 currently doesn't mean anything and I somehow don't
 expect we're likely to add it for other reasons (unless there's a
 proposal I'm missing?). Just like we use %a instead of
 %r to remind people that it's not quite the same (since it applies
 .encode('ascii', 'backslashreplace')), shouldn't we
 use anything *but* %s to remind people that that is also not the same
 (not at all, in fact)? The PEP's argument against
 %b (rejected as not adding any value either in clarity or simplicity)
 is hardly a good reason.


 The biggest reason to use %s is to support a common code base for 2/3
 endeavors.  The biggest reason to not include %b
 is that it means binary number in format(); given that each type can
 invent it's own mini-language, this probably isn't
 a very strong argument against it.

 I have moderate feelings for keeping %s as a synonym for %b for backwards
 compatibility with Py2 code (when it's
 appropriate).


 Changed to:

 --
 ``%b`` will insert a series of bytes.  These bytes are collected in one of
 two
 ways:

   - input type supports ``Py_buffer`` [4]_?

 use it to collect the necessary bytes

   - input type is something else?
 use its ``__bytes__`` method [5]_ ; if there isn't one, raise a
 ``TypeError``

 In particular, ``%b`` will not accept numbers nor ``str``.  ``str`` is
 rejected
 as the string to bytes conversion requires an encoding, and we are
 refusing to
 guess; numbers are rejected because:

   - what makes a number is fuzzy (float? Decimal? Fraction? some user
 type?)

   - allowing numbers would lead to ambiguity between numbers and textual
 representations of numbers (3.14 vs '3.14')

   - given the nature of wire formats, explicit is definitely better than
 implicit

 ``%s`` is included as a synonym for ``%b`` for the sole purpose of making
 2/3 code
 bases easier to maintain.  Python 3 only code should use ``%b``.

 Examples::

  b'%b' % b'abc'
 b'abc'

  b'%b' % 'some string'.encode('utf8')
 b'some string'

  b'%b' % 3.14

 Traceback (most recent call last):
 ...
 TypeError: b'%b' does not accept 'float'

  b'%b' % 'hello world!'

 Traceback (most recent call last):
 ...
 TypeError: b'%b' does not accept 'str'

 --


 --
 ~Ethan~
 ___
 Python-Dev mailing list
 Python-Dev@python.org
 https://mail.python.org/mailman/listinfo/python-dev
 Unsubscribe:
 https://mail.python.org/mailman/options/python-dev/guido%40python.org




 --
 --Guido van Rossum (python.org/~guido)

 ___
 Python-Dev mailing list
 Python-Dev@python.org
 https://mail.python.org/mailman/listinfo/python-dev
 Unsubscribe:
 https://mail.python.org/mailman/options/python-dev/dholth%40gmail.com

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 3

[Daniel Holth]

On Thu, Mar 27, 2014 at 2:53 PM, Guido van Rossum gu...@python.org wrote:
 So what's the use case for Python 2/3 compatible code? IMO the main use case
 for the PEP is simply to be able to construct bytes from a combination of a
 template and some input that may include further bytes and numbers. E.g. in
 asyncio when you write an HTTP client or server you have to construct bytes
 to write to the socket, and I'd be happy if I could write b'HTTP/1.0 %d
 %b\r\n' % (status, message) rather than having to use
 str(status).encode('ascii') and concatenation or join().

It seems to be notoriously difficult to understand or explain why
Unicode can still be very hard in Python 3 or in code that is in the
middle of being ported or has to run in both interpreters. As far as I
can tell part of it is when a symbol has type(str or bytes) depending
(declared as if we had a static type system with union types); some of
it is because incorrect mixing can happen without an exception, only
to be discovered later and far away in space and time from the error
(worse of all in a serialized file), and part of it is all of the not
easily checkable types a particular Unicode object has depending on
whether it contains surrogates or codes  n. Sometimes you might
simply disagree about whether an API should be returning bytes or
Unicode in mildly ambiguous cases like base64 encoding. Sometimes
Unicode is just intrinsically complicated.

For me this PEP holds the promise of being able to do work in the
bytes domain, with no accidental mixing ever, when I *really* want
bytes. For 2+3 I would get exceptions sometimes in Python 2 and
exceptions all the time in Python 3 for mistakes. I hope this is less
error prone in strict domains than for example ustring
processing.encode('latin1'). And I hope that there is very little
type(str or int) in HTTP for example or other legitimate bytes
domains but I don't know; I suspect that if you have a lot of problems
with bytes' %s then it's a clue you should use (u%s %
(argument)).encode() instead.

sprintf()'s version of %s just takes a char* and puts it in without
doing any type conversion of course. IANACL (I am not a C lawyer).
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 3

[Daniel Holth]

I love it.

On Tue, Mar 25, 2014 at 6:37 PM, Ethan Furman et...@stoneleaf.us wrote:
 Okay, I included that last round of comments (from late February).

 Barring typos, this should be the final version.

 Final comments?

 -
 PEP: 461
 Title: Adding % formatting to bytes and bytearray
 Version: $Revision$
 Last-Modified: $Date$
 Author: Ethan Furman et...@stoneleaf.us
 Status: Draft
 Type: Standards Track
 Content-Type: text/x-rst
 Created: 2014-01-13
 Python-Version: 3.5
 Post-History: 2014-01-14, 2014-01-15, 2014-01-17, 2014-02-22, 2014-03-25
 Resolution:


 Abstract
 

 This PEP proposes adding % formatting operations similar to Python 2's
 ``str``
 type to ``bytes`` and ``bytearray`` [1]_ [2]_.


 Rationale
 =

 While interpolation is usually thought of as a string operation, there are
 cases where interpolation on ``bytes`` or ``bytearrays`` make sense, and the
 work needed to make up for this missing functionality detracts from the
 overall
 readability of the code.


 Motivation
 ==

 With Python 3 and the split between ``str`` and ``bytes``, one small but
 important area of programming became slightly more difficult, and much more
 painful -- wire format protocols [3]_.

 This area of programming is characterized by a mixture of binary data and
 ASCII compatible segments of text (aka ASCII-encoded text).  Bringing back a
 restricted %-interpolation for ``bytes`` and ``bytearray`` will aid both in
 writing new wire format code, and in porting Python 2 wire format code.

 Common use-cases include ``dbf`` and ``pdf`` file formats, ``email``
 formats, and ``FTP`` and ``HTTP`` communications, among many others.


 Proposed semantics for ``bytes`` and ``bytearray`` formatting
 =

 %-interpolation
 ---

 All the numeric formatting codes (such as ``%x``, ``%o``, ``%e``, ``%f``,
 ``%g``, etc.) will be supported, and will work as they do for str, including
 the padding, justification and other related modifiers.  The only difference
 will be that the results from these codes will be ASCII-encoded text, not
 unicode.  In other words, for any numeric formatting code `%x`::

b%x % val

 is equivalent to

(%x % val).encode(ascii)

 Examples::

 b'%4x' % 10
b'   a'

 b'%#4x' % 10
' 0xa'

 b'%04X' % 10
'000A'

 ``%c`` will insert a single byte, either from an ``int`` in range(256), or
 from
 a ``bytes`` argument of length 1, not from a ``str``.

 Examples::

  b'%c' % 48
 b'0'

  b'%c' % b'a'
 b'a'

 ``%s`` is included for two reasons:  1) `b` is already a format code for
 ``format`` numerics (binary), and 2) it will make 2/3 code easier as Python
 2.x
 code uses ``%s``; however, it is restricted in what it will accept::

   - input type supports ``Py_buffer`` [6]_?
 use it to collect the necessary bytes

   - input type is something else?
 use its ``__bytes__`` method [7]_ ; if there isn't one, raise a
 ``TypeError``

 In particular, ``%s`` will not accept numbers (use a numeric format code for
 that), nor ``str`` (encode it to ``bytes``).

 Examples::

  b'%s' % b'abc'
 b'abc'

  b'%s' % 'some string'.encode('utf8')
 b'some string'

  b'%s' % 3.14
 Traceback (most recent call last):
 ...
 TypeError: b'%s' does not accept numbers, use a numeric code instead

  b'%s' % 'hello world!'
 Traceback (most recent call last):
 ...
 TypeError: b'%s' does not accept 'str', it must be encoded to `bytes`


 ``%a`` will call ``ascii()`` on the interpolated value.  This is intended
 as a debugging aid, rather than something that should be used in production.
 Non-ASCII values will be encoded to either ``\xnn`` or ``\u``
 representation.  Use cases include developing a new protocol and writing
 landmarks into the stream; debugging data going into an existing protocol
 to see if the problem is the protocol itself or bad data; a fall-back for a
 serialization format; or even a rudimentary serialization format when
 defining ``__bytes__`` would not be appropriate [8].

 .. note::

 If a ``str`` is passed into ``%a``, it will be surrounded by quotes.


 Unsupported codes
 -

 ``%r`` (which calls ``__repr__`` and returns a ``str``) is not supported.


 Proposed variations
 ===

 It was suggested to let ``%s`` accept numbers, but since numbers have their
 own
 format codes this idea was discarded.

 It has been suggested to use ``%b`` for bytes as well as ``%s``.  This was
 rejected as not adding any value either in clarity or simplicity.

 It has been proposed to automatically use ``.encode('ascii','strict')`` for
 ``str`` arguments to ``%s``.

   - Rejected as this would lead to intermittent failures.  Better to have
 the
 operation always fail so the trouble-spot can be correctly fixed.

 It has been proposed to 

Re: [Python-Dev] Making proxy types easier to write and maintain

[Daniel Holth]

pypy's transparent proxy feature:
http://pypy.readthedocs.org/en/latest/objspace-proxies.html#transparent-proxies



On Thu, Mar 20, 2014 at 1:56 PM, Larry Hastings la...@hastings.org wrote:
 On 03/20/2014 12:49 AM, Nick Coghlan wrote:

 So long as Graham's willing to go along with it, he doesn't have to to
 be the one to write the PEP.


 PEP? Why does it need a PEP?  I didn't think it'd even be a new top-level
 library module, it'd be an addition / additions to functools.


 /arry

 ___
 Python-Dev mailing list
 Python-Dev@python.org
 https://mail.python.org/mailman/listinfo/python-dev
 Unsubscribe:
 https://mail.python.org/mailman/options/python-dev/dholth%40gmail.com

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 460 reboot

[Daniel Holth]

On Tue, Jan 14, 2014 at 1:52 PM, Guido van Rossum gu...@python.org wrote:
 On Tue, Jan 14, 2014 at 9:45 AM, Chris Barker chris.bar...@noaa.gov wrote:
 On Tue, Jan 14, 2014 at 9:29 AM, Yury Selivanov yselivanov...@gmail.com
 wrote:

  - Try str(), and do .encode(‘ascii’, ‘stcict’)” on the result.


 please no -- that's the source of a lot of pain in py2 now.

 having a failure as a result of the value, rather than the type, of an
 object just makes hard-to-test for bugs. Everything will be hunky dory for
 development and testing, then in deployment some idiot ( ;-) ) will pass in
 some non-ascii compatible string and you get  failure. And the person who
 gets the failure doesn't understand why, or they wouldn't have passed in
 non-ascii values in the first place...

 Ease of porting is nice, but let's not make it easy to port bug-prone code.

 Right. This is a big red flag to me as well.

 I think there is some inherent conflict between the extensible design
 of str.format() and the practical needs of people who are actually
 going to use formatting operations (either % or .format()) with bytes.

 The *practical* needs are mostly limited to supporting basic number
 formatting (decimal, hex, padding) and interpolation of anything that
 supports the buffer interface. It would also be nice if you didn't
 have to specify the type at all in the format string, i.e. {} should
 do the right thing for numbers and (all sorts of) bytes.

 But the way to arrive at this behavior without duplicating a whole lot
 of code seems to be to call the existing text-based __format__ API and
 convert the result to bytes -- for numbers this should be safe (their
 formatting produces just ASCII digits and a selected few other ASCII
 characters) but leads to an undesirable outcome for other types -- not
 just str but also e.g. lists or dicts containing str instances, since
 those call __repr__ on the contained items, and repr() may produce
 non-ASCII bytes.

 This is why my earlier proposal used ascii(), which is a nerfed(*)
 version of repr(). This does the right thing for numbers as well as
 for many other types (e.g. None, bool) and does something unpleasant
 for text strings that is perhaps better than the alternative.

 Which reminds me. Quite a few people have spoken out in favor of loud
 failures rather than silent wrong output. But I think that in the
 specific context of formatting output, there is a long and IMO good
 tradition of producing (slightly) wrong output in favor of more strict
 behavior. Consider for example what to do when a number doesn't fit in
 the given width. Would you rather raise an exception, truncate the
 value, or mess up the formatting? All languages newer than Fortran
 that I've used have chosen the latter, and I still agree it's a good
 idea. Similar with infinities, NaN, or None. (Yes, it's embarrassing
 to have a website displaying 'null'. But isn't a 500 even *more*
 embarrassing?)

 This doesn't mean I'm insensitive to the argument in favor of loud and
 early failure. It's just that I can see both sides of the coin, and
 I'm still deciding which argument is more important.

 (*) Gamer slang for a weapon made less dangerous. :-)

I think loud and early failure is important for porting while you
might still be trying to pound out the previously blurry encode/decode
boundaries. In this code str and bytes will be wrong everywhere. Some
APIs might return either str or bytes based on the input. Let it fail,
find the boundaries, and fix it until it does something useful without
failing. And it kindof depends on the context whether it is worse to
display weird ephemeral output or write the same weird output to long
term storage.

I'm not sure what to think about content-dependent failures on
protocols that are supposed to be ASCII-only-without-repr-noise.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 460 reboot

[Daniel Holth]

On Mon, Jan 13, 2014 at 12:42 PM, R. David Murray rdmur...@bitdance.com wrote:
 On Mon, 13 Jan 2014 12:41:18 +0100, Antoine Pitrou solip...@pitrou.net 
 wrote:
 On Sun, 12 Jan 2014 18:11:47 -0800
 Guido van Rossum gu...@python.org wrote:
  On Sun, Jan 12, 2014 at 5:27 PM, Ethan Furman et...@stoneleaf.us wrote:
   On 01/12/2014 04:47 PM, Guido van Rossum wrote:
   %s seems the trickiest: I think with a bytes argument it should just
   insert those bytes (and the padding modifiers should work too), and
   for other types it should probably work like %a, so that it works as
   expected for numeric values, and with a string argument it will return
   the ascii()-variant of its repr(). Examples:
  
   b'%s' % 42 == b'42'
   b'%s' % 'x' == b'x' (i.e. the three-byte string containing an 'x'
   enclosed in single quotes)
  
   I'm not sure about the quotes.  Would anyone ever actually want those in 
   the
   byte stream?
 
  Perhaps not, but it's a hint that you should probably think about an
  encoding. It's symmetric with how '%s' % b'x' returns b'x'. Think of
  it as payback time. :-)

 What is the use case for embedding a quoted ASCII-encoded representation
 in a byte stream?

 There is no use case in the sense you are asking, just like there is no
 real use case for '%s' % b'x' producing b'x'.  But the real use case
 is exactly the same: to let you know your code is screwed up without
 actually blowing up with a encoding Exception.

 For the record, I like Guido's logic and proposal.  I don't understand
 Nick's objection, since I don't see the difference between the situation
 here where a string gets interpolated into bytes as 'xxx' and the
 corresponding situation where bytes gets interpolated into a string
 as b'xxx'.  Why struggle to keep bytes interpolation pure if string
 interpolation isn't?

 Guido's proposal makes the language more symmetric, and thus more
 consistent and less surprising.  Exactly the hallmarks of Python's design
 sense, IMO.  (Big surprise, right? :)

 Of course, this point of view *is* based on the idea that when you are
 doing interpolation using %/.format, you are in fact primarily concerned
 with ASCII compatible byte streams.  This is a Practicality sort of
 argument.  It is, after all, by far the most common use case when
 doing interpolation[*].

 If you wanted to do a purist version of this symmetry, you'd have bytes(x)
 calling __bytes__ if it was defined and falling back to calling a
 __brepr__ otherwise.

 But what would __brepr__ implement?  The variety of format codes in
 the struct module argues that there is no one obvious binary
 repr for most types.  (Those that have one would implement __bytes__).
 And what would be the __brepr__ of an arbitrary 'object'?

 Faced with the impracticality of defining __brepr__ usefully in any pure
 bytes form, it seems sensible to admit that the most useful __brepr__
 is the ascii() encoding of the __repr__.  Which naturally produces 'xxx'
 as the __brepr__ of a string.

 This does cause things to get a little un-pretty when you are operating
 at the python prompt:

  b'%s' % object
 b'class \\\'object\\\''

 But then again that is most likely really not what you mean to do, so
 it becomes a big red flag...just like b'xxx' is a small red flag when
 you accidentally interpolate unencoded bytes into a string.

 --David

 PS: When I first read Guido's remark that the result of interpolating a
 string should be 'xxx', I went Wah?  I had to reason my way through to
 it as above, but to him it was just the natural answer.  Guido isn't
 always right, but this kind of automatic language design consistency
 is one reason he's the BDFL.

 [*] I still think that you mostly want to design your library so that
 you are handling the text parts as text and the bytes parts as bytes,
 and encoding/gluing them as appropriate at the IO boundary.  But if Guido
 says his real code would benefit by being able to interpolate ASCII into
 bytes at certain points, I'll believe him.

elided rant/

If you think corrupted data is easier or more pleasant to track down
than encoding exceptions then I think you are strange. It makes
porting really difficult while you are still trying to figure out
where the bytes/str boundaries are. I am now deeply suspicious of all
% formatting.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 460 reboot

[Daniel Holth]

I see it now. bfoo%sbar % b'baz' should also expand to bfoob'foo'bar

Instead of %b could %j mean I should have used + or join() here
but was too lazy and work on str too?

On Mon, Jan 13, 2014 at 2:51 PM, Terry Reedy tjre...@udel.edu wrote:
 On 1/13/2014 1:40 PM, Brett Cannon wrote:

  So bytes formatting really needn't (and shouldn't, IMO) mirror str
  formatting.


 This was my presumption in writing byteformat().


 I think one of the things about Guido's proposal that bugs me is that it
 breaks the mental model of the .format() method from str in terms of how
 the mini-language works. For str.format() you have the conversion and
 the format spec (e.g. {!r} and {:d}, respectively). You apply the
 conversion by calling the appropriate built-in, e.g. 'r' calls repr().
 The format spec semantically gets passed with the object to format()
 which calls the object's __format__() method: ``format(number, 'd')``.

 Now Guido's suggestion has two parts that affect the mini-language for
 .format(). One is that for bytes.format() the default conversion is
 bytes() instead of str(), which is fine (probably want to add 'b' as a
 conversion value as well to be consistent). But the other bit is that
 the format spec goes from semantically meaning ``format(thing,
 format_spec)`` to ``format(thing, format_spec).encode('ascii',
 'strict')`` for at least numbers. That implicitness bugs me as I have
 always thought of format specs just leading to a call to format(). I
 think I can live with it, though, as long as it is **consistently**
 applied across the board for bytes.format(); every use of a format spec
 leads to calling ``format(thing, format_spec).encode('ascii',
 'strict')`` no matter what type 'thing' would be and it is clearly
 documented that this is done to ease porting and handle the common case
 then I can live with it.


 This is how my byteformat function works, except that when no format_spec is
 given, byte and bytearrary objects are left unchanged rather than being
 decoded and encoded again.


 This even gives people in-place ASCII encoding for strings by always
 using '{:s}' with text which they can do when they port their code to
 run under both Python 2 and 3. So you should be able to do
 ``b'Content-Type: {:s}'.format('image/jpeg')`` and have it give ASCII.
 If you want more explicit encoding to latin-1 then you need to do it
 explicitly and not rely on the mini-language to do tricks for you.

 IOW I want to treat the format mini-language as a language and thus not
 have any special-casing or massive shifts in meaning between
 str.format() and bytes.format() so my mental model doesn't have to
 contort based on whether it's str or bytes. My preference is not have
 any, but if Guido is going say PBP here then I want absolute consistency
 across the board in how bytes.format() tweaks things.

 As for %s for the % operator calling ascii(), I think that will be a
 porting nightmare of finding out why your bytes suddenly stopped being
 formatted properly and then having to crawl through all of your code for
 that one use of %s which is getting bytes in. By raising a TypeError you
 will very easily detect where your screw-up occurred thanks to the
 traceback; do so otherwise feels too much like implicit type conversion
 and ask any JavaScript developer how that can be a bad thing.


 I personally would not add 'bytes % whatever'.

 --
 Terry Jan Reedy


 ___
 Python-Dev mailing list
 Python-Dev@python.org
 https://mail.python.org/mailman/listinfo/python-dev
 Unsubscribe:
 https://mail.python.org/mailman/options/python-dev/dholth%40gmail.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 460 reboot

[Daniel Holth]

On Mon, Jan 13, 2014 at 3:11 PM, Yury Selivanov yselivanov...@gmail.com wrote:
 On January 13, 2014 at 3:08:43 PM, Daniel Holth (dho...@gmail.com) wrote:

 I see it now. bfoo%sbar % b'baz' should also expand to bfoob'foo'bar

 Instead of %b could %j mean I should have used + or join()
 here
 but was too lazy and work on str too?

 Isn’t this just error prone? Since it’s a new format character, many,
 probably, would write %s by mistake. And, besides, there was no %j
 in python2.

Merely a flesh wound.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 460 reboot

[Daniel Holth]

On Mon, Jan 13, 2014 at 4:59 PM, Guido van Rossum gu...@python.org wrote:
 On Mon, Jan 13, 2014 at 1:29 PM, Glenn Linderman v+pyt...@g.nevcal.com 
 wrote:
 On 1/13/2014 12:09 PM, Guido van Rossum wrote:

 Yeah, the %s behavior with a string argument was a messy attempt at
 compromise. I was hoping to mimick a common use of %s in Python 2,
 where it can be used with either an 8-bit string or a number as
 argument, acting like %b in the former case and like %d in the latter
 case. Not having %s at all in Python 3 means that porting requires
 more thinking (== more opportunity for mistakes when you're converting
 in bulk) and there's no easy way to write code that works in Python 2
 and 3.

 If we have %b for strictly interpolating bytes, I'm fine with adding
 %a for calling ascii() on the argument and then interpolating the
 result after ASCII-encoding it.

 If somehow (unlikely though it seems) we end up keeping %s (e.g.
 strictly to ease porting), we could also keep %r as an alias for %a.


 %s for strictly interpolating bytes eases porting. Sad name, but good for
 compatibility. When the blowup happens, due to having a str type passed, the
 porter adds the appropriate .encode(...) to the parameter, so it doesn't
 blow up on Py 3, and it'll be OK for Py 2 as well, will it not?

 Lots of code uses %s with numbers too, and probably the occasional
 None or list (relying on the Python 2 near-guarantee that most
 objects' str() is their repr() and that repr() nearly guarantees to
 return only ASCII).

 E.g. I'm sure you can find live code doing something like

 headers.append('Content-Length: %s\r\n' % len(body))

But if the alternative is spurious quotes then the choice is clear...
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 460 reboot

[Daniel Holth]

On Sun, Jan 12, 2014 at 8:27 PM, Ethan Furman et...@stoneleaf.us wrote:
 On 01/12/2014 04:47 PM, Guido van Rossum wrote:


 %s seems the trickiest: I think with a bytes argument it should just
 insert those bytes (and the padding modifiers should work too), and
 for other types it should probably work like %a, so that it works as
 expected for numeric values, and with a string argument it will return
 the ascii()-variant of its repr(). Examples:

 b'%s' % 42 == b'42'
 b'%s' % 'x' == b'x' (i.e. the three-byte string containing an 'x'
 enclosed in single quotes)


 I'm not sure about the quotes.  Would anyone ever actually want those in the
 byte stream?

 --
 ~Ethan~

Is there a formatting character that means anything except a unicode
string to prevent accidentally interpolating a Unicode string into a
bytes string without [a sane] encoding?
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 460 reboot

[Daniel Holth]

On Sun, Jan 12, 2014 at 9:18 PM, Guido van Rossum gu...@python.org wrote:
 On Sun, Jan 12, 2014 at 6:07 PM, Daniel Holth dho...@gmail.com wrote:
 Is there a formatting character that means anything except a unicode
 string to prevent accidentally interpolating a Unicode string into a
 bytes string without [a sane] encoding?

 No, and we shouldn't introduce one. An operation should either work
 for no type, one type, a few specific types, or all types. Something
 that works for all but one type will *appear* to work for all types to
 a casually experimenting user and may pass extensive unittests,
 leaving a bomb that can detonate when you least expect it.

That pretty much describes how I feel about str(bytes). I would accept
only a bytes or only a string as consolation formatting characters
:-)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] RFC: PEP 460: Add bytes % args and bytes.format(args) to Python 3.5

[Daniel Holth]

On Sat, Jan 11, 2014 at 4:28 PM, Terry Reedy tjre...@udel.edu wrote:
 On 1/11/2014 1:44 PM, Stephen J. Turnbull wrote:

 We already *have* a type in Python 3.3 that provides text
 manipulations on arrays of 8-bit objects: str (per PEP 393).

BTW: I don't know why so many people keep asking for use cases.
Isn't it obvious that text data without known (but ASCII compatible)
encoding or multiple different encodings in a single data chunk
is part of life ?

 Isn't it equally obvious that if you create or read all such ASCII-
 compatible chunks as (encoding='ascii', errors='surrogateescape') that
 you *don't need* string APIs for bytes?

 Why do these text chunks need to be bytes in the first place?
 That's why we ask for use cases.  AFAICS, reading and writing ASCII-
 compatible text data as 'latin1' is just as fast as bytes I/O.  So
 it's not I/O efficiency, and (since in this model we don't do any
 en/decoding on bytes/str), it's not redundant en/decoding of bytes to
 str and back.


 The problem with some criticisms of using 'unicode in Python 3' is that
 there really is no such thing. Unicode in 3.0 to 3.2 used the old internal
 model inherited from 2.x. Unicode in 3.3+ uses a different internal model
 that is a game changer with respect to certain issues of space and time
 efficiency (and cross-platform correctness and portability). So at least
 some the valid criticisms based on the old model are out of date and no
 longer valid.

-1 on adding more surrogateesapes by default. It's a pain to track
down where the encoding errors came from.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python3 complexity (was RFC: PEP 460: Add bytes...)

[Daniel Holth]

So the customer you're looking for is the person who cares a lot about
encodings, knows how to do Unicode correctly, and has noticed that
certain valid cases not limited to imperialist simpletons (dealing
with specific common things invented before 1996, dealing with mixed
encodings, doing what Nick describes as ASCII compatible binary
protocols) are *more complicated to do correctly* in Python 3 because
Python 3 undeniably has more complicated though probably better
*Unicode* support. N.b. WSGI, email, url parsing etc. The same person
loves Python, all the other Python 3 features, and probably you
personally, but mostly does not write programs in the domains that
Python 3 makes easier. They emphatically do not want the Python 2
model especially not implicit coercion. They only want additional
tools for text or string processing in Python 3.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] RFC: PEP 460: Add bytes % args and bytes.format(args) to Python 3.5

[Daniel Holth]

On Tue, Jan 7, 2014 at 10:36 AM, Stephen J. Turnbull step...@xemacs.org wrote:
 Daniel Holth writes:

   Isn't it true that if you have bytes  127 or surrogate escapes then
   encoding to latin1 is no longer as fast as memcpy?

 Be careful.  As phrased, the question makes no sense.  You don't have
 bytes when you are encoding, you have characters.

 If you mean what happens when my str contains characters in the range
 128-255?, the answer is encoding a str in 8-bit representation to
 latin1 is effectively memcpy.  If you read in latin1, it's memcpy all
 the way (unless you combine it with a non-latin1 string, in which case
 you're in the cases below).

 If you mean what happens when my str contains characters in the range
 255, you have to truncate 16-bit units to 8 bit units; no memcpy.

 Surrogates require = 16 bits; no memcpy.

That is neat.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] RFC: PEP 460: Add bytes % args and bytes.format(args) to Python 3.5

[Daniel Holth]

+1

I have always been delighted that it is possible to manipulate binary
data in Python using string operations. It's not just immoral
non-Unicode text processing. A poor man's ASN.1 generator is an
example of a very non-text thing that might be convenient to write
with a few %s fill-in-the-blanks.

Isn't it true that if you have bytes  127 or surrogate escapes then
encoding to latin1 is no longer as fast as memcpy?

On Tue, Jan 7, 2014 at 8:22 AM, Serhiy Storchaka storch...@gmail.com wrote:
 Most popular formatting codes in Mercurial sources:

2519 %s
 493 %d
 102 %r
  48 %Y
  47 %M
  41 %H
  39 %S
  38 %m
  33 %i
  29 %b
  23 %ld
  19 %ln
  12 %.3f
  10 %a
  10 %.1f
   9 %(val)r
   9 %p
   9 %.2f
   8 %I
   6 %n
   5 %(val)s
   5 %.0f
   5 %02x
   4 %f
   4 %c
   4 %12s
   3 %(user)s
   3 %(id)s
   3 %h
   3 %(bzdir)s
   3 %0.2f
   3 %02d



 ___
 Python-Dev mailing list
 Python-Dev@python.org
 https://mail.python.org/mailman/listinfo/python-dev
 Unsubscribe:
 https://mail.python.org/mailman/options/python-dev/dholth%40gmail.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Backward-incompatible change to random.randrange in 2.7.6

[Daniel Holth]

But who could forget njzrs' wasp UAV software line 107, using
int=float? 
https://github.com/nzjrs/wasp/blob/master/sw/groundstation/wasp/__init__.py#L107

On Wed, Dec 18, 2013 at 10:49 AM, Guido van Rossum gu...@python.org wrote:
 BTW, I bet a lavish dinner at PyCon that it is *only* Zope/ZODB that
 does this. In the time we added this bogus dependency on undocumented
 parameters, the PythonLabs team was at Zope and we didn't always get
 our boundaries straight.

 On Wed, Dec 18, 2013 at 2:51 AM, Nick Coghlan ncogh...@gmail.com wrote:
 On 18 December 2013 20:17, Chris Angelico ros...@gmail.com wrote:
 On Wed, Dec 18, 2013 at 8:43 PM, Serhiy Storchaka storch...@gmail.com 
 wrote:
 18.12.13 04:40, Benjamin Peterson написав(ла):

 Mostly yes, but at least you could tell people to upgrade straight to
 2.7.7 and skip 2.7.6.


 It'll make the people to postpone the upgrade to 2.7.6 (which fixes many
 security bugs) until 2.7.7 release,  instead of correcting their
 morally-broken programs.

 If this is considered enough breakage to be a problem, would it be
 possible to issue a 2.7.6.1 or 2.7.6+fixed release that's identical to
 2.7.6 but with this change reverted? It'd be a minor mess, but then
 people would still get those security fixes, and it means not breaking
 stuff in a point release.

 If we revert it, it means we do 2.7.7 ASAP so that *everyone* can just
 skip straight to 2.7.7.

 That kind of user visible change shouldn't have been made in a point
 release, regardless of what the docs said. It just isn't worth the
 risk of breaking the code of people that are relying on what's
 possible rather than what the docs say.

 Cheers,
 Nick.


 --
 Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
 ___
 Python-Dev mailing list
 Python-Dev@python.org
 https://mail.python.org/mailman/listinfo/python-dev
 Unsubscribe: 
 https://mail.python.org/mailman/options/python-dev/guido%40python.org



 --
 --Guido van Rossum (python.org/~guido)
 ___
 Python-Dev mailing list
 Python-Dev@python.org
 https://mail.python.org/mailman/listinfo/python-dev
 Unsubscribe: 
 https://mail.python.org/mailman/options/python-dev/dholth%40gmail.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Backward-incompatible change to random.randrange in 2.7.6

[Daniel Holth]

On Dec 18, 2013 11:54 AM, Tim Peters tim.pet...@gmail.com wrote:

 [Daniel Holth]
  But who could forget njzrs' wasp UAV software line 107, using
  int=float?
 https://github.com/nzjrs/wasp/blob/master/sw/groundstation/wasp/__init__.py#L107

 I could forget it ;-)  The remarkable thing about the two instances of:

  random.randrange(0.0,1.0, int=float)

 in that file is that they're obscure and inefficient ways to spell:

  random.random()


You can keep your premature optimizations thank you :-)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 0404 and VS 2010

[Daniel Holth]

On Thu, Nov 21, 2013 at 4:12 PM, Paul Moore p.f.mo...@gmail.com wrote:
 On 21 November 2013 21:02, Greg Ewing greg.ew...@canterbury.ac.nz wrote:
 Is that much different from package authors having to
 release binaries for different versions of Python,
 if they want to support older versions?

 Having multiple binaries for the same x.y version
 is different from what's been done before, but it
 seems to me an unavoidable consequence of supporting
 one x.y version for longer than usual.

 None of the currently available binary distribution formats
 distinguish Windows binaries by anything other than minor version. For
 wheels (and I think eggs), this is a showstopper as the name is
 essential metadata (compatibility tags) for the other formats (wininst
 and msi) the name is merely informational - packagers could rename,
 but (a) they will forget, and (b) the users won't know if they have or
 not.

 Before we can cleanly support multiple ABIs for a single minor version
 on Windows, we need to have a resolution of this dilemma (which may be
 nothing more than only binaries for the python.org builds are allowed
 on PyPI...)

 Paul

As for wheel it actually does support an ABI tag that is separate from
the Python version and the architecture. It's the second one
pyversion-abi-architecture as in py27-none-any or py27-cp27-linux_i386
(spelling?). The build tool and installer would have to be modified to
be aware of any newly defined ABI tags.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] flaky tests caused by repr() sort order

[Daniel Holth]

+1 on unsorted repr(). It makes it obvious that the collection is not sorted.

On Thu, Nov 21, 2013 at 1:10 PM, Christian Heimes christ...@python.org wrote:
 Am 21.11.2013 18:57, schrieb Tim Peters:
 Best to change the failing tests.  For example, _they_ can sort the
 dict keys if they rely on a fixed order.  Sorting in general is a
 dubious idea because it can be a major expense with no real benefit
 for most uses.

 I don't consider repr() as a performance critical function. It's mostly
 used for debugging.

 ___
 Python-Dev mailing list
 Python-Dev@python.org
 https://mail.python.org/mailman/listinfo/python-dev
 Unsubscribe: 
 https://mail.python.org/mailman/options/python-dev/dholth%40gmail.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] non-US zip archives support in zipfile.py

[Daniel Holth]

FYI zipfile does do UTF-8
(http://hg.python.org/cpython/file/d7ebe03fa752/Lib/zipfile.py#l368).
Only the non-Unicode encodings might need some help. I like that the
patch is only concerned with decoding. Is it necessary to support
writing non-UTF8 encodings?

On Tue, Oct 15, 2013 at 10:20 AM, Victor Stinner
victor.stin...@gmail.com wrote:
 Both issues are related and should be fixed in the *same* commit.
 Python should provide an option to encode explicitly filenames to
 UTF-8. Or maybe better: encode to UTF-8 by default, with an option to
 encode to cp437, and another to encode to an arbitrary encoding.

 Victor

 2013/10/15 Alexander Belopolsky alexander.belopol...@gmail.com:
 On Mon, Oct 14, 2013 at 6:13 PM, Victor Stinner
 victor.stin...@gmail.com wrote:
 I opened an issue proposing exactly the same change, but I didn't provide a
 patch.

 I found this:

 http://bugs.python.org/issue10614

 but it has (seemingly incorrect) patch.

 Also related:

 http://bugs.python.org/issue10972

 Victor - is any of these what you had in mind?
 ___
 Python-Dev mailing list
 Python-Dev@python.org
 https://mail.python.org/mailman/listinfo/python-dev
 Unsubscribe: 
 https://mail.python.org/mailman/options/python-dev/dholth%40gmail.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] non-US zip archives support in zipfile.py

[Daniel Holth]

On Tue, Oct 15, 2013 at 8:02 AM, Steven D'Aprano st...@pearwood.info wrote:
 On Tue, Oct 15, 2013 at 10:20:03AM +0200, Victor Stinner wrote:
 Both issues are related and should be fixed in the *same* commit.
 Python should provide an option to encode explicitly filenames to
 UTF-8. Or maybe better: encode to UTF-8 by default, with an option to
 encode to cp437, and another to encode to an arbitrary encoding.

 Why single out cp437 for special treatment? Surely it's enough to use
 UTF-8 by default, or specify an arbitrary encoding.

It is part of the ZIP specification. CP437 or UTF-8 are the two
official choices, but other encodings happen on Russian, Japanese
systems.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] On the dangers of giving developers the best resources

[Daniel Holth]

Sounds like you are suggesting we get a raspberry pi. All sorts of dumb
waste shows up when you run code on those.
El oct 8, 2013 4:46 p.m., Guido van Rossum gu...@python.org escribió:

 Let's agree to disagree then. I see your methodology used all around me
 with often problematic results. Maybe devs should have two machines -- one
 monster that is *only* usable to develop fast, one small that where they
 run their own apps (email, web browser etc.).


 On Tue, Oct 8, 2013 at 1:30 PM, Tim Delaney 
 timothy.c.dela...@gmail.comwrote:

 On 9 October 2013 03:35, Guido van Rossum gu...@python.org wrote:

 On Tue, Oct 8, 2013 at 8:33 AM, R. David Murray 
 rdmur...@bitdance.comwrote:

 PS: I have always thought it sad that the ready availability of memory,
 CPU speed, and disk space tends to result in lazy programs.  I
 understand
 there is an effort/value tradeoff, and I make those tradeoffs myself
 all the time...but it still makes me sad.  Then, again, in my early
 programming days I spent a fair amount of time writing and using Forth,
 and that probably colors my worldview. :)


 I never used or cared for Forth, but I have the same worldview. I
 remember getting it from David Rosenthal, an early Sun reviewer. He stated
 that engineers should be given the smallest desktop computer available, not
 the largest, so they would feel their users' pain and optimize
 appropriately. Sadly software vendors who are also hardware vendors have
 incentives going in the opposite direction -- they want users to feel the
 pain so they'll buy a new device.


 I look at it a different way. Developers should be given powerful
 machines to speed up the development cycle (especially important when
 prototyping and in the code/run unit test cycle), but everything should be
 tested on the smallest machine available.

 It's also a good idea for each developer to have a resource-constrained
 machine for developer testing/profiling/etc. Virtual machines work quite
 well for this - you can modify the resource constraints (CPU, memory, etc)
 to simulate different scenarios.

 I find that this tends to better promote the methodology of make it
 right, then make it fast (small, etc), which I subscribe to. Optimising
 too early leads to all your code being complicated, rather than just the
 bits that need it.

 Tim Delaney




 --
 --Guido van Rossum (python.org/~guido)

 ___
 Python-Dev mailing list
 Python-Dev@python.org
 https://mail.python.org/mailman/listinfo/python-dev
 Unsubscribe:
 https://mail.python.org/mailman/options/python-dev/dholth%40gmail.com


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 453 (pip bootstrapping) ready for pronouncement?

[Daniel Holth]

On Fri, Sep 27, 2013 at 12:15 PM, R. David Murray rdmur...@bitdance.com wrote:
 On Fri, 27 Sep 2013 15:26:41 +0100, Paul Moore p.f.mo...@gmail.com wrote:
 On 27 September 2013 15:08, Stephen J. Turnbull step...@xemacs.org wrote:
  New users on Windows and Mac OS X. I've heard many more complaints
from folks running tutorials about the pip bootstrapping process than
I ever have from the community at large about the GIL :P
 
  I bet those users are *not* running third-party distros, but rather
  are sitting in front of pretty close to plain vanilla factory installs
  of the OS, no?  And new users on Mac OS X already have old installs
  of Python, no?

 Windows users who don't use a third-party distro like Enthought,
 generally download the python.org installer. At the moment, that
 doesn't give them a pip command. So if they want to install any
 third party package, they have to start by installing pip. The
 instructions for that are reasonably clear, but non-trivial, largely
 because tools like curl are not commonly available on Windows, and by
 default running a Python script may not do what you expect.

 Rather than try to fix these problems (which are *hard*) the intent is
 to have the pip command installed by the python.org installer.
 [...]
 I can't speak for Linux distros or OSX users, but for Windows I do
 believe that this is a significant improvement, and worth the (IMO,
 negligible) risk involved in adding this to a maintenance release.

 I'm not an OS X user, and probably most people on this list use macports
 or something similar, which essentially puts them in the same boat as
 the linux users...and there's a section in the PEP about that (that's
 where the message about installing pip if you run pip and the distro
 didn't include it with python is supposed to come from).

 For OS X users *not* using something like macports, I'm pretty sure they
 are going to be in a similar boat to the Windows users, with just a touch
 of added confusion coming from the fact that an older version of Python
 is already installed.  But the instructions they will find on the web
 for installing package X (once this change hits the field) will be to
 install the newest version of 2.7 (or 3) using the python.org installer,
 and then they will have the pip command and can go from there.

 --David

OS X and Linux change Python in ways that can be confusing to new and
experienced users, like not installing the stdlib source code by
default, or not installing the profiler, or simply by being out of
date. On these platforms attempting to use the system Python for
development can be a costly mistake instead of a convenience.

Users get a more consistent experience by starting with the installers
from python.org. Hopefully the tutorials will reflect the consistency
added by this PEP.

This PEP only gets pip, an operation which is a recurring
inconvenience for me even though I use Linux. It does not solve any of
the problems you may have after pip has been installed.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tweaking PEP 8 guidelines for use of leading underscores

[Daniel Holth]

https://pypi.python.org/pypi/apipkg provides a much more effective way
to denote API than an _

On Wed, Jul 17, 2013 at 12:20 PM, R. David Murray rdmur...@bitdance.com wrote:
 On Tue, 16 Jul 2013 17:46:34 -0400, Terry Reedy tjre...@udel.edu wrote:
 On 7/16/2013 9:39 AM, R. David Murray wrote:
  On Tue, 16 Jul 2013 23:19:21 +1000, Steven D'Aprano st...@pearwood.info 
  wrote:

  For example, pkgutil includes classes with single-underscore methods, 
  which I take as private. It also has a function simplegeneric, which is 
  undocumented and not listed in __all__. In in the absence of even a 
  comment saying Don't use this, I take it as an oversight, not policy 
  that simplegeneric is private.
 
  I think you'd be wrong about that, though.  simplegeneric should really be
  treated as private.  I'm speaking here not about the general principle of
  the thing, but about my understanding of simplegeneric's specific history.

 I think Steven (valid) point is Why not, then, say it is internal
 either in docs or name?-- which in this case would be in the docs.

 I don't think that's what he was saying; but yes, we should do that :)

 --David
 ___
 Python-Dev mailing list
 Python-Dev@python.org
 http://mail.python.org/mailman/listinfo/python-dev
 Unsubscribe: 
 http://mail.python.org/mailman/options/python-dev/dholth%40gmail.com
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] stat module in C -- what to do with stat.py?

[Daniel Holth]

cffi makes this kind of constant-grabbing very easy. However this is a
tiny module so no problem with having a C version.

On Thu, Jun 20, 2013 at 11:04 AM, Thomas Wouters tho...@python.org wrote:



 On Thu, Jun 20, 2013 at 8:01 AM, Brett Cannon br...@python.org wrote:




 On Thu, Jun 20, 2013 at 10:14 AM, Eric V. Smith e...@trueblade.com
 wrote:

 On 6/20/2013 9:40 AM, Nick Coghlan wrote:
  On 20 June 2013 23:29, Christian Heimes christ...@python.org wrote:
  Am 20.06.2013 15:21, schrieb Florent:
  we already have _pyio.py, we could have _pystat.py.
 
  Works for me.
 
  I suggest following the guidelines in PEP 399 for cross implementation
  compatibility of the standard library:
  http://www.python.org/dev/peps/pep-0399/#details
 
  1. Keep stat.py
  2. Make the C version _stat
  3. Add the following to the end of stat.py:
 
  try:
  from _stat import *
  except ImportError:
  pass
 
  Run the tests with and without the C module in the test suite (again,
  as per PEP 399).

 Agreed with the above.

 But isn't the real problem with this module in Python the fact that the
 constants might be wrong? I'm not sure what, if anything, we can do
 about that.


 There isn't anything we can do beyond at least trying to provide
 reasonable defaults when something better isn't available (which is what the
 stats module has done all this time). It might make testing difficult for
 the Python code when the C code has the right values, but I don't think it's
 necessarily worth tossing out the Python code as backup.


 If the .py file is going to be wrong or incomplete, why would we want to
 keep it -- or use it as fallback -- at all? If we're dead set on having a
 .py file instead of requiring it to be part of the interpreter (whichever
 that is, however it was built), it should be generated as part of the build
 process. Personally, I don't see the value in it; other implementations will
 need to do *something* special to use it anyway.

 --
 Thomas Wouters tho...@python.org

 Hi! I'm an email virus! Think twice before sending your email to help me
 spread!

 ___
 Python-Dev mailing list
 Python-Dev@python.org
 http://mail.python.org/mailman/listinfo/python-dev
 Unsubscribe:
 http://mail.python.org/mailman/options/python-dev/dholth%40gmail.com

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Bilingual scripts

[Daniel Holth]

On Tue, May 28, 2013 at 1:30 PM, Barry Warsaw ba...@python.org wrote:
 On May 25, 2013, at 03:12 AM, Chris McDonough wrote:

You probably already know this, but I'll mention it anyway.  This
probably matters a lot for nose and pyflakes, but I'd say that for tox
it should not, it basically just scripts execution of shell commands.
I'd think maybe in cases like tox (and others that are compatible with
both Python 2 and 3) the hashbang should just be set to
#!/usr/bin/python unconditionally.

 Unfortunately, not entirely so:

 https://bitbucket.org/hpk42/tox/issue/96/cant-have-a-python-3-setuppy

Maybe we could also think about modifying pyflakes so that it can
validate both 2 and 3 code (choosing one or the other based on a header
line in the validated files and defaulting to the version of Python
being run).  This is kind of the right thing anyway.

 Agreed.  Auto-detection may need to be accompanied by a command line option to
 override in some cases.  But I agree, that in general, it would be very nice
 if the script itself were actually bilingual.  (But then, see my previous
 comment about cross-interpreter dependencies.)

Nose is a bit of a special case.  I personally never run nosetests
directly, I always use setup.py nosetests, which makes it not matter.

 Which is morally equivalent to `$python -m nose`.

In general, I'd like to think that scripts that get installed to global
bindirs will execute utilities that are useful independent of the
version of Python being used to execute them.

 Agreed.  I'm trying to tease out some conventions we can recommend for when
 this can't be the case for whatever reason.

 -Barry

Wheel has no mechanism for renaming scripts (or any file) based on the
Python version used to install. Instead you would have to build
python-version-specific packages for each desired script name.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Bilingual scripts

[Daniel Holth]

On Tue, May 28, 2013 at 2:04 PM, Barry Warsaw ba...@python.org wrote:
 On May 28, 2013, at 01:57 PM, Daniel Holth wrote:

Wheel has no mechanism for renaming scripts (or any file) based on the
Python version used to install. Instead you would have to build
python-version-specific packages for each desired script name.

 Note that I'm not trying to borrow any implementation details from wheels,
 just the file naming conventions (compatibility tags) described in PEP 425.
 It would still be up to upstream package or distro tools to fiddle the
 installed file names.

I'm just saying that I prefer a setup.py without too many
Python-version-specific differences, since it would look pretty silly
to install a wheel of nose generated on Python 3.2 on Python 3.3 and
have the wrong version suffix on the scripts. I like the plainly named
scripts without version suffixes.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 4XX: pyzaa Improving Python ZIP Application Support

[Daniel Holth]

Everyone seems to like the first half of this simple PEP adding the
extensions. The 3-letter extension for windowed apps can be pzw
while the pyz extension for console apps stays the same.

The second half, the tool
https://bitbucket.org/dholth/pyzaa/src/tip/pyzaa.py?at=default is less
mature, but there's not a whole lot to do in a simple tool that may
serve more as an example: you can open any file with ZipFile in append
mode, even one that is not a zip file and just contains the #!python
shebang line.

Thanks,

Daniel
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 4XX: pyzaa Improving Python ZIP Application Support

[Daniel Holth]

As the PEP author I declare we can have 3-letter extensions. It is not
a big deal.

Daniel Holth

On Mon, May 6, 2013 at 4:30 PM, Richard Oudkerk shibt...@gmail.com wrote:
 So the bug would just cause .pyzw files to be opened with py instead of pyw?
 Won't this be harmless?

 I think the worst that would happen would be that you get a redundant
 console window if you are not already running powershell inside a console.

 --
 Richard


 ___
 Python-Dev mailing list
 Python-Dev@python.org
 http://mail.python.org/mailman/listinfo/python-dev
 Unsubscribe:
 http://mail.python.org/mailman/options/python-dev/dholth%40gmail.com
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Why can't I encode/decode base64 without importing a module?

[Daniel Holth]

On Thu, Apr 25, 2013 at 10:07 AM, Barry Warsaw ba...@python.org wrote:
 On Apr 25, 2013, at 03:34 PM, Lennart Regebro wrote:

In the case of JSON objects, they are intended for data exchange, and
hence in the end need to be byte strings.

 Except that they're not.

 http://bugs.python.org/issue10976

 -Barry
 ___
 Python-Dev mailing list
 Python-Dev@python.org
 http://mail.python.org/mailman/listinfo/python-dev
 Unsubscribe: 
 http://mail.python.org/mailman/options/python-dev/dholth%40gmail.com

What am I doing wrong in this JSON crypto signature verification
snippet that features many conversions between binary and text?

recipients = jwsjs[recipients]
encoded_payload = binary(jwsjs[payload])
headers = []
for recipient in recipients:
h = binary(recipient[header])
s = binary(recipient[signature])
header = json.loads(native(urlsafe_b64decode(h)))
vk = urlsafe_b64decode(binary(header[jwk][vk]))
secured_input = b..join((h, encoded_payload))
sig = urlsafe_b64decode(s)
sig_msg = sig+secured_input
verified_input = native(ed25519ll.crypto_sign_open(sig_msg, vk))
verified_header, verified_payload = verified_input.split('.')
verified_header = binary(verified_header)
decoded_header = native(urlsafe_b64decode(verified_header))
headers.append(json.loads(decoded_header))

verified_payload = binary(verified_payload)

# only return header, payload that have passed through the crypto library.
payload = json.loads(native(urlsafe_b64decode(verified_payload)))

return headers, payload
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 4XX: pyzaa Improving Python ZIP Application Support

[Daniel Holth]

On Wed, Apr 3, 2013 at 1:26 AM, Stefan Behnel stefan...@behnel.de wrote:
 Brett Cannon, 02.04.2013 19:28:
 On Tue, Apr 2, 2013 at 1:20 PM, Steve Dower steve.do...@microsoft.comwrote:

 python -m pyzaa pack [-o path/name] [-m module.submodule:callable] [-c]
 [-w] [-p interpreter] directory:

ZIP the contents of directory as directory.pyz or [-w]
 directory.pyzw. Adds the executable flag to the archive.

 ...

-p interpreter include #!interpreter as the first line of the archive

 What happens when -p is omitted? I'd hope it would add the interpreter
 used to create the zip (or at least the major version), but that may not be
 ideal for some reason that I haven't thought of yet.

 Question is whether ``/usr/bin/python3.3`` is better or ``/usr/bin/env
 python3.3``. I vote for the latter since it gets you the right thing
 without having to care about whether the interpreter moved or is being
 hidden by a user-installed interpreter.

 It can't work properly from within a virtualenv when you write
 /usr/bin/python, so using /usr/bin/env instead is actually required.

 Stefan

Pushed as Draft PEP 441, tooling prototyped (with less than awesome
CLI) at https://bitbucket.org/dholth/pyzaa or
https://crate.io/packages/pyzaa

Thanks,

Daniel Holth
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] What code in Python creates .exe launchers for 'entry_points' in 'setup.py'?

[Daniel Holth]

The new distlib also provides that feature
On Apr 1, 2013 4:58 PM, Éric Araujo e...@netwok.org wrote:

 Hello,

 Le 01/04/2013 16:51, Grigory Petrov a écrit :
  But what python code is responsible for creation of this bootstrapper
  executables? I have searched python 2.7.3 source code for some time, but
  can't find a place.

 Setuptools is not in the standard library, so you would need to search
 the setuptools or distribute codebases.

 Regards
 ___
 Python-Dev mailing list
 Python-Dev@python.org
 http://mail.python.org/mailman/listinfo/python-dev
 Unsubscribe:
 http://mail.python.org/mailman/options/python-dev/dholth%40gmail.com

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] PEP 4XX: pyzaa Improving Python ZIP Application Support

[Daniel Holth]

https://docs.google.com/document/d/1MKXgPzhWD5wIUpoSQX7dxmqgTZVO6l9iZZis8dnri78/edit?usp=sharing

PEP: 4XX

Title: Improving Python ZIP Application Support

Author: Daniel Holth dho...@gmail.com

Status: Draft

Type: Standards Track

Python-Version: 3.4

Created: 30 March 2013

Post-History: 30 March 2013, 1 April 2013


Improving Python ZIP Application Support


Python has had the ability to execute directories or ZIP-format
archives as scripts since version 2.6. When invoked with a zip file or
directory as its first argument the interpreter adds that directory to
sys.path and executes the __main__ module. These archives provide a
great way to publish software that needs to be distributed as a single
file script but is complex enough to need to be written as a
collection of modules.


This feature is not as popular as it should be, mainly because no
one’s heard of it because it wasn’t promoted as part of Python 2.6,
but also because Windows users don’t have a file extension (other than
.py) to associate with the launcher.


This PEP proposes to fix these problems by re-publicising the feature,
defining the .pyz and .pyzw extensions as “Python ZIP Applications”
and “Windowed Python ZIP Applications”, and providing some simple
tooling to manage the format.

A New Python ZIP Application Extension


The Python 3.4 installer will associate .pyz and .pyzw “Python ZIP
Applications” with the platform launcher so they can be executed. A
.pyz archive is a console application and a .pyzw archive is a
windowed application, indicating whether the console should appear
when running the app.


Why not use .zip or .py? Users expect a .zip file would be opened with
an archive tool, and users expect .py to be opened with a text editor.
Both would be confusing for this use case.


For UNIX users, .pyz applications should be prefixed with a #! line
pointing to the correct Python interpreter and an optional
explanation.


#!/usr/bin/env python3

# This is a Python application stored in a ZIP archive.

(binary contents of archive)


As background, ZIP archives are defined with a footer containing
relative offsets from the end of the file. They remain valid when
concatenated to the end of any other file. This feature is completely
standard and is how self-extracting ZIP archives and the bdist_wininst
installer format work.

Minimal Tooling: The pyzaa Module

This PEP also proposes including a simple application for working with
these archives: The Python Zip Application Archiver “pyzaa” (rhymes
with “huzzah” or “pizza”). “pyzaa” can archive or extract these files,
compile bytecode, and can write the __main__ module if it is not
present.

Usage

python -m pyzaa (pack | compile)


python -m pyzaa pack [-o path/name] [-m module.submodule:callable]
[-c] [-w] [-p interpreter] directory:

ZIP the contents of directory as directory.pyz or [-w]
directory.pyzw. Adds the executable flag to the archive.

-c compile .pyc files and add them to the archive

-p interpreter include #!interpreter as the first line of the archive

-o path/name archive is written to path/name.pyz[w] instead of
dirname. The extension is added if not specified.

-m module.submodule:callable __main__.py is written as “import
module.submodule; module.submodule.callable()”


pyzaa pack will warn if the directory contains C extensions or if
it doesn’t contain __main__.py.


python -m pyzaa compile arcname.pyz[w]

The Python files in arcname.pyz[w] are compiled and appended to
the ZIP file.


A standard ZIP utility or Python’s zipfile module can unpack the archives.

FAQ

Q. Isn’t pyzaa just a very thin wrapper over zipfile and compileall?

A. Yes.


Q. How does this compete with existing sdist/bdist formats?

A. There is some overlap, but .pyz files are especially interesting as
a way to distribute an installer. They may also prove useful as a way
to deliver applications when users shouldn’t be asked to perform
virtualenv + “pip install”.

References

[1] http://bugs.python.org/issue1739468 “Allow interpreter to execute
a zip file”

[2] http://bugs.python.org/issue17359 “Feature is not documented”

Copyright

This document has been placed into the public domain.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Accepting PEP 434, Idle Enhancement Exception

[Daniel Holth]

Yes, it would probably make more sense to split the editor and shell
processes as many Python IDEs do, with IDLE running in CPython and the
user's computation running in the chosen interpreter.

On Sat, Mar 30, 2013 at 8:35 AM, Maciej Fijalkowski fij...@gmail.com wrote:
 On Sat, Mar 30, 2013 at 5:26 AM, Antoine Pitrou solip...@pitrou.net wrote:
 On Sat, 30 Mar 2013 08:33:38 +0200
 Simon Cross hodgestar+python...@gmail.com wrote:

 Having a standalone version of IDLE might be really useful to
 alternative Python implementations.

 Why?

 I don't think it's worth discussing - tkinter does not work on any
 other implementation than CPython and it seems it won't work. It's a
 bit pity, but I guess if I felt really bad about it, I should just
 make it work.

 PS. are there idle projects in SoC? Maybe we should put a more
 pypy-friendly one there too?

 Cheers,
 fijal
 ___
 Python-Dev mailing list
 Python-Dev@python.org
 http://mail.python.org/mailman/listinfo/python-dev
 Unsubscribe: 
 http://mail.python.org/mailman/options/python-dev/dholth%40gmail.com
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Safely importing zip files with C extensions

[Daniel Holth]

Jim Fulton is right that weird failures are a characteristic of zipped
eggs, so one of the #1 requests for setuptools is how to prohibit
zipping from ever happening. This is an important reason why wheel is
billed as an installation format -- fewer users with pitchforks. It's
very cool that it works though. Debugging is slightly easier than it
was in the old days because pdb can now read the source code from the
zip.

An unzipped wheel as a directory with the same name as the wheel would
be a more reliable solution that might be interesting to work with. It
would work the same as egg unless you had important files in the
.data/ (currently mostly used for console scripts and include files)
directory. However it was always confusing to have more than one kind
(zipped, unzipped) of egg.

On Wed, Mar 27, 2013 at 4:41 PM, Vinay Sajip vinay_sa...@yahoo.co.uk wrote:
 Amaury Forgeot d'Arc amauryfa at gmail.com writes:


 Better: just put the wheel path to sys.path
 sys.path.append('/tmp/simplejson-3.1.2-cp27-none-linux_x86_64.whl')
 and let a sys.path_hook entry do the job.

 That's what the mount() actually does - adds the wheel to a registry that an
 import hook uses. You also need a place to check that the wheel being mounted
 is compatible with the Python doing the mounting - I'm not sure whether what
 the import hook should do if e.g. there is a compatibility problem with the
 wheel (e.g. is it clear that it should always raise an ImportError? Or ignore
 the wheel - seems wrong? Or do something else?)

 Regards,

 Vinay Sajip

 ___
 Python-Dev mailing list
 Python-Dev@python.org
 http://mail.python.org/mailman/listinfo/python-dev
 Unsubscribe: 
 http://mail.python.org/mailman/options/python-dev/dholth%40gmail.com
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] IDLE in the stdlib

[Daniel Holth]

I showed IDLE to my 6-year-old on the Raspberry Pi and I'm convinced
it is cool. Gave up on trying to (slowly) install bpython. We were
multiplying large numbers and counting to 325,000 in no time. It might
not be for *me* but I'm not going to teach my daughter a large IDE any
time soon.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] IDLE in the stdlib

[Daniel Holth]

On Wed, Mar 20, 2013 at 3:54 PM, Barry Warsaw ba...@python.org wrote:
 On Mar 20, 2013, at 12:40 PM, Guido van Rossum wrote:

I didn't hear any at the sprint here.

 JFDI! :)

 -Barry

+1 why are we still talking show me the patches
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] built-in Python test runner (was: Python Language Summit at PyCon: Agenda)

[Daniel Holth]

On Tue, Mar 5, 2013 at 2:49 PM, Barry Warsaw ba...@python.org wrote:
 On Mar 05, 2013, at 02:11 AM, Donald Stufft wrote:

Doesn't setuptools/distribute already have a setup.py test command? That
seems like the easiest way forward?

 Yes, and in theory it can make `python setup.py test` work well.  But there
 are lots of little details (such as API differences for ensuring that doctests
 run, additional tests discovery, etc.) that make this often not work so well
 in practice.  Some of that is social and some of it is technical.  I still
 claim that including test suite information in a package's metadata would be a
 win, but maybe that's just too much to hope for right now.

It would be a win, but parsing the metadata is just not what happens
right now, let alone writing anything about which and where the
modules are defined in the sdist. We can barely install packages by
using the dependency metadata from PKG-INFO; pip always re-generates
it from setup.py egg_info.

Your testing metadata prototype would only have to write two lines to
the metadata instead of one a-la: Extension: flufl; flufl/test_suite:
nose.collector; document the extension; write some tool to actually
parse the metadata and invoke the tests; it may become a core feature
in the next version, or having a monolithic specification may become
less important.

Thanks,

Daniel Holth
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python Language Summit at PyCon: Agenda

[Daniel Holth]

On Mon, Mar 4, 2013 at 2:26 PM, Berker Peksağ berker.pek...@gmail.com wrote:
 On Mon, Mar 4, 2013 at 9:14 PM, Barry Warsaw ba...@python.org wrote:
 On Mar 04, 2013, at 07:41 PM, Antoine Pitrou wrote:

 $ python -m unittest discover
 $ python setup.py test
 $ python setup.py nosetests
 $ python -m nose test
 $ nosetests-X.Y

 Besides having a multitude of choices, there's almost no way to
 automatically discover (e.g. by metadata inspection or some such) how to
 invoke the tests.  You're often lucky if there's a README.test and it's
 still accurate.

I hope we can have a pytest utility that does the right thing in 3.4 :-)
Typing python -m unittest discover is too cumbersome.

 Where is this work being done (e.g. is there a PEP)?

 There is an open issue on the tracker: http://bugs.python.org/issue14266

 --Berker

setup.py's setup(test_suite=x)... not sure if this is a distutils or
setuptools feature. PEP 426 has an extension mechanism that could do
the job.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python Language Summit at PyCon: Agenda

[Daniel Holth]

On Mon, Mar 4, 2013 at 3:14 PM, Barry Warsaw ba...@python.org wrote:
 On Mar 04, 2013, at 03:02 PM, Daniel Holth wrote:

setup.py's setup(test_suite=x)... not sure if this is a distutils or
setuptools feature. PEP 426 has an extension mechanism that could do
the job.

 Shouldn't testing be a first order feature?

Unfortunately there are so many potential first-order features that
we've had to leave some out in order to save time. How to run the
tests is not something that you need to know when searching PyPI for
a distribution and its dependencies.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] running tests; mebs

[Daniel Holth]

 As a community, we know how important tests are, so I think our tools should
 reflect that and make it easy for those tests to be expressed.  As a selfish
 side-effect, I want to reduce the amount of guesswork I need to perform in
 order to know how to run a package's test when I `$vcs clone` their
 repository. ;)



 Distutils2 had a way of specifying this in the metadata. It looks like this 
 hasn't made it into the reboot:

  http://alexis.notmyidea.org/distutils2/distutils/newcommands.html

 Michael

 Cheers,
 -Barry

I'm not aware of a reboot of the setup.py replacement / improvement
effort. The work that has been done has proceeded backwards from the
installer end of things.

I had a proposal called mebs, not an actual project. A completely
plugin-based system would recognize any sdist format and provide a
minimal, consistent interface. Add tests to the below text from
October.

...

A very simple meta-build system mebs is used to recognize sdists and
build binary packages. Build systems provide plugins having three
methods,

.recognize()
.metadata()
.build()

An installer downloads an sdist. For each installed build plugin,
.recognize(dir) is called. The first plugin to return True is used.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cffi in stdlib

[Daniel Holth]

On Wed, Feb 27, 2013 at 4:18 AM, Ronald Oussoren ronaldousso...@mac.com wrote:

 On 27 Feb, 2013, at 10:06, Maciej Fijalkowski fij...@gmail.com wrote:

 On Wed, Feb 27, 2013 at 9:29 AM, Ronald Oussoren ronaldousso...@mac.com 
 wrote:

 On 26 Feb, 2013, at 16:13, Maciej Fijalkowski fij...@gmail.com wrote:

 Hello.

 I would like to discuss on the language summit a potential inclusion
 of cffi[1] into stdlib.

 The API in general looks nice, but I do have some concens w.r.t. including 
 cffi in the stdlib.

 1. Why is cffi completely separate from ctypes, instead of layered on top 
 of it? That is, add a utility module to ctypes that can parse C 
 declarations and generate the right ctypes definitions.

 Because ctypes API is a mess and magic. We needed a cleaner (and much
 smaller) model.

 The major advantages of starting over is probably that you can hide the 
 complexity and that opens opportunities for optimizations. That said, I'm not 
 convinced that ctypes is unnecessarily complex.

cffi actually does have a ctypes backend in addition to the ffi and
compile a CPython extension backends. But the ctypes backend is
guaranteed to be slow and messy because it is ctypes.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cffi in stdlib

[Daniel Holth]

+1


On Tue, Feb 26, 2013 at 12:14 PM, Maciej Fijalkowski fij...@gmail.comwrote:

 On Tue, Feb 26, 2013 at 7:07 PM, Eli Bendersky eli...@gmail.com wrote:
 
 
 
  On Tue, Feb 26, 2013 at 8:39 AM, Maciej Fijalkowski fij...@gmail.com
  wrote:
 
  On Tue, Feb 26, 2013 at 6:34 PM, Eli Bendersky eli...@gmail.com
 wrote:
  
   On Tue, Feb 26, 2013 at 7:42 AM, Nick Coghlan ncogh...@gmail.com
   wrote:
  
   On Wed, Feb 27, 2013 at 1:13 AM, Maciej Fijalkowski 
 fij...@gmail.com
   wrote:
Hello.
   
I would like to discuss on the language summit a potential
 inclusion
of cffi[1] into stdlib.
  
   I think cffi is well worth considering as a possible inclusion for
   Python 3.4. (In particular, I'm a fan of the fact it just uses C
   syntax to declare what you're trying to talk to)
  
  
   I'm cautiously +0.5 because I'd really like to see a strong comparison
   case
   being made vs. ctypes. I've used ctypes many times and it was easy and
   effortless (well, except the segfaults when wrong argument types are
   declared :-). I'll be really interesting in seeing concrete examples
   that
   demonstrate how CFFI is superior.
 
  My main issue with ctypes, other than confusing API, which is up to
  taste, is that you just cannot wrap some libraries, like OpenSSL
  because of API vs ABI. OpenSSL uses macros extensively. Another point
  is that even C POSIX stdlib gives you incomplete structs and you have
  to guess where to put what fields.
 
 
  Yep, I can think of some reasons too. But just mentioning it so you
 remember
  explicitly listing the advantages when it comes to writing a PEP or some
  other sort of formal proposal. An FWIW, I think there's already enough
  positive feedback to at least start drafting a PEP. It can always be
  rejected later ;-)
 
  Eli
 

 It's copy-pasted in the original mail and it's in the docs. Anyway,
 this is just to familizarize people with the library. We're going to
 discuss it on the language summit at pycon.
 ___
 Python-Dev mailing list
 Python-Dev@python.org
 http://mail.python.org/mailman/listinfo/python-dev
 Unsubscribe:
 http://mail.python.org/mailman/options/python-dev/dholth%40gmail.com

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Fwd: PEP 426 is now the draft spec fordistribution metadata 2.0

[Daniel Holth]

On Feb 19, 2013 6:57 AM, a.cava...@cavallinux.eu wrote:

  I've never seen environment markers being used or supported
  in the wild.
 
  I'm not against modernizing the format, but given that version 1.2
  has been out for around 8 years now, without much following,
  I think we need to make the implementation bit a requirement
  before accepting the PEP.

 Elephant in the room? Somethin's been around for so long and hasn't got
 any sort of
 mainstream support is *not* a good candidate for a requirement but maybe
 it's me
 being naive.


I think 1.2 was started and then amended over a long period of time to
become what it is today. I wanted to edit it again just to add
Provides-Extra. It turned out to be more practical to make a new PEP. Nick
explains that for complicated reasons the implementation of Metadata 1.2
was not successful. For Metadata 2.0 we had a different strategy: put it
into distribute first. You will find that this aids adoption. distlib will
provide another implementation that is in many ways interchangeable.

We also have three implementations of an installer for the related wheel
format, three implementations of the wheel builder, and two surprisingly
short implementations of environment markers. Yes, the wheel project has
the same name as the wheel format.

Metadata 2.0's Provides-Extra field is there to represent the setuptools
extras feature which is necessary for a very large number of
distributions on pypi. For a while the most common environment markers will
look like just extra == 'foo'. So it should not be surprising that
distribute is the first tool to recognize Provides-Extra. Previously
distribute did not use METADATA / PKG-INFO for dependencies, instead
reading from requires.txt

If you can get around the fact that we've incorporated most of another PEP
386 into v2.0, rewritten and explained everything to death, I would argue
that 2.0 is actually simpler than 1.2. Only four headers are actually
required in a valid Metadata 2.0 file and a number of the fields are
informational and do not have to be understood by the installer.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Fwd: PEP 426 is now the draft spec fordistribution metadata 2.0

[Daniel Holth]

On Tue, Feb 19, 2013 at 11:26 AM, Paul Moore p.f.mo...@gmail.com wrote:

 On 19 February 2013 13:59, Nick Coghlan ncogh...@gmail.com wrote:
  It's OK if people don't want to read the detailed rationale provided
  for each of the major changes as part of the PEP, or if they want to
  dispute a particular piece of that rationale. But merely going it's
  too complicated! or metadata 1.2 failed, so 2.0 will fail as well!
  is not a reasonable response. Software distribution is complicated -
  trying to oversimplify it is one of the reasons setuptools became
  necessary.

 Nevertheless, the landscape is confusing. PEPs 241, 314, 345, 426,
 390, 376 and 386 are all relevant to one extent or another, and only
 PEPs 426 (Metadata 2.0) and 390 (Static Metadata for Distutils -
 setup.cfg) are still in Draft format. The others are all Accepted or
 Final. And yet PEPs 345, 390 and 386 are unused and likely to remain
 so.

 I believe that the only ones that are *actually* of use are 241, 314
 and 426 (Metadata 1.0, 1.1 and 2.0, but not 1.2) and 376 (Database of
 Installed Python Distributions) although 376 still has a number of
 flaws. I'd suggest that these should be marked as Final, and the
 others as Rejected, so that we have a clear statement of what is
 actually supported.

 Making sure that users have the means to write code that *uses* these
 standards using functionality available in the stdlib is then the next
 step as you say. It is critical that this is done, because packaging
 tools are unique in that the barrier to using external dependencies is
 particularly high for them - for example, pip's reliance on
 distribute/setuptools is necessary, but has been problematic at times.

 Paul

 PS Apologies for using a load of PEP numbers without explanation.
 Here's a glossary:

 Metadata 1.0 - PEP 241
 Metadata 1.1 - PEP 314
 Metadata 1.2 - PEP 345
 Metadata 1.3 - PEP 426
 Static Metadata for Distutils - PEP 390
 Database of Installed Python Distributions - PEP 376
 Changing the version comparison module in Distutils - PEP 386


The documentation is bad.

The standard library will almost certainly grow code that can interpret
these PEPs. Packaging tool authors can handle these specs but no sane
person reads PEPs to figure out how to submit a simple library to pypi. No
end user cares about metadata versions either or even notices that sdists
contain PKG-INFO at their root. You get what setup() produces and that's
it.

Wheel installers can work without being installed into the target
environment at all. This is very hard to understand if you have been using
the distutils model (the installed package imports the installer and
packages are always rebuilt from source) for more than a decade. This
feature does something to remedy the setuptools chicken/egg problem. We
have eliminated the egg ;-)

Once the installer doesn't need to be installed, repeatedly, in every
virtualenv, hundreds of times a day, users could just have a single
up-to-date standalone copy of pip invoked as python pip.zip install x.
Additional packages needed only for builds (even setuptools) may be
installed automatically by a build system (like pip) when building rather
than using packages.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Fwd: PEP 426 is now the draft spec for distribution metadata 2.0

[Daniel Holth]

On Tue, Feb 19, 2013 at 3:25 PM, Paul Moore p.f.mo...@gmail.com wrote:

 On 19 February 2013 13:40, Nick Coghlan ncogh...@gmail.com wrote:
  If a tools wants to support metadata 2.0, it has to support all
  the complicated stuff as well, i.e. handle the requires fields,
  the environment markers and version comparisons/sorting.
 
  Which is what distutils2 can be used for now, and what distlib will
  provide without the unwanted build system infrastructure in
  distutils2.

 One particular item in Metadata 2.0 which needs to be addressed is
 requirements specifiers, as used in (for example) Requires-Dist. These
 take forms like DistName (= ver). There are a number of potential
 issues here:

 1. This differs from the pkg_resources format (which omits the
 parentheses). Having 2 subtly different formats is not a good idea in
 the long term. At the moment, pkg_resources format is used in pip
 requirements (on the command line and in requirement files) as well as
 in setuptools/distribute and possibly elsewhere.
 2. There is currently no code that I am aware of that implements this
 spec. I don't believe distlib does (yet - give Vinay 5 minutes and who
 knows? :-)), pkg_resources as I said implements a different format,
 and distutils2, apart from being a big dependency to cover just this
 point, only matches the version (not the project name) and presumably
 does so using the distutils2 version ordering (which is incompatible
 with Metadata 2.0).
 3. The format is fiddly to parse and process - nobody is likely to
 implement it for themselves without a library that does it (especially
 not when using pkg_resources is so easy).

 The PEP needs a rationale as to why the pkg_resources format wasn't
 used, and should suggest a migration path for projects which currently
 use the pkg_resources style (such as pip's requirements files).

 Paul


We did add full support for Metadata 2.0 Requires-Dist in pkg_resources.
The fiddling happens here:
https://bitbucket.org/tarek/distribute/src/tip/pkg_resources.py?at=default#cl-2469

The code removes the parenthesis and adds == when no operator was specified
to convert us back to the pkg_resources format. The rest of pkg_resources
carries on as before. Wheel installs would not pull in dependencies
otherwise.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Fwd: PEP 426 is now the draft spec for distribution metadata 2.0

[Daniel Holth]

On Tue, Feb 19, 2013 at 4:34 PM, M.-A. Lemburg m...@egenix.com wrote:

 On 19.02.2013 14:40, Nick Coghlan wrote:
  On Tue, Feb 19, 2013 at 11:23 PM, M.-A. Lemburg m...@egenix.com wrote:
  * PEP 426 doesn't include any mention of the egg distribution format,
even though it's the most popular distribution format at the moment.
It should at least include the location of the metadata file
in eggs (EGG-INFO/PKG-INFO) and egg installations
(eggdir/EGG-INFO/PKG-INFO).
 
  Other tools involved in Python distribution may also use this format.
 
  The egg format has never been, and never will be, officially endorsed
  by python-dev. The wheel format is the standard format for binary
  distribution, and PEP 376 defines the standard location for metadata
  on installed distributions.

 Oh, come on, Nick, that's just silly. setuptools was included in stdlib
 for a short while, so the above is simply wrong. Eggs are the most
 widely used binary distribution format for Python package on PyPI:

 # wc *files.csv
   25585   25598 1431013 2013-02-19-egg-files.csv
46194640  236694 2013-02-19-exe-files.csv
 254 255   13402 2013-02-19-msi-files.csv
  104691  104853 5251962 2013-02-19-tar-gz-files.csv
  24  241221 2013-02-19-whl-files.csv
   17937   18022  905913 2013-02-19-zip-files.csv
  153110  153392 7840205 total

 (based on todays PyPI stats)

 It doesn't really help ignoring realities... and I'm saying
 that as one of the core devs who got setuptools kicked out of
 the stdlib again.

 --
 Marc-Andre Lemburg
 eGenix.com


The wheel philosophy is that it should be supported by both python-dev and
setuptools and that you should feel happy about using setuptools if you
like it whether or not python-dev (currently) endorses that. If you are
using setuptools (distribute's pkg_resources) then you can use both at the
same time.

Distribute, distutils and setuptools' problems have not been well
understood which I think is why there has been a need to discredit
setuptools by calling it non-standard. It is the defacto standard. If your
packages have dependencies there is no other choice. Wheel tries to solve
the real problem by allowing you to build a package with setuptools while
giving the end-user the choice of installing setuptools or not.

Of course eggs are the most popular right now. The wheel format is very
egg-like while avoiding some of egg's problems. See the comparison in the
PEP or read the story on wheel's rtfd. The wheel project includes tools to
losslessly convert eggs or bdist_wininst to wheel.

I am confident distlib can thrive outside of the standard library! Why the
rush to kill it before its prime?
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Fwd: PEP 426 is now the draft spec for distribution metadata 2.0

[Daniel Holth]

On Tue, Feb 19, 2013 at 5:10 PM, M.-A. Lemburg m...@egenix.com wrote:

 On 19.02.2013 23:01, Daniel Holth wrote:
  On Tue, Feb 19, 2013 at 4:34 PM, M.-A. Lemburg m...@egenix.com wrote:
 
  On 19.02.2013 14:40, Nick Coghlan wrote:
  On Tue, Feb 19, 2013 at 11:23 PM, M.-A. Lemburg m...@egenix.com
 wrote:
  * PEP 426 doesn't include any mention of the egg distribution format,
even though it's the most popular distribution format at the moment.
It should at least include the location of the metadata file
in eggs (EGG-INFO/PKG-INFO) and egg installations
(eggdir/EGG-INFO/PKG-INFO).
 
  Other tools involved in Python distribution may also use this format.
 
  The egg format has never been, and never will be, officially endorsed
  by python-dev. The wheel format is the standard format for binary
  distribution, and PEP 376 defines the standard location for metadata
  on installed distributions.
 
  Oh, come on, Nick, that's just silly. setuptools was included in stdlib
  for a short while, so the above is simply wrong. Eggs are the most
  widely used binary distribution format for Python package on PyPI:
 
  # wc *files.csv
25585   25598 1431013 2013-02-19-egg-files.csv
 46194640  236694 2013-02-19-exe-files.csv
  254 255   13402 2013-02-19-msi-files.csv
   104691  104853 5251962 2013-02-19-tar-gz-files.csv
   24  241221 2013-02-19-whl-files.csv
17937   18022  905913 2013-02-19-zip-files.csv
   153110  153392 7840205 total
 
  (based on todays PyPI stats)
 
  It doesn't really help ignoring realities... and I'm saying
  that as one of the core devs who got setuptools kicked out of
  the stdlib again.
 
  --
  Marc-Andre Lemburg
  eGenix.com
 
 
  The wheel philosophy is that it should be supported by both python-dev
 and
  setuptools and that you should feel happy about using setuptools if you
  like it whether or not python-dev (currently) endorses that. If you are
  using setuptools (distribute's pkg_resources) then you can use both at
 the
  same time.
 
  Distribute, distutils and setuptools' problems have not been well
  understood which I think is why there has been a need to discredit
  setuptools by calling it non-standard. It is the defacto standard. If
 your
  packages have dependencies there is no other choice. Wheel tries to solve
  the real problem by allowing you to build a package with setuptools while
  giving the end-user the choice of installing setuptools or not.
 
  Of course eggs are the most popular right now. The wheel format is very
  egg-like while avoiding some of egg's problems. See the comparison in the
  PEP or read the story on wheel's rtfd. The wheel project includes tools
 to
  losslessly convert eggs or bdist_wininst to wheel.

 That's all fine, but it doesn't explain the refusal to add the
 documentation of the location of the PKG-INFO file in eggs ?


It would just be a sentence, I wouldn't have a problem with it but I also
don't see why it would be necessary. Even setuptools doesn't touch the file
usually. Right now distribute's pkg_resources currently only understands
Requires-Dist if it is inside a .dist-info directory.

Sorry, Chris must have meant http://hg.python.org/distlib/ . I was
struggling to imagine a world where that is more visible than something on
bitbucket. Half the comments have been about putting something in stdlib
right away, something that I do not understand at all as a voracious PyPI
or https://crate.io/ user.

You should secure both the metadata and the packages but it doesn't have
anything to do with the serialization format.

Wheel is designed so that the metadata is at the end of the .zip and can be
retrieved with an HTTP partial request. The feature may or may not be
useful.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 426 is now the draft spec for distribution metadata 2.0

[Daniel Holth]

On Sun, 17 Feb 2013 20:11:23 +1000
Nick Coghlan ncogh...@gmail.com wrote:

 Major changes since the last draft:

 1. Metadata-Version is 2.0 rather than 1.3, and the field now has the
 same major.minor semantics as are defined for wheel versions in PEP
 427 (i.e. if a tool sees a major version number it doesn't recognise,
 it should give up rather than trying to guess what to do with it,
 while it's OK to process a higher minor version)

So is it possible to bundle several versioned metadata files in order
to maximize compatibility? Or does a distribution with 2.0 metadata
have to abandon all hope of being compatible with all tools?

Regards

Antoine.

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/dholth%40gmail.com

Not likely to matter for a while as the current md v1 tools don't
understand this new obsolescence rule :-)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL delegation for PEP 426 (PyPI metadata 1.3)

[Daniel Holth]

On Fri, Feb 15, 2013 at 12:27 PM, Erik Bray erik.m.b...@gmail.com wrote:

 On Sun, Feb 3, 2013 at 5:24 PM, Vinay Sajip vinay_sa...@yahoo.co.uk
 wrote:
  Éric Araujo merwok at netwok.org writes:
 
  Looks like we agree that a basic tool able to bootstrap the packaging
  story is needed :)
 
  Agreed. Just because distutils can't easily/reliably build things that
 are
  better built with SCons/WAF/tup/whatever, doesn't mean that we shouldn't
 have
  the ability to build pure-Python distributions and distributions
 including C
  libs and extensions, with the ability to extend easily by third-party
 tools. It
  just needs to be done in a way which is easy to build on, so the included
  battery stays small and simple. Easier said than done, I know :-)
 
  Regards,
 
  Vinay Sajip

 Sorry to revive an old-ish discussion--I'm just catching up on things.
  But I just wanted to add that distutils is still pretty okay for
 building reasonably complex projects.  Although it does not rise to
 the level of complexity of Numpy or SciPy, the Astropy project
 (https://github.com/astropy/astropy) has managed to put together a
 pretty nice build system on top of mostly-plain distutils (it does use
 distribute but primarily just for 2to3 support).


 This has necessitated a number of hacks to overcome shortcomings and
 bugs in distutils, but most of those shortcomings could probably be
 fixed in distutils within the framework of a slightly lifted freeze.
 But in any case I haven't found it worthwhile to switch to something
 like bento when the batteries included in the stdlib have been mostly
 Good Enough. Having fewer installation dependencies has also made it
 significantly easier for non-advanced users to install. Even the
 distribute requirement doesn't add too much overhead, as most users
 have it on their systems by default now, and for those who don't
 distribute_setup.py works okay.

 TL;DR, strong -1 on the stdlib getting out of the build business.
 Also as I think Nick already mentioned one of the wins of
 Setup-Requires-Dist is to have a standard way to bring in extra build
 requirements (such as bento) so if we have better support for a
 feature like that it's not necessary to bless any preferred tool.


Distutils is not really going away. We need it to build the existing 28,000
packages. However empirically it seems if you try to write a significant
extension to or improvement of distutils then you are likely to get burnt
out and switch careers.

Instead of literally killing distutils we hope to make it very easy to use
other build tools when you need them and not use any build tools at all
when you don't. As a thought experiment: what if one of those third party
build tools hosted on pypi was distutils itself? What would you need to do
to make that happen?

The packaging peps PEP-376 and so on are brilliant because they are simple
enough to be implemented twice. If we had better ways to separate interface
from implementation in Python I'd like to see two of whatever else we come
up with for packaging.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Submitting PEP 425 (compatibility tags)

[Daniel Holth]

This is the improved compatibility tags PEP 425, specifying how part of the
Wheel PEP 427 filenames work. Last time we discussed whether replacing .
with _ was ugly but I concluded it was harmless.

Submitted for your consideration,

PEP: 425
Title: Compatibility Tags for Built Distributions
Version: $Revision$
Last-Modified: 07-Aug-2012
Author: Daniel Holth dho...@fastmail.fm
BDFL-Delegate: Nick Coghlan ncogh...@gmail.com
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 27-Jul-2012
Python-Version: 3.4
Post-History: 8-Aug-2012, 18-Oct-2012, 15-Feb-2013


Abstract


This PEP specifies a tagging system to indicate with which versions of
Python a built or binary distribution is compatible.  A set of three
tags indicate which Python implementation and language version, ABI,
and platform a built distribution requires.  The tags are terse because
they will be included in filenames.


PEP Editor's Note
=

While the naming scheme described in this PEP will not be supported directly
in the standard library until Python 3.4 at the earliest, draft
implementations may be made available in third party projects.


Rationale
=

Today python setup.py bdist generates the same filename on PyPy
and CPython, but an incompatible archive, making it inconvenient to
share built distributions in the same folder or index.  Instead, built
distributions should have a file naming convention that includes enough
information to decide whether or not a particular archive is compatible
with a particular implementation.

Previous efforts come from a time where CPython was the only important
implementation and the ABI was the same as the Python language release.
This specification improves upon the older schemes by including the Python
implementation, language version, ABI, and platform as a set of tags.

By comparing the tags it supports with the tags listed by the
distribution, an installer can make an educated decision about whether
to download a particular built distribution without having to read its
full metadata.

Overview


The tag format is {python tag}-{abi tag}-{platform tag}

python tag
‘py27’, ‘cp33’
abi tag
‘cp32dmu’, ‘none’
platform tag
‘linux_x86_64’, ‘any’

For example, the tag py27-none-any indicates compatible with Python 2.7
(any Python 2.7 implementation) with no abi requirement, on any platform.

Use
===

The `wheel` built package format includes these tags in its filenames,
of the form ``{distribution}-{version}(-{build tag})?-{python tag}-{abi
tag}-{platform tag}.whl``.  Other package formats may have their own
conventions.

Details
===

Python Tag
--

The Python tag indicates the implementation and version required by
a distribution.  Major implementations have abbreviated codes, initially:

* py: Generic Python (does not require implementation-specific features)
* cp: CPython
* ip: IronPython
* pp: PyPy
* jy: Jython

Other Python implementations should use `sys.implementation.name`.

The version is `py_version_nodot`.  CPython gets away with no dot,
but if one is needed the underscore `_` is used instead.  PyPy should
probably use its own versions here `pp18`, `pp19`.

The version can be just the major version `2` or `3` `py2`, `py3` for
many pure-Python distributions.

Importantly, major-version-only tags like `py2` and `py3` are not
shorthand for `py20` and `py30`.  Instead, these tags mean the packager
intentionally released a cross-version-compatible distribution.

A single-source Python 2/3 compatible distribution can use the compound
tag `py2.py3`.  See `Compressed Tag Sets`, below.

ABI Tag
---

The ABI tag indicates which Python ABI is required by any included
extension modules.  For implementation-specific ABIs, the implementation
is abbreviated in the same way as the Python Tag, e.g. `cp33d` would be
the CPython 3.3 ABI with debugging.

The CPython stable ABI is `abi3` as in the shared library suffix.

Implementations with a very unstable ABI may use the first 6 bytes (as
8 base64-encoded characters) of the SHA-256 hash of ther source code
revision and compiler flags, etc, but will probably not have a great need
to distribute binary distributions. Each implementation's community may
decide how to best use the ABI tag.

Platform Tag


The platform tag is simply `distutils.util.get_platform()` with all
hyphens `-` and periods `.` replaced with underscore `_`.

* win32
* linux_i386
* linux_x86_64

Use
===

The tags are used by installers to decide which built distribution
(if any) to download from a list of potential built distributions.
The installer maintains a list of (pyver, abi, arch) tuples that it
will support.  If the built distribution's tag is `in` the list, then
it can be installed.

For example, an installer running under CPython 3.3 on a linux_x86_64
system might support::

 1. cp33-cp33m-linux_x86_64
 2. cp33-abi3-linux_x86_64
 3. cp33-none-linux_x86_64
 4. cp33-none-any
 5. cp3-none-any
 6

Re: [Python-Dev] Usage of += on strings in loops in stdlib

[Daniel Holth]

On Wed, Feb 13, 2013 at 7:10 AM, Serhiy Storchaka storch...@gmail.comwrote:

 On 13.02.13 10:52, Larry Hastings wrote:

 I've always hated the .join(array) idiom for fast string
 concatenation--it's ugly and it flies in the face of TOOWTDI.  I think
 everyone should use x = a + b + c + d for string concatenation, and we
 should just make that fast.


 I prefer x = '%s%s%s%s' % (a, b, c, d) when string's number is more than
 3 and some of them are literal strings.


Fixed: x = ('%s' *  len(abcd)) % abcd
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL delegation for PEP 426 (PyPI metadata 1.3)

[Daniel Holth]

The neat thing about wheel is that you can install them without having the
software used to build them. So we might try to provide a very simple wheel
installer script with Python that did not even depend on DistUtils. You
would be able to install pip etc with that tool. There is no need to put
wheel directly into DistUtils.
On Feb 3, 2013 7:54 AM, Antoine Pitrou solip...@pitrou.net wrote:

 On Sun, 3 Feb 2013 12:34:36 +
 Paul Moore p.f.mo...@gmail.com wrote:
 
  So it's perfectly possible to use wheels right now, without the pip
  integration. But the pip developers don't want to integrate the wheel
  format just because it exists - they want the assurance that it's an
  accepted format supported by PEPs, hence the interest in getting the 3
  wheel PEPs (of which the metadata PEP is the first) accepted.

 But it's an accepted format by whom? If we accept the PEP but refuse
 to integrate support in the stdlib, it doesn't make much sense.

 I vote for removing the distutils is frozen principle. It was already
 destructive enough when packaging was still alive - bug fixes were
 actively turned down for the dubious reason that fixing bugs would
 break compatibility. Its rationale has become completely obsolete now
 that packaging has failed.

 Regards

 Antoine.
 ___
 Python-Dev mailing list
 Python-Dev@python.org
 http://mail.python.org/mailman/listinfo/python-dev
 Unsubscribe:
 http://mail.python.org/mailman/options/python-dev/dholth%40gmail.com

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL delegation for PEP 426 (PyPI metadata 1.3)

[Daniel Holth]

My position is that these days distutils doesn't belong in the standard
library any more than Django does. So I am mildly opposed to supporting it
when you should be using better designed third party tools like Bento or
setuptools. Wheel makes it possible for Python to get out of the build tool
business. Just install your preferred tools with a concise bootstrap
installer.
On Sun, 3 Feb 2013 09:41:29 -0500
Daniel Holth dho...@gmail.com wrote:
 The neat thing about wheel is that you can install them without having the
 software used to build them. So we might try to provide a very simple
wheel
 installer script with Python that did not even depend on DistUtils. You
 would be able to install pip etc with that tool. There is no need to put
 wheel directly into DistUtils.

But you still need to *produce* wheels, don't you? It would be more
useful for everyone if that facility was bundled with Python.

Regards

Antoine.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/dholth%40gmail.com
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL delegation for PEP 426 (PyPI metadata 1.3)

[Daniel Holth]

Bento is the only available packaging tool to heap praise onto and it is
impressive. I am reacting to all the hate heaped on setup tools when I
think the underlying DistUtils design is a big part of the problem. My
feeling is that stdlib packaging tools should be for bootstrapping and
reference, more like wsgiref than django.

Wheel is very similar to egg but tries to solve the original setup tools
sin of installing something you don't want (the entire setuptools build
system) by omitting the runtime component. The result is a very reviewable
couple hundred lines bootstrap installer or unzip in a pinch. Its also
built upon Python standards that were not available when easy_install was
introduced.
On Feb 3, 2013 2:09 PM, Ralf Schmitt r...@systemexit.de wrote:

 Daniel Holth dho...@gmail.com writes:

  Wheel makes it possible for Python to get out of the build tool
  business. Just install your preferred tools with a concise bootstrap
  installer.

 If this is true, it would also have been possible with eggs, yet it
 didn't happen. Why do you think it will happen now or am I missing
 something?
 ___
 Python-Dev mailing list
 Python-Dev@python.org
 http://mail.python.org/mailman/listinfo/python-dev
 Unsubscribe:
 http://mail.python.org/mailman/options/python-dev/dholth%40gmail.com

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL delegation for PEP 426 + distutils freeze

[Daniel Holth]

I did think that updating distutils to have basic support for the packaging
PEPs was a decent idea, but then it wound up being more or less rewritten
entirely like I've been cajoled into doing with PEP 426 (Metadata). I don't
know whether distutils(1) can survive the minimum changes required for
wheel (.dist-info folders and a bdist_wheel command) without breaking a
significant number of its clients or not. History suggests no. It may still
be worth a shot.

bdist_wheel started out as a distutils2 plugin but it was inconvenient to
develop it inside the CPython repository and it was not practical to
convert projects over from setuptools to distutils2.

The pypi metadata has had a standardlib generator for a long time without
having anything in the standard library to make use of the metadata. Wheels
are just .zip files of one package's share of the site-packages directory.
Why not support them asymmetrically. I value the PEP and consensus for the
format more than the implementation.

One idea that's sure to be controversial would be to include a bootstrap
installer as a script without committing to a Python API (just a
command-line API). Since Python 2.6, wheel's own wheel archive can be used
to install wheels:

$ python wheel-0.14.0-py2.py3-none-any.whl/wheel -h
usage: wheel [-h] {keygen,sign,verify,unpack,install,convert,help} ...



On Sun, Feb 3, 2013 at 2:40 PM, Chris Jerdonek chris.jerdo...@gmail.comwrote:

 On Sun, Feb 3, 2013 at 10:33 AM, Éric Araujo mer...@netwok.org wrote:
  Le 03/02/2013 07:48, Antoine Pitrou a écrit :
  I vote for removing the distutils is frozen principle.
  I’ve also been thinking about that.  There have been two exceptions to
  the freeze, for ABI flags in extension module names and for pycache
  directories.  When the stable ABI was added and MvL wanted to change
  distutils (I don’t know to do what exactly), Tarek stood firm on the
  freeze and asked for any improvement to go into distutils2, and after
  MvL said that he would not contibute to an outside project, we merged d2
  into the stdlib.  Namespace packages did not impact distutils either.
  Now that we’ve removed packaging from the stdlib, we have two Python
  features that are not supported in the standard packaging system, and I
  agree that it is a bad thing for our users.
 
  I’d like to propose a reformulation of the freeze:

 This could be common knowledge, but is the current formulation of the
 freeze spelled out somewhere?

 --Chris
 ___
 Python-Dev mailing list
 Python-Dev@python.org
 http://mail.python.org/mailman/listinfo/python-dev
 Unsubscribe:
 http://mail.python.org/mailman/options/python-dev/dholth%40gmail.com

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com