from:"M.\-A. Lemburg"

On 14.10.2020 17:59, Antoine Pitrou wrote:
> 
> Le 14/10/2020 à 17:25, M.-A. Lemburg a écrit :
>>
>> Well, there's a trend here:
>>
>> [...]
>>
>> Those two benchmarks were somewhat faster in Py3.7 and got slower in 3.8
>> and then again in 3.9, so this is more than just an artifact.
> 
> unpack-sequence is a micro-benchmark.  It's useful if you want to
> investigate the cause of a regression witnessed elsewhere (or if you're
> changing things in precisely that part of the interpreter), but it's not
> relevant in itself to measure Python performance.

Since unpacking is done a lot in Python applications, this particular
micro benchmark does have an effect on overall performance and there
was some recent discussion about exactly this part of the code slowing
down (even though the effects were related to macOS only AFAIR).

As with most micro benchmarks, you typically don't see the effect
of one slowdown or speedup in applications. Only if several such
changes come together, you notice a change.

That said, it's still good practice to keep an eye on such performance
regressions and also to improve upon micro benchmarks.

The latter was my main motiviation for writing pybench back in 1997,
which focuses on such micro benchmarks, rather than higher level
benchmarks, where it's much harder to find out why performance
changed.

> regex-dna is a "mini"-benchmark. I suppose someone could look if there
> were any potentially relevant changes done in the regex engine, that
> would explain the changes.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Oct 14 2020)
>>> Python Projects, Coaching and Support ...https://www.egenix.com/
>>> Python Product Development ...https://consulting.egenix.com/

::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   https://www.egenix.com/company/contact/
 https://www.malemburg.com/
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/S5RDUVCUER2GIAOBNTLOZ7QUNCRDDIWJ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: [python-committers] Re: Performance benchmarks for 3.9

On 14.10.2020 16:14, Antoine Pitrou wrote:
> Le 14/10/2020 à 15:16, Pablo Galindo Salgado a écrit :
>> Hi!
>>
>> I have updated the branch benchmarks in the pyperformance server and now
>> they include 3.9. There are
>> some benchmarks that are faster but on the other hand some benchmarks
>> are substantially slower, pointing
>> at a possible performance regression in 3.9 in some aspects. In
>> particular some tests like "unpack sequence" are
>> almost 20% slower. As there are some other tests were 3.9 is faster, is
>> not fair to conclude that 3.9 is slower, but
>> this is something we should look into in my opinion.
>>
>> You can check these benchmarks I am talking about by:
>>
>> * Go here: https://speed.python.org/comparison/
>> * In the left bar, select "lto-pgo latest in branch '3.9'" and "lto-pgo
>> latest in branch '3.8'"
>> * To better read the plot, I would recommend to select a "Normalization"
>> to the 3.8 branch (this is in the top part of the page)
>>    and to check the "horizontal" checkbox.
> Those numbers tell me that it's a wash.  I wouldn't worry about a small
> regression on a micro- or mini-benchmark while the overall picture is
> stable.

Well, there's a trend here:



Those two benchmarks were somewhat faster in Py3.7 and got slower in 3.8 and
then again in 3.9, so this is more than just an artifact.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Oct 14 2020)
>>> Python Projects, Coaching and Support ...https://www.egenix.com/
>>> Python Product Development ...https://consulting.egenix.com/


::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   https://www.egenix.com/company/contact/
 https://www.malemburg.com/

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/XICRZGNDU6TOG3VNFHKTOCVBQO3ZEZ5L/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: [python-committers] Performance benchmarks for 3.9

On 14.10.2020 16:00, Pablo Galindo Salgado wrote:
>> Would it be possible to get the data for older runs back, so that
> it's easier to find the changes which caused the slowdown ?
> 
> Unfortunately no. The reasons are that that data was misleading because
> different points were computed with a different version of pyperformance and
> therefore with different packages (and therefore different code). So the 
> points
> could not be compared among themselves.
> 
> Also, past data didn't include 3.9 commits because the data gathering was not
> automated and it didn't run in a long time :(

Make sense.

Would it be possible rerun the tests with the current
setup for say the last 1000 revisions or perhaps a subset of these
(e.g. every 10th revision) to try to binary search for the revision which
introduced the change ?

> On Wed, 14 Oct 2020 at 14:57, M.-A. Lemburg  <mailto:m...@egenix.com>> wrote:
> 
> Hi Pablo,
> 
> thanks for pointing this out.
> 
> Would it be possible to get the data for older runs back, so that
> it's easier to find the changes which caused the slowdown ?
> 
> Going to the timeline, it seems that the system only has data
> for Oct 14 (today):
> 
> 
> https://speed.python.org/timeline/#/?exe=12=regex_dna=1=1000=off=on=on=none
> 
> In addition to unpack_sequence, the regex_dna test has slowed
> down a lot compared to Py3.8.
> 
> 
> https://github.com/python/pyperformance/blob/master/pyperformance/benchmarks/bm_unpack_sequence.py
> 
> https://github.com/python/pyperformance/blob/master/pyperformance/benchmarks/bm_regex_dna.py
> 
> Thanks.
> 
> On 14.10.2020 15:16, Pablo Galindo Salgado wrote:
> > Hi!
> >
> > I have updated the branch benchmarks in the pyperformance server and 
> now they
> > include 3.9. There are
> > some benchmarks that are faster but on the other hand some benchmarks 
> are
> > substantially slower, pointing
> > at a possible performance regression in 3.9 in some aspects. In 
> particular
> some
> > tests like "unpack sequence" are
> > almost 20% slower. As there are some other tests were 3.9 is faster, is
> not fair
> > to conclude that 3.9 is slower, but
> > this is something we should look into in my opinion.
> >
> > You can check these benchmarks I am talking about by:
> >
> > * Go here: https://speed.python.org/comparison/
> > * In the left bar, select "lto-pgo latest in branch '3.9'" and "lto-pgo 
> latest
> > in branch '3.8'"
> > * To better read the plot, I would recommend to select a "Normalization"
> to the
> > 3.8 branch (this is in the top part of the page)
> >    and to check the "horizontal" checkbox.
> >
> > These benchmarks are very stable: I have executed them several times 
> over the
> > weekend yielding the same results and,
> > more importantly, they are being executed on a server specially 
> prepared to
> > running reproducible benchmarks: CPU affinity,
> > CPU isolation, CPU pinning for NUMA nodes, CPU frequency is fixed, CPU
> governor
> > set to performance mode, IRQ affinity is
> > disabled for the benchmarking CPU nodes...etc so you can trust these 
> numbers.
> >
> > I kindly suggest for everyone interested in trying to improve the 3.9 
> (and
> > master) performance, to review these benchmarks
> > and try to identify the problems and fix them or to find what changes
> introduced
> > the regressions in the first place. All benchmarks
> > are the ones being executed by the pyperformance suite
> > (https://github.com/python/pyperformance) so you can execute them
> > locally if you need to.
> >
> > ---
> >
> > On a related note, I am also working on the speed.python.org
> <http://speed.python.org>
> > <http://speed.python.org> server to provide more automation and
> > ideally some integrations with GitHub to detect performance 
> regressions. For
> > now, I have done the following:
> >
> > * Recompute benchmarks for all branches using the same version of
> > pyperformance (except master) so they can
> >    be compared with each other. This can only be seen in the 
> "Comparison"
> > tab: https://speed.python.org/comparison/
> > * I am setting daily builds of the master branch so we can detect 
> performance
> > regressions with dai

[Python-Dev] Re: [python-committers] Performance benchmarks for 3.9

Hi Pablo,

thanks for pointing this out.

Would it be possible to get the data for older runs back, so that
it's easier to find the changes which caused the slowdown ?

Going to the timeline, it seems that the system only has data
for Oct 14 (today):

https://speed.python.org/timeline/#/?exe=12=regex_dna=1=1000=off=on=on=none

In addition to unpack_sequence, the regex_dna test has slowed
down a lot compared to Py3.8.

https://github.com/python/pyperformance/blob/master/pyperformance/benchmarks/bm_unpack_sequence.py
https://github.com/python/pyperformance/blob/master/pyperformance/benchmarks/bm_regex_dna.py

Thanks.

On 14.10.2020 15:16, Pablo Galindo Salgado wrote:
> Hi!
> 
> I have updated the branch benchmarks in the pyperformance server and now they
> include 3.9. There are
> some benchmarks that are faster but on the other hand some benchmarks are
> substantially slower, pointing
> at a possible performance regression in 3.9 in some aspects. In particular 
> some
> tests like "unpack sequence" are
> almost 20% slower. As there are some other tests were 3.9 is faster, is not 
> fair
> to conclude that 3.9 is slower, but
> this is something we should look into in my opinion.
> 
> You can check these benchmarks I am talking about by:
> 
> * Go here: https://speed.python.org/comparison/
> * In the left bar, select "lto-pgo latest in branch '3.9'" and "lto-pgo latest
> in branch '3.8'"
> * To better read the plot, I would recommend to select a "Normalization" to 
> the
> 3.8 branch (this is in the top part of the page)
>    and to check the "horizontal" checkbox.
> 
> These benchmarks are very stable: I have executed them several times over the
> weekend yielding the same results and,
> more importantly, they are being executed on a server specially prepared to
> running reproducible benchmarks: CPU affinity,
> CPU isolation, CPU pinning for NUMA nodes, CPU frequency is fixed, CPU 
> governor
> set to performance mode, IRQ affinity is
> disabled for the benchmarking CPU nodes...etc so you can trust these numbers.
> 
> I kindly suggest for everyone interested in trying to improve the 3.9 (and
> master) performance, to review these benchmarks
> and try to identify the problems and fix them or to find what changes 
> introduced
> the regressions in the first place. All benchmarks
> are the ones being executed by the pyperformance suite
> (https://github.com/python/pyperformance) so you can execute them
> locally if you need to.
> 
> ---
> 
> On a related note, I am also working on the speed.python.org
>  server to provide more automation and
> ideally some integrations with GitHub to detect performance regressions. For
> now, I have done the following:
> 
> * Recompute benchmarks for all branches using the same version of
> pyperformance (except master) so they can
>    be compared with each other. This can only be seen in the "Comparison"
> tab: https://speed.python.org/comparison/
> * I am setting daily builds of the master branch so we can detect performance
> regressions with daily granularity. These
>    daily builds will be located in the "Changes" and "Timeline" tabs
> (https://speed.python.org/timeline/).
> * Once the daily builds are working as expected, I plan to work on trying to
> automatically comment or PRs or on bpo if
> we detect that a commit has introduced some notable performance regression.
> 
> Regards from sunny London,
> Pablo Galindo Salgado.
> 
> ___
> python-committers mailing list -- python-committ...@python.org
> To unsubscribe send an email to python-committers-le...@python.org
> https://mail.python.org/mailman3/lists/python-committers.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-committ...@python.org/message/G3LB4BCAY7T7WG22YQJNQ64XA4BXBCT4/
> Code of Conduct: https://www.python.org/psf/codeofconduct/
> 

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Oct 14 2020)
>>> Python Projects, Coaching and Support ...https://www.egenix.com/
>>> Python Product Development ...https://consulting.egenix.com/


::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   https://www.egenix.com/company/contact/
 https://www.malemburg.com/
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/P7Y2HFD62P6FYLNJFE6BZ7XHQ2NEN6J5/
Code of Conduct:

[Python-Dev] Re: [python-committers] Farewell, Python 3.5

2020-10-01 Thread M.-A. Lemburg

Thank you, Larry and the whole release team, for putting so much
work into this !

On 01.10.2020 19:49, Larry Hastings wrote:
> 
> At last!  Python 3.5 has now officially reached its end-of-life.  Since there
> have been no checkins or PRs since I tagged 3.5.10, 3.5.10 will stand as the
> final release in the 3.5 series.
> 
> As with a similar announcement I wrote about eighteen months ago, I know we 
> can
> all look back fondly on Python 3.5.  3.5 added many new asynchronous I/O
> programming features, the "typing" module, and even a new operator ("@").  
> Plus
> many and varied quality-of-life improvements for the Python programmer, in 
> both
> the language, the library, the core implementation, and even the installers. 
> Python 3.5.0 was the best version of the best language at the time, and since
> then it's gotten even better!
> 
> My thanks to all the members of the Python 3.5 release team.  In alphabetical 
> order:
> 
> Georg Brandl
> 
> Julian Palard
> 
> Ned Deily
> 
> Steve Dower
> 
> Terry Reedy
> 
> My thanks also to the Python infrastructure team.
> 
> 
> The end of Python 3.5 support also ends my tenure as a Python Release 
> Manager. 
> Congratulations, you survived me and my frequent mistakes!  (Special shouts 
> out
> to Ned and Benjamin for running around behind the scenes quietly cleaning up 
> my
> messes--and not even telling me most of the time.)  Rest assured that I leave
> you in /much/ better hands with the current crop of RMs: Ned, Łukasz, and 
> Pablo.
> 
> One amusing note.  During my tenure as a Python release manager, I had to deal
> with /three/ different revision control systems.  Although we'd switched 
> CPython
> itself to Mercurial  by the time 3.4 alpha 0 was released, there were still 
> many
> supporting repositories still on Subversion.  (I remember having to do
> Subversion branch merges as part of my 3.4 release work... what a pain.)  And 
> of
> course these days we're on Git (-hub).  This straddling of three different
> workflows certainly complicated the lives of us Release Managers.  So, my
> friends, please... make up your minds!  ;-)
> 
> 
> It's been my honor to serve you,
> 
> 
> //arry/
> 
> p.s. As of today, every supported version of Python supports f-strings.  The
> only remaining excuse for "we can't use f-strings" is no longer viable!
> 
> 
> ___
> python-committers mailing list -- python-committ...@python.org
> To unsubscribe send an email to python-committers-le...@python.org
> https://mail.python.org/mailman3/lists/python-committers.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-committ...@python.org/message/YKZON55BE5JMK6355KPD53HRUXOOYTYN/
> Code of Conduct: https://www.python.org/psf/codeofconduct/
> 

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Oct 01 2020)
>>> Python Projects, Coaching and Support ...https://www.egenix.com/
>>> Python Product Development ...https://consulting.egenix.com/


::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   https://www.egenix.com/company/contact/
 https://www.malemburg.com/
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/JRSWYO3X4JMBG3B6SWI6KWEDUYW5W5SY/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 624: Remove Py_UNICODE encoder APIs

2020-08-04 Thread M.-A. Lemburg

Hi Inada-san,

thanks for attending EuroPython. I won't be back online until
next Wednesday. Would it be possible to wait until then to continue
the discussion ?

Thanks,
-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...   http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...   http://zope.egenix.com/


::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
  http://www.malemburg.com/


On 04.08.2020 05:13, Inada Naoki wrote:
> Hi, Lemburg.
> 
> Thank you for organizing the EuroPython 2020.
> I enjoyed watching some sessions from home.
> 
> I think current PEP 624 covers all your points and ready for Steering
> Council discussion.
> Would you like to review the PEP before it?
> 
> Regards,
> 
> 
> On Thu, Jul 9, 2020 at 8:19 AM Inada Naoki  wrote:
>>
>> On Thu, Jul 9, 2020 at 5:46 AM M.-A. Lemburg  wrote:
>>> - the fact that the encode APIs encoding from a Unicode buffer
>>>   to a bytes object; this is an important fact, since the removal
>>>   removes access to this codec functionality for extensions
>>>
>>> - PyUnicode_AsEncodedString() is not a proper alternative, since
>>>   it requires to create a temporary PyUnicode object, which is
>>>   inefficient and wastes memory
>>
>> I wrote your points in the "Alternative Idea > Replace Py_UNICODE*
>> with Py_UCS4* "
>> section. I wrote "User can encode UCS-4 string in C without creating
>> Unicode object." in it.
>>
>> https://www.python.org/dev/peps/pep-0624/#replace-py-unicode-with-py-ucs4
>>
>> Note that the current Py_UNICODE* encoder APIs create temporary
>> PyUnicode objects.
>> They are inefficient and wastes memory now. Py_UNICODE* may be UTF-16 on some
>> platforms (e.g. Windows) and builtin codecs don't support UTF-16 input.
>>
>>
>>>
>>> - the maintenance effect mentioned in the PEP does not really
>>>   materialize, since the underlying functionality still exists
>>>   in the codecs - only access to the functionality is removed
>>>
>>
>> In the same section, I described the maintenance cost as below.
>>
>> * Other Python implementations may not have builtin codec for UCS-4.
>> * If we change the Unicode internal representation to UTF-8, we need
>> to keep UCS-4 support only for these APIs.
>>
>>> - keeping just the generic PyUnicode_Encode() API would be a
>>>   compromise
>>>
>>> - if we remove the codec specific PyUnicode_Encode*() APIs, why
>>>   are we still keeping the specisl PyUnicde_Decode*() APIs ?
>>>
>>
>> OK, I will add "Discussions" section. (I don't like "FAQ" because some 
>> question
>> are important even if it is not "frequently" asked.)
>>
>> Quick answer is:
>>
>> * They are stable ABI. (Py_UNICODE is excluded from stable ABI).
>> * Decoding from char* is more common and generic use case than encoding from
>>   Py_UNICODE*.
>> * Other Python implementations using UTF-8 as internal representation
>> can implement
>>   it easily.
>>
>> But I'm not opposite to remove it (especially for minor UTF-7 codec).
>> It is just out of scope of this PEP.
>>
>>
>>> - the deprecations were just done because the Py_UNICODE data
>>>   type was replaced by a hybrid type. Using this as an argument
>>>   for removing functionality is not really good practice, when
>>>   these are ways to continue exposing the functionality using other
>>>   data types.
>>
>> I hope the "Replace Py_UNICODE* with Py_UCS4* " section describe this.
>>
>> Regards,
>>
>> --
>> Inada Naoki  
> 
> 
> 
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/WZYG5X3MMJX6B7LWO6FXIJEORSYJSQYK/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 624: Remove Py_UNICODE encoder APIs

2020-07-08 Thread M.-A. Lemburg

Hi Inada-san,

I am currently too busy with EuroPython to participate in longer
discussions. FWIW: I intend to continue after EuroPython.

In any case, thanks for writing up the PEP. Could you please add my
points about:

- the fact that the encode APIs encoding from a Unicode buffer
  to a bytes object; this is an important fact, since the removal
  removes access to this codec functionality for extensions

- PyUnicode_AsEncodedString() is not a proper alternative, since
  it requires to create a temporary PyUnicode object, which is
  inefficient and wastes memory

- the maintenance effect mentioned in the PEP does not really
  materialize, since the underlying functionality still exists
  in the codecs - only access to the functionality is removed

- keeping just the generic PyUnicode_Encode() API would be a
  compromise

- if we remove the codec specific PyUnicode_Encode*() APIs, why
  are we still keeping the specisl PyUnicde_Decode*() APIs ?

- the deprecations were just done because the Py_UNICODE data
  type was replaced by a hybrid type. Using this as an argument
  for removing functionality is not really good practice, when
  these are ways to continue exposing the functionality using other
  data types.

I am still strongly -1 on removing all encoding APIs without
at least some upgrade path for existing code to use and keeping
the API symmetric.

Cheers,
-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...   http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...   http://zope.egenix.com/


::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
  http://www.malemburg.com/


On 07.07.2020 17:17, Inada Naoki wrote:
> Hi, folks.
> 
> Since the previous discussion was suspended without consensus, I wrote
> a new PEP for it. (Thank you Victor for reviewing it!)
> 
> This PEP looks very similar to PEP 623 "Remove wstr from Unicode",
> but for encoder APIs, not for Unicode object APIs.
> 
> URL (not available yet): https://www.python.org/dev/peps/pep-0624/
> 
> ---
> 
> PEP: 624
> Title: Remove Py_UNICODE encoder APIs
> Author: Inada Naoki 
> Status: Draft
> Type: Standards Track
> Content-Type: text/x-rst
> Created: 06-Jul-2020
> Python-Version: 3.11
> 
> 
> Abstract
> 
> 
> This PEP proposes to remove deprecated ``Py_UNICODE`` encoder APIs in
> Python 3.11:
> 
> * ``PyUnicode_Encode()``
> * ``PyUnicode_EncodeASCII()``
> * ``PyUnicode_EncodeLatin1()``
> * ``PyUnicode_EncodeUTF7()``
> * ``PyUnicode_EncodeUTF8()``
> * ``PyUnicode_EncodeUTF16()``
> * ``PyUnicode_EncodeUTF32()``
> * ``PyUnicode_EncodeUnicodeEscape()``
> * ``PyUnicode_EncodeRawUnicodeEscape()``
> * ``PyUnicode_EncodeCharmap()``
> * ``PyUnicode_TranslateCharmap()``
> * ``PyUnicode_EncodeDecimal()``
> * ``PyUnicode_TransformDecimalToASCII()``
> 
> .. note::
> 
>`PEP 623  `_ propose to remove
>Unicode object APIs relating to ``Py_UNICODE``. On the other hand, this PEP
>is not relating to Unicode object. These PEPs are split because they have
>different motivation and need different discussion.
> 
> 
> Motivation
> ==
> 
> In general, reducing the number of APIs that have been deprecated for
> a long time and have few users is a good idea for not only it
> improves the maintainability of CPython, but it also helps API users
> and other Python implementations.
> 
> 
> Rationale
> =
> 
> Deprecated since Python 3.3
> ---
> 
> ``Py_UNICODE`` and APIs using it are deprecated since Python 3.3.
> 
> 
> Inefficient
> ---
> 
> All of these APIs are implemented using ``PyUnicode_FromWideChar``.
> So these APIs are inefficient when user want to encode Unicode
> object.
> 
> 
> Not used widely
> ---
> 
> When searching from top 4000 PyPI packages [1]_, only pyodbc use
> these APIs.
> 
> * ``PyUnicode_EncodeUTF8()``
> * ``PyUnicode_EncodeUTF16()``
> 
> pyodbc uses these APIs to encode Unicode object into bytes object.
> So it is easy to fix it. [2]_
> 
> 
> Alternative APIs
> 
> 
> There are alternative APIs to accept ``PyObject *unicode`` instead of
> ``Py_UNICODE *``. Users can migrate to them.
> 
> 
> =
> ==
> Deprecated APIAlternative APIs
> =
> ==
> ``PyUnicode_Encode()``

[Python-Dev] Re: Plan to remove Py_UNICODE APis except PEP 623.

2020-07-01 Thread M.-A. Lemburg

On 30.06.2020 15:17, Victor Stinner wrote:
> Le mar. 30 juin 2020 à 13:53, M.-A. Lemburg  a écrit :
>>> I would prefer to analyze the list on a case by case basis. I don't
>>> think that it's useful to expose every single encoding supported by
>>> Python as a C function.
>>
>> (...)
>> This does not mean we have to give up the symmetry in the C API,
>> or that the encoding APIs are now suddenly useless. It only means
>> that we have to replace Py_UNICODE with one of the supported data
>> for storing Unicode.
> 
> Let's agree to disagree :-)
> 
> I don't think that completeness is a good rationale to design the C API.

Oh, if that's your opinion, then we definitely disagree :-)

I strongly believe that the success of Python was in major parts
built on the fact that Python does have a complete and easily
usable C API.

Without this, Python would never have convinced
the "Python is slow" advocates that you can actually build
fast applications in Python by using Python to orchestrate and
integrate with low level C libraries, and we'd be regarded as
yet another Tcl.

> The C API is too large, we have to make it smaller.

That*s a different discussion, but disagree on that perspective
as well: we have to refactor parts of the Python C API to make it
more consistent and remove hacks which developers sometimes added
as helper functions without considering the big picture approach.

The Unicode API has over the year grown a lot of such helpers
and there's certainly room for improvement, but simply ripping out
things is not always the right answer, esp. not when you touch the
very core of the design.

> A specialized
> function, like PyUnicode_AsUTF8String(), can be justified by different
> reasons:
> 
> * It is a very common use case and so it helps to write C extensions
> * It is significantly faster than the alternative generic function
> 
> In C, you can execute arbitrary Python code by calling methods on
> Python str objects. For example, "abc".encode("utf-8",
> "surrogateescape") in Python becomes PyObject_CallMethod(obj,
> "encode", "ss", "utf-8", "surrogatepass") in C. Well, there is already
> a more specialized and generic PyUnicode_AsEncodedObject() function.

You know as well as I do, that the Python call mechanism is by far
the slowest part in the Python C API, so telling developers to
use this as the main way to run tasks which can be run much faster,
easier and with less memory overhead or copying of data by directly
calling a simple C API, is not a good way to advocate for a useful
Python C API.

> We must not add a C API function for every single Python feature,
> otherwise it would be too expensive to maintain, and it would become
> impossible for other Python implementations to implement the fully C
> API. Well, even today, PyPy already only implements a small subset of
> the C API.

I honestly don't think that other Python implementations should
even try to implement the Python C API. Instead, they should build
a bridge to use the CPython runtime and integrate this into their
system.
>> Since the C world has adopted wchar_t for this purpose, it's the
>> natural choice.
> 
> In my experience, in C extensions, there are two kind of data:
> 
> * bytes is used as a "char*": array of bytes
> * Unicode is used as a Python object

Uhm, what about all those applications, libraries and OS calls
producing Unicode data ? It is not always feasible or even desired
to first convert this into a Python Unicode object.

> For the very rare cases involving wchar_t*, PyUnicode_FromWideChar()
> can be used. I don't think that performance justifies to duplicate
> each function, once for a Python str object, once for wchar_t*. I
> mostly saw code involving wchar_t* to initialize Python. But this code
> was wrong since it used PyUnicode function *before* Python was
> initialized. That's bad and can now crash in recent Python versions.

But that*s an entirely unrelated issue, right ? The C lib has
full support for wchar_t and provides plenty of APIs for using
it. The main() invocation is just one small part of the lib C
Unicode system.

> The new PEP 587 has a different design and avoids Python objects and
> anything related to the Python runtime:
> https://docs.python.org/dev/c-api/init_config.html#c.PyConfig_SetString
> 
> Moreover, CPython implements functions taking wchar_t* string by
> calling PyUnicode_FromWideChar() internally...

I mentioned wchar_t as buffer input replacement for the
PyUnicode_Encode*() API as alternative to the deprecated
Py_UNICODE.

Of course, you can convert all whcar_t data into a Python Unicode
object first and then apply operations on this, but the point of
the encode APIs is to have a low-level ac

[Python-Dev] Re: Plan to remove Py_UNICODE APis except PEP 623.

2020-07-01 Thread M.-A. Lemburg

On 28.06.2020 16:24, Inada Naoki wrote:
> Hi, Lamburg.
> 
> Thank you for quick response.
> 
>>
>> We can't just remove access to one half of a codec (the decoding
>> part) without at least providing an alternative for C extensions
>> to use.
>>
>> Py_UNICODE can be removed from the API, but only if there are
>> alternative APIs which C extensions can use to the same effect.
>>
>> Given PEP 393, this would be APIs which use wchar_t instead of
>> Py_UNICODE.
>>
> 
> Decoding part is implemented as `const char *` -> `PyObject*` (Unicode 
> object).
> I think this is reasonable since `const char *` is perfect to abstract
> the encoded string,
> 
> In case of encoding part, `wchar_t *` is not perfect abstraction for
> (decoded) unicode
> string.

Note that the PyUnicode_Encode*() APIs are meant to be make the
codec's encoding machinery available to C extensions, so that they
don't have to implement this again.

In that sense, their purpose is not to encode an existing Unicode
object, but instead, to provide access to the low-level buffer to
bytes object encoding.

The reasoning here is the same as for decoding: you have the original
data you want to process available in some array and want to turn
this into the Python object.

The path Victor suggested requires always going via a Python Unicode
object, but that it very expensive and not really an appropriate
way to address the use case.

As an example application, think of a database module which provides
the Unicode data as Py_UNICODE buffer. You want to write this as UTF-8
data to a file or a socket, so you have the PyUnicode_EncodeUTF8() API
decode this for you into a bytes object which you can then write out
using the Python C APIs for this.

>  Converting from Unicode object into `wchar_t *` is not zero-cost.
> I think `PyObject *` (Unicode object) -> `PyObject *` (bytes object)
> looks better signature than
> `wchar_t *` -> `Pyobject *` (bytes object) because for encoders.

See above. The motivation for these APIs is different. They are
not about taking a Unicode object and converting it into bytes,
they are deliberately about taking a data buffer as input and
producing the Python bytes object as output (to implement symmetry
between decoding and encoding).

> * Unicode object is more important than `wchar_t *` in Python.

Right, but as I tried to explain in my reply to Victor, I designed
the Unicode API in Python to be a rich API, which provides all
necessary tools to easily work with Unicode in C extensions as
well as in the CPython interpreter.

The API is not only focused on what the CPython interpreter needs.
It's an API which implements a concise interface to Unicode as
used in Python.

> * All PyUnicode_EncodeXXX APIs are implemented with PyUnicode_FromWideChar.
> 
> For example, we have these private encode APIs:
> 
> * PyObject* _PyUnicode_AsAsciiString(PyObject *unicode, const char *errors)
> * PyObject* _PyUnicode_AsLatin1String(PyObject *unicode, const char *errors)
> * PyObject* _PyUnicode_AsUTF8String(PyObject *unicode, const char *errors)
> * PyObject* _PyUnicode_EncodeUTF16(PyObject *unicode, const char
> *errors, int byteorder)
> ...
> 
> So how about making them public, instead of undeprecate Py_UNICODE* encode 
> APIs?

I'd be fine with keeping just a generic PyUnicode_Encode() API,
but this should then be encoding from a buffer to a bytes object.

The above all take Unicode objects as input and create the same
problem as I described above, with the temporary Unicode object being
created and all the associated malloc and scanning overhead needed
for this.

The reason I mention wchar_t as new basis for the PyUnicde_Encode()
API is that whcar_t has grown to be accepted as the standard for
Unicode buffers in C. If you don't believe that this is good enough,
we could also force Py_UCS4, but this would alienate Windows extension
writers.

> 1. Add PyUnicode_AsXXXBytes public APIs in Python 3.10.
>Current private APIs can become macro (e.g. #define
> _PyUnicode_AsAsciiString PyUnicode_AsAsciiBytes),
>or deprecated static inline function.
> 2. Remove Py_UNICODE* encode APIs in Python 3.12.

FWIW: I don't object to deprecating Py_UNICODE. I just don't
want to lose the symmetry in decoding/encoding and add the cost
of having to go via a Python Unicode object just to decode
to bytes.

Thanks,
-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...   http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...   http://zope.egenix.com/

::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611

[Python-Dev] Re: Plan to remove Py_UNICODE APis except PEP 623.

2020-06-30 Thread M.-A. Lemburg

On 29.06.2020 11:57, Victor Stinner wrote:
> Le dim. 28 juin 2020 à 11:22, M.-A. Lemburg  a écrit :
>> as you may remember, I wasn't happy with the deprecations of the
>> APIs in PEP 393, since there are no C API alternatives for
>> the encoding APIs deprecated in the PEP, which allow direct
>> encoding provided by these important codecs.
>>
>> AFAIK, the situation hasn't changed since then.
> 
> I would prefer to analyze the list on a case by case basis. I don't
> think that it's useful to expose every single encoding supported by
> Python as a C function.

I designed the Unicode C API as a rich API, so that it's easy
to use from C extensions and the interpreter as well.

The main theme was to have symmetric API for both encoding and
decoding. The PEP now suggests to remove the API on the basis of
deprecating Py_UNICODE, which is a change in data type.

This does not mean we have to give up the symmetry in the C API,
or that the encoding APIs are now suddenly useless. It only means
that we have to replace Py_UNICODE with one of the supported data
for storing Unicode.

Since the C world has adopted wchar_t for this purpose, it's the
natural choice.

> I would prefer to only have a fast-path for the most common encodings:
> ASCII, Latin1, UTF-8, Windows ANSI code page. That's all.
> 
> For any other encodings, the general PyUnicode_AsEncodedString() and
> PyUnicode_Decode() function are good enough.

PyUnicode_AsEncodedString() converts Unicode objects to a
bytes object. This is not an symmetric replacement for the
PyUnicode_Encode*() APIs, since those go from Py_UNICODE to
a bytes object.

> If someone expects an overhead of passing a string, please prove it
> with a benchmark. But IMO a small overhead is acceptable for rare
> encodings.
> 
> Note: PyUnicode_AsEncodedString() and PyUnicode_Decode() also have
> "fast paths" for most common encodings: ASCII, UTF-8, "mbcs" (Python
> alias of the Windows ANSI code page), Latin1. But also UTF-16 and
> UTF-32: I'm not if it's really worth it to have these ones, but it was
> cheap to have them :-)
> 
> 
>> We can't just remove access to one half of a codec (the decoding
>> part) without at least providing an alternative for C extensions

Sorry, I meant the "encoding part".

>> to use.
> 
> I disagree, we can. The alternative exists since Python 2:
> PyUnicode_AsEncodedString() and PyUnicode_Decode().

See above.

If we remove the direct encoding/decoding C APIs we should at the
very least provide generic alternatives which can be used as drop-in
replacement for the PyUnicde_Encode*() APIs.

>> Given PEP 393, this would be APIs which use wchar_t instead of
>> Py_UNICODE.
> 
> Using wchar_t is inefficient on all platforms using 16-bit wchar_t
> since surrogate pairs need a special code path. For example,
> PyUnicode_FromWideChar() has to scan the string twice: the first time
> to count the number of surrogate pairs, to allocate the exact buffer
> size.

If you want full UCS4 compatibility, that's true, but those platforms
suffer from this deficiency platform wide, so Python is in no way
special.

The main point is that wchar_t is the standard in C to represent
Unicode code points, so it's a natural choice as replacement for
Py_UNICODE.

Since the C API is not only meant to be used by the CPython interpreter,
we should stick to standards rather than expecting the world to adapt
to our implementations. This also makes the APIs future proof, e.g.
in case we make another transition from the current hybrid internal
data type for Unicode towards UTF-8 buffers as internal data type.

Cheers,
-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...   http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...   http://zope.egenix.com/

::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
  http://www.malemburg.com/
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/XWYHZK6LPAM2MV3E7AXGKZSIPJ43MMFX/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Plan to remove Py_UNICODE APis except PEP 623.

2020-06-28 Thread M.-A. Lemburg

Hi Inada-san,

as you may remember, I wasn't happy with the deprecations of the
APIs in PEP 393, since there are no C API alternatives for
the encoding APIs deprecated in the PEP, which allow direct
encoding provided by these important codecs.

AFAIK, the situation hasn't changed since then.

We can't just remove access to one half of a codec (the decoding
part) without at least providing an alternative for C extensions
to use.

Py_UNICODE can be removed from the API, but only if there are
alternative APIs which C extensions can use to the same effect.

Given PEP 393, this would be APIs which use wchar_t instead of
Py_UNICODE.

Thanks,
-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Jun 28 2020)
>>> Python Projects, Coaching and Support ...https://www.egenix.com/
>>> Python Product Development ...https://consulting.egenix.com/


::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   https://www.egenix.com/company/contact/
 https://www.malemburg.com/


On 28.06.2020 04:35, Inada Naoki wrote:
> Hi, all.
> 
> I proposed PEP 623 to remove Unicode APIs deprecated by PEP 393.
> 
> In this thread, I am proposing removal of Py_UNICODE (not Unicode
> objects) APIs deprecated by PEP 393.
> Please reply for any comments.
> 
> 
> ## Undocumented, have Py_DEPRECATED
> 
> There is no problem to remove them in Python 3.10.  I will just do it.
> 
> * Py_UNICODE_str*** functions -- already removed in
> https://github.com/python/cpython/pull/21164
> * PyUnicode_GetMax()
> 
> 
> ## Documented and have Py_DEPRECATED
> 
> * PyLong_FromUnicode
> * PyUnicode_AsUnicodeCopy
> * PyUnicode_Encode
> * PyUnicode_EncodeUTF7
> * PyUnicode_EncodeUTF8
> * PyUnicode_EncodeUTF16
> * PyUnicode_EncodeUTF32
> * PyUnicode_EncodeUnicodeEscape
> * PyUnicode_EncodeRawUnicodeEscape
> * PyUnicode_EncodeLatin1
> * PyUnicode_EncodeASCII
> * PyUnicode_EncodeCharmap
> * PyUnicode_TranslateCharmap
> * PyUnicode_EncodeMBCS
> 
> These APIs are documented.  The document has ``.. deprecated:: 3.3
> 4.0`` directive.
> They have been `Py_DEPRECATED` since Python 3.6 too.
> 
> Plan: Change the document to ``.. deprecated:: 3.0 3.10`` and remove
> them in Python 3.10.
> 
> 
> ## PyUnicode_EncodeDecimal
> 
> It is not documented.  It has not been deprecated by Py_DEPRECATED.
> 
> Plan: Add Py_DEPRECATED in Python 3.9 and remove it in 3.11.
> 
> 
> ## PyUnicode_TransformDecimalToASCII
> 
> It is documented, but doesn't have ``deprecated`` directive. It is not
> deprecated by Py_DEPRECATED.
> 
> Plan: Add Py_DEPRECATED and ``deprecated 3.3 3.11`` directive in 3.9,
> and remove it in 3.11.
> 
> 
> ## _PyUnicode_ToLowercase, _PyUnicode_ToUppercase
> 
> They are not deprecated by PEP 393, but bpo-12736.
> They are documented as deprecated, but don't have ``Py_DEPRECATED``.
> 
> Plan: Add Py_DEPRECATED in 3.9, and remove them in 3.11.
> 
> Note: _PyUnicode_ToTitlecase has Py_DEPRECATED. It can be removed in 3.10.
> 
> 
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/RFVZQJIM3AZ4IJDC6QKWFH4PQZYQGRQD/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 622: Structural Pattern Matching

2020-06-24 Thread M.-A. Lemburg

On 24.06.2020 20:08, Guido van Rossum wrote:
> On Wed, Jun 24, 2020 at 7:27 AM M.-A. Lemburg  <mailto:m...@egenix.com>> wrote:
> 
> Wow, so 19 years after PEP 275, we are indeed getting a switch
> statement. Nice :-)
> 
> 
> Indeed. Fortunately there are now some better ideas to steal from other
> languages than C's switch. :-)

Your PEP certainly is a lot more powerful than the good ol' C
switch :-)

Something I know the Perl camp is always very fond of is the
matching on regexps. Is this possible using the proposal (sorry,
I don't quite understand the __match__() protocol yet) ?

> Something which struck me as odd when first scanning through the PEP
> is the default case compared to other Python block statements:
> 
> match something:
>     case 0 | 1 | 2:
>         print("Small number")
>     case [] | [_]:
>         print("A short sequence")
>     case str() | bytes():
>         print("Something string-like")
>     case _:
>         print("Something else")
> 
> rather than what a Pythonista would probably expect:
> 
> match something:
>     case 0 | 1 | 2:
>         print("Small number")
>     case [] | [_]:
>         print("A short sequence")
>     case str() | bytes():
>         print("Something string-like")
>     else:
>         print("Something else")
> 
> Was there a reason for using a special value "_" as match-all value ?
> I couldn't find any explanation for this in the PEP.
> 
> 
> Nearly every other language whose pattern matching syntax we've examined
> uses _ as the wildcard.
> 
> The authors don't feel very strongly about whether to use `else:` or
> `case _:`. The latter would be possible even if we added an explicit
> `else` clause, and we like TOOWTDI. But it's clear that a lot of people
> *expect* to see `else`, and maybe seeing `case _:` is not the best
> introduction to wildcards for people who haven't seen a match statement
> before.
> 
> A wrinkle with `else` is that some of the authors would prefer to see it
> aligned with `match` rather than with the list of cases, but for others
> it feels like a degenerate case and should be aligned with those. (I'm
> in the latter camp.)
> 
> There still is a lively internal discussion going on, and we'll get back
> here when we have a shared opinion.

Great. Thanks for considering it.

I'd make "else" match its use in other statements, which would
mean aligning it with "match", but don't really feel strong about
either way.

The problem I see with "case _" is that it's just too easy to
miss when looking at the body of "match", even more so, since
people will not necessarily put it at the end, or add it
as or'ed add-on to some other case, e.g.

match something:
case 0 | 1 | 2 | _:
print("Small number or something else")
case [] | [_]:
print("A short sequence")
case _:
print("Not sure what this is")
case str() | bytes():
print("Something string-like")

You could even declare the above stand-alone or or'ed
use of "_" illegal and force use of "else" instead to push
for TOOWTDI.

Oh, and thanks for not having continue / break in the switch cases !
Those tend to often cause subtle bugs in C applications (ie. a
missing break).

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Jun 24 2020)
>>> Python Projects, Coaching and Support ...https://www.egenix.com/
>>> Python Product Development ...https://consulting.egenix.com/

::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   https://www.egenix.com/company/contact/
 https://www.malemburg.com/
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/UABXTHJR6O3PB4ARF5QDLBQYXZ2PA3QL/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 622: Structural Pattern Matching

2020-06-24 Thread M.-A. Lemburg

On 24.06.2020 16:27, M.-A. Lemburg wrote:
> Wow, so 19 years after PEP 275, we are indeed getting a switch
> statement. Nice :-)
> 
> Something which struck me as odd when first scanning through the PEP
> is the default case compared to other Python block statements:
> 
> match something:
> case 0 | 1 | 2:
> print("Small number")
> case [] | [_]:
> print("A short sequence")
> case str() | bytes():
> print("Something string-like")
> case _:
> print("Something else")
> 
> rather than what a Pythonista would probably expect:
> 
> match something:
> case 0 | 1 | 2:
> print("Small number")
> case [] | [_]:
> print("A short sequence")
> case str() | bytes():
> print("Something string-like")
> else:
> print("Something else")
> 
> Was there a reason for using a special value "_" as match-all value ?
> I couldn't find any explanation for this in the PEP.

To clarify:

The Python compiler could turn the "else:" into what
"case _:" would produce. The syntax would just look
more intuitive, IMO.

The question was not about using "_" as match-all in general.

> Cheers.
> 
> 
> 
> On 23.06.2020 18:01, Guido van Rossum wrote:
>> I'm happy to present a new PEP for the python-dev community to review.
>> This is joint work with Brandt Bucher, Tobias Kohn, Ivan Levkivskyi and
>> Talin.
>>
>> Many people have thought about extending Python with a form of pattern
>> matching similar to that found in Scala, Rust, F#, Haskell and other
>> languages with a functional flavor. The topic has come up regularly on
>> python-ideas (most recently yesterday :-).
>>
>> I'll mostly let the PEP speak for itself:
>> - Published: https://www.python.org/dev/peps/pep-0622/ (*)
>> - Source: https://github.com/python/peps/blob/master/pep-0622.rst
>>
>> (*) The published version will hopefully be available soon.
>>
>> I want to clarify that the design space for such a match statement is
>> enormous. For many key decisions the authors have clashed, in some cases
>> we have gone back and forth several times, and a few uncomfortable
>> compromises were struck. It is quite possible that some major design
>> decisions will have to be revisited before this PEP can be accepted.
>> Nevertheless, we're happy with the current proposal, and we have
>> provided ample discussion in the PEP under the headings of Rejected
>> Ideas and Deferred Ideas. Please read those before proposing changes!
>>
>> I'd like to end with the contents of the README of the repo where we've
>> worked on the draft, which is shorter and gives a gentler introduction
>> than the PEP itself:
>>
>>
>> # Pattern Matching
>>
>> This repo contains a draft PEP proposing a `match` statement.
>>
>> Origins
>> ---
>>
>> The work has several origins:
>>
>> - Many statically compiled languages (especially functional ones) have
>>   a `match` expression, for example
>>  
>> [Scala](http://www.scala-lang.org/files/archive/spec/2.11/08-pattern-matching.html),
>>   [Rust](https://doc.rust-lang.org/reference/expressions/match-expr.html),
>>  
>> [F#](https://docs.microsoft.com/en-us/dotnet/fsharp/language-reference/pattern-matching);
>> - Several extensive discussions on python-ideas, culminating in a
>>   summarizing
>>   [blog
>> post](https://tobiaskohn.ch/index.php/2018/09/18/pattern-matching-syntax-in-python/)
>>   by Tobias Kohn;
>> - An independently developed [draft
>>  
>> PEP](https://github.com/ilevkivskyi/peps/blob/pattern-matching/pep-.rst)
>>   by Ivan Levkivskyi.
>>
>> Implementation
>> --
>>
>> A full reference implementation written by Brandt Bucher is available
>> as a [fork]((https://github.com/brandtbucher/cpython/tree/patma)) of
>> the CPython repo.  This is readily converted to a [pull
>> request](https://github.com/brandtbucher/cpython/pull/2)).
>>
>> Examples
>> 
>>
>> Some [example
>> code](https://github.com/gvanrossum/patma/tree/master/examples/) is
>> available from this repo.
>>
>> Tutorial
>> 
>>
>> A `match` statement takes an expression and compares it to successive
>> patterns given as one or more `case` blocks.  This is superficially
>> similar to a `switch` statement in C, Java or JavaScript (an many
>> other languages), but much more powerful.
>>
>>

[Python-Dev] Re: PEP 622: Structural Pattern Matching

2020-06-24 Thread M.-A. Lemburg

Wow, so 19 years after PEP 275, we are indeed getting a switch
statement. Nice :-)

Something which struck me as odd when first scanning through the PEP
is the default case compared to other Python block statements:

match something:
case 0 | 1 | 2:
print("Small number")
case [] | [_]:
print("A short sequence")
case str() | bytes():
print("Something string-like")
case _:
print("Something else")

rather than what a Pythonista would probably expect:

match something:
case 0 | 1 | 2:
print("Small number")
case [] | [_]:
print("A short sequence")
case str() | bytes():
print("Something string-like")
else:
print("Something else")

Was there a reason for using a special value "_" as match-all value ?
I couldn't find any explanation for this in the PEP.

Cheers.



On 23.06.2020 18:01, Guido van Rossum wrote:
> I'm happy to present a new PEP for the python-dev community to review.
> This is joint work with Brandt Bucher, Tobias Kohn, Ivan Levkivskyi and
> Talin.
> 
> Many people have thought about extending Python with a form of pattern
> matching similar to that found in Scala, Rust, F#, Haskell and other
> languages with a functional flavor. The topic has come up regularly on
> python-ideas (most recently yesterday :-).
> 
> I'll mostly let the PEP speak for itself:
> - Published: https://www.python.org/dev/peps/pep-0622/ (*)
> - Source: https://github.com/python/peps/blob/master/pep-0622.rst
> 
> (*) The published version will hopefully be available soon.
> 
> I want to clarify that the design space for such a match statement is
> enormous. For many key decisions the authors have clashed, in some cases
> we have gone back and forth several times, and a few uncomfortable
> compromises were struck. It is quite possible that some major design
> decisions will have to be revisited before this PEP can be accepted.
> Nevertheless, we're happy with the current proposal, and we have
> provided ample discussion in the PEP under the headings of Rejected
> Ideas and Deferred Ideas. Please read those before proposing changes!
> 
> I'd like to end with the contents of the README of the repo where we've
> worked on the draft, which is shorter and gives a gentler introduction
> than the PEP itself:
> 
> 
> # Pattern Matching
> 
> This repo contains a draft PEP proposing a `match` statement.
> 
> Origins
> ---
> 
> The work has several origins:
> 
> - Many statically compiled languages (especially functional ones) have
>   a `match` expression, for example
>  
> [Scala](http://www.scala-lang.org/files/archive/spec/2.11/08-pattern-matching.html),
>   [Rust](https://doc.rust-lang.org/reference/expressions/match-expr.html),
>  
> [F#](https://docs.microsoft.com/en-us/dotnet/fsharp/language-reference/pattern-matching);
> - Several extensive discussions on python-ideas, culminating in a
>   summarizing
>   [blog
> post](https://tobiaskohn.ch/index.php/2018/09/18/pattern-matching-syntax-in-python/)
>   by Tobias Kohn;
> - An independently developed [draft
>  
> PEP](https://github.com/ilevkivskyi/peps/blob/pattern-matching/pep-.rst)
>   by Ivan Levkivskyi.
> 
> Implementation
> --
> 
> A full reference implementation written by Brandt Bucher is available
> as a [fork]((https://github.com/brandtbucher/cpython/tree/patma)) of
> the CPython repo.  This is readily converted to a [pull
> request](https://github.com/brandtbucher/cpython/pull/2)).
> 
> Examples
> 
> 
> Some [example
> code](https://github.com/gvanrossum/patma/tree/master/examples/) is
> available from this repo.
> 
> Tutorial
> 
> 
> A `match` statement takes an expression and compares it to successive
> patterns given as one or more `case` blocks.  This is superficially
> similar to a `switch` statement in C, Java or JavaScript (an many
> other languages), but much more powerful.
> 
> The simplest form compares a target value against one or more literals:
> 
> ```py
> def http_error(status):
>     match status:
>         case 400:
>             return "Bad request"
>         case 401:
>             return "Unauthorized"
>         case 403:
>             return "Forbidden"
>         case 404:
>             return "Not found"
>         case 418:
>             return "I'm a teapot"
>         case _:
>             return "Something else"
> ```
> 
> Note the last block: the "variable name" `_` acts as a *wildcard* and
> never fails to match.
> 
> You can combine several literals in a single pattern using `|` ("or"):
> 
> ```py
>         case 401|403|404:
>             return "Not allowed"
> ```
> 
> Patterns can look like unpacking assignments, and can be used to bind
> variables:
> 
> ```py
> # The target is an (x, y) tuple
> match point:
>     case (0, 0):
>         print("Origin")
>     case (0, y):
>         print(f"Y={y}")
>     case (x, 0):
>         print(f"X={x}")
>     case (x, y):
>         print(f"X={x}, Y={y}")
>     case _:
>

[Python-Dev] Re: [capi-sig] Re: Removal of _Py_ForgetReference from public header in 3.9 issue

2020-06-15 Thread M.-A. Lemburg

On 15.06.2020 11:02, Petr Viktorin wrote:
> On 2020-06-14 22:10, cpyt...@nicwatson.org wrote:
>> I maintain an open source Python module in C. I'm trying to verify for
>> the first time that the module still works with cpython 3.9. This
>> module does *not* use the "limited" C API.
>>
>> In building my module against 3.9b3, I'm getting a missing declaration
>> warning on _Py_ForgetReference. My module builds and passes test fine,
>> this is just a compiler warning issue.
> 
> What does the _Py_ForgetReference function do? The [documentation] says
> it's only for use in the interpereter core, so I'd assume it's .
> 
> [documentation]: https://docs.python.org/3/c-api/refcounting.html

This is typically used in error handlers of constructors where you
want to avoid having the deallocator run as result of the
DECREF, e.g. because the object has not been full initialized
yet.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Jun 15 2020)
>>> Python Projects, Coaching and Support ...https://www.egenix.com/
>>> Python Product Development ...https://consulting.egenix.com/

::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   https://www.egenix.com/company/contact/
 https://www.malemburg.com/
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/PEFTCHDRJQDATHG5LKUCBA6PTS3HOHJP/
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-Dev] Adding a tzidx cache to datetime

2019-05-10 Thread M.-A. Lemburg

On 10.05.2019 02:58, Paul Ganssle wrote:
> This is only "only" for dateutil in the sense that no one other than
> dateutil implements tzinfo with the interface provided. If dateutil were
> /not/ already implemented with a list of offsets and their indexes, I
> would still propose this, and just re-write dateutil to take advantage
> of it. From a cursory glance at pendulum, it seems that they could take
> advantage of it as well (though they use their own datetime subclass, so
> they have always had the ability to add this).
> 
>> What do you think of adding a private "_cache" attribute which would
>> be an arbitrary Python object? (None by default)
> 
> We cannot use a private attribute (other than to do the actual storage,
> since the thing that gets stored is not directly accessible anyway and
> is instead mediated by a layer that manages the cache) because this is a
> feature explicitly being added for use by tzinfo, /not/ by datetime. If
> it's private then it's not safe for implementations of tzinfo to
> actually use it, which defeats the purpose.
> 
> Regarding the use of an arbitrary Python object: What I'm proposing is
> that we offer a bit of the "free" storage space in the alignment bits to
> tzinfo objects to use as a cache. In /most/ cases this will be very
> useful to someone implementing a tzinfo, because there are really only
> so many ways to accomplish this task, and most time zones are
> expressible as a very short list of offset/name/dst combinations, plus
> some rule for which applies when, which is why a small integer cache is
> sufficient and more or less universal (i.e. not specific to dateutil's
> implementation).
> 
> I will also note that in my design, it is still possible for `tzinfo` to
> return something other than [0, 254], it's just that that information
> will not be cached, so it won't get the benefit of any optimization, but
> the same interface / implementation can be used.
> 
> In my test with gcc, adding an additional PyObject* to the end of the
> PyDateTime_DateTime struct increased the size of the `datetime.datetime`
> object from 64 to 72 bytes, whereas adding an `unsigned char` after the
> `fold` leaves it unchanged. Given that the expansion to arbitrary Python
> objects is speculative and doesn't have any particular use case, I would
> prefer to leave the feature as is, and reconsider the possibility of
> storing arbitrary Python objects on the datetime if there's some
> compelling reason to do so (it would be a backwards-compatible change at
> that point anyway).

Given that many datetime objects in practice don't use timezones
(e.g. in large data stores you typically use UTC and naive datetime
objects), I think that making the object itself larger to accommodate
for a cache, which will only be used a smaller percentage of the use
cases, isn't warranted. Going from 64 bytes to 72 bytes also sounds
like this could have negative effects on cache lines.

If you need a per object cache, you can either use weakref
objects or maintain a separate dictionary in dateutil or other
timezone helpers which indexes objects by id(obj).

That said, if you only add a byte field which doesn't make the object
larger in practice (you merely use space that alignments would
use anyway), this shouldn't be a problem. The use of that field
should be documented, though, so that other implementations can
use/provide it as well.

Thanks,
-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, May 10 2019)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...   http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...   http://zope.egenix.com/


::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
  http://www.malemburg.com/


> On 5/9/19 8:14 PM, Victor Stinner wrote:
>> Hi Paul,
>>
>> The change is basically an optimization. I'm uncomfortable to design
>> it "only" for dateutil. What if tomorrow someone has to store an
>> arbitrary Python object, rather than just an integer (in range [0;
>> 254]), into a datetime for a different optimization?
>>
>> Moreover, I dislike adding a *public* method for an *internal* cache.
>>
>> Right now, it is not possible to create a weak reference to a
>> datetime. If we make it possible, it would be possible to have an
>> external cache implemented with weakref.WeakSet to clear old entries
>> when a datetime object is detroyed.
>>
>> What do you think of adding a private "_cache" attribute which would
>> be an arbitrary Python object? (None by default)
>>
>> Victor
>>
>> Le mar. 7 mai

Re: [Python-Dev] Farewell, Python 3.4

2019-05-08 Thread M.-A. Lemburg

Thank you for having been 3.4 release manager, Larry !

On 08.05.2019 17:36, Larry Hastings wrote:
> 
> It's with a note of sadness that I announce the final retirement of
> Python 3.4.  The final release was back in March, but I didn't get
> around to actually closing and deleting the 3.4 branch until this morning.
> 
> Python 3.4 introduced many features we all enjoy in modern Python--the
> asyncio, ensurepip, and enum packages, just to name three.  It's a
> release I hope we all remember fondly.
> 
> My eternal thanks to all the members of the release team that worked on
> Python 3.4:
> 
> Georg Brandl
> 
> Julien Palard
> 
> Martin von Löwis
> 
> Ned Deily
> 
> Steve Dower
> 
> Terry Reedy
> 
> and all the engineers of the Python infrastructure team.
> 
> Special thanks to Benjamin Peterson and Ned Deily, who frequently
> scurried around behind the scenes cleaning up the messes I cluelessly
> left in my wake.
> 
> Having closed 3.4, I am now retired as Python 3.4 Release Manager.  I
> regret to inform all of you that you're still stuck with me as Python
> 3.5 Release Manager until sometime next year.
> 
> 
> My very best wishes,
> 
> 
> //arry/
> 
> 
> 
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/mal%40egenix.com
> 

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, May 08 2019)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...   http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...   http://zope.egenix.com/


::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
  http://www.malemburg.com/

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Experiment an opt-in new C API for Python? (leave current API unchanged)

2018-11-19 Thread M.-A. Lemburg

On 19.11.2018 11:53, Antoine Pitrou wrote:
> On Mon, 19 Nov 2018 11:28:46 +0100
> Victor Stinner  wrote:
>> Python internals rely on internals to implement further optimizations,
>> than modifying an "immutable" tuple, bytes or str object, because you
>> can do that at the C level. But I'm not sure that I would like 3rd
>> party extensions to rely on such things.
> 
> I'm not even talking about *modifying* tuples or str objects, I'm
> talking about *accessing* their value without going through an abstract
> API that does slot lookups, indirect function calls and object unboxing.
> 
> For example, people may need a fast way to access the UTF-8
> representation of a unicode object.  Without making indirect function
> calls, and ideally without making a copy of the data either.  How do
> you do that using the generic C API?

Something else you need to consider is creating instances of
types, e.g. a tuple. In C you will have to be able to put
values into the data structure before it is passed outside
the function in order to build the tuple.

If you remove this possibility to have to copy data all the
time, losing the advantages of having a rich C API.
 --
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Nov 19 2018)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...   http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...   http://zope.egenix.com/


::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
  http://www.malemburg.com/

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Microsoft to acquire GitHub for $7.5 billion

2018-06-05 Thread M.-A. Lemburg

Something that may change is the way they treat Github
accounts, after all, MS is very much a sales driven company.

But then there's always the possibility to move to Gitlab
as alternative (hosted or run on PSF VMs), so I would
worry too much.

Do note, however, that the value in Github is not so much with
the products they have, but with the data. Their databases
know more about IT developer than anyone else and given
that Github is put under the AI umbrella in MS should tell
us something :-)


On 04.06.2018 19:02, Antoine Pitrou wrote:
> 
> That's true, but Microsoft has a lot of stakes in the ecosystem.
> For example, since it has its own CI service that it tries to promote
> (VSTS), is it in Microsoft's best interest to polish and improve
> integrations with other CI services?
> 
> Regards
> 
> Antoine.
> 
> 
> On Mon, 4 Jun 2018 09:06:28 -0700
> Guido van Rossum  wrote:
>> On Mon, Jun 4, 2018 at 8:40 AM, Antoine Pitrou  wrote:
>>
>>>
>>> On Mon, 4 Jun 2018 17:03:27 +0200
>>> Victor Stinner  wrote:  

 At this point, I have no opinion about the event :-) I just guess that
 it should make GitHub more sustainable since Microsoft is a big
 company with money and interest in GitHub. I'm also confident that
 nothing will change soon. IMHO there is no need to worry about
 anything.  
>>>
>>> It does spell uncertainty on the long term.  While there is no need to
>>> worry for now, I think it gives a different colour to the debate about
>>> moving issues to Github.
>>>  
>>
>> I don't see how this *increases* the uncertainty. Surely if GitHub had
>> remained independent there would have been be similar concerns about how it
>> would make enough money to stay in business.
>>
> 
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/mal%40egenix.com
> 

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Jun 05 2018)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...   http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...   http://zope.egenix.com/


::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
  http://www.malemburg.com/

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python startup time

2018-05-14 Thread M.-A. Lemburg


On 14.05.2018 18:26, Chris Barker via Python-Dev wrote:
> 
> 
> On Fri, May 11, 2018 at 11:05 AM, Ryan Gonzalez  > wrote:
> 
>  https://refi64.com/uprocd/ 
> 
> 
> very cool -- but *nix only, of course :-(
> 
> But it seems that there is a demand for this sort of thing, and a few
> major projects are rolling their own. So maybe it makes sense to put
> something into the standard library that everyone could contribute to
> and use.
> 
> With regard to forking -- is there another way? I don't have the
> expertise to have any idea if this is possible, but:
> 
> start up python
> 
> capture the entire runtime image as a single binary blob.
> 
> could that blob be simply loaded into memory and run?
> 
> (hmm -- probably not -- memory addresses would be hard-coded then, yes?)
> or is memory virtualized enough these days?

You might want to look into combining this with PyRun:

https://www.egenix.com/products/python/PyRun/

which takes care of mmap'ing the byte code of the stdlib into
memory.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...   http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...   http://zope.egenix.com/


::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
  http://www.malemburg.com/

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python initialization and embedded Python

2017-11-23 Thread M.-A. Lemburg

On 18.11.2017 01:01, Victor Stinner wrote:
> Hi,
> 
> The CPython internals evolved during Python 3.7 cycle. I would like to
> know if we broke the C API or not.
> 
> Nick Coghlan and Eric Snow are working on cleaning up the Python
> initialization with the "on going" PEP 432:
> https://www.python.org/dev/peps/pep-0432/
> 
> Many global variables used by the "Python runtime" were move to a new
> single "_PyRuntime" variable (big structure made of sub-structures).
> See Include/internal/pystate.h.
> 
> A side effect of moving variables from random files into header files
> is that it's not more possible to fully initialize _PyRuntime at
> "compilation time". For example, previously, it was possible to refer
> to local C function (functions declared with "static", so only visible
> in the current file). Now a new "initialization function" is required
> to must be called.
> 
> In short, it means that using the "Python runtime" before it's
> initialized by _PyRuntime_Initialize() is now likely to crash. For
> example, calling PyMem_RawMalloc(), before calling
> _PyRuntime_Initialize(), now calls the function NULL: dereference a
> NULL pointer, and so immediately crash with a segmentation fault.

To prevent a complete crash, would it be possible to initialize
the struct entries to a generic function (or set of such functions
with the right signatures), which then issue a message to stderr
hinting to the missing call to _PyRuntime_Initialize()
before terminating ?

> I'm writing this email to ask if this change is an issue or not to
> embedded Python and the Python C API. Is it still possible to call
> "all" functions of the C API before calling Py_Initialize()?
> 
> I was bitten by the bug while reworking the Py_Main() function to
> split it into subfunctions and cleanup the code to handle the command
> line arguments and environment variables. I fixed the issue in main()
> by calling _PyRuntime_Initialize() as soon as possible: it's now the
> first instruction of main() :-) (See Programs/python.c)
> 
> To give a more concrete example: Py_DecodeLocale() is the recommanded
> function to decode bytes from the operating system, but this function
> calls PyMem_RawMalloc() which does crash before
> _PyRuntime_Initialize() is called. Is Py_DecodeLocale() used to
> initialize Python?
> 
> For example, "void Py_SetProgramName(wchar_t *);" expects a text
> string, whereas main() gives argv as bytes. Calling
> Py_SetProgramName() from argv requires to decode bytes... So use
> Py_DecodeLocale()...
> 
> Should we do something in Py_DecodeLocale()? Maybe crash if
> _PyRuntime_Initialize() wasn't called yet?
> 
> Maybe, the minimum change is to expose _PyRuntime_Initialize() in the
> public C API?
> 
> Victor
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/mal%40egenix.com
> 

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Nov 23 2017)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...   http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...   http://zope.egenix.com/


::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
  http://www.malemburg.com/

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Aligning the packaging.python.org theme with the rest of the docs

2017-05-30 Thread M.-A. Lemburg

On 30.05.2017 13:49, Nick Coghlan wrote:
> Here's an alternate wording for the README that would focus on those
> considerations without explicitly asking folks not to use the theme:
> 
> "Note that when adopting this theme, you're also borrowing an element
> of the trust and credibility established by the CPython core
> developers over the years, as well as the legal credibility arising
> from their close association with the Python Software Foundation.
> That's fine, and you're welcome to do so for other Python community
> projects if you so choose, but please keep in mind that in doing so
> you're also choosing to become a co-steward of that collective trust
> :)"

Much better. Thanks :-)

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, May 30 2017)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...   http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...   http://zope.egenix.com/


::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
  http://www.malemburg.com/

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Aligning the packaging.python.org theme with the rest of the docs

2017-05-28 Thread M.-A. Lemburg

I'm -1 on going down the suggested route of Apple et al. for an
open source language.

We don't need more trademarks to "protect" ourselves against
fellow open source projects.

I see this whole trademark business that OSS projects are getting
into in recent years in a more and more critical light.

On one hand you have the open source idea, where you want to enable
anyone to reuse your code. The OSS copyright licenses implement this
idea nicely.

On the other hand, you have trademark law which limits
this reusability by imposing strict rules on how you can reuse the
names, designs, colors, etc. trademarked by the OSS project (or
else - TM lawyers tell you - you lose the rights to the TMs).

The latter has to be handled with great care, since while you
do want to protect the trademarks against malicious use by third
parties (e.g. a company registering the same mark and then preventing
use of the trademark in their jurisdiction), you don't want to
hinder the original intent of having an open source license. In
particular, you don't want to bully community projects which try to
further the use of our open source software by using similar marks.

In my role as PSF TM committee member, it's often painful to have to
tell community members that they cannot use e.g. really nice looking
variants of the Python logo for their projects. Let's not add more
pain.

If you want to make PSF related projects be recognized as such,
simply add the PSF logo to the footer of the site and provide
some blurb explaining the relationships to the about page.

We could also have a special "PSF Project" logo for such a
purpose. Use of the logo would then be subject to the TM
requirements, but not the looks and UI of the site.

On 28.05.2017 07:31, Nick Coghlan wrote:
> On 28 May 2017 at 06:54, Guido van Rossum  wrote:
>> Are you also going to stop others from using the psf theme?
> 
> I think it would definitely make sense to discourage the use of this
> particular theme for projects that aren't relatively directly
> affiliated with the PSF - there are plenty of other pip-installable
> high contrast themes out there that aren't closely associated with a
> particular backing organisation.
> 
> The one Jon was originally considering for the PyPA docs was
> Alabaster, which is now the default theme in Sphinx 1.3+:
> https://alabaster.readthedocs.io/en/latest/
> 
> So I think it would be appropriate to include a sentence like the
> following in the README for psf-docs-theme:
> 
> "Please limit use of this theme to projects which are closely
> affiliated with the Python Software Foundation, and have permission
> from either the CPython core development team or the PSF itself for
> such use. For other projects looking for a simple, high contrast, pip
> installable Sphinx theme, we recommend the Alabaster theme used by
> default in Sphinx 1.3+."
> 
> Cheers,
> Nick.
> 

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, May 28 2017)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...   http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...   http://zope.egenix.com/

::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
  http://www.malemburg.com/

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Is this a bug or a feature?

2017-02-17 Thread M.-A. Lemburg

Please report such problems on our bug tracker:

http://bugs.python.org/

In your case, Python doesn't appear to find the (right) libz
shared library, which is a bit odd, but it's better to track this
down on the tracker :-)

On 16.02.2017 21:33, Patrick Wallinger wrote:
> I'm brand new to python and was trying to install the newest version
> (3.6.0) on my ubuntu 16.04.1 but couldn't get it to build error free. I
> tried searching the bugs but didn't see a match for what I am seeing.
> 
> Thanks for any help,
> Patrick
> 
> Here is the report:
> patrick@ubuntu-MacBook:~/Programs/Python-3.6.0$ ./python -m test -v
> test_venv
> == CPython 3.6.0 (default, Feb 16 2017, 13:20:29) [GCC 5.4.0 20160609]
> ==   Linux-4.4.0-62-generic-x86_64-with-debian-stretch-sid little-endian
> ==   hash algorithm: siphash24 64bit
> ==  cwd: /home/patrick/Programs/Python-3.6.0/build/test_python_20536
> ==  encodings: locale=UTF-8, FS=utf-8
> Testing with flags: sys.flags(debug=0, inspect=0, interactive=0,
> optimize=0, dont_write_bytecode=0, no_user_site=0, no_site=0,
> ignore_environment=0, verbose=0, bytes_warning=0, quiet=0,
> hash_randomization=1, isolated=0)
> Run tests sequentially
> 0:00:00 [1/1] test_venv
> test_defaults (test.test_venv.BasicTest) ... ok
> test_executable (test.test_venv.BasicTest) ... ok
> test_executable_symlinks (test.test_venv.BasicTest) ... ok
> test_isolation (test.test_venv.BasicTest) ... ok
> test_overwrite_existing (test.test_venv.BasicTest) ... ok
> test_prefixes (test.test_venv.BasicTest) ... ok
> test_prompt (test.test_venv.BasicTest) ... ok
> test_symlinking (test.test_venv.BasicTest) ... ok
> test_unoverwritable_fails (test.test_venv.BasicTest) ... ok
> test_upgrade (test.test_venv.BasicTest) ... ok
> test_devnull (test.test_venv.EnsurePipTest) ... ok
> test_explicit_no_pip (test.test_venv.EnsurePipTest) ... ok
> test_no_pip_by_default (test.test_venv.EnsurePipTest) ... ok
> test_with_pip (test.test_venv.EnsurePipTest) ... FAIL
> 
> ==
> FAIL: test_with_pip (test.test_venv.EnsurePipTest)
> --
> Traceback (most recent call last):
>   File "/home/patrick/Programs/Python-3.6.0/Lib/test/test_venv.py", line
> 372, in test_with_pip
> with_pip=True)
> subprocess.CalledProcessError: Command '['/tmp/tmp51me85og/bin/python',
> '-Im', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit
> status 1.
> 
> During handling of the above exception, another exception occurred:
> 
> Traceback (most recent call last):
>   File "/home/patrick/Programs/Python-3.6.0/Lib/test/test_venv.py", line
> 378, in test_with_pip
> self.fail(msg.format(exc, details))
> AssertionError: Command '['/tmp/tmp51me85og/bin/python', '-Im',
> 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 1.
> 
> **Subprocess Output**
> Traceback (most recent call last):
>   File "/home/patrick/Programs/Python-3.6.0/Lib/runpy.py", line 193, in
> _run_module_as_main
> "__main__", mod_spec)
>   File "/home/patrick/Programs/Python-3.6.0/Lib/runpy.py", line 85, in
> _run_code
> exec(code, run_globals)
>   File "/home/patrick/Programs/Python-3.6.0/Lib/ensurepip/__main__.py",
> line 4, in 
> ensurepip._main()
>   File "/home/patrick/Programs/Python-3.6.0/Lib/ensurepip/__init__.py",
> line 189, in _main
> default_pip=args.default_pip,
>   File "/home/patrick/Programs/Python-3.6.0/Lib/ensurepip/__init__.py",
> line 102, in bootstrap
> _run_pip(args + [p[0] for p in _PROJECTS], additional_paths)
>   File "/home/patrick/Programs/Python-3.6.0/Lib/ensurepip/__init__.py",
> line 27, in _run_pip
> import pip
> zipimport.ZipImportError: can't decompress data; zlib not available
> 
> 
> --
> Ran 14 tests in 1.278s
> 
> FAILED (failures=1)
> test test_venv failed
> test_venv failed
> 
> 1 test failed:
> test_venv
> 
> Total duration: 1 sec
> Tests result: FAILURE
> 
> 
> 
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/mal%40egenix.com
> 

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Feb 17 2017)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...   http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...   http://zope.egenix.com/


::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611

Re: [Python-Dev] Someons's put a "Python 2.8" on GitHub

2016-12-10 Thread M.-A. Lemburg

On 10.12.2016 10:05, David Mertz wrote:
> I'm forwarding this to the PSF Trademarks committee. If there is a
> violation, it's a misuse of trademark, not copyright on the code which has
> the Python license stack.
> 
> I'm on that committee and agree this is improper use. Let's see what other
> members think.

Our trademark policy for the word mark Python does allow
for such use and without asking for permissions:

https://www.python.org/psf/trademarks/
"""
...
As such, stating accurately that software is written in the Python
programming language, that it is compatible with the Python programming
language, or that it contains the Python programming language, is always
allowed. In those cases, you may use the word "Python" or the unaltered
logos to indicate this, without our prior approval. This is true both
for non-commercial and commercial uses.

This clause overrides other clauses of this policy.
...
"""

and this is on purpose, since Python is BSD software which
anyone can use, modify, fork, etc.

The fork also contains a list of differences compared to
Python 2.7 as shipped by the PSF, so the license is fulfilled
as well:

https://github.com/naftaliharris/python2.8

All that said, I still believe we should contact the author
and ask for a name change to make it clear to our users that
the PSF is not endorsing this fork, nor does it provide
support for it:

https://github.com/naftaliharris
https://www.naftaliharris.com/contact/

Regardless of the name, it'll be interesting to see whether
there's demand for such a fork. Without a website, binaries
to download, documentation, etc. it's still in the very early
stages.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Dec 10 2016)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...   http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...   http://zope.egenix.com/

::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
  http://www.malemburg.com/

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Supported versions of OpenSSL

On 31.08.2016 14:02, Nick Coghlan wrote:
> On 31 August 2016 at 20:20, M.-A. Lemburg <m...@egenix.com> wrote:
>> ... which would then mean: Python's compatibility roadmap will
>> be dictated by OpenSSL.
>>
>> I won't buy into that, sorry. Crypto is a helper in certain
>> situations, it's not what Python is all about. We should not
>> let OpenSSL dictate how and when we deprecate platforms or
>> OS versions.
> 
> It won't dictate general support for those platforms, it will dictate
> support for the *ssl module* on those platforms. If someone isn't
> making secure network connections from Python, things will work fine.
> If a redistributor is stepping in to provide the assertion that the
> network connection is secure despite our upstream misgivings, things
> will work fine.
> 
> Connections will only fail in cases where neither we nor a
> redistributor are prepared to make the assertion that a requested
> secure network connection will actually be secure.

Yes, you can view it that way. Your car works, but access
to fuel is severely limited ;-)

The way things are developing around Python, if pip doesn't
work, your Python installation cannot be considered
working.

Hmm, perhaps pip ought to fallback to curl or wget if there's
no ssl module. Or we move away entirely from HTTPS and use
properly signed Python packages - ah, nevermind, it's not the
first time that have different views than many other core
devs. That's fine.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Aug 31 2016)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...   http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...   http://zope.egenix.com/

::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
  http://www.malemburg.com/

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Supported versions of OpenSSL

On 31.08.2016 12:05, Christian Heimes wrote:
> This was my last reply to your mails on this topic. It's clear to me
> that you are not open to Cory's, Nick's or my arguments and that you
> won't change your position. More replies are just a waste of my limited
> time.

I *am* open to arguments, but so far I have not seen a compelling
one which strikes the right balance.

And if you have read my replies, I'm only suggesting to postpone
the switch to 1.0.2 by one release and even there I said that
it's not all that dramatic if this doesn't happen due to the way
the timelines for 3.6 and 3.7 work.

What I am seriously worried about, is the next step ...

> Instead I'm going to focus on a PEP to define OpenSSL support and to
> auto-deprecate unsupported OpenSSL versions. The PEP is a very high
> chance to get accepted. Everybody except you support the plan.

... which would then mean: Python's compatibility roadmap will
be dictated by OpenSSL.

I won't buy into that, sorry. Crypto is a helper in certain
situations, it's not what Python is all about. We should not
let OpenSSL dictate how and when we deprecate platforms or
OS versions.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Aug 31 2016)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...   http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...   http://zope.egenix.com/

::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
  http://www.malemburg.com/

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Supported versions of OpenSSL

On 31.08.2016 10:43, Antoine Pitrou wrote:
> On Wed, 31 Aug 2016 10:31:12 +0200
> "M.-A. Lemburg" <m...@egenix.com> wrote:
>>
>> I am thinking of Python users out there who are running on LTS
>> OS releases simply because their IT doesn't let them run anything
>> else.
> 
> There is a solution nowadays, which is to use Anaconda (or Miniconda).

Sure, or use ActivePython or eGenix PyRun :-)

But is that really what we want to tell people ?

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Aug 31 2016)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...   http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...   http://zope.egenix.com/


::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
  http://www.malemburg.com/

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Supported versions of OpenSSL

On 31.08.2016 10:50, Christian Heimes wrote:
> On 2016-08-31 10:31, M.-A. Lemburg wrote:
>> In all this discussion I have yet to find a compelling security
>> relevant argument for using an 1.0.2 API which is so important
>> that we cannot make this optional at runtime.
>>
>> The only argument Christian reported was this one:
>>
>> """
>>> BTW: Are there any features in 1.0.2 that we need and would warrant
>>> dropping support for 1.0.1 earlier than Ubuntu 14.04 LTS ?
>>
>> Yes, there are features I want to use, e.g. proper hostname
>> verification. Python's post-handshake verification is a hack and leads
>> to information disclosure.
>> """
>>
>> Regarding that argument: hostname validation can be done
>> in 1.0.1 by providing a verification hook handler. That's
>> intended and by design, not a hack. 1.0.2 comes with
>> support for hostname validation making this a little easier
>> (you still have to set this up, though).
> 
> Are you willing to do implement and maintain this callback? Are you
> willing to do all work?

Maintain: yes, if needed.

It is already implemented, so that part isn't hard :-)

> Are you aware how many security bugs we had in our own verification
> code? I'm aware of at least two critical bugs.

Not that many, given that the host name validation is more
a best practices art rather than one where all participants
implement the standards:

http://bugs.python.org/issue?%40columns=id%2Cactivity%2Ctitle%2Ccreator%2Cassignee%2Cstatus%2Ctype&%40sort=-activity&%40filter=status&%40action=searchid=file%3Acontent&%40search_text=match_hostname=search=-1%2C1%2C2%2C3

The only critical bug I could find was this one (NUL bytes
in subjectAltName):

http://bugs.python.org/issue18709

but as I understand, the true origin of the bug was an OpenSSL
function, not the host name matching code in Python.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Aug 31 2016)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...   http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...   http://zope.egenix.com/


::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
  http://www.malemburg.com/

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Supported versions of OpenSSL

On 31.08.2016 01:55, Gregory P. Smith wrote:
> On Tue, Aug 30, 2016 at 1:08 PM M.-A. Lemburg <m...@egenix.com> wrote:
>>> On 29.08.2016 22:16, Christian Heimes wrote:
>>> In my
>>> opinion it is more than reasonable to ditch 1.0.1 and earlier.
>>
>> I want you to consider the consequences of doing this carefully.
>>
> 
> We have. The proposal now stands at: We're free to ditch 1.0.1 and earlier
> support in 3.7 as soon as we have a need for a 1.0.2 specific API.
> 
> There don't seem to be any important negative consequences of doing so.

Uhm, what about not being able to run Python unless you are willing
to stop taking benefit of OS vendor supplied OpenSSL security
fixes ?

In all this discussion I have yet to find a compelling security
relevant argument for using an 1.0.2 API which is so important
that we cannot make this optional at runtime.

The only argument Christian reported was this one:

"""
> BTW: Are there any features in 1.0.2 that we need and would warrant
> dropping support for 1.0.1 earlier than Ubuntu 14.04 LTS ?

Yes, there are features I want to use, e.g. proper hostname
verification. Python's post-handshake verification is a hack and leads
to information disclosure.
"""

Regarding that argument: hostname validation can be done
in 1.0.1 by providing a verification hook handler. That's
intended and by design, not a hack. 1.0.2 comes with
support for hostname validation making this a little easier
(you still have to set this up, though).

So essentially, the only argument so far has been that
we can remove some support code and let 1.0.2 take care of
this.

That's a maintenance argument, not a security one. And the
code in question has been rather stable, so it doesn't add
much maintenance overhead:

https://hg.python.org/cpython/annotate/default/Lib/ssl.py#l195

For dropping Python support on older platforms, I'd expect to
at least see an argument such as: 1.0.1 has a security flaw
which cannot be addressed by design (e.g. a missing crypto
feature which is essential for working with modern SSL
servers).

>> Crypto is important to have, but at the same time it's not
>> essentially for everything you do in Python, e.g. you can
>> easily run data analysis scripts or applications without ever
>> touching the ssl module.
> 
> While technically true, the ssl module is required to fetch and install
> software via pip which for casual users means it is essential. People can
> always operate without it if they are willing to download and build the
> libraries they need manually.
>
>> Yet, a move to require OpenSSL 1.0.2 for Python 3.7 will make
>> it impossible to run such apps on systems that still use OpenSSL
>> 1.0.1, e.g. Ubuntu 14.04 or CentOS 7.
>>
> 
> Not important. That isn't something we need to worry about. Compiling a new
> libssl is easy.  People using systems that are 4+ years old by the time 3.7
> comes out who expect new software to compile and just work are expecting
> too much.

Perhaps, but industry OSes are not upgraded that often, so it's
not unusual to run into a system where you'd like to get Python
working on an LTS OS version which is still maintained but doesn't
use the latest and greatest software versions.

> I find that users of such systems either use only what their distro itself
> supplies (ie: ancient versions at that point) or are fully comfortable
> building any dependencies their own software needs. If they are comfortable
> building a CPython runtime in the first place, they should be comfortable
> building required libraries. Nothing new there.

Why do you assume that people will compile their own CPython
on such systems ? It's well possible, and probably more likely,
that they want to run a binary version of the application on
such a platform, only to find that it doesn't run because of
a missing OpenSSL 1.0.2 API (the libssl.so.1.0 will still be
available, but not expose the requested API - the lib version
of OpenSSL did not change between 1.0.1 and 1.0.2).

>> Why not make the 1.0.2 and 1.1.0 support optional as we do
>> in so many other situations for various systems and libs ?
>>
> 
> In general, conditional compilation complicates code and adds a maintenance
> burden. The sooner we can ditch older versions, the cleaner and easier to
> maintain our code is.

I think we need to find the right balance here. A few lines of
conditional code vs. Python not running on a system at all.

And just to be clear, since some people appear to think that
I want to make them work for me for free (something I find a bit
bizarre given how much time I have invested in Python development -
you are benefiting from the code I've written just as much as
I am from the code you have written):

We are using our own egenix-pyopenssl package which comes wi

Re: [Python-Dev] Supported versions of OpenSSL

2016-08-30 Thread M.-A. Lemburg

On 29.08.2016 22:16, Christian Heimes wrote:
> On 2016-08-29 21:31, M.-A. Lemburg wrote:
>> On 29.08.2016 18:33, Cory Benfield wrote:
>>>
>>>> On 29 Aug 2016, at 04:09, M.-A. Lemburg <m...@egenix.com> wrote:
>>>>
>>>> On 28.08.2016 22:40, Christian Heimes wrote:
>>>>> ...
>>>>> I like to reduce the maintenance burden and list of supported OpenSSL
>>>>> versions ASAP. OpenSSL has deprecated 0.9.8 and 1.0.0 last year. 1.0.1
>>>>> will reach EOL by the end of this year,
>>>>> https://www.openssl.org/policies/releasestrat.html . However OpenSSL
>>>>> 0.9.8 is still required for some platforms (OSX).
>>>>> ...
>>>>> For upcoming 3.6 I would like to limit support to 1.0.2+ and require
>>>>> 1.0.2 features for 3.7.
>>>>> ...
>>>>
>>>> Hmm, that last part would mean that Python 3.7 will no longer compile
>>>> on e.g. Ubuntu 14.04 LTS which uses OpenSSL 1.0.1 as default version.
>>>> Since 14.04 LTS is supported until 2019, I think it would be better
>>>> to only start requiring 1.0.2 in Python 3.8.
>>>
>>> Can someone explain to me why this is a use-case we care about?
>>
>> Ubuntu 14.04 is a widely deployed system and newer Python version
>> should run on such widely deployed systems without having to
>> replace important vendor maintained system libraries such as
>> OpenSSL.
> 
> "Widely deployed" is true for a lot of old operating systems including
> Windows XP.
> 
>> Python 3.7 starts shipping around June 2018 (assuming the 18 month
>> release cycle). Ubuntu 14.04 EOL is April 2019, so in order to
>> be able to use Python 3.7 on such a system, you'd have to upgrade
>> to a more recent LTS version 10 months before the EOL date (with
>> all the associated issues) or lose vendor maintenance support and
>> run with your own copy of OpenSSL.
> 
> Why would you deploy an unsupported Python version on a LTS release? Why
> should compatibility be our concern?
> 
>> Sure, but Ubuntu will continue to support OpenSSL 1.0.1
>> until 2019, backporting important security fixes as necessary and
>> that's what's important.
> 
> I see an easy solution here: either pay or make Canonical backport all
> required features to OpenSSL 1.0.1. 
> 
>> It's unfortunate that Python has to rely on a 3rd party library
>> for security, but we should at least make sure that our users
>> can rely on OS vendor support to keep the lib up to date with
>> security fixes.
> 
> No, it is a good thing that we can rely on 3rd party libraries for
> security. Crypto and security is not our domain. It is incredible hard
> to develop and maintain crypto code. Also my proposal enforces OS
> vendors to supply up to date OpenSSL versions.

That was not my point. It's unfortunate that Python depends on
a library which is inevitably going to need updates frequently,
and which then may have the implication that Python won't compile on
systems which don't ship with more recent OpenSSL libs - even
if your application doesn't even need ssl at all.

>> On 29.08.2016 10:24, Christian Heimes wrote:
>>> By the way I knew that something like this would come up from you.
>>> Thank you that you satisfied my expectation. :p
>>
>> Sure, I want Python to be used on as many systems as possible,
>> both in terms of architecture and OS. The more the better.
>> If we don't have to drop support early, why should we ?
> 
> MAL, I don't like your attitude. It feels like you want me and other
> contributors to waste time on this topic. That is not how this
> discussion is going to end. If *you* want to keep support for outdated
> OpenSSL versions, than it is *your* responsibility and *your* time. You
> cannot and will not put this burden on me.

Please reread what I suggested: to postpone the switch to require
OpenSSL 1.0.2 by one Python release version. And in my reply I then
put this into more context, saying that your schedule will likely
work out.

Postponing this should not introduce more work for anyone; if you'd
like to add support for 1.0.2 feature early this can also easily be
done by making such support optional depending on which OpenSSL
lib Python is compiled against. This takes a few #ifdefs, nothing
more.

> Python is running out of developers with OpenSSL expertise. It's Alex,
> Antoine, Benjamin, Victor and me. Antoine and me haven't been active for
> a while. Victor and Benjamin are mostly working on other topics. As far
> as I can judge Alex, he rather works on PyCA than CPython stdlib.
> 
> I'm both interested and willing t

Re: [Python-Dev] Supported versions of OpenSSL

2016-08-29 Thread M.-A. Lemburg

On 29.08.2016 18:33, Cory Benfield wrote:
> 
>> On 29 Aug 2016, at 04:09, M.-A. Lemburg <m...@egenix.com> wrote:
>>
>> On 28.08.2016 22:40, Christian Heimes wrote:
>>> ...
>>> I like to reduce the maintenance burden and list of supported OpenSSL
>>> versions ASAP. OpenSSL has deprecated 0.9.8 and 1.0.0 last year. 1.0.1
>>> will reach EOL by the end of this year,
>>> https://www.openssl.org/policies/releasestrat.html . However OpenSSL
>>> 0.9.8 is still required for some platforms (OSX).
>>> ...
>>> For upcoming 3.6 I would like to limit support to 1.0.2+ and require
>>> 1.0.2 features for 3.7.
>>> ...
>>
>> Hmm, that last part would mean that Python 3.7 will no longer compile
>> on e.g. Ubuntu 14.04 LTS which uses OpenSSL 1.0.1 as default version.
>> Since 14.04 LTS is supported until 2019, I think it would be better
>> to only start requiring 1.0.2 in Python 3.8.
> 
> Can someone explain to me why this is a use-case we care about?

Ubuntu 14.04 is a widely deployed system and newer Python version
should run on such widely deployed systems without having to
replace important vendor maintained system libraries such as
OpenSSL.

Python 3.7 starts shipping around June 2018 (assuming the 18 month
release cycle). Ubuntu 14.04 EOL is April 2019, so in order to
be able to use Python 3.7 on such a system, you'd have to upgrade
to a more recent LTS version 10 months before the EOL date (with
all the associated issues) or lose vendor maintenance support and
run with your own copy of OpenSSL.

OTOH, the situations isn't all that bad: people on such systems
will likely not go straight for the 3.7.0 release of Python, but
instead wait for 3.7.2+ for the dust to settle.

> The nature of a stable distribution is that all the packages are guaranteed 
> to work together. Once you start compiling your own software, you are out in 
> the cold: you are no longer covered by that guarantee. Why, then, should 
> someone compiling a version of Python that was barely conceived when Ubuntu 
> 14.04 was released expect that they can compile it from source using only 
> dependencies available in their mainline distribution?
> 
> I absolutely understand wanting to support Ubuntu 14.04 for any release of 
> Python that already compiles on Ubuntu 14.04: that makes sense. But new 
> releases should not be shackled to what is in LTS releases of operating 
> systems. If it were, a more coherent argument would be that we cannot drop 
> 0.9.8 support in any Python released before the middle of 2017 because RHEL 5 
> ships it, and we will not be able to require OpenSSL 1.0.2 until almost 2021 
> because RHEL 6 ships 1.0.1. After all, why does Ubuntu 14.04 get privileged 
> over RHEL? Both are supported LTS releases, after all.
> 
> I don’t believe that the Python dev team has ever promised that future 
> versions of Python will compile on a given LTS release. Am I mistaken?
> 
> While I’m here, I should note that Ubuntu 14.04 goes EOL in 2019, while 
> OpenSSL 1.0.1 loses support from upstream at the end of this year, which is 
> probably a good reason to stop supporting it in new Python versions. OpenSSL 
> 1.0.0 and 0.9.8 are already unsupported by upstream.

Sure, but Ubuntu will continue to support OpenSSL 1.0.1
until 2019, backporting important security fixes as necessary and
that's what's important.

It's unfortunate that Python has to rely on a 3rd party library
for security, but we should at least make sure that our users
can rely on OS vendor support to keep the lib up to date with
security fixes.

On 29.08.2016 10:24, Christian Heimes wrote:
> By the way I knew that something like this would come up from you.
> Thank you that you satisfied my expectation. :p

Sure, I want Python to be used on as many systems as possible,
both in terms of architecture and OS. The more the better.
If we don't have to drop support early, why should we ?

>> BTW: Are there any features in 1.0.2 that we need and would warrant
>> dropping support for 1.0.1 earlier than Ubuntu 14.04 LTS ?
>
> Yes, there are features I want to use, e.g. proper hostname
> verification. Python's post-handshake verification is a hack and leads
> to information disclosure.

This doesn't sound like a feature worth breaking compatibility
to me.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Aug 29 2016)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...   http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...   http://zope.egenix.com/


::: We implement business ideas - efficientl

Re: [Python-Dev] Supported versions of OpenSSL

2016-08-29 Thread M.-A. Lemburg

On 28.08.2016 22:40, Christian Heimes wrote:
> ...
> I like to reduce the maintenance burden and list of supported OpenSSL
> versions ASAP. OpenSSL has deprecated 0.9.8 and 1.0.0 last year. 1.0.1
> will reach EOL by the end of this year,
> https://www.openssl.org/policies/releasestrat.html . However OpenSSL
> 0.9.8 is still required for some platforms (OSX).
> ...
> For upcoming 3.6 I would like to limit support to 1.0.2+ and require
> 1.0.2 features for 3.7.
> ...

Hmm, that last part would mean that Python 3.7 will no longer compile
on e.g. Ubuntu 14.04 LTS which uses OpenSSL 1.0.1 as default version.
Since 14.04 LTS is supported until 2019, I think it would be better
to only start requiring 1.0.2 in Python 3.8.

BTW: Are there any features in 1.0.2 that we need and would warrant
dropping support for 1.0.1 earlier than Ubuntu 14.04 LTS ?

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Aug 29 2016)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...   http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...   http://zope.egenix.com/

::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
  http://www.malemburg.com/

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

2016-06-10 Thread M.-A. Lemburg

On 10.06.2016 20:55, Donald Stufft wrote:
> Ok, so you’re looking for how would you replicate the blocking behavior of 
> os.urandom that exists in 3.5.0 and 3.5.1?
> 
> In that case, it’s hard. I don’t think linux provides any way to externally 
> determine if /dev/urandom has been initialized or not. Probably the easiest 
> thing to do would be to interface with the getrandom() function using a 
> c-ext, CFFI, or ctypes. If you’re looking for a way of doing this without 
> calling the getrandom() function.. I believe the answer is you can’t.

Well, you can see the effect by running Python early in the boot process.

See e.g. http://bugs.python.org/issue26839#msg267749

and if you look at the system log file, you'll find a notice
entry "random: %s pool is initialized" which gets written once the
pool is initialized:

http://lxr.free-electrons.com/source/drivers/char/random.c#L684

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Jun 10 2016)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...   http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...   http://zope.egenix.com/


::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
  http://www.malemburg.com/

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] New hash algorithms: SHA3, SHAKE, BLAKE2, truncated SHA512

2016-05-29 Thread M.-A. Lemburg

On 28.05.2016 23:13, Christian Heimes wrote:
> On 2016-05-27 14:41, M.-A. Lemburg wrote:
>> On 27.05.2016 22:58, Ryan Gonzalez wrote:
>>> On May 27, 2016 3:04 PM, "Victor Stinner" <victor.stin...@gmail.com> wrote:
>>>>
>>>> Le vendredi 27 mai 2016, M.-A. Lemburg <m...@egenix.com> a écrit :
>>>>>
>>>>> The current patch is 1.2MB for SHA-3 - that's pretty heavy for just
>>>>> a few hash functions, which aren't in any wide spread use yet and
>>>>> probably won't be for quite a few years ahead.
>>>>
>>>>
>>>> Oh wow, it's so fat? Why is it so big? Can't we use a lighter version?
>>>>
>>>
>>> The stark majority of the patch is Lib/test/vectors/sha3_224.txt, which
>>> seems to be (as the file path implies) just test data. A whopping >1k LOC
>>> of really long hashes.
>>
>> Right. There's about 1MB test data in the patch, but even
>> without that data, the patch adds more than 6400 lines of code.
> 
> The KeccakCodePackage is rather large. I already removed all unnecessary
> files and modified some files so more code is shared between 32 and
> 64bit optimized variants. Please keep in mind that the KCP contains
> multiple implementations with different optimizations for CPU
> architectures. I already removed the ARM NEON optimization.
> I also don't get your obsession with lines of code. The gzip and expat
> are far bigger than the KeccakCodePackage.

For a small piece of code, it's fine to have a copy in the
stdlib, but for larger chunks such as this one, I think we
ought to consider alternative options, since I don't think
it's good to have to carry around this baggage forever.

OpenSSL will eventually have good enough support for what
most Python users will need from these new hash functions.
That's why I think it's better to have a discussion of whether
we need to full package in the stdlib or better only provide
limited support built into the stdlib and refer people to
PyPI packages for things that you don't need every day.

Regarding the stories for zlib and expat, I only remember
that expat was essentially unmaintained when we added it
and the existing version at the time had known bugs (but
could be wrong).

For zlib, I have no clue as to why we have a copy in the stdlib.
That lib is available on all systems nowadays. Perhaps it wasn't
when we added it; don't remember. If so, it's a good example of
why adding copies to the stdlib is not such a good idea :-)

>> If we add this now, there should at least be an exit strategy
>> to remove the code again, when OpenSSL ships with the same
>> code, IMO.
>>
>> Aside: BLAKE2 has already landed in OpenSSL 1.1.0:
>>
>> https://github.com/openssl/openssl/tree/master/crypto/blake2
> 
> Except BLAKE2 in OpenSSL is severely castrated and tailored towards a
> very limited use case. The implementation does not support any of the
> useful advanced features like keyed hashing (MAC), salt,
> personalization, tree hashing and variable hash length.

I bet that the use cases they put into OpenSSL is what
most people will eventually use, so essentially the same
reasoning we use for putting stuff into the stdlib.

Besides, the code just landed in OpenSSL. It's likely they'll
continue to optimize it and possibly also add the variants
they left out initially.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, May 29 2016)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...   http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...   http://zope.egenix.com/

::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
  http://www.malemburg.com/

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] New hash algorithms: SHA3, SHAKE, BLAKE2, truncated SHA512

On 27.05.2016 23:46, Donald Stufft wrote:
> 
>> On May 27, 2016, at 5:41 PM, M.-A. Lemburg <m...@egenix.com> wrote:
>>
>> If we add this now, there should at least be an exit strategy
>> to remove the code again, when OpenSSL ships with the same
>> code, IMO.
> 
> I think it is a clear win to have the fallback implementations in cases where 
> people either don’t have OpenSSL or don’t have a new enough OpenSSL for those 
> implementations. Not having the fallback just makes it more difficult for 
> people to rely on those hash functions.

This will only be needed once the stdlib itself starts requiring
support for some of these hashes and for that we could add
a pure Python implementation, eg.

https://github.com/coruus/py-keccak

In all other cases, you can simply add the support via a
package such as Björn's or Christian's.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, May 27 2016)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...   http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...   http://zope.egenix.com/

::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
  http://www.malemburg.com/

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] New hash algorithms: SHA3, SHAKE, BLAKE2, truncated SHA512

On 27.05.2016 22:58, Ryan Gonzalez wrote:
> On May 27, 2016 3:04 PM, "Victor Stinner" <victor.stin...@gmail.com> wrote:
>>
>> Le vendredi 27 mai 2016, M.-A. Lemburg <m...@egenix.com> a écrit :
>>>
>>> The current patch is 1.2MB for SHA-3 - that's pretty heavy for just
>>> a few hash functions, which aren't in any wide spread use yet and
>>> probably won't be for quite a few years ahead.
>>
>>
>> Oh wow, it's so fat? Why is it so big? Can't we use a lighter version?
>>
> 
> The stark majority of the patch is Lib/test/vectors/sha3_224.txt, which
> seems to be (as the file path implies) just test data. A whopping >1k LOC
> of really long hashes.

Right. There's about 1MB test data in the patch, but even
without that data, the patch adds more than 6400 lines of code.

If we add this now, there should at least be an exit strategy
to remove the code again, when OpenSSL ships with the same
code, IMO.

Aside: BLAKE2 has already landed in OpenSSL 1.1.0:

https://github.com/openssl/openssl/tree/master/crypto/blake2

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, May 27 2016)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...   http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...   http://zope.egenix.com/


::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
  http://www.malemburg.com/

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] New hash algorithms: SHA3, SHAKE, BLAKE2, truncated SHA512

On 27.05.2016 18:41, Chris Barker wrote:
> On Fri, May 27, 2016 at 9:35 AM, M.-A. Lemburg <m...@egenix.com> wrote:
> 
>>> So if ( and that's a big if) it's possible to anticipate what will be
>>> in widespread use in a couple years, getting it in now would be a good
>>> thing.
>>
>> You cut away the important part of what I said:
>> "The current patch is 1.2MB for SHA-3 - that's pretty heavy for
>> just a few hash functions, ..."
>>
>> If people want to use the hashes earlier, this is already possible
>> via a separate package, so we're not delaying their use.
>>
> 
> That's true for ANY addition to the stdlib -- it could always be made
> available in a third party lib. (unless you want to use it in another part
> of the stdlib...)

Well, any addition for which someone already wrote a package,
but yes...

>> It is clear that SHA-3 will get more traction in coming years (*),
>> but I'm pretty sure that OpenSSL will have good implementations by
>> the time people will actively start using the new hash algorithm
>> and then hashlib will automatically make that available (hashlib
>> uses the OpenSSL EVP abstraction, so will be able to use any
>> new algorithms added to OpenSSL).
>>
>> However, if we add the reference implementation now, we'd then be
>> left with 1.2MB unnecessary code in the stdlib.
>>
> 
> I'm probably showing my ignorance here, but couldn't we swap in the OpenSSL
> implementation when that becomes available?

We could, but only if we don't expose separate interfaces
for the hashes and not add them to hashlib.

hashlib.algorithms
hashlib.algorithms_guaranteed

> -CHB
> 
> 
> (*) People are just now starting to move from SHA-1 to SHA-2
>> and SHA-2 was standardized in 2001. Python received SHA-2 support
>> in 2006. So there's plenty of time to decide :-)
> 
> 
> can't deny the history, nor the inertia -- but that doesn't make it a good
> thing...
> 
> 
> 
> 
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/mal%40egenix.com
> 

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, May 27 2016)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...   http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...   http://zope.egenix.com/


::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
  http://www.malemburg.com/

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] New hash algorithms: SHA3, SHAKE, BLAKE2, truncated SHA512

On 27.05.2016 17:44, Chris Barker - NOAA Federal wrote:
>>> , which aren't in any wide spread use yet and
>> probably won't be for quite a few years ahead.
> 
> Anything added to the stdlib now will be in py3.6+, yes?
> 
> Which won't be in widespread use for quite a few years yet, either.
> 
> So if ( and that's a big if) it's possible to anticipate what will be
> in widespread use in a couple years, getting it in now would be a good
> thing.

You cut away the important part of what I said:
"The current patch is 1.2MB for SHA-3 - that's pretty heavy for
just a few hash functions, ..."

If people want to use the hashes earlier, this is already possible
via a separate package, so we're not delaying their use.

It is clear that SHA-3 will get more traction in coming years (*),
but I'm pretty sure that OpenSSL will have good implementations by
the time people will actively start using the new hash algorithm
and then hashlib will automatically make that available (hashlib
uses the OpenSSL EVP abstraction, so will be able to use any
new algorithms added to OpenSSL).

However, if we add the reference implementation now, we'd then be
left with 1.2MB unnecessary code in the stdlib.

The question is not so much: is SHA-3 useful or not, it's
whether we want to maintain this forever going forward or
not.

(*) People are just now starting to move from SHA-1 to SHA-2
and SHA-2 was standardized in 2001. Python received SHA-2 support
in 2006. So there's plenty of time to decide :-)

> -CHB
> 
> 
> 
>>
>> IMO, relying on OpenSSL is a better strategy than providing
>> (and maintaining) our own compatibility versions. Until OpenSSL
>> has them, people can use Björn's package:
>>
>> https://github.com/bjornedstrom/python-sha3
>>
>> Perhaps you could join forces with Björn to create a standard
>> SHA-3 standalone package on PyPI based on your two variants
>> which we could recommend to people in the docs ?!
>>
>> --
>> Marc-Andre Lemburg
>> eGenix.com
>>
>> Professional Python Services directly from the Experts (#1, May 27 2016)
> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
> Python Database Interfaces ...   http://products.egenix.com/
> Plone/Zope Database Interfaces ...   http://zope.egenix.com/
>> 
>>
>> ::: We implement business ideas - efficiently in both time and costs :::
>>
>>   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
>>D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
>>   Registered at Amtsgericht Duesseldorf: HRB 46611
>>   http://www.egenix.com/company/contact/
>>  http://www.malemburg.com/
>>
>> ___
>> Python-Dev mailing list
>> Python-Dev@python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe: 
>> https://mail.python.org/mailman/options/python-dev/chris.barker%40noaa.gov

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, May 27 2016)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...   http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...   http://zope.egenix.com/

::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
  http://www.malemburg.com/

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] New hash algorithms: SHA3, SHAKE, BLAKE2, truncated SHA512

On 27.05.2016 13:03, Donald Stufft wrote:
> 
>> On May 27, 2016, at 6:54 AM, M.-A. Lemburg <m...@egenix.com> wrote:
>>
>> IMO, relying on OpenSSL is a better strategy than providing
>> (and maintaining) our own compatibility versions. Until OpenSSL
>> has them, people can use Björn's package:
> 
> Even now, hashlib doesn’t rely on OpenSSL if I recall, I mean it will
> use it if OpenSSL is available but otherwise it has internal implementations
> too.

I know, but still don't think that's a good idea. It makes
sense in case you don't want to carry around OpenSSL all the
time, but how often does that happen nowadays ?

BTW: If I recall correctly, those hash implementations predate
the deeper support for OpenSSL we now have in Python.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, May 27 2016)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...   http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...   http://zope.egenix.com/

::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
  http://www.malemburg.com/

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] New hash algorithms: SHA3, SHAKE, BLAKE2, truncated SHA512

On 27.05.2016 06:54, Raymond Hettinger wrote:
> 
>> On May 25, 2016, at 3:29 AM, Christian Heimes  wrote:
>>
>> I have three hashing-related patches for Python 3.6 that are waiting for
>> review. Altogether the three patches add ten new hash algorithms to the
>> hashlib module: SHA3 (224, 256, 384, 512), SHAKE (SHA3 XOF 128, 256),
>> BLAKE2 (blake2b, blake2s) and truncated SHA512 (224, 256).
> 
> Do we really need ten?  I don't think the standard library is the place to 
> offer all variants of hashing.  And we should avoid getting in a cycle of 
> "this was just released by NIST" and "nobody uses that one anymore".  Is any 
> one of them an emergent best practice (i.e. starting to be commonly used in 
> network protocols because it is better, faster, stronger, etc)?
> 
> Your last message on https://bugs.python.org/issue16113 suggests that these 
> aren't essential and that there is room for debate about whether some of them 
> are standard-library worthy (i.e. we will have them around forever).

I can understand your eagerness to get this landed, since it's
been 4 years since work started, but I think we should wait with
the addition until OpenSSL has them:

https://github.com/openssl/openssl/issues/439

The current patch is 1.2MB for SHA-3 - that's pretty heavy for just
a few hash functions, which aren't in any wide spread use yet and
probably won't be for quite a few years ahead.

IMO, relying on OpenSSL is a better strategy than providing
(and maintaining) our own compatibility versions. Until OpenSSL
has them, people can use Björn's package:

https://github.com/bjornedstrom/python-sha3

Perhaps you could join forces with Björn to create a standard
SHA-3 standalone package on PyPI based on your two variants
which we could recommend to people in the docs ?!

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, May 27 2016)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...   http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...   http://zope.egenix.com/


::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
  http://www.malemburg.com/

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] What does a double coding cookie mean?

2016-03-20 Thread M.-A. Lemburg

On 17.03.2016 15:02, Serhiy Storchaka wrote:
> On 17.03.16 15:14, M.-A. Lemburg wrote:
>> On 17.03.2016 01:29, Guido van Rossum wrote:
>>> Should we recommend that everyone use tokenize.detect_encoding()?
>>
>> I'd prefer a separate utility for this somewhere, since
>> tokenize.detect_encoding() is not available in Python 2.
>>
>> I've attached an example implementation with tests, which works
>> in Python 2.7 and 3.
> 
> Sorry, but this code doesn't match the behaviour of Python interpreter,
> nor other tools. I suggest to backport tokenize.detect_encoding() (but
> be aware that the default encoding in Python 2 is ASCII, not UTF-8).

Yes, I got the default for Python 3 wrong. I'll fix that. Thanks
for the note.

What other aspects are different than what Python implements ?

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Mar 17 2016)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...   http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...   http://zope.egenix.com/

2016-03-07: Released eGenix pyOpenSSL 0.13.14 ... http://egenix.com/go89
2016-02-19: Released eGenix PyRun 2.1.2 ...   http://egenix.com/go88

::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
  http://www.malemburg.com/

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] What does a double coding cookie mean?

2016-03-19 Thread M.-A. Lemburg

On 17.03.2016 18:53, Serhiy Storchaka wrote:
> On 17.03.16 19:23, M.-A. Lemburg wrote:
>> On 17.03.2016 15:02, Serhiy Storchaka wrote:
>>> On 17.03.16 15:14, M.-A. Lemburg wrote:
>>>> On 17.03.2016 01:29, Guido van Rossum wrote:
>>>>> Should we recommend that everyone use tokenize.detect_encoding()?
>>>>
>>>> I'd prefer a separate utility for this somewhere, since
>>>> tokenize.detect_encoding() is not available in Python 2.
>>>>
>>>> I've attached an example implementation with tests, which works
>>>> in Python 2.7 and 3.
>>>
>>> Sorry, but this code doesn't match the behaviour of Python interpreter,
>>> nor other tools. I suggest to backport tokenize.detect_encoding() (but
>>> be aware that the default encoding in Python 2 is ASCII, not UTF-8).
>>
>> Yes, I got the default for Python 3 wrong. I'll fix that. Thanks
>> for the note.
>>
>> What other aspects are different than what Python implements ?
> 
> 1. If there is a BOM and coding cookie, the source encoding is "utf-8-sig".

Ok, that makes sense (even though it's not mandated by the PEP;
the utf-8-sig codec didn't exist yet).

> 2. If there is a BOM and coding cookie is not 'utf-8', this is an error.

It's an error for Python, but why should a detection function
always raise an error for this case ? It would probably be a good
idea to have an errors parameter to leave this to the use to decide.

Same for unknown encodings.

> 3. If the first line is not blank or comment line, the coding cookie is
> not searched in the second line.

Hmm, the PEP does allow having the coding cookie in the
second line, even if the first line is not a comment. Perhaps
that's not really needed.

> 4. Encoding name should be canonized. "UTF8", "utf8", "utf_8" and
> "utf-8" is the same encoding (and all are changed to "utf-8-sig" with BOM).

Well, that's cosmetics :-) The codec system will take care of
this when needed.

> 5. There isn't the limit of 400 bytes. Actually there is a bug with
> handling long lines in current code, but even with this bug the limit is
> larger.

I think it's a reasonable limit, since shebang lines may only be
127 long on at least Linux (and probably several other Unix systems
as well).

But just in case, I made this configurable :-)

> 6. I made a mistake in the regular expression, missed the underscore.

I added it.

> tokenize.detect_encoding() is the closest imitation of the behavior of
> Python interpreter.

Probably, but that doesn't us on Python 2, right ?

I'll upload the script to github later today or tomorrow to
continue development.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Mar 17 2016)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...   http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...   http://zope.egenix.com/

2016-03-07: Released eGenix pyOpenSSL 0.13.14 ... http://egenix.com/go89
2016-02-19: Released eGenix PyRun 2.1.2 ...   http://egenix.com/go88

::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
  http://www.malemburg.com/

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] What does a double coding cookie mean?

2016-03-19 Thread M.-A. Lemburg

On 17.03.2016 01:29, Guido van Rossum wrote:
> I've updated the PEP. Please review. I decided not to update the
> Unicode howto (the thing is too obscure). Serhiy, you're probably in a
> better position to fix the code looking for cookies to pick the first
> one if there are two on the same line (or do whatever you think should
> be done there).

Thanks, will do.

> Should we recommend that everyone use tokenize.detect_encoding()?

I'd prefer a separate utility for this somewhere, since
tokenize.detect_encoding() is not available in Python 2.

I've attached an example implementation with tests, which works
in Python 2.7 and 3.

> On Wed, Mar 16, 2016 at 5:05 PM, Guido van Rossum <gu...@python.org> wrote:
>> On Wed, Mar 16, 2016 at 12:59 AM, M.-A. Lemburg <m...@egenix.com> wrote:
>>> The only reason to read up to two lines was to address the use of
>>> the shebang on Unix, not to be able to define two competing
>>> source code encodings :-)
>>
>> I know. I was just surprised that the PEP was sufficiently vague about
>> it that when I found that mypy picked the second if there were two, I
>> couldn't prove to myself that it was violating the PEP. I'd rather
>> clarify the PEP than rely on the reasoning presented earlier here.

I suppose it's a rather rare case, since it's the first time
that I heard about anyone thinking that a possible second line
could be picked - after 15 years :-)

>> I don't like erroring out when there are two different cookies on two
>> lines; I feel that the spirit of the PEP is to read up to two lines
>> until a cookie is found, whichever comes first.
>>
>> I will update the regex in the PEP too (or change the wording to avoid 
>> "match").
>>
>> I'm not sure what to do if there are two cooking on one line. If
>> CPython currently picks the latter we may want to preserve that
>> behavior.
>>
>> Should we recommend that everyone use tokenize.detect_encoding()?
>>
>> --
>> --Guido van Rossum (python.org/~guido)
> 
> 
> 

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Mar 17 2016)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...   http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...   http://zope.egenix.com/

2016-03-07: Released eGenix pyOpenSSL 0.13.14 ... http://egenix.com/go89
2016-02-19: Released eGenix PyRun 2.1.2 ...   http://egenix.com/go88

::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
  http://www.malemburg.com/

#!/usr/bin/python
"""
Utility to detect the source code encoding of a Python file.

Marc-Andre Lemburg, 2016.

Supports Python 2.7 and 3.

"""
import sys
import re
import codecs

# Debug output ?
_debug = True

# PEP 263 RE
PEP263 = re.compile(b'^[ \t]*#.*?coding[:=][ \t]*([-.a-zA-Z0-9]+)',
re.MULTILINE)

###

def detect_source_encoding(code, buffer_size=400):

""" Detect and return the source code encoding of the Python code
given in code.

code must be given as bytes.

The function uses a buffer to determine the first two code lines
with a default size of 400 bytes/code points.  This can be adjusted
using the buffer_size parameter.

"""
# Get the first two lines
first_two_lines = b'\n'.join(code[:buffer_size].splitlines()[:2])
# BOMs override any source code encoding comments
if first_two_lines.startswith(codecs.BOM):
return 'utf-8'
# .search() picks the first occurrance
m = PEP263.search(first_two_lines)
if m is None:
return 'ascii'
return m.group(1).decode('ascii')

# Tests

def _test():

l = (
  (b"""\
# No encoding
""", 'ascii'),
  (b"""\
# coding: latin-1
""", 'latin-1'),
  (b"""\
#!/usr/bin/python
# coding: utf-8
""", 'utf-8'),
  (b"""\
coding=123
# The above could be detected as source code encoding
""", 'ascii'),
  (b"""\
# coding: latin-1
# coding: utf-8
""", 'latin-1'),
  (b"""\
# No encoding on first line
# No encoding on second line
# coding: utf-8
""", 'ascii'),
  (codecs.BOM + b"""\
# No encoding
"&qu

Re: [Python-Dev] What does a double coding cookie mean?

2016-03-19 Thread M.-A. Lemburg

On 17.03.2016 15:55, Guido van Rossum wrote:
> On Thu, Mar 17, 2016 at 5:04 AM, Serhiy Storchaka  wrote:
>>> Should we recommend that everyone use tokenize.detect_encoding()?
>>
>> Likely. However the interface of tokenize.detect_encoding() is not very
>> simple.
> 
> I just found that out yesterday. You have to give it a readline()
> function, which is cumbersome if all you have is a (byte) string and
> you don't want to split it on lines just yet. And the readline()
> function raises SyntaxError when the encoding isn't right. I wish
> there were a lower-level helper that just took a line and told you
> what the encoding in it was, if any. Then the rest of the logic can be
> handled by the caller (including the logic of trying up to two lines).

I've uploaded the code I posted yesterday, modified to address
some of the issues it had to github:

https://github.com/malemburg/python-snippets/blob/master/detect_source_encoding.py

I'm pretty sure the two-lines read can be optimized away and
put straight into the regular expression used for matching.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Mar 18 2016)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...   http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...   http://zope.egenix.com/

2016-03-07: Released eGenix pyOpenSSL 0.13.14 ... http://egenix.com/go89
2016-02-19: Released eGenix PyRun 2.1.2 ...   http://egenix.com/go88

::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
  http://www.malemburg.com/

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

From openstack-dev-bounces+archive=mail-archive@lists.openstack.org Sat Mar 
19 08:12:23 2016
Return-path: 

Envelope-to: arch...@mail-archive.com
Delivery-date: Sat, 19 Mar 2016 08:12:23 -0700
Received: from bolt10a.mxthunder.net ([209.105.224.168])
by mail-archive.com with esmtp (Exim 4.76)
(envelope-from 
)
id 1ahIYA-0002fr-Cy
for arch...@mail-archive.com; Sat, 19 Mar 2016 08:12:22 -0700
Received: by bolt10a.mxthunder.net (Postfix, from userid 12345)
id 3qRVGK3w2Rz19ktG; Fri, 18 Mar 2016 08:56:36 -0700 (PDT)
Received: from lists.openstack.org (lists.openstack.org [50.56.173.222])
(using TLSv1 with cipher AES256-SHA (256/256 bits))
(No client certificate requested)
by bolt10a.mxthunder.net (Postfix) with ESMTPS id 3qRVFs4wjPz19kcC
for ; Fri, 18 Mar 2016 08:56:33 -0700 (PDT)
Received: from localhost ([127.0.0.1] helo=lists.openstack.org)
by lists.openstack.org with esmtp (Exim 4.76)
(envelope-from )
id 1agwhl-0005a9-59; Fri, 18 Mar 2016 15:52:49 +
Received: from g4t3426.houston.hp.com ([15.201.208.54])
 by lists.openstack.org with esmtp (Exim 4.76)
 (envelope-from ) id 1agwhj-0005YM-O5
 for openstack-...@lists.openstack.org; Fri, 18 Mar 2016 15:52:47 +
Received: from G4W9121.americas.hpqcorp.net (g4w9121.houston.hp.com
 [16.210.21.16]) (using TLSv1.2 with cipher AES256-SHA (256/256 bits))
 (No client certificate requested)
 by g4t3426.houston.hp.com (Postfix) with ESMTPS id 605A664
 for ; Fri, 18 Mar 2016 15:52:47 + (UTC)
Received: from G4W9121.americas.hpqcorp.net (16.210.21.16) by
 G4W9121.americas.hpqcorp.net (16.210.21.16) with Microsoft SMTP Server (TLS)
 id 15.0.1076.9; Fri, 18 Mar 2016 15:52:38 +
Received: from G4W6304.americas.hpqcorp.net (16.210.26.229) by
 G4W9121.americas.hpqcorp.net (16.210.21.16) with Microsoft SMTP Server (TLS)
 id 15.0.1076.9 via Frontend Transport; Fri, 18 Mar 2016 15:52:38 +
Received: from G9W0750.americas.hpqcorp.net ([169.254.9.246]) by
 G4W6304.americas.hpqcorp.net ([16.210.26.229]) with mapi id 14.03.0169.001;
 Fri, 18 Mar 2016 15:52:38 +
From: "Hayes, Graham" 
To: "OpenStack Development Mailing List (not for usage questions)"
 
Thread-Topic: [openstack-dev] [all][infra][ptls] tagging reviews, making
 tags searchable
Thread-Index: AQHRgSbdQRQMpxFOeUKGrubFuR+3RA==
Date: Fri, 18 Mar 2016 15:52:37 +
Message-ID:

Re: [Python-Dev] What does a double coding cookie mean?

2016-03-16 Thread M.-A. Lemburg

On 16.03.2016 01:28, Guido van Rossum wrote:
> I agree that the spirit of the PEP is to stop at the first coding
> cookie found. Would it be okay if I updated the PEP to clarify this?
> I'll definitely also update the docs.

+1

The only reason to read up to two lines was to address the use of
the shebang on Unix, not to be able to define two competing
source code encodings :-)

> On Tue, Mar 15, 2016 at 2:04 PM, Brett Cannon  wrote:
>>
>>
>> On Tue, 15 Mar 2016 at 13:31 Guido van Rossum  wrote:
>>>
>>> I came across a file that had two different coding cookies -- one on
>>> the first line and one on the second. CPython uses the first, but mypy
>>> happens to use the second. I couldn't find anything in the spec or
>>> docs ruling out the second interpretation. Does anyone have a
>>> suggestion (apart from following CPython)?
>>>
>>> Reference: https://github.com/python/mypy/issues/1281
>>
>>
>> I think the spirit of PEP 263 is for the first specified encoding to win as
>> the support of two lines is to support shebangs and not multiple encodings
>> :) . I also think the fact that tokenize.detect_encoding() doesn't
>> automatically read two lines from its input also suggests the intent is
>> "first encoding wins" (and that is the semantics of the function).
> 
> 
> 

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Mar 16 2016)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...   http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...   http://zope.egenix.com/

2016-03-07: Released eGenix pyOpenSSL 0.13.14 ... http://egenix.com/go89
2016-02-19: Released eGenix PyRun 2.1.2 ...   http://egenix.com/go88

::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
  http://www.malemburg.com/

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Very old git mirror under github user "python-git"

2016-02-28 Thread M.-A. Lemburg

On 28.02.2016 18:46, Georg Brandl wrote:
> On 02/27/2016 11:45 PM, Matthias Bussonnier wrote:
>> Hi all, 
>>
>>
>>> On Feb 27, 2016, at 14:21, Alexander Walters >> > wrote:
>>>
>>> Can we even ask github to pull it down and reasonably expect them to comply?
>>>  Their entire model is built on everyone forking everyone else.
>>
>> Why the model is everyone forking, some of the help page of GitHub actually 
>> tell
>> you to contact GitHub support, like if you desire to "detach" a fork. 
>>
>> Every reasonable requests I made to GitHub and the few interactions I had 
>> with
>> the support always went well. 
>> This did include asking GitHub to contact user as their pages were confusing,
>> and might be misleading others.
>>
>> So I would suggest 
>>
>> 1) asking GitHub to contact author, potentially forwarding him/her a message
>> from this list asking him/her to bring that down or transfer the control to 
>> you.
>> That should be easy to do as it will not force GitHub to provide anyone with 
>> the
>> emails of the the owner of python-git.
>>
>> 2) in the case of no response from author ask politely GitHub that the repo 
>> is
>> confusing for user, and ask what they can do about that.
> 
> These are both fine.  Although I don't see much confusion; there's bound to be
> hundreds of forks of CPython, if not already, then definitely once we move to
> GitHub.
> 
>> 3) If still nothing can be done make a DMCA request. You can likely argue 
>> that
>> the logo/name are used without PSF content. 
>> https://help.github.com/articles/dmca-takedown-policy/ 
> 
> Please no.  There is absolutely no call using such a blunt instrument, just 
> for
> a case of minor inconvenience.  It could also be blown up into a PR disaster,
> probably rightly so.

I frankly don't understand what all the fuzz is about. The repo
in question hasn't been touched in 7 years. It refers to Python 2.7
alpha 0.

It also clearly reads "Unofficial Python SVN auto-updating mirror",
so there's no confusion either.

The talk about DMCA requests really doesn't apply. Python is
open-source. Anyone can fork it, at any version they like, as
long as the license is respected.

The trademark use is also perfectly in line with our TM policy.
The logo is a bit blurred, but that's really the only nit I could
find.

Asking the owner to take the repo down is still a good thought,
but there's definitely nothing wrong with it per se.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Feb 28 2016)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...   http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...   http://zope.egenix.com/

2016-02-19: Released eGenix PyRun 2.1.2 ...   http://egenix.com/go88

::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
  http://www.malemburg.com/

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 493: HTTPS verification migration tools for Python 2.7

2016-02-24 Thread M.-A. Lemburg

On 24.02.2016 21:39, Cory Benfield wrote:
> 
>> On 24 Feb 2016, at 12:19, M.-A. Lemburg <m...@egenix.com> wrote:
>>
>> On 24.02.2016 12:28, Cory Benfield wrote:
>>>
>>>> On 24 Feb 2016, at 10:32, Nick Coghlan <ncogh...@gmail.com> wrote:
>>>>
>>>> Security Considerations
>>>> ---
>>>>
>>>> Relative to the behaviour in Python 3.4.3+ and Python 2.7.9->2.7.11, this
>>>> approach does introduce a new downgrade attack against the default security
>>>> settings that potentially allows a sufficiently determined attacker to 
>>>> revert
>>>> Python to the default behaviour used in CPython 2.7.8 and earlier releases.
>>>> However, such an attack requires the ability to modify the execution
>>>> environment of a Python process prior to the import of the ``ssl`` module,
>>>> and any attacker with such access would already be able to modify the
>>>> behaviour of the underlying OpenSSL implementation.
>>>>
>>>
>>> I’m not entirely sure this is accurate. Specifically, an attacker that is 
>>> able to set environment variables but nothing else (no filesystem access) 
>>> would be able to disable hostname validation. To my knowledge this is the 
>>> only environment variable that could be set that would do that.
>>
>> An attacker with access to the OS environment of a process would
>> be able to do lots of things. I think disabling certificate checks
>> is not one of the highest ranked attack vectors you'd use, given
>> such capabilities :-)
>>
>> Think of LD_PRELOAD attacks, LD_LIBRARY_PATH manipulations, shell PATH
>> manipulations (think spawned processes), compiler flag manipulations
>> (think "pip install sourcepkg"), OpenSSL reconfiguration, etc.
>>
>> Probably much easier than an active attack would be to simply extract
>> sensitive information from the environ and use this for more direct
>> attacks, e.g. accessing databases, payment services, etc.
> 
> To be clear, I’m not suggesting that this represents a reason not to do any 
> of this, just that we should not suggest that there is no risk here: there 
> is, and it is a new attack vector.

Fair enough :-)

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Feb 24 2016)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...   http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...   http://zope.egenix.com/

2016-02-19: Released eGenix PyRun 2.1.2 ...   http://egenix.com/go88

::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
  http://www.malemburg.com/

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 493: HTTPS verification migration tools for Python 2.7

2016-02-24 Thread M.-A. Lemburg

On 24.02.2016 12:28, Cory Benfield wrote:
> 
>> On 24 Feb 2016, at 10:32, Nick Coghlan  wrote:
>>
>> Security Considerations
>> ---
>>
>> Relative to the behaviour in Python 3.4.3+ and Python 2.7.9->2.7.11, this
>> approach does introduce a new downgrade attack against the default security
>> settings that potentially allows a sufficiently determined attacker to revert
>> Python to the default behaviour used in CPython 2.7.8 and earlier releases.
>> However, such an attack requires the ability to modify the execution
>> environment of a Python process prior to the import of the ``ssl`` module,
>> and any attacker with such access would already be able to modify the
>> behaviour of the underlying OpenSSL implementation.
>>
> 
> I’m not entirely sure this is accurate. Specifically, an attacker that is 
> able to set environment variables but nothing else (no filesystem access) 
> would be able to disable hostname validation. To my knowledge this is the 
> only environment variable that could be set that would do that.

An attacker with access to the OS environment of a process would
be able to do lots of things. I think disabling certificate checks
is not one of the highest ranked attack vectors you'd use, given
such capabilities :-)

Think of LD_PRELOAD attacks, LD_LIBRARY_PATH manipulations, shell PATH
manipulations (think spawned processes), compiler flag manipulations
(think "pip install sourcepkg"), OpenSSL reconfiguration, etc.

Probably much easier than an active attack would be to simply extract
sensitive information from the environ and use this for more direct
attacks, e.g. accessing databases, payment services, etc.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Feb 24 2016)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...   http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...   http://zope.egenix.com/

2016-02-19: Released eGenix PyRun 2.1.2 ...   http://egenix.com/go88

::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
  http://www.malemburg.com/

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Modify PyMem_Malloc to use pymalloc for performance

2016-02-12 Thread M.-A. Lemburg

On 12.02.2016 12:18, Victor Stinner wrote:
> ping?

Sorry, your email must gotten lost in my inbox.

> 2016-02-08 15:18 GMT+01:00 Victor Stinner <victor.stin...@gmail.com>:
>> 2016-02-04 15:05 GMT+01:00 M.-A. Lemburg <m...@egenix.com>:
>>> Sometimes, yes, but we also do allocations for e.g.
>>> parsing values in Python argument tuples (e.g. using
>>> "es" or "et"):
>>>
>>> https://docs.python.org/3.6/c-api/arg.html
>>>
>>> We do document to use PyMem_Free() on those; not sure whether
>>> everyone does this though.
>>
>> It's well documented. If programs start to crash, they must be fixed.
>>
>> I don't propose to "break the API" for free, but to get a speedup on
>> the overall Python.
>>
>> And I don't think that we can say that it's an API change, since we
>> already stated that PyMem_Free() must be used.
>>
>> If your program has bugs, you can use a debug build of Python 3.5 to
>> detect misusage of the API.

Yes, but people don't necessarily do this, e.g. I have
for a very long time ignored debug builds completely
and when I started to try them, I found that some of the
things I had been doing with e.g. free list implementations
did not work in debug builds.

>>> The Python test suite doesn't test Python C extensions,
>>> so it's not surprising that it passes :-)
>>
>> What do you mean by "C extensions"? Which modules?
>>
>> Many modules in the stdlib have "C accelerators" and the PEP 399 now
>> *require* to test the C and Python implementations.

Yes, but those are part of the stdlib. You'd need to check
a few C extensions which are not tested as part of the stdlib,
e.g. numpy, scipy, lxml, pillow, etc. (esp. ones which implement custom
types in C since these will often need the memory management
APIs).

It may also be a good idea to check wrapper generators such
as cython, swig, cffi, etc.

>>>> Instead of teaching developers that well, in fact, PyObject_Malloc()
>>>> is unrelated to object programming, I think that it's simpler to
>>>> modify PyMem_Malloc() to reuse pymalloc ;-)
>>>
>>> Perhaps if you add some guards somewhere :-)
>>
>> We have runtime checks but only implemented in debug mode for efficiency.
>>
>> By the way, I proposed once to add an environment variable to allow to
>> enable these checks without having to recompile Python.  Since the PEP
>> 445, it became easy to implement this. What do you think?
>> https://www.python.org/dev/peps/pep-0445/#add-a-new-pydebugmalloc-environment-variable
>>
>> "This alternative was rejected because a new environment variable
>> would make Python initialization even more complex. PEP 432 tries to
>> simplify the CPython startup sequence."
>>
>> The PEP 432 looks stuck, so I don't think that we should block
>> enhancements because of this PEP. Anyway, my idea should be easy to
>> implement.

I suppose such a flag would create a noticeable runtime
performance hit, since the compiler would no longer be
able to inline the PyMem_*() APIs if you redirect those
APIs to other sets at runtime.

I also don't see much point in carrying around such
baggage in production builds of Python, since you'd most
likely only want to use the tools to debug C extensions during
their development.

>>> Seriously, this may work if C extensions use the APIs
>>> consistently, but in order to tell, we'd need to check
>>> few.
>>
>> Can you suggest me names of projects that must be tested?

See above for a list of starters :-)

It would be good to add a few more that work on text or
larger chunks of memory, since those will most likely utilize
the memory allocators more than other extensions which mostly
wrap (sets of) C variables.

Some of them may also have benchmarks, so in addition to
checking whether they work with the change, you could also
test performance.

>>> I guess the main question then is whether pymalloc is good enough
>>> for general memory allocation needs; and the answer may well be
>>> "yes".
>>
>> What do you mean by "good enough"? For the runtime performance,
>> pymalloc looks to be faster than malloc(). What are your other
>> criterias? Memory fragmentation?

Runtime performance, difference in memory consumption (arenas
cannot be freed if there are still small chunks allocated),
memory locality. I'm no expert in this, so can't really
comment much.

I suspect that lib C and OS provided allocators will have
advantages as well, but since pymalloc redirects to them for
all larger memory chunks, it's probably an overall win for
Python C extensio

Re: [Python-Dev] Modify PyMem_Malloc to use pymalloc for performance

2016-02-04 Thread M.-A. Lemburg

On 03.02.2016 22:03, Victor Stinner wrote:
> Hi,
> 
> There is an old discussion about the performance of PyMem_Malloc()
> memory allocator. CPython is stressing a lot memory allocators. Last
> time I made statistics, it was for the PEP 454:
> "For example, the Python test suites calls malloc() , realloc() or
> free() 270,000 times per second in average."
> https://www.python.org/dev/peps/pep-0454/#log-calls-to-the-memory-allocator
> 
> I proposed a simple change: modify PyMem_Malloc() to use the pymalloc
> allocator which is faster for allocation smaller than 512 bytes, or
> fallback to malloc() (which is the current internal allocator of
> PyMem_Malloc()).
> 
> This tiny change makes Python up to 6% faster on some specific (macro)
> benchmarks, and it doesn't seem to make Python slower on any
> benchmark:
> http://bugs.python.org/issue26249#msg259445
> 
> Do you see any drawback of using pymalloc for PyMem_Malloc()?

Yes: You cannot free memory allocated using pymalloc with the
standard C lib free().

It would be better to go through the list of PyMem_*() calls
in Python and replace them with PyObject_*() calls, where
possible.

> Does anyone recall the rationale to have two families to memory allocators?

The PyMem_*() APIs were needed to have a cross-platform malloc()
implementation which returns standard C lib free()able memory,
but also behaves well when passing 0 as size.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Feb 04 2016)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...   http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...   http://zope.egenix.com/


::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
  http://www.malemburg.com/

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Modify PyMem_Malloc to use pymalloc for performance

2016-02-04 Thread M.-A. Lemburg

On 04.02.2016 13:29, Victor Stinner wrote:
> Hi,
> 
> 2016-02-04 11:17 GMT+01:00 M.-A. Lemburg <m...@egenix.com>:
>>> Do you see any drawback of using pymalloc for PyMem_Malloc()?
>>
>> Yes: You cannot free memory allocated using pymalloc with the
>> standard C lib free().
> 
> That's not completly new.
> 
> If Python is compiled in debug mode, you get a fatal error with a huge
> error message if you free the memory allocated by PyMem_Malloc() using
> PyObject_Free() or PyMem_RawFree().
> 
> But yes, technically it's possible to use free() when Python is *not*
> compiled in debug mode.

Debug mode is a completely different beast ;-)

>> It would be better to go through the list of PyMem_*() calls
>> in Python and replace them with PyObject_*() calls, where
>> possible.
> 
> There are 536 calls to the functions PyMem_Malloc(), PyMem_Realloc()
> and PyMem_Free().
> 
> I would prefer to modify a single place having to replace 536 calls :-/

You have a point there, but I don't think it'll work out
that easily, since we are using such calls to e.g. pass
dynamically allocated buffers to code in extensions (which then
have to free the buffers again).

>>> Does anyone recall the rationale to have two families to memory allocators?
>>
>> The PyMem_*() APIs were needed to have a cross-platform malloc()
>> implementation which returns standard C lib free()able memory,
>> but also behaves well when passing 0 as size.
> 
> Yeah, PyMem_Malloc() & PyMem_Free() help to have a portable behaviour.
> But, why not PyObject_Malloc() & PObject_Free() were not used in the
> first place?

Good question. I guess developers simply thought of PyObject_Malloc()
being for PyObjects, not arbitrary memory buffers, most likely
because pymalloc was advertised as allocator for Python objects,
not random chunks of memory.

Also: PyObject_*() APIs were first introduced with pymalloc, and
no one really was interested in going through all the calls to
PyMem_*() APIs and convert those to use the new pymalloc at the
time.

All this happened between Python 1.5.2 and 2.0.

One of the reasons probably also was that pymalloc originally
did not return memory back to the system malloc(). This was
changed only some years ago.

> An explanation can be that PyMem_Malloc() can be called without the
> GIL held. But it wasn't true before Python 3.4, since PyMem_Malloc()
> called (indirectly) PyObject_Malloc() when Python was compiled in
> debug mode, and PyObject_Malloc() requires the GIL to be held.
> 
> When I wrote the PEP 445, there was a discussion about the GIL. It was
> proposed to allow to call PyMem_xxx() without the GIL:
> https://www.python.org/dev/peps/pep-0445/#gil-free-pymem-malloc
> 
> This option was rejected.

AFAIR, the GIL was not really part of the consideration at the time.
We used pymalloc for PyObject allocation, that's all.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Feb 04 2016)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...   http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...   http://zope.egenix.com/

::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
  http://www.malemburg.com/

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Modify PyMem_Malloc to use pymalloc for performance

2016-02-04 Thread M.-A. Lemburg

On 04.02.2016 14:25, Victor Stinner wrote:
> Thanks for your feedback, you are asking good questions :-)
> 
> 2016-02-04 13:54 GMT+01:00 M.-A. Lemburg <m...@egenix.com>:
>>> There are 536 calls to the functions PyMem_Malloc(), PyMem_Realloc()
>>> and PyMem_Free().
>>>
>>> I would prefer to modify a single place having to replace 536 calls :-/
>>
>> You have a point there, but I don't think it'll work out
>> that easily, since we are using such calls to e.g. pass
>> dynamically allocated buffers to code in extensions (which then
>> have to free the buffers again).
> 
> Ah, interesting. But I'm not sure that we delegate the responsability
> of freeing the memory to external libraries. Usually, it's more the
> opposite: a library gives us an allocated memory block, and we have to
> free it. No?

Sometimes, yes, but we also do allocations for e.g.
parsing values in Python argument tuples (e.g. using
"es" or "et"):

https://docs.python.org/3.6/c-api/arg.html

We do document to use PyMem_Free() on those; not sure whether
everyone does this though.

> I checked if we call directly malloc() to pass the buffer to a
> library, but I failed to find such case.
>
> Again, in debug mode, calling free() on a memory block allocated by
> PyMem_Malloc() will likely crash. Since we run the Python test suite
> with a Python compiled in debug mode, we would already have detected
> such bug, no?

The Python test suite doesn't test Python C extensions,
so it's not surprising that it passes :-)

> See also my old issue http://bugs.python.org/issue18203 which replaced
> almost all direct calls to malloc() with PyMem_Malloc() or
> PyMem_RawMalloc().
> 
>> Good question. I guess developers simply thought of PyObject_Malloc()
>> being for PyObjects,
> 
> Yeah, I also understood that, but in practice, it looks like
> PyMem_Malloc() is slower than so using it makes the code less
> efficient than it can be.
> 
> Instead of teaching developers that well, in fact, PyObject_Malloc()
> is unrelated to object programming, I think that it's simpler to
> modify PyMem_Malloc() to reuse pymalloc ;-)

Perhaps if you add some guards somewhere :-)

Seriously, this may work if C extensions use the APIs
consistently, but in order to tell, we'd need to check
few. I know that I switched over all mx Extensions to
use PyObject_*() instead of PyMem_*() or native malloc()
several years ago and have not run into any issues.

I guess the main question then is whether pymalloc is good enough
for general memory allocation needs; and the answer may well be
"yes".

BTW: Tuning pymalloc for commonly used object sizes is
another area where Python could gain better performance,
i.e. reserve more / pre-allocate space for often used block
sizes. pymalloc will also only work well for small blocks
(up to 512 bytes). Everything else is routed to the
system malloc().

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Feb 04 2016)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...   http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...   http://zope.egenix.com/

::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
  http://www.malemburg.com/

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] More optimisation ideas

2016-01-31 Thread M.-A. Lemburg

On 30.01.2016 20:15, Steve Dower wrote:
> Brett tried freezing the entire stdlib at one point (as we do for parts of 
> importlib) and reported no significant improvement. Since that rules out code 
> compilation as well as the OS calls, it'd seem the priority is to execute 
> less code on startup.
> 
> Details of that work were posted to python-dev about twelve months ago, IIRC. 
> Maybe a little longer.

Freezing the entire stdlib does improve the startup time,
simply because it removes stat calls, which dominate the startup
time at least on Unix.

It also allows sharing the stdlib byte code in memory, since it gets
stored in static C structs which the OS will happily mmap into
multiple processes for you without any additional effort.

Our eGenix PyRun does exactly that. Even though the original
motivation is a different one, the gained improvement in
startup time is a nice side effect:

http://www.egenix.com/products/python/PyRun/

Aside: The encodings don't really make much difference here. The
dictionaries aren't all that big, so generating them on the fly doesn't
really create much overhead. The trade off in terms of maintainability/speed
definitely leans toward maintainability. For the larger encoding
tables we already have C implementations with appropriate data
structures to make lookup speed vs. storage needs efficient.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Jan 31 2016)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...   http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...   http://zope.egenix.com/

::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
  http://www.malemburg.com/

> Top-posted from my Windows Phone
> 
> -Original Message-
> From: "Serhiy Storchaka" 
> Sent: ‎1/‎30/‎2016 10:22
> To: "python-dev@python.org" 
> Subject: Re: [Python-Dev] More optimisation ideas
> 
> On 30.01.16 18:31, Steve Dower wrote:
>> On 30Jan2016 0645, Serhiy Storchaka wrote:
>>> $ ./python -m timeit -s "import codecs; from encodings.cp437 import
>>> decoding_table" -- "codecs.charmap_build(decoding_table)"
>>> 10 loops, best of 3: 4.36 usec per loop
>>>
>>> Getting rid from charmap_build() would save you at most 4.4 microseconds
>>> per encoding. 0.0005 seconds if you have imported *all* standard
>>> encodings!
>>
>> Just as happy to be proven wrong. Perhaps I misinterpreted my original
>> profiling and then, embarrassingly, ran with the result for a long time
>> without retesting.
> 
> AFAIK the most time is spent in system calls like stat or open. 
> Archiving the stdlib into the ZIP file and using zipimport can decrease 
> Python startup time (perhaps there is an open issue about this).
> 
> 
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/steve.dower%40python.org
> 
> 
> 
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/mal%40egenix.com
> 

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Update PEP 7 to require curly braces in C

2016-01-19 Thread M.-A. Lemburg

On 19.01.2016 00:20, Brett Cannon wrote:
> On Sun, 17 Jan 2016 at 11:10 Brett Cannon  wrote:
> 
>> While doing a review of http://bugs.python.org/review/26129/ I asked to
>> have curly braces put around all `if` statement bodies. Serhiy pointed out
>> that PEP 7 says curly braces are optional:
>> https://www.python.org/dev/peps/pep-0007/#id5. I would like to change
>> that.
>>
>> My argument is to require them to prevent bugs like the one Apple made
>> with OpenSSL about two years ago:
>> https://www.imperialviolet.org/2014/02/22/applebug.html. Skipping the
>> curly braces is purely an aesthetic thing while leaving them out can lead
>> to actual bugs.
>>
>> Anyone object if I update PEP 7 to remove the optionality of curly braces
>> in PEP 7?
>>
> 
> Currently this thread stands at:

Make that:

> +1
>   Brett
>   Ethan
>   Robert
>   Georg
>   Nick
>   Maciej Szulik
> +0
>   Guido
> -0
>   Serhiy
> -1
MAL
>   Victor (maybe; didn't specifically vote)
>   Larry
>   Stefan

There are plenty other cases where typos can ruin the flow
of your code in C; the discussed case is not a very common one.

I find the whole discussion a bit strange: In Python we're
advocating not having to use curly braces, because whitespace
already provides the needed semantics for us, yet you are
now advocating that without adding extra curly braces
we'd be in danger of writing wrong code.

The Apple bug can happen in Python just as well:

if a:
run_if_true()
else:
run_if_false()

can turn into (say by hitting a wrong key in the editor):

if a:
run_if_true()
run_if_false()

(run_if_false is now run when a is true, and nothing is
done in case a is false)

So what's the correct way to address this ?

It's having a test for both branches (a is true, a is false),
not starting to write e.g.

if a:
run_if_true()
if not a:
run_if false()

to feel more "secure".

Also note that the extra braces won't necessarily help you.
If you remove "else" from:

if (a) {
run_if_true();
}
else {
run_if_false();
}

you get exactly the same Apply bug as before, only with more
curly braces.

This all sounds a bit like security theater to me ;-)

I'd say: people who feel better using the extra braces can use
them, while others who don't need them can go ahead as always
... and both groups should write more tests :-)

BTW: There are other things in PEP 7 which should probably be updated
instead, e.g. it still says we should use C89, but we're having more
and more C99 code (mostly new library functions) in CPython.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Jan 19 2016)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...   http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...   http://zope.egenix.com/

::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
  http://www.malemburg.com/

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Update PEP 7 to require curly braces in C

2016-01-18 Thread M.-A. Lemburg

On 18.01.2016 08:00, Victor Stinner wrote:
> I like if without braces when the body is only one line, especially when
> there is no else block.

Same here.

Compilers warn about these things today, so I don't think
we need to go paranoid ;-)

> Victor
> 
> 
> Le dimanche 17 janvier 2016, Brett Cannon  a écrit :
> 
>> While doing a review of http://bugs.python.org/review/26129/ I asked to
>> have curly braces put around all `if` statement bodies. Serhiy pointed out
>> that PEP 7 says curly braces are optional:
>> https://www.python.org/dev/peps/pep-0007/#id5. I would like to change
>> that.
>>
>> My argument is to require them to prevent bugs like the one Apple made
>> with OpenSSL about two years ago:
>> https://www.imperialviolet.org/2014/02/22/applebug.html. Skipping the
>> curly braces is purely an aesthetic thing while leaving them out can lead
>> to actual bugs.
>>
>> Anyone object if I update PEP 7 to remove the optionality of curly braces
>> in PEP 7?


-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Jan 18 2016)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...   http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...   http://zope.egenix.com/


::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
  http://www.malemburg.com/

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Branches in which to fix the SSL tests

2016-01-07 Thread M.-A. Lemburg

On 07.01.2016 04:06, Martin Panter wrote:
> Currently some SSL tests in the test suite are broken by a recent
> certificate change at https://svn.python.org/; see
>  for the bug report. The tests are
> broken when the test suite is run with the “-unetwork” option enabled,
> and most of the buildbots appear to be affected. (In 3.6 the tests
> have temporarily been disabled as a workaround.) I have a simple patch
> that subsitutes the old root certificate for the new which I would
> like to commit, but I’m not sure which branches to apply it to, or
> even which branches are open to normal maintainence and bug fixes.

As mentioned on the issue tracker I'm not convinced that your
patch is a good solution going forward. Rather than basing
the test on svn.python.org, which can change again in the
future, we should use the domain we have specifically
reserved for stdlib tests, namely pythontest.net and get this
set up in a way so that we can run SSL tests again.

I can help with getting SSL certificates for pythontest.net,
since I'm managing the PSF SSL certificates (well, trying to
keep those managed anyway ;-) - I was not involved in the
choice of new CA for svn.python.org):

https://wiki.python.org/psf/PSF%20SSL%20Certificates

Regarding which branches to apply the final fix, others have
already chimed in. Essentially, any branch which we'll need
to run tests on in the foreseeable future will need to be fixed,
since the change in svn.python.org's SSL setup (the expired
certificate which caused the tests to fail was replaced with
a new one from a different CA, again causing tests to fail)
affects all released test suites.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Jan 07 2016)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...   http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...   http://zope.egenix.com/

::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
  http://www.malemburg.com/

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Do windows 10 users, like windows 7 users need to install a SP before installing Python will work?

2015-12-07 Thread M.-A. Lemburg

On 07.12.2015 21:50, Laura Creighton wrote:
> As webmaster, I am dealing with 3 unhappy would-be python users who have
> windows 10.
> 
> Right now their first problem is that when they click on the big
> yellow button here: https://www.python.org/downloads/
> 
> instead of getting a download of 3.5.1 they get a redirect to
> https://www.python.org/downloads/windows/
> 
> I've tried them on both the 
> Download Windows x86 web-based installer
> 
> and
>  
> Download Windows x86-64 web-based installer
> 
> but still no go, they get the  Modify/Repair/Uninstall screen
> like: http://www2.openend.se/~lac/5796.2.png
> 
> I do not know how to help them now.

Have they already tried the regular installers (as opposed to
the web installers) ?

Cheers,
-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Dec 07 2015)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...   http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...   http://zope.egenix.com/


::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
  http://www.malemburg.com/

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python Language Reference has no mention of list comprehensions

On 03.12.2015 14:37, Paul Moore wrote:
> On 3 December 2015 at 12:51, Laura Creighton  wrote:
>> Intentional or Oversight?
> 
> Hard to find :-)
> 
> https://docs.python.org/3/reference/expressions.html#displays-for-lists-sets-and-dictionaries
> 
> I went via "Atoms" in the expression section, then followed the links
> in the actual grammar spec.

Strange that the doc search facility doesn't find this:

https://docs.python.org/3/search.html?q=comprehension

The human readable documentation is in the tutorial:

https://docs.python.org/3/tutorial/datastructures.html#list-comprehensions

(this is found by the doc search)

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Dec 03 2015)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...   http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...   http://zope.egenix.com/


::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
  http://www.malemburg.com/

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python Language Reference has no mention of list comÃprehensions

On 03.12.2015 17:09, Ryan Gonzalez wrote:
> 
> 
> On December 3, 2015 8:26:23 AM CST, Laura Creighton  wrote:
>> In a message of Thu, 03 Dec 2015 13:37:17 +, Paul Moore writes:
>>> On 3 December 2015 at 12:51, Laura Creighton  wrote:
 Intentional or Oversight?
>>>
>>> Hard to find :-)
>>>
>>> https://docs.python.org/3/reference/expressions.html#displays-for-lists-sets-and-dictionaries
>>>
>>> I went via "Atoms" in the expression section, then followed the links
>>> in the actual grammar spec.
>>>
>>> Paul
>>
>> I think the whole use of the language displays as in
>>  
>>  6.2.4. Displays for lists, sets and dictionaries
>>
>>  For constructing a list, a set or a dictionary Python provides 
>>  special syntax called “displays”, each of them in two flavors:
>>
>>either the container contents are listed explicitly, or
>>they are computed via a set of looping and filtering instructions, 
>>called a comprehension.
>>
>> is very odd.  I don't know anybody who talks of 'displays'.  They
>> talk of 'two ways to construct a'.  
>>
>> Who came up with the word 'display' and what does it have going for
>> it that I have missed?  Right now I think its chief virtue is that
>> it is a meaningless noun.  (But not meaningless enough, as I
>> associate displays with output, not construction).
>>
>> I think that 
>>
>>6.2.4 Constructing lists, sets and dictionaries
>>
>> would be a much more useful title, and
>>
>> 6.2.4 Constructing lists, sets and dictionaries -- explicitly or
>> through the use of comprehensions
>>
> 
> What about:
> 
> 6.2.4 Constricting lists, sets, and dictionaries (including comprehensions)
> 
> or something to that effect?
> 
>> an even better one.
>>
>> Am I missing something important about the 'display' language?

I don't think changing a single header is useful in this case.

The grammar uses the token term "display" to mean "representation
of an object". While you normally only think of output when talking
of the representation of an object, it can also refer to the visual
definition of an object when passed to the parser.

A list comprehension is an example of such a visual definition of
an object, hence the token name.

If we were to change the term, we'd have to change it throughout
the reference, grammar and parser implementation.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Dec 03 2015)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...   http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...   http://zope.egenix.com/


::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
  http://www.malemburg.com/

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python Language Reference has no mention of list comÃprehensions

On 03.12.2015 18:30, Laura Creighton wrote:
> What I would like is if it were a lot easier for a person who just
> saw a list comprehension for the very first time, and was told what it
> is, to have a much, much easier time finding it in the Reference Manual.

Such a person should more likely be directed to the tutorial
rather than the very technical language spec :-)

> Would a section on comprehensions in general, defining what a comprehension
> is be appropriate?

We already have this in the tutorial:

https://docs.python.org/3/tutorial/datastructures.html#list-comprehensions

Cheers,
-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Dec 03 2015)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...   http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...   http://zope.egenix.com/


::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
  http://www.malemburg.com/

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python Language Reference has no mention of list comÃprehensions

On 03.12.2015 19:27, Laura Creighton wrote:
> So how do we get search to work so that people in the Language
> Reference who type in 'List Comprehension' get a hit?

It seems that the search index is broken for at least a few
documentation file releases:

ok: 
https://docs.python.org/2.6/search.html?q=comprehension_keywords=yes=default
not ok: 
https://docs.python.org/2.7/search.html?q=comprehension_keywords=yes=default
ok: 
https://docs.python.org/3.2/search.html?q=comprehension_keywords=yes=default
not ok: 
https://docs.python.org/3.3/search.html?q=comprehension_keywords=yes=default
not ok: 
https://docs.python.org/3.4/search.html?q=comprehension_keywords=yes=default
not ok: 
https://docs.python.org/3.5/search.html?q=comprehension_keywords=yes=default

(ok = "/reference/expressions.html is found")

Interestingly, these URLs give different results, e.g.

ok: 
https://docs.python.org/release/2.7.1/search.html?q=comprehension_keywords=yes=default
ok: 
https://docs.python.org/release/2.7.2/search.html?q=comprehension_keywords=yes=default
ok: 
https://docs.python.org/release/2.7.3/search.html?q=comprehension_keywords=yes=default
ok: 
https://docs.python.org/release/2.7.4/search.html?q=comprehension_keywords=yes=default
ok: 
https://docs.python.org/release/2.7.5/search.html?q=comprehension_keywords=yes=default
ok: 
https://docs.python.org/release/2.7.6/search.html?q=comprehension_keywords=yes=default
ok: 
https://docs.python.org/release/2.7.7/search.html?q=comprehension_keywords=yes=default
ok: 
https://docs.python.org/release/2.7.8/search.html?q=comprehension_keywords=yes=default
not ok:
https://docs.python.org/release/2.7.9/search.html?q=comprehension_keywords=yes=default
not ok:
https://docs.python.org/release/2.7.10/search.html?q=comprehension_keywords=yes=default

Looks like something changed between the 2.7.8 and 2.7.9 release.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Dec 03 2015)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...   http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...   http://zope.egenix.com/


::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
  http://www.malemburg.com/

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Deleting with setting C API functions

2015-12-02 Thread M.-A. Lemburg

On 02.12.2015 13:29, Serhiy Storchaka wrote:
> On 02.12.15 12:06, Victor Stinner wrote:
>> 2015-12-02 9:42 GMT+01:00 Serhiy Storchaka :
>>> You have enough time to update your projects, and you can update them
>>> uniformly for all versions. And may be you will found few weird bugs related
>>> to misuse of Set* API.
>>
>> Did you check popular projects using C extensions to check if they
>> call Set*() functions to delete attributes/items?
> 
> I have checked following projects.
> 
> regex, simplejson, Pillow, PyQt4, LibreOffice, PyGTK, PyICU, pyOpenSSL, 
> libxml2, Boost, psutil,
> mercurial don't use PyObject_SetAttr at all.
> 
> NumPy, pgobject don't use PyObject_SetAttr for deleting.
> 
> PyYAML and lxml use PyObject_SetAttr only in code generated by Cython and 
> never use it for deleting.

While I don't think deleting attributes is a very common thing
to do in any Python code base (unless you need to break circular
references or explicitly want to free resources), the
fact that PyObject_DelAttr() itself is implemented as macro
using the NULL attribute value clearly creates an API incompatibility
when removing this functionality or generating warnings, since
all code using the correct PyObject_DelAttr() at the moment,
would then trigger the warning as well.

As a result, the deprecation would have to be extended across
more releases than the usual cycle.

A first step would be to replace the macro with a proper function
to avoid false positive warnings, even when using the correct API.

Then we could add a warning to the PyObject_SetAttr() function and
hope that not too many projects use the stable ABI as basis to
have C extensions work across several releases.

Overall, I'm not sure whether it's worth the trouble. Documenting
the feature and adding a deprecation notice to just the documentation
would likely be better. We could then remove the functionality
in Python 4.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Dec 02 2015)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...   http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...   http://zope.egenix.com/

::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
  http://www.malemburg.com/

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS verification backport guidance)

2015-11-24 Thread M.-A. Lemburg

I think the PEP is a good step forward to compromise between
the crypto purists (use whatever technologies makes us more
secure even if it breaks things) and those who cannot upgrade
their Python 2.7 because of the PEP 476 change, since it causes their
applications to fail (e.g. because the embedded devices they want to
interface to only support self signed certs).

I would still find having built-in support for the recommendations
in the Python stdlib a better approach, but PEP 493 is good enough
in at least solving real problems people are having.

PS: Would be great to have a PyPI package which implements these
recommendations so that you can simply add it as dependency - and
then please for Python 3 as well, since people with embedded
devices will want to be able to use Python 3 as well ;-)

On 24.11.2015 15:27, Laura Creighton wrote:
> In a message of Tue, 24 Nov 2015 14:05:53 +, Paul Moore writes:
>> Simply adding "people who have no control over their broken
>> infrastructure" with a note that this PEP helps them, would be
>> sufficient here (and actually helps the case for the PEP, so why not?
>> ;-))
> 
> But does it help them?  Or does it increase the power of those who
> hand out certificates and who are intensely security conscious over
> those who would like to get some work done this afternoon?
> 
> Laura
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/mal%40egenix.com
> 

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Nov 24 2015)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...   http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...   http://zope.egenix.com/

::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
  http://www.malemburg.com/

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python stdlib ssl.SSLContext is missing mode setting ability

2015-11-19 Thread M.-A. Lemburg

On 19.11.2015 09:14, Cory Benfield wrote:
> 
>> On 19 Nov 2015, at 03:53, Ben Bangert  wrote:
>>
>> In Python 2 and 3, the ssl module's SSLContext object has a way to set
>> SSL options, but not to set SSL modes.
>>
>> The set_mode command and some of the available modes:
>> https://www.openssl.org/docs/manmaster/ssl/SSL_CTX_set_mode.html
>>
>> The most critical mode is SSL_MODE_RELEASE_BUFFERS, which can drop the
>> SSL overhead *per connection* from around 25kb to ~7kb. The pyopenssl
>> library allows the setting of SSLContext modes, it seems very odd that
>> the Python 2/3 ssl modules do not. Though I could understand that
>> perhaps not all SSL libraries Python might build against would have
>> this mode thing available.
>>
> 
> Ben,
> 
> Do we need the ability to set arbitrary modes? Most of the modes mentioned in 
> the OpenSSL documentation are things we actively don’t want the user to set 
> because stuff will randomly break. With that in mind, and with the fact that 
> SSL_MODE_RELEASE_BUFFERS is so obviously better than the standard, should we 
> just instead have the ssl module automatically set SSL_MODE_RELEASE_BUFFERS 
> unconditionally?
> 
> If so, I’m happy to submit a bug/patch to get that to happen.

The mode should only be enabled for OpenSSL versions which are
not affected by this vulnerability:

https://www.rapid7.com/db/vulnerabilities/http-openssl-cve-2014-0198

Other than that it seems like a good way forward. Plenty other
projects have had this enabled per default for years:

http://www.dovecot.org/list/dovecot/2011-October/131381.html
https://svn.boost.org/trac/boost/changeset/71706
https://community.openvpn.net/openvpn/ticket/157

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Nov 19 2015)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...   http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...   http://zope.egenix.com/

2015-10-23: Released mxODBC Connect 2.1.5 ... http://egenix.com/go85

::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
  http://www.malemburg.com/

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Reading Python source file

2015-11-19 Thread M.-A. Lemburg

On 17.11.2015 16:22, Guido van Rossum wrote:
> On Tue, Nov 17, 2015 at 1:59 AM, M.-A. Lemburg <m...@egenix.com> wrote:
>>> [moving from read source line by line to reading all in one go]
>> We use the same simplification in eGenix PyRun's emulation of
>> the Python command line interface and it has so far not
>> caused any problems.
> 
> Curious how you do it? I'd actually be quite disappointed if the
> amount of parsing done by the standard REPL went down.

Oh, that's easy:

elif sys.argv[0] == '-' and not (pyrun_as_string or pyrun_as_module):
# Read the script from stdin
pyrun_as_string = True
pyrun_script = sys.stdin.read()

and then, later on:

# Run the script
try:
pyrun_execute_script(pyrun_script, mode)
except Exception as reason:
if pyrun_interactive:
import traceback
traceback.print_exc()
pyrun_prompt(banner='')
else:
raise
else:
# Enter interactive mode, in case wanted
if pyrun_interactive:
pyrun_prompt()

The REPL is not affected by this, since we use the standard
code.interact() for the prompt. This reads the entry line
by line, joins the lines and tries to compile the entry every
time it receives a new line until it succeeds or fails.

Serhiy's proposed change should not affect this mode of
operation.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Nov 19 2015)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...   http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...   http://zope.egenix.com/

2015-10-23: Released mxODBC Connect 2.1.5 ... http://egenix.com/go85

::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
  http://www.malemburg.com/

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Reading Python source file

2015-11-17 Thread M.-A. Lemburg

On 17.11.2015 02:53, Serhiy Storchaka wrote:
> I'm working on rewriting Python tokenizer (in particular the part that reads 
> and decodes Python
> source file). The code is complicated. For now there are such cases:
> 
> * Reading from the string in memory.
> * Interactive reading from the file.
> * Reading from the file:
>   - Raw reading ignoring encoding in parser generator.
>   - Raw reading UTF-8 encoded file.
>   - Reading and recoding to UTF-8.
> 
> The file is read by the line. It makes hard to check correctness of the first 
> line if the encoding
> is specified in the second line. And it makes very hard problems with null 
> bytes and with
> desynchronizing buffered C and Python files. All this problems can be easily 
> solved if read all
> Python source file in memory and then parse it as string. This would allow to 
> drop a large complex
> and buggy part of code.
> 
> Are there disadvantages in this solution? As for memory consumption, the 
> source text itself will
> consume only small part of the memory consumed by AST tree and other 
> structures. As for performance,
> reading and decoding all file can be faster then by the line.

A problem with this approach is that you can no
longer fail early and detect indentation errors et al. while
parsing the data (which may well come from a pipe).

Another related problem is that you have to wait for the full
input data before you can start compiling the code.

I don't think these situations are all that common, though,
so reading in the full source code before compiling it
sounds like a reasonable approach.

We use the same simplification in eGenix PyRun's emulation of
the Python command line interface and it has so far not
caused any problems.

> [1] http://bugs.python.org/issue25643

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Nov 17 2015)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...   http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...   http://zope.egenix.com/

2015-10-23: Released mxODBC Connect 2.1.5 ... http://egenix.com/go85

::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
  http://www.malemburg.com/

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Support of UTF-16 and UTF-32 source encodings

2015-11-15 Thread M.-A. Lemburg

On 14.11.2015 23:56, Victor Stinner wrote:
> These encodings are rarely used. I don't think that any text editor use
> them. Editors use ascii, latin1, utf8 and... all locale encoding. But I
> don't know any OS using UTF-16 as a locale encoding. UTF-32 wastes disk
> space.

UTF-16 is used a lot for Windows text files, e.g. Unicode
CSV files (the save as "Unicode text file" option writes
UTF-16).

However, nowadays, all text editors also support UTF-8 and
many of these recognize the UTF-8 BOM as identifier to detect
Unicode text files.

> Ok, even if it exists, Python already accepts a very wide range of
> encoding. It is not worth to make the parser much more complex just to
> support encodings which are also never used (for .py files).

Agreed. In Python 2 we decided to only allow ASCII super-sets
for Python source files, which out ruled multi-byte encodings
such as UTF-16 and -32. I don't think we need to make the parser
more complex just to support them. UTF-8 works fine as Python
source code encoding.

> Victor
> Le 14 nov. 2015 20:20, "Serhiy Storchaka"  a écrit :
> 
>> For now UTF-16 and UTF-32 source encodings are not supported. There is a
>> comment in Parser/tokenizer.c:
>>
>> /* Disable support for UTF-16 BOMs until a decision
>>is made whether this needs to be supported.  */
>>
>> Can we make a decision whether this support will be added in foreseeable
>> future (say in near 10 years), or no?
>>
>> Removing commented out and related code will help to refactor the
>> tokenizer, and that can help to fix some existing bugs (e.g. issue14811,
>> issue18961, issue20115 and may be others). Current tokenizing code is too
>> tangled.
>>
>> If the support of UTF-16 and UTF-32 is planned, I'll take this to
>> attention during refactoring. But in many places besides the tokenizer the
>> ASCII compatible encoding of source files is expected.
>>
>> ___
>> Python-Dev mailing list
>> Python-Dev@python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com
>>
> 
> 
> 
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/mal%40egenix.com
> 

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Nov 15 2015)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...   http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...   http://zope.egenix.com/

2015-10-23: Released mxODBC Connect 2.1.5 ... http://egenix.com/go85

::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
  http://www.malemburg.com/

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Translate Python language

2015-11-11 Thread M.-A. Lemburg

On 11.11.2015 17:20, Donald Stufft wrote:
> On November 11, 2015 at 11:19:07 AM, Paul Moore (p.f.mo...@gmail.com) wrote:
>> On 11 November 2015 at 15:13, Christophe Bal wrote:
>>> Hello.
>>>
>>> I'm a french teacher and I would like to use Python with young child but
>>> I've a big problem. All the keyword are in english. So I would like to know
>>> if there is a way to hack the AST tools such as to use french keywords and
>>> then translate a list of french keywords to their english regular version.
>>>
>>> Where should I start ? My idea is not to build a new language but simply
>>> translate words using an hacked version of the AST tools.
>>>
>>> Hoping to be clear and I do not pollute the message in this list.
>>
>> Do you really just mean keywords? Most of the English words used in
>> Python (for example "open") aren't actually keywords but are names of
>> functions defined in the builtins and/or standard library. Those are
>> not exposed as distinct objects in the AST, but are simply name
>> lookups.
>>
> 
> You might be able to use an encoding to do this.

Or use Andrew Dalke's fun project LOLPython as basis:

http://www.dalkescientific.com/writings/diary/archive/2007/06/01/lolpython.html

It uses a parser which then translates the source code
into regular Python source code, which it then runs.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Nov 11 2015)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...   http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...   http://zope.egenix.com/

2015-10-23: Released mxODBC Connect 2.1.5 ... http://egenix.com/go85

::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
  http://www.malemburg.com/

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 0484 - the Numeric Tower

2015-10-14 Thread M.-A. Lemburg

On 14.10.2015 01:37, Raymond Hettinger wrote:
> 
>> On Oct 13, 2015, at 9:16 AM, Random832  wrote:
>>
>>> ## 
>>> ## Decimal has all of the methods specified by the Real abc, but it should
>>> ## not be registered as a Real because decimals do not interoperate with
>>> ## binary floats (i.e.  Decimal('3.14') + 2.71828 is undefined).  But,
>>> ## abstract reals are expected to interoperate (i.e. R1 + R2 should be
>>> ## expected to work if R1 and R2 are both Reals).
>>
>> Why?
> 
> Q.  Why is Python the way it is?
> A.   Because Guido said so ;-)
> 
> IIRC, the answer is that we were being conservative with possibly unintended 
> operations between types with differing precision and with differing notions 
> of what numbers could be exactly representable.
> 
> We could have (and still could) make the choice to always coerce to decimal 
> (every float is exactly representable in decimal).  Further, any decimal 
> float or binary float could be losslessly coerced to a Fraction, but that 
> probably isn't what you really want most of the time.  I think people who 
> work in decimal usually want to stay there and people who work with binary 
> floating point want to stay there as well (invisible coercions being more 
> likely to cause pain than relieve pain).

I can only underline this. Conversion to decimals or fractions should
not be implicit. People needing these types will know when they need
them and apply the required explicit conversions to fit their use case.

E.g. in accounting you'll likely use decimal, in finance and science
you stick to floats.

>From a theoretical point of view, it would make sense to add coercion
to these types, but not from a practical point of view.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...   http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...   http://zope.egenix.com/

: Try our mxODBC.Connect Python Database Interface for free ! ::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [python-committers] How are we merging forward from the Bitbucket 3.5 repo?

2015-08-16 Thread M.-A. Lemburg

On 16.08.2015 16:08, Guido van Rossum wrote:
 I presume the issue here is that Hg is so complicated that everyone knows a
 different subset of the commands and semantics.
 
 I personally don't know what the commands for cherry-picking a revision
 would be.
 
 I also don't know exactly what happens when you merge a PR using bitbucket.
 (I'm only familiar with the GitHub PR flow, and I don't like its behavior,
 which seems to always create an extra merge revision for what I consider as
 logically a single commit.)
 
 BTW When I go to https://bitbucket.org/larry/cpython350 the first thing I
 see (in a very big bold font) is This is Python version 3.6.0 alpha 1.
 What's going on here? It doesn't inspire confidence.

You are probably looking at the default branch within that repo fork.

This is the 3.5 branch:

https://bitbucket.org/larry/cpython350/branch/3.5

 On Sun, Aug 16, 2015 at 10:13 AM, Larry Hastings la...@hastings.org wrote:
 


 So far I've accepted two pull requests into bitbucket.com/larry/cpython350
 in the 3.5 branch, what will become 3.5.0rc2.  As usual, it's the
 contributor's responsibility to merge forward; if their checkin goes in to
 3.5, it's their responsibility to also merge it into the
 hg.python.org/cpython 3.5 branch (which will be 3.5.1) and default branch
 (which right now is 3.6).

 But... what's the plan here?  I didn't outline anything specific, I just
 assumed we'd do the normal thing, pulling from 3.5.0 into 3.5.1 and
 merging.  But of the two pull requests so far accepted, one was merged this
 way, though it cherry-picked the revision (skipping the pull request merge
 revision Bitbucket created), and one was checked in to 3.5.1 directly (no
 merging).

 I suppose we can do whatever we like.  But it'd be helpful if we were
 consistent.  I can suggest three approaches:

1. After your push request is merged, you cherry-pick your revision
from bitbucket.com/larry/cpython350 into hg.python.org/cpython and
merge.  After 3.5.0 final is released I do a big null merge from
bitbucket.com/larry/cpython350 into hg.python.org/cpython.
2. After your push request is merged, you manually check in a new
equivalent revision into hg.python.org/cpython in the 3.5 branch.  No
merging necessary because from Mercurial's perspective it's unrelated to
the revision I accepted.  After 3.5.0 final is released I do a big null
merge from bitbucket.com/larry/cpython350 into hg.python.org/cpython.
3. After your push request is merged, you pull from
bitbucket.com/larry/cpython350 into hg.python.org/cpython and merge
into 3.5.  In this version I don't have to do a final null merge!

 I'd prefer 3; that's what we normally do, and that's what I was
 expecting.  So far people have done 1 and 2.

 Can we pick one approach and stick with it?  Pretty-please?


 */arry*

 ___
 python-committers mailing list
 python-committ...@python.org
 https://mail.python.org/mailman/listinfo/python-committers


 
 
 
 
 ___
 python-committers mailing list
 python-committ...@python.org
 https://mail.python.org/mailman/listinfo/python-committers
 

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Aug 16 2015)
 Python Projects, Coaching and Consulting ...  http://www.egenix.com/
 mxODBC Plone/Zope Database Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/

2015-08-12: Released mxODBC 3.3.4 ... http://egenix.com/go80
2015-08-22: FrOSCon 2015 ...6 days to go

: Try our mxODBC.Connect Python Database Interface for free ! ::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Backporting the 3.5+ Windows build project files to 2.7

2015-06-25 Thread M.-A. Lemburg

On 22.06.2015 19:03, Zachary Ware wrote:
 Hi,
 
 As you may know, Steve Dower put significant effort into rewriting the
 project files used by the Windows build as part of moving to VC14 as
 the official compiler for Python 3.5.  Compared to the project files
 for 3.4 (and older), the new project files are smaller, cleaner,
 simpler, more easily extensible, and in some cases quite a bit more
 correct.
 
 I'd like to backport those new project files to 2.7, and Intel is
 willing to fund that work as part of making Python ICC compilable on
 all platforms they support since it makes building Python 2.7 with ICC
 much easier.  I have no intention of changing the version of MSVC used
 for official 2.7 builds, it *will* remain at MSVC 9.0 (VS 2008) as
 determined the last time we had a thread about it.  VS 2010 and newer
 can access older compilers (back to 2008) as a 'PlatformToolset' if
 they're installed, so all we have to do is set the PlatformToolset in
 the projects at 'v90'.  Backporting the projects would make it easier
 to build 2.7 with a newer compiler, but that's already possible if
 somebody wants to put enough work into it, the default will be the old
 compiler, and we can emit big scary warnings if somebody does use a
 compiler other than v90.
 
 With the stipulation that the officially supported compiler won't
 change, I want to make sure there's no major opposition to replacing
 the old project files in PCbuild.  The old files would move to
 PC\VS9.0, so they'll still be available and usable if necessary.
 
 Using the backported project files to build 2.7 would require two
 versions of Visual Studio to be installed; VS2010 (or newer) would be
 required in addition to VS2008.  All Windows core developers should
 already have VS2010 for Python 3.4 (and/or VS2015 for 3.5) and I
 expect that anyone else who cares enough to still have VS2008 probably
 has (or can easily get) one of the free editions of VS 2010 or newer,
 so I don't consider this to be a major issue.

To understand this correctly:

In order to build Python 2.7.11 (or a later version), we'd
now need VS 2008 *and* VS 2010 installed ?

This doesn't sound like it would make things easier for
people who need to build their own Python on Windows, e.g.
because they are freezing their apps for distribution, or
using their own internal special builds.

For VS 2008 we now have a long-term solution thanks to MS.
The same is not true for VS 2010 (download links are not official
anymore), so we'd complicate things again by requiring the
mixed compiler setup.

 The backported files could be added alongside the old files in
 PCbuild, in a better-named 'NewPCbuild' directory, or in a
 subdirectory of PC. I would rather replace the old project files in
 PCbuild, though; I'd like for the backported files to be the
 recommended way to build, complete with support from PCbuild/build.bat
 which would make the new project files the default for the buildbots.

If you keep the old VS 2008 build files around, I'd be
fine with having an optional upgrade path to newer
compilers :-)

It's rather unlikely that the project files will change much
in coming years, so there shouldn't be much maintenance
effort.

 I have a work-in-progress branch with the backported files in PCbuild,
 which you can find at
 https://hg.python.org/sandbox/zware/log/project_files_backport.  There
 are still a couple bugs to work out (and a couple unrelated changes to
 PC/pyconfig.h), but most everything works as expected.
 
 Looking forward to hearing others' opinions,

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jun 25 2015)
 Python Projects, Coaching and Consulting ...  http://www.egenix.com/
 mxODBC Plone/Zope Database Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/

2015-06-25: Released mxODBC 3.3.3 ... http://egenix.com/go79
2015-06-16: Released eGenix pyOpenSSL 0.13.10 ... http://egenix.com/go78
2015-07-20: EuroPython 2015, Bilbao, Spain ... 25 days to go

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Backporting the 3.5+ Windows build project files to 2.7

2015-06-25 Thread M.-A. Lemburg

On 25.06.2015 17:12, Zachary Ware wrote:
 On Thu, Jun 25, 2015 at 8:54 AM, M.-A. Lemburg m...@egenix.com wrote:
 On 22.06.2015 19:03, Zachary Ware wrote:
 Using the backported project files to build 2.7 would require two
 versions of Visual Studio to be installed; VS2010 (or newer) would be
 required in addition to VS2008.  All Windows core developers should
 already have VS2010 for Python 3.4 (and/or VS2015 for 3.5) and I
 expect that anyone else who cares enough to still have VS2008 probably
 has (or can easily get) one of the free editions of VS 2010 or newer,
 so I don't consider this to be a major issue.

 To understand this correctly:

 In order to build Python 2.7.11 (or a later version), we'd
 now need VS 2008 *and* VS 2010 installed ?
 
 Using the backported project files, yes, two versions of VS would be
 required.  However, the second version doesn't have to be VS 2010, it
 could be 2010, 2012, 2013, 2015, or any from the future (until
 Microsoft changes the format of the project files again).
 
 This doesn't sound like it would make things easier for
 people who need to build their own Python on Windows, e.g.
 because they are freezing their apps for distribution, or
 using their own internal special builds.

 For VS 2008 we now have a long-term solution thanks to MS.
 The same is not true for VS 2010 (download links are not official
 anymore), so we'd complicate things again by requiring the
 mixed compiler setup.
 
 For anyone building 2.7 and any other (current) version of Python on
 the same machine, all the necessary pieces are already in place.
 
 The main motivation for backporting is to make it easier to build
 Python with ICC.  Pre-backport, Intel gave me a list of 9 steps they
 were doing to build 2.7.10 with ICC.  Post-backport, it's just
 `PCbuild\build.bat -e [-p x64] [-d] /p:PlatformToolset=Intel C++
 Compiler XE 15.0 /p:BasePlatformToolset=v90` (the bracketed options
 being optional, for 64-bit and debug builds, respectively).

Sounds good.

BTW: I remember there was some discussion a while ago to
get ICC licenses to core devs. Has there been any progress
on this ?

 There are some benefits for MSVC builds too, though: it's easier to do
 a parallel build, and OpenSSL and Tcl/Tk/Tix are built by the project
 files, so the whole build can be done much quicker.  The new project
 files for Tcl and Tk also take care of copying their DLLs into PCbuild
 so that _tkinter can find them without having to mess around with
 PATH.  My patch also fixes a couple of long-standing bugs with finding
 the right libraries for 64-bit Tcl/Tk and the test suite undoing the
 work of FixTk.py after any test that imports Tkinter, both of which
 were exacerbated by the rest of the patch.
 
 If you keep the old VS 2008 build files around, I'd be
 fine with having an optional upgrade path to newer
 compilers :-)
 
 The old files are moved to PC/VS9.0, and they work as expected as far
 as I've tested them. 

So it's still possible to build with just VS 2008 installed
or will the VS 2010 (or later) be required for those old
files as well ?

 Btw, the upgrade path to newer compilers is just
 a side-effect which I'm kind of torn about.  On one hand, making it
 easier to build 2.7 with the wrong compiler is bound to lead to
 issues somewhere somehow.  On the other hand, people are already doing
 so anyway, with their own upgraded project files.

I guess will have the discussion about switching compilers
for Python 2.7 again at some point. People writing extensions
will sooner or later want to use features from more recent
compilers for Python 2.7 as well :-)

 It's rather unlikely that the project files will change much
 in coming years, so there shouldn't be much maintenance
 effort.
 
 Hopefully not :) 

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jun 25 2015)
 Python Projects, Coaching and Consulting ...  http://www.egenix.com/
 mxODBC Plone/Zope Database Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/

2015-06-25: Released mxODBC 3.3.3 ... http://egenix.com/go79
2015-06-16: Released eGenix pyOpenSSL 0.13.10 ... http://egenix.com/go78
2015-07-20: EuroPython 2015, Bilbao, Spain ... 25 days to go

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] speed.python.org

2015-06-23 Thread M.-A. Lemburg

On 23.06.2015 03:58, Zachary Ware wrote:
On Thu, Jun 4, 2015 at 10:51 AM, Maciej Fijalkowski fij...@gmail.com wrote:
On Thu, Jun 4, 2015 at 4:32 PM, R. David Murray rdmur...@bitdance.com
wrote:
OK, so what you are saying is that speed.python.org will run a buildbot
slave so that when a change is committed to cPython, a speed run will be
triggered? Is the runner a normal buildbot slave, or something
custom? In the normal case the master controls what the slave
runs...but regardless, you'll need to let us know how the slave
invocation needs to be configured on the master.

Ideally nightly (benchmarks take a while). The setup for pypy looks like
this:

https://bitbucket.org/pypy/buildbot/src/5fa1f1a4990f842dfbee416c4c2e2f6f75d451c4/bot2/pypybuildbot/builds.py?at=default#cl-734

so fairly easy. This already generates a json file that you can plot.
We can setup an upload automatically too.

I've been looking at what it will take to set up the buildmaster for
this, and it looks like it's just a matter of checking out the
benchmarks, building Python, testing it, and running the benchmarks.
There is the question of which benchmark repo to use:
https://bitbucket.org/pypy/benchmarks or
https://hg.python.org/benchmarks; ideally, we should use
hg.python.org/benchmarks for CPython benchmarks, but it looks like
pypy/benchmarks has the necessary runner, so I suppose we'll be using
it for now. Is there interest from both sides to merge those
repositories?

The big question for the buildmaster is what options to pass to the
benchmark runner. I suppose most of them should match the
CPythonBenchmark BuildFactory from the PyPy buildbot master
configuration, but otherwise I'm not sure.

The other big question is where the benchmarks will be run. The
speed.python.org page makes it sound like there's a box intended for
that purpose (the one running the speed.python.org page?); can anyone
with access to it contact me to get the build slave set up?

Yes, I believe the machine is currently only running that page.

I've pinged the PSF Infra Team to get you access to it.

Thank you for looking into this !

Cheers,
--
Marc-Andre Lemburg
Director
Python Software Foundation
http://www.python.org/psf/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 490: Chain exceptions at C level

2015-06-20 Thread M.-A. Lemburg

On 20.06.2015 09:30, Victor Stinner wrote:
 Hi,
 
 I didn't get much feedback on this PEP. Since the Python 3.6 branch is
 open (default), it's probably better to push such change in the
 beginning of the 3.6 cycle, to catch issues earlier.
 
 Are you ok to chain exceptions at C level by default?

I think it's a good idea to make C APIs available to
simplify chaining exceptions at the C level, but don't
believe that always doing this by default is a good idea.
It should really be a case-by-case decision, IMO.

Note that Python exceptions are cheap to raise in C
(very much unlike in Python), so making this more
expensive by default would introduce a significant
overhead - without much proven benefit.

More below...

 PEP: 490
 Title: Chain exceptions at C level
 Version: $Revision$
 Last-Modified: $Date$
 Author: Victor Stinner victor.stin...@gmail.com
 Status: Draft
 Type: Standards Track
 Content-Type: text/x-rst
 Created: 25-March-2015
 Python-Version: 3.6
 
 
 Abstract
 
 
 Chain exceptions at C level, as already done at Python level.
 
 
 Rationale
 =
 
 Python 3 introduced a new killer feature: exceptions are chained by
 default, PEP 3134.
 
 Example::
 
 try:
 raise TypeError(err1)
 except TypeError:
 raise ValueError(err2)
 
 Output::
 
 Traceback (most recent call last):
   File test.py, line 2, in module
 raise TypeError(err1)
 TypeError: err1
 
 During handling of the above exception, another exception occurred:
 
 Traceback (most recent call last):
   File test.py, line 4, in module
 raise ValueError(err2)
 ValueError: err2
 
 Exceptions are chained by default in Python code, but not in
 extensions written in C.
 
 A new private ``_PyErr_ChainExceptions()`` function was introduced in
 Python 3.4.3 and 3.5 to chain exceptions. Currently, it must be called
 explicitly to chain exceptions and its usage is not trivial.
 
 Example of ``_PyErr_ChainExceptions()`` usage from the ``zipimport``
 module to chain the previous ``OSError`` to a new ``ZipImportError``
 exception::
 
 PyObject *exc, *val, *tb;
 PyErr_Fetch(exc, val, tb);
 PyErr_Format(ZipImportError, can't open Zip file: %R, archive);
 _PyErr_ChainExceptions(exc, val, tb);
 
 This PEP proposes to also chain exceptions automatically at C level to
 stay consistent and give more information on failures to help
 debugging. The previous example becomes simply::
 
 PyErr_Format(ZipImportError, can't open Zip file: %R, archive);
 
 
 Proposal
 
 
 Modify PyErr_*() functions to chain exceptions
 --
 
 Modify C functions raising exceptions of the Python C API to
 automatically chain exceptions: modify ``PyErr_SetString()``,
 ``PyErr_Format()``, ``PyErr_SetNone()``, etc.
 
 
 Modify functions to not chain exceptions
 
 
 Keeping the previous exception is not always interesting when the new
 exception contains information of the previous exception or even more
 information, especially when the two exceptions have the same type.
 
 Example of an useless exception chain with ``int(str)``::
 
 TypeError: a bytes-like object is required, not 'type'
 
 During handling of the above exception, another exception occurred:
 
 Traceback (most recent call last):
   File stdin, line 1, in module
 TypeError: int() argument must be a string, a bytes-like object or
 a number, not 'type'
 
 The new ``TypeError`` exception contains more information than the
 previous exception. The previous exception should be hidden.
 
 The ``PyErr_Clear()`` function can be called to clear the current
 exception before raising a new exception, to not chain the current
 exception with a new exception.
 
 
 Modify functions to chain exceptions
 
 
 Some functions save and then restore the current exception. If a new
 exception is raised, the exception is currently displayed into
 sys.stderr or ignored depending on the function.  Some of these
 functions should be modified to chain exceptions instead.
 
 Examples of function ignoring the new exception(s):
 
 * ``ptrace_enter_call()``: ignore exception
 * ``subprocess_fork_exec()``: ignore exception raised by enable_gc()
 * ``t_bootstrap()`` of the ``_thread`` module: ignore exception raised
   by trying to display the bootstrap function to ``sys.stderr``
 * ``PyDict_GetItem()``, ``_PyDict_GetItem_KnownHash()``: ignore
   exception raised by looking for a key in the dictionary
 * ``_PyErr_TrySetFromCause()``: ignore exception
 * ``PyFrame_LocalsToFast()``: ignore exception raised by
   ``dict_to_map()``
 * ``_PyObject_Dump()``: ignore exception. ``_PyObject_Dump()`` is used
   to debug, to inspect a running process, it should not modify the
   Python state.
 * ``Py_ReprLeave()``: ignore exception because there is no way to
   report them
 * ``type_dealloc()``: ignore exception raised by

Re: [Python-Dev] speed.python.org (was: 2.7 is here until 2020, please don't call it a waste.)

2015-06-04 Thread M.-A. Lemburg

On 04.06.2015 04:08, Tetsuya Morimoto wrote:
 If someone were to volunteer to set up and run speed.python.org, I think
 we could add some additional focus on performance regressions. Right now,
 we don't have any way of reliably and reproducibly testing Python
 performance.
 
 I'm very interested in speed.python.org and feel regret that the project is
 standing still. I have a mind to contribute something ...

On 03.06.2015 18:59, Maciej Fijalkowski wrote: On Wed, Jun 3, 2015 at 3:49 PM, 
R. David Murray
 I think we should look into getting speed.python.org up and
 running for both Python 2 and 3 branches:

  https://speed.python.org/

 What would it take to make that happen ?

 I guess ideal would be some cooperation from some of the cpython devs,
 so say someone can setup cpython buildbot

 What does set up cpython buildbot mean in this context?

 The way it works is dual - there is a program running the benchmarks
 (the runner) which is in the pypy case run by the pypy buildbot and
 the web side that reports stuff. So someone who has access to cpython
 buildbot would be useful.

Ok, so there's interest and we have at least a few people who are
willing to help.

Now we need someone to take the lead on this and form a small
project group to get everything implemented. Who would be up
to such a task ?

The speed project already has a mailing list, so you could use
that for organizing the details.

We could also create a PSF work group and assign a budget to it,
if that helps.

If you need help with all this, let me know.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jun 04 2015)
 Python Projects, Coaching and Consulting ...  http://www.egenix.com/
 mxODBC Plone/Zope Database Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/


: Try our mxODBC.Connect Python Database Interface for free ! ::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] 2.7 is here until 2020, please don't call it a waste.

2015-06-03 Thread M.-A. Lemburg

On 02.06.2015 21:07, Maciej Fijalkowski wrote:
 Hi
 
 There was a PSF-sponsored effort to improve the situation with the
 https://bitbucket.org/pypy/codespeed2/src being written (thank you
 PSF). It's not better enough than codespeed that I would like, but
 gives some opportunities.
 
 That said, we have a benchmark machine for benchmarking cpython and I
 never deployed nightly benchmarks of cpython for a variety of reasons.
 
 * would be cool to get a small VM to set up the web front
 
 * people told me that py3k is only interesting, so I did not set it up
 for py3k because benchmarks are mostly missing
 
 I'm willing to set up a nightly speed.python.org using nightly build
 on python 2 and possible python 3 if there is an interest. I need
 support from someone maintaining python buildbot to setup builds and a
 VM to set up stuff, otherwise I'm good to go
 
 DISCLAIMER: I did facilitate in codespeed rewrite that was not as
 successful as I would have hoped. I did not receive any money from the
 PSF on that though.

I think we should look into getting speed.python.org up and
running for both Python 2 and 3 branches:

 https://speed.python.org/

What would it take to make that happen ?

 Cheers,
 fijal
 
 
 On Mon, Jun 1, 2015 at 1:14 PM, M.-A. Lemburg m...@egenix.com wrote:
 On 01.06.2015 12:44, Armin Rigo wrote:
 Hi Larry,

 On 31 May 2015 at 01:20, Larry Hastings la...@hastings.org wrote:
 p.s. Supporting this patch also helps cut into PyPy's reported performance
 lead--that is, if they ever upgrade speed.pypy.org from comparing against
 Python *2.7.2*.

 Right, we should do this upgrade when 2.7.11 is out.

 There is some irony in your comment which seems to imply PyPy is
 cheating by comparing with an old Python 2.7.2: it is inside a thread
 which started because we didn't backport performance improvements to
 2.7.x so far.

 Just to convince myself, I just ran a performance comparison.  I ran
 the same benchmark suite as speed.pypy.org, with 2.7.2 against 2.7.10,
 both freshly compiled with no configure options at all.  The
 differences are usually in the noise, but range from +5% to... -60%.
 If anything, this seems to show that CPython should take more care
 about performance regressions.  If someone is interested:

 * raytrace-simple is 1.19 times slower
 * bm_mako is 1.29 times slower
 * spitfire_cstringio is 1.60 times slower
 * a number of other benchmarks are around 1.08.

 The 7.0x faster number on speed.pypy.org would be significantly
 *higher* if we upgraded the baseline to 2.7.10 now.

 If someone were to volunteer to set up and run speed.python.org,
 I think we could add some additional focus on performance
 regressions. Right now, we don't have any way of reliably
 and reproducibly testing Python performance.

 Hint: The PSF would most likely fund such adventures :-)

 --
 Marc-Andre Lemburg
 eGenix.com

 Professional Python Services directly from the Source  (#1, Jun 01 2015)
 Python Projects, Coaching and Consulting ...  http://www.egenix.com/
 mxODBC Plone/Zope Database Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/
 

 : Try our mxODBC.Connect Python Database Interface for free ! ::

eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
Registered at Amtsgericht Duesseldorf: HRB 46611
http://www.egenix.com/company/contact/
 ___
 Python-Dev mailing list
 Python-Dev@python.org
 https://mail.python.org/mailman/listinfo/python-dev
 Unsubscribe: 
 https://mail.python.org/mailman/options/python-dev/fijall%40gmail.com
 ___
 Python-Dev mailing list
 Python-Dev@python.org
 https://mail.python.org/mailman/listinfo/python-dev
 Unsubscribe: 
 https://mail.python.org/mailman/options/python-dev/mal%40egenix.com
 

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jun 03 2015)
 Python Projects, Coaching and Consulting ...  http://www.egenix.com/
 mxODBC Plone/Zope Database Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/


: Try our mxODBC.Connect Python Database Interface for free ! ::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] 2.7 is here until 2020, please don't call it a waste.

2015-06-01 Thread M.-A. Lemburg

On 01.06.2015 12:44, Armin Rigo wrote:
 Hi Larry,
 
 On 31 May 2015 at 01:20, Larry Hastings la...@hastings.org wrote:
 p.s. Supporting this patch also helps cut into PyPy's reported performance
 lead--that is, if they ever upgrade speed.pypy.org from comparing against
 Python *2.7.2*.
 
 Right, we should do this upgrade when 2.7.11 is out.
 
 There is some irony in your comment which seems to imply PyPy is
 cheating by comparing with an old Python 2.7.2: it is inside a thread
 which started because we didn't backport performance improvements to
 2.7.x so far.
 
 Just to convince myself, I just ran a performance comparison.  I ran
 the same benchmark suite as speed.pypy.org, with 2.7.2 against 2.7.10,
 both freshly compiled with no configure options at all.  The
 differences are usually in the noise, but range from +5% to... -60%.
 If anything, this seems to show that CPython should take more care
 about performance regressions.  If someone is interested:
 
 * raytrace-simple is 1.19 times slower
 * bm_mako is 1.29 times slower
 * spitfire_cstringio is 1.60 times slower
 * a number of other benchmarks are around 1.08.
 
 The 7.0x faster number on speed.pypy.org would be significantly
 *higher* if we upgraded the baseline to 2.7.10 now.

If someone were to volunteer to set up and run speed.python.org,
I think we could add some additional focus on performance
regressions. Right now, we don't have any way of reliably
and reproducibly testing Python performance.

Hint: The PSF would most likely fund such adventures :-)

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jun 01 2015)
 Python Projects, Coaching and Consulting ...  http://www.egenix.com/
 mxODBC Plone/Zope Database Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/


: Try our mxODBC.Connect Python Database Interface for free ! ::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Single-file Python executables (was: Computed Goto dispatch for Python 2)

2015-05-28 Thread M.-A. Lemburg

You might want to have a look at eGenix PyRun, which gives you
an almost complete Python runtime in 4-13MB (depending on what
startup performance needs you have):

http://www.egenix.com/products/python/PyRun/

On 28.05.2015 17:58, Barry Warsaw wrote:
 On May 28, 2015, at 11:39 AM, Donald Stufft wrote:
 
 You don’t need a fully functioning Python for a single file binary, you
 only need enough to actually run your application. For example, if you're
 making an application that can download files over HTTP, you don't need to
 include parts of the stdlib like xmlrpc, pickle, shelve, marshall, sqlite,
 csv, email, mailcap, mailbox, imaplib, nntplib, etc.
 
 There are actually two related but different use cases to single file
 executables.
 
 The first is nicely solved by tools like pex, where you don't need to include
 a fully functional Python at the head of the zip file because the environment
 you're deploying it into will have enough Python to make the zip work.  This
 can certainly result in smaller zip files.  This is the approach I took with
 Snappy Ubuntu Core support for Python 3, based on the current situation that
 the atomic upgrade client is written in Python 3.  If that changes and Python
 3 is removed from the image, then this approach won't work.
 
 pex (and others) does a great job at this, so unless there are things better
 refactored into upstream Python, I don't think we need to do much here.
 
 The second use case is as you describe: put a complete functional Python
 environment at the head of the zip file so you don't need anything in the
 target deployment environment.  Complete can easily mean the entire stdlib,
 and although that would usually be more bloated than you normally need, hey,
 it's just some extra unused bits so who cares? wink.  I think this would be
 an excellent starting point which can be optimized to trim unnecessary bits
 later, maybe by third party tools.

See above. This is what eGenix PyRun provides.

Our main motivation is to have a binary which works on all
Unix platforms, without having to rely on the way too many
system dependent Python distribution (with all their quirks
and whistles ;-)).

On Windows, we use py2exe at the moment, but a port of PyRun
to Windows would be possible as well. You'd still need the
separate Python DLL, though in order to stay compatible to
C extensions which link against this.

As for application packaging: we don't have a need to put
everything into a single ZIP file or even concatenate such
a ZIP file to PyRun (which is possible: just add sys.executable to
sys.path to import from the executable).

We have plans to create a tool to make such packaging possible,
though, since it's handy to have for building small executable
apps, e.g. to drive installations or larger applications.

 Of course deciding which pieces you include in the zip file you're appending
 to the end of Python is up to whatever tool builds this executable which
 doesn't need to be part of Python itself. If Python itself gained the ability
 to operate in that manner than third party tools could handle trying to do
 the optimizations where it only includes the things it actually needs in the
 stdlib and excludes things it doesn't. The key thing here is that since
 you're doing a single file binary, you don't need to have a Python which is
 suitable to execute random Python code, you only need one that is suitable to
 execute this particular code so you can specialize what that includes.
 
 I'd love to see Python itself gain such a tool, but if it had the critical
 pieces to execute in this way, that would enable a common approach to
 supporting this in third party tools, on a variety of platforms.
 
 I do think single-file executables are an important piece to Python's
 long-term competitiveness.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, May 28 2015)
 Python Projects, Coaching and Consulting ...  http://www.egenix.com/
 mxODBC Plone/Zope Database Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/


: Try our mxODBC.Connect Python Database Interface for free ! ::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Computed Goto dispatch for Python 2

2015-05-28 Thread M.-A. Lemburg

On 28.05.2015 02:17, Parasa, Srinivas Vamsi wrote:
 Hi All,
 
 This is Vamsi from Server Scripting Languages Optimization team at Intel 
 Corporation.
 
 Would like to submit a request to enable the computed goto based dispatch in 
 Python 2.x (which happens to be enabled by default in Python 3 given its 
 performance benefits on a wide range of workloads). We talked about this 
 patch with Guido and he encouraged us to submit a request on Python-dev 
 (email conversation with Guido shown at the bottom of this email).

+1.

It's been successful for Python 3, doesn't change semantics and
increases performance.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, May 28 2015)
 Python Projects, Coaching and Consulting ...  http://www.egenix.com/
 mxODBC Plone/Zope Database Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/


: Try our mxODBC.Connect Python Database Interface for free ! ::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PYTHONHTTPSVERIFY env var

On 12.05.2015 05:03, Nick Coghlan wrote:
 On 12 May 2015 at 04:49, M.-A. Lemburg m...@egenix.com wrote:
 On 11.05.2015 12:15, Nick Coghlan wrote:
 By contrast, the configuration file shouldn't provide a new attack
 vector (or simplify any existing attack vector), as if you have the
 permissions needed to modify the config file, you likely also have the
 permissions needed to modify the system certificate store, and the
 latter is a *far* more interesting attack vector than a downgrade
 attack solely against Python.

 Thus the environment variable based off switch is neither necessary
 (as an administrator controlled configuration file can do the job),
 nor sufficient (it can't handle the -E switch), *and* it represents an
 increase in the attack surface area relative to a Python
 implementation without the capability.

 Whether or not -E will have an effect on the env var depends
 on the implementation. At the moment, -E only has an effect
 on the C runtime, while the stdlib happily reads from os.environ
 without taking the flag into account.
 
 I had an off-list discussion with Christian Heimes about that in
 relation to the OpenSSL flags, and he pointed out the reason -E
 specifically needs to be a command line switch is because that is the
 only way to affect how environment variables are processed during
 interpreter startup. Once an application is up and running, further
 environment variable sanitisation can be handled at an application
 level by whitelisting entries in os.environ and deleting everything
 else.
 
 As proposed, the PYTHONHTTPSVERIFY would only affect the ssl
 module and only be checked when loading this module, i.e. not
 at Python startup time.
 
 Right, the same is true for the configuration file proposal.
 
 It would be nice to hear from ActiveState, Enthought  Continuum
 Analytics as well, but if they don't chime in here, I don't see any
 particular need to go chasing them explicitly.

 I think the approach to only consider a subset of redistributors
 as viable targets for such a switch is a bit too narrow.

 You are leaving out all the parties which use custom
 Python installations to run their applications, e.g.
 the Plone and Zope community, the ZenOSS community,
 the many Windows applications built on Python, etc.
 
 No, they already have a solution: monkeypatch (or just plain patch)
 the SSL module. That's an upstream supported technique, which is why
 it's documented in the PEP.

Again, this is not a proper solution for your friendly
sys admin. They won't go in and patch Python, but rather
consider it broken, if it doesn't provide a configuration
option and simply stay with 2.7.8.

We've had that discussion already, so I won't go into details
again.

 The problem we (as in Red Hat) ran into was that that technique
 *doesn't work* for the case of backporting PEP 476 to Python 2.7.5 as
 an opt-in feature.

And neither does it work for people deploying Windows apps
built on Python (where the code is usually hidden away
in a ZIP archive or even compiled into a DLL), or people
deploying Plone or ZenOSS.

It's not only Red Hat's customers that are affected :-)

 to fix this regression in 2.7.9.

 We made the decision when PEP 476 was accepted that this change turned
 a silent security failure into a noisy one, rather than being a
 regression in its own right. PEP 493 isn't about disagreeing with that
 decision, it's about providing a smoother upgrade path in contexts
 where letting the security failure remain silent is deemed to be
 preferred in the near term.

 The change wasn't regression. The missing downgrade path
 is a regression.
 
 It's a shame we don't have -X options in Python 2, as that would be
 a nice hard-to-attack option (although it wouldn't play well with
 subprocesses)
 
 Some other comments on PEP 493:

 * I don't think we really want to add the overhead of
   having to parse an INI file every time Python starts up.
   Please remember that we just parsing of the sysconfig
   data not long ago because we wanted to avoid this startup
   time.
 
 Compared to the overhead of reading from the system cert database,
 reading a config file at ssl module import time should be trivial.

The cert database is only read when importing the ssl
module, not with each Python startup, so that makes a
big difference.

The INI file would have to parsed at ssl module load time
as well to work around this.

 * I don't see why the attack surface of using an INI file
   somewhere in the system should be smaller than e.g. using
   sitecustomize.py
 
 You can put sitecustomize.py in a user directory, and if there's no
 system wide sitecustomize, Python will read it automatically (unless
 user site directories are turned off).

In a user based installation (which most applications shipping
their own Python installation are), you can always do this
provided you can gain the application user permissions.

For a system installed Python, you'd always have to run
Python with -E -S -s to avoid

Re: [Python-Dev] PYTHONHTTPSVERIFY env var

On 12.05.2015 13:21, Donald Stufft wrote:
 
 On May 12, 2015, at 7:17 AM, Nick Coghlan ncogh...@gmail.com wrote:

 On 12 May 2015 at 21:09, Donald Stufft don...@stufft.io wrote:
 If you control the app you don't need to do that. All relevant api accept 
 the context parameter. The shims are only useful when you don't control the 
 app. So an app shipping their own python doesn't fall under that.

 I think the bundled Python scenario MAL is interested in is this one:

 1. An application with a bundled CPython runtime is using the
 verification defaults
 2. Upgraded the bundled Python to 2.7.9
 3. Didn't provide new configuration settings to disable certificate 
 verification
 4. Is being upgraded in an environment where verifying certificates
 makes the app unusable for environmental reasons related to
 certificate management

 The PyRun single-file Python interpreter has a similar need, where
 some apps than ran fine under 2.7.8 will need a way to disable cert
 verification in 2.7.9+ on a per-application basis, *without* modifying
 the applications.

 Both of those make sense to me as cases where the environment variable
 based security downgrade approach is the least bad answer available,
 which is why I eventually agreed it should be one of the
 recommendations in the PEP.

 
 Why is without modifying the app a reasonable goal? If Python is bundled
 with the app then you have direct control over when that upgrade happens,
 so you can delay the upgrade to 2.7.9 until your application which is
 bundling Python has the relevant switches. This is distinctly different
 from a situation like downstream distributors where the version of Python
 being provided is being provided by a group different than the person
 providing the application.

Take a Plone Intranet as example:

The unified installer downloads and installs Python 2.7 for you.
As of Plone 4.3.3 the version is Python 2.7.6.

Now say you are a sys admin and your Intranet users are affected by
some bug in 2.7.6 which is fixed in 2.7.9. The natural approach would
be to upgrade the bundled Python to 2.7.9.

Because it's an Intranet and Plone is used to aggregate
information from other systems which use self-signed
certificates, you don't want to risk breaking your Plone
installation and need a way to disable the cert checks.

The best way to do this is by configuring the bundled
Python to disable the checks, since you would not want
to mess with the Plone application itself.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, May 12 2015)
 Python Projects, Coaching and Consulting ...  http://www.egenix.com/
 mxODBC Plone/Zope Database Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/


: Try our mxODBC.Connect Python Database Interface for free ! ::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PYTHONHTTPSVERIFY env var

On 12.05.2015 12:04, Donald Stufft wrote:
 
 On May 12, 2015, at 3:57 AM, M.-A. Lemburg m...@egenix.com wrote:

 In a user based installation (which most applications shipping
 their own Python installation are), you can always do this
 provided you can gain the application user permissions.
 
 Of course, if the application is shipping it’s own Python then
 it has to actually do something to update to 2.7.9 and it can
 add it’s own option to disable TLS verification. I personally
 think that the application providing that option is the *right* way
 and all these other things are, at best, just temporary shims until
 the applications do that.

I still believe that requiring to monkeypatch Python is a very poor
approach in terms of quality software design. We can do better and
we should.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, May 12 2015)
 Python Projects, Coaching and Consulting ...  http://www.egenix.com/
 mxODBC Plone/Zope Database Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/


: Try our mxODBC.Connect Python Database Interface for free ! ::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PYTHONHTTPSVERIFY env var

On 12.05.2015 13:19, Nick Coghlan wrote:
 On 12 May 2015 at 21:17, Nick Coghlan ncogh...@gmail.com wrote:
 Both of those make sense to me as cases where the environment variable
 based security downgrade approach is the least bad answer available,
 which is why I eventually agreed it should be one of the
 recommendations in the PEP.
 
 It occurs to me that the subtitle of PEP 493 could be All software is
 terrible, but it's often a system administrator's job to make it run
 anyway :)

+1 :-)

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, May 12 2015)
 Python Projects, Coaching and Consulting ...  http://www.egenix.com/
 mxODBC Plone/Zope Database Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/


: Try our mxODBC.Connect Python Database Interface for free ! ::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PYTHONHTTPSVERIFY env var

On 11.05.2015 12:47, Nick Coghlan wrote:
 On 11 May 2015 at 20:23, Donald Stufft don...@stufft.io wrote:
 On May 11, 2015, at 6:15 AM, Nick Coghlan ncogh...@gmail.com wrote:
 We made the decision when PEP 476 was accepted that this change turned
 a silent security failure into a noisy one, rather than being a
 regression in its own right. PEP 493 isn't about disagreeing with that
 decision, it's about providing a smoother upgrade path in contexts
 where letting the security failure remain silent is deemed to be
 preferred in the near term.

 I don't really agree that the decision to disable TLS is an environment one,
 it's really a per application decision. This is why I was against having some
 sort of global off switch for all of Python because just because one
 application needs it turned off doesn't mean you want it turned off for 
 another
 Python application.
 
 The scenario I'm interested in is the one where it *was* off globally
 (i.e. you were already running Python 2.7.8 or earlier) and you want
 to manage a global rollout of a new Python version that supports being
 configured to verify HTTPS certificates by default, while making the
 decision on whether or not to enable HTTPS certificate verification on
 a server-by-server basis, rather than having that decision be coupled
 directly to the rollout of the updated version of Python.

I guess we're approaching this from different perspectives :-)

I'm mostly interested in having a switch that can be set on
a per application basis, not globally.

I think the global default is fine and I'm just looking for a way to have
admins disable it on a case-by-case basis for those applications which
have problems with the new default. Hence the env var approach
- the admin would simply edit the application's startup shell
script, add the env var and that's it.

For pip et al. which don't use the ssl module, the admins can simply
continue using older versions for those applications - ones which
don't implement the extra verification. In many cases, this is not
necessary, since production environments typically don't use PyPI
at all: they use a local directory with the needed distribution
files, which is both more secure and reliable.

 I agree that the desired end state is where Python 3 is, and where
 upstream Python 2.7.9+ is, this is solely about how to facilitate
 folks getting from point A to point B without an intervening window of
 I broke the world and now my boss is yelling at me about it :)

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, May 11 2015)
 Python Projects, Coaching and Consulting ...  http://www.egenix.com/
 mxODBC Plone/Zope Database Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/


: Try our mxODBC.Connect Python Database Interface for free ! ::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PYTHONHTTPSVERIFY env var

On 10.05.2015 05:04, Robert Collins wrote:
 On 10 May 2015 at 11:44, Chris Angelico ros...@gmail.com wrote:
 On Sun, May 10, 2015 at 4:13 AM, M.-A. Lemburg m...@egenix.com wrote:
 By providing a way to intentionally switch off the new default,
 we do make people aware of the risks and that's good enough,
 while still maintaining the contract people rightly expect of
 patch level releases of Python.

 Just as long as it's the sysadmin, and NOT some random attacker over
 the internet, who has the power to downgrade security. Environment
 variables can be attacked in various ways.
 
 They can, and the bash fun was very good evidence of that.
 
 OTOH if someones environment is at risk, PATH and PYTHONPATH are
 already very effective attack vectors.

If an attacker has access to the process environment, you're doomed
anyway, so that's not really an argument for or against using
environment variables :-)

You'd just need to create a file os.py and point PYTHONPATH at it.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, May 11 2015)
 Python Projects, Coaching and Consulting ...  http://www.egenix.com/
 mxODBC Plone/Zope Database Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/


: Try our mxODBC.Connect Python Database Interface for free ! ::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PYTHONHTTPSVERIFY env var

On 11.05.2015 11:13, Nick Coghlan wrote:
 On 11 May 2015 at 18:04, M.-A. Lemburg m...@egenix.com wrote:
 On 10.05.2015 05:04, Robert Collins wrote:
 On 10 May 2015 at 11:44, Chris Angelico ros...@gmail.com wrote:
 On Sun, May 10, 2015 at 4:13 AM, M.-A. Lemburg m...@egenix.com wrote:
 By providing a way to intentionally switch off the new default,
 we do make people aware of the risks and that's good enough,
 while still maintaining the contract people rightly expect of
 patch level releases of Python.

 Just as long as it's the sysadmin, and NOT some random attacker over
 the internet, who has the power to downgrade security. Environment
 variables can be attacked in various ways.

 They can, and the bash fun was very good evidence of that.

 OTOH if someones environment is at risk, PATH and PYTHONPATH are
 already very effective attack vectors.

 If an attacker has access to the process environment, you're doomed
 anyway, so that's not really an argument for or against using
 environment variables :-)
 
 The core issue lies in managing the user vs administrator
 permissions split. Even for self-administered systems, it's
 recommended practice to run *without* administrative permissions
 normally, and only elevate when you need them. One of the things
 you're watching out for in such cases is that it shouldn't be possible
 for an attacker to make a change to the user environment, and have
 that propagate to have an effect on a process running with
 administrative access. One of the recommended hardening measures
 against that kind of attack vector is to *turn off* Python's
 environment variable processing when launching Python processes with
 administrative access.

The env var would not be read at Python startup time, only
when loading the ssl module, so the -E switch would not have
the effect of disabling it - unlike the hash seed logic, which
is run (and has to be run) at Python startup time.

 We didn't care about that in the hash randomisation case, as the
 compatibility concern there applied on a per application basis, and
 caring about hash order was technically a bug in its own right. By
 contrast, in the situation we're worried about for certificate
 verification compatibility, the issue is environmental: certificate
 management in many private intranets isn't yet to the same standard as
 that on the public internet, so administrators may have a valid reason
 for defaulting Python back to the old behaviour, and redistributors
 may feel obliged to provide an opt-in period prior to switching the
 default behaviour to opt-out. Having the new setting be ignored in
 Python processes run under a hardened configuration means that an
 environment variable based solution won't have the desired effect in
 providing that smoother migration path to the more hardened
 configuration.
 
 I've now written a draft recommendations to redistributors PEP for
 Robert's configuration file based design:
 https://www.python.org/dev/peps/pep-0493/ (exact file names  config
 setting names TBD)

The Fastly cache seems to be having problems again. I only get:
503 Backend is unhealthy - Details: cache-fra1225-FRA 1431335851 2631441948

 I wouldn't be opposed to seeing that as an upstream Python 2.7.x
 feature, but agreement amongst redistributors on using a file-based
 approach is the main outcome I'm looking for.

Can't we have both ?

I don't think that we can wait for a whole PEP process to
run through to fix this regression in 2.7.9.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, May 11 2015)
 Python Projects, Coaching and Consulting ...  http://www.egenix.com/
 mxODBC Plone/Zope Database Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/


: Try our mxODBC.Connect Python Database Interface for free ! ::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PYTHONHTTPSVERIFY env var