Re: [Python-Dev] Make the stable API-ABI usable

[Victor Stinner]

Le 18 nov. 2017 08:32, "Serhiy Storchaka"  a écrit :

Making PyTuple_GET_ITEM() a function will destroy the half of the benefit.
And this will make the ABI larger.


Sorry if I wasn't explicit about it: my idea of changing the API has an
obvious impact on performance. That's why the first step on the second
milestone of my plan is to spend time to measure the slowdown.

The other part of the my overall plan is to experiement new optimizations.
See my draft PEP for ideas.

The idea behind adding PyTuple_GET_ITEM() is to be able to compile C
extensions using it, without having to modify the code. If you require to
modify the code, I don't expect that you will be able to compile more than
half of C extensions on PyPI, like old code with no active maintainer...

Anyway, the PyTuple_GET_ITEM() will remain a macro in the default API for
Python 3.7.

See also my blog post which explains why the fact that it is a macro
prevents us from optimizing it, like having specialized compact tuple for
small integers.

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] unittest isolation and warnings

[Christian Tismer]

Thanks a lot!
Good to know.

Ciao -- Chris

On 17.11.17 16:44, Brett Cannon wrote:
> Tests are not isolated from the warnings system, so things will leak
> out. Your best option is to use the context manager in the warnings
> module to temporarily make all warnings raise exceptions and test for
> the exception (I'm at the airport, hence why I don't know the name of
> the context manager; the warnings module docs actually have a sample on
> how best to write tests the involve warnings).
> 
> 
> On Fri, Nov 17, 2017, 01:34 Christian Tismer,  > wrote:
> 
> Hi guys,
> 
> when writing tests, I suddenly discovered that unittest
> is not isolated to warnings.
> 
> Example:
> One of my tests emits warnings when a certain condition is
> met. Instead of reporting the error immediately, it uses
> warnings, and at the end of the test, an error is produced
> if there were warnings.
> 
>         if hasattr(__main__, "__warningregistry__"):
>             raise RuntimeError("There are errors, see above.")
> 
> By chance, I discovered that an error was suddenly triggered without
> a warning. That must mean the warning existed already from
> another test as a left-over.
> 
> My question:
> Is that known, and is that intended?
> To what extent are the test cases isolated from each other?
> 
> I do admit that my usage of warnings is somewhat special.
> But it is very convenient to report many errors on remote servers.
> 
> Cheers -- Chris
> 
> --
> Christian Tismer             :^)   tis...@stackless.com
> 
> Software Consulting          :     http://www.stackless.com/
> Karl-Liebknecht-Str. 121     :     https://github.com/PySide
> 14482 Potsdam                :     GPG key -> 0xFB7BEE0E
> phone +49 173 24 18 776  fax +49 (30) 700143-0023
> 
> ___
> Python-Dev mailing list
> Python-Dev@python.org 
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
> 


-- 
Christian Tismer :^)   tis...@stackless.com
Software Consulting  : http://www.stackless.com/
Karl-Liebknecht-Str. 121 : https://github.com/PySide
14482 Potsdam: GPG key -> 0xFB7BEE0E
phone +49 173 24 18 776  fax +49 (30) 700143-0023



signature.asc
Description: OpenPGP digital signature
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Make the stable API-ABI usable

[Serhiy Storchaka]

18.11.17 11:13, Victor Stinner пише:
The idea behind adding PyTuple_GET_ITEM() is to be able to compile C 
extensions using it, without having to modify the code.


The simplest way to do this:

#define PyTuple_GET_ITEM PyTuple_GetItem

This will not add new names to ABI. Such defines can be added in a 
separate header file included for compatibility.


In any case making PyTuple_GET_ITEM() a function will break the 
following code:


PyObject **items = &PyTuple_GET_ITEM(tuple, 0);
Py_ssize_t size = PyTuple_GET_SIZE(tuple, 0);
foo(items, size);

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] unittest isolation and warnings

[Antoine Pitrou]

Hi Christian,

On Fri, 17 Nov 2017 10:15:24 +0100
Christian Tismer  wrote:
> 
> Example:
> One of my tests emits warnings when a certain condition is
> met. Instead of reporting the error immediately, it uses
> warnings, and at the end of the test, an error is produced
> if there were warnings.

I suggest you try using subtests.  This would allow you to report
several errors from a given test method.
https://docs.python.org/3/library/unittest.html#distinguishing-test-iterations-using-subtests

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Make the stable API-ABI usable

[Antoine Pitrou]

I agree with Serhiy.  It doesn't make sense to add PyTuple_GET_ITEM to
the stable ABI.  People who want to benefit from the stable ABI should
use PyTuple_GetItem.  That's not very complicated.

Regards

Antoine.



On Sat, 18 Nov 2017 11:42:36 +0200
Serhiy Storchaka  wrote:
> 18.11.17 11:13, Victor Stinner пише:
> > The idea behind adding PyTuple_GET_ITEM() is to be able to compile C 
> > extensions using it, without having to modify the code.  
> 
> The simplest way to do this:
> 
> #define PyTuple_GET_ITEM PyTuple_GetItem
> 
> This will not add new names to ABI. Such defines can be added in a 
> separate header file included for compatibility.
> 
> In any case making PyTuple_GET_ITEM() a function will break the 
> following code:
> 
>  PyObject **items = &PyTuple_GET_ITEM(tuple, 0);
>  Py_ssize_t size = PyTuple_GET_SIZE(tuple, 0);
>  foo(items, size);
> 
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/python-python-dev%40m.gmane.org



___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Make the stable API-ABI usable

[Antoine Pitrou]

On Sat, 18 Nov 2017 10:13:36 +0100
Victor Stinner  wrote:
> 
> Anyway, the PyTuple_GET_ITEM() will remain a macro in the default API for
> Python 3.7.
> 
> See also my blog post which explains why the fact that it is a macro
> prevents us from optimizing it, like having specialized compact tuple for
> small integers.

I'm not sure this would be an optimization.  You'll add checks to
PyTuple_GET_ITEM() to select the "kind" of tuple at runtime, and this
may very well make things slower.  You would need a JIT with type
constraints to remove the overhead and truly gain from the optimization.

Besides, PyTuple_GET_ITEM() can't really get any faster if all you care
about is the *objects* in the tuple, not their value.

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python initialization and embedded Python

[Antoine Pitrou]

On Sat, 18 Nov 2017 01:01:47 +0100
Victor Stinner  wrote:
> 
> Maybe, the minimum change is to expose _PyRuntime_Initialize() in the
> public C API?

+1.  Also a symmetric PyRuntime_Finalize() function (even if it's a
no-op currently).

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Make the stable API-ABI usable

[Nick Coghlan]

On 18 November 2017 at 11:05, Victor Stinner  wrote:
> Hi,
>
> tl; dr I propose to extend the existing "stable API" to make it almost
> as complete as the current API. For example, add back
> PyTuple_GET_ITEM() to be stable API, but it becomes a function call
> rather than a macro. The final question is if it's not too late to
> iterate on an implementation of this idea for Python 3.7? Knowing that
> the stable API doesn't affect the "current API" at all, since the "new
> C API" (extended stable API) would only be accessible using an
> *opt-in* flag.

I'm -1 on expanding the stable API/ABI in 3.7 (especially without a
PEP), but I'm +1 on refactoring the way we maintain it, with a view to
expanding it (with function calls substituting in for the macros in
Py_LIMITED_API mode) in 3.8.

This isn't an urgent change, but the strict backwards compatibility
policy means it's one where we'll be stuck with any mistakes we make
for a long time. (Proper use of symbol versioning might offer a
subsequent escape clause, but that introduces its own cross-platform
compatibility problems).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Make the stable API-ABI usable

[Nick Coghlan]

On 18 November 2017 at 23:50, Nick Coghlan  wrote:
> On 18 November 2017 at 11:05, Victor Stinner  wrote:
>> Hi,
>>
>> tl; dr I propose to extend the existing "stable API" to make it almost
>> as complete as the current API. For example, add back
>> PyTuple_GET_ITEM() to be stable API, but it becomes a function call
>> rather than a macro. The final question is if it's not too late to
>> iterate on an implementation of this idea for Python 3.7? Knowing that
>> the stable API doesn't affect the "current API" at all, since the "new
>> C API" (extended stable API) would only be accessible using an
>> *opt-in* flag.
>
> I'm -1 on expanding the stable API/ABI in 3.7 (especially without a
> PEP), but I'm +1 on refactoring the way we maintain it, with a view to
> expanding it (with function calls substituting in for the macros in
> Py_LIMITED_API mode) in 3.8.

Expanding on this concept: I think PEP 432 may be a reasonable model
here, whereby you can write a mostly-core-developer focused PEP that
sets out your vision for where you'd like to get to eventually, and
then that PEP provides context for issue-level refactorings that might
otherwise seem like code churn with no apparent purpose.

Each refactoring will still need to stand as beneficial on its own
(usually on grounds of code clarity, or otherwise making future
maintenance easier, even if it makes near term backports harder), but
that process tends to run more smoothly when there's a shared
understanding of the overarching goal.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Make the stable API-ABI usable

[Koos Zevenhoven]

Your email didn't compile. The compiler says that it's a naming conflict,
but actually I think you forgot a semicolon! (Don't worry, it happens to
all of us, whether we be happy or not :-)

--Koos


On Sat, Nov 18, 2017 at 3:05 AM, Victor Stinner 
wrote:

> Hi,
>
> tl; dr I propose to extend the existing "stable API" to make it almost
> as complete as the current API. For example, add back
> PyTuple_GET_ITEM() to be stable API, but it becomes a function call
> rather than a macro. The final question is if it's not too late to
> iterate on an implementation of this idea for Python 3.7? Knowing that
> the stable API doesn't affect the "current API" at all, since the "new
> C API" (extended stable API) would only be accessible using an
> *opt-in* flag.
>
>
> Since I failed to find time to write a proper PEP (sorry about that),
> and the incoming deadline for Python 3.7 features is getting closer, I
> decided to dump my dump into this email. Sorry about the raw
> formatting.
>
> It's a third turn of feedback. The first one was done in python-ideas
> last July, the second was at the CPython sprint last September at
> Instagram. Both were positive.
>
>https://mail.python.org/pipermail/python-ideas/2017-July/046399.html
>
> The C API of CPython is amazing. It made CPython as powerful as we
> know it today. Without the C API, Python wouldn't have numpy. Without
> numpy, Python wouldn't be as popular as it is nowadays. Ok, enough for
> the introduction.
>
> The C API is awful. I hate it so much! The C API is old, error prone,
> too big, and expose every single CPython implementation detail... The
> C API is likely the first reason why faster implementation of Python
> (PyPy?) are not as popular as they should be.
>
> The fact that the C API is so widely used prevents many evolutions of
> CPython. CPython is stuck by its own (public) C API.
>
> PyPy developers spent a lot of time to make cffi great and advertize
> it all around the world. While it's a great project, the fact is that
> the C API *is* still used and the motivation to rewrite old code
> written with the C API to cffi, Cython or whateer else is too low.
> Maybe because the carrot is not big enough. Let's make a bigger
> carrot! (bigger than PyPy amazing performances? I'm not sure that it's
> doable :-( we will see)
>
> Using Cython, it's simpler to write C extensions and Cython code is
> more "portable" on different Python versions, since the C API evolved
> the last 10 years. For example, Python 3 renamed PyString with PyBytes
> and dropped the PyInt type. Writing a C extension working on Python 2
> and Python 3 requires to "pollute" the code with many #ifdef. Cython
> helps to reduce them (or even avoid them? sorry, I don't know well
> Cython).
>
> The C *API* is tidely linked to the *ABI*. I tried to explain it with
> an example in my article:
>
>https://vstinner.github.io/new-python-c-api.html
>
> A known ABI issue is that it's not possible to load C extensions
> compiled in "release mode" on a Python compiled in "debug mode". The
> debug mode changes the API in a subtle way which changes the ABI and
> so makes C extensions incompatible. This issue prevents many people to
> use a Python debug build, whereas debug builds are very helpful to
> detect bugs earlier.
>
> Another issue is that C extensions must be recompiled for each Python
> release (3.5, 3.6, 3.7, etc.). Linux vendors like Red Hat cannot
> provide a single binary for multiple Python versions which prevents to
> upgade the "system" Python in the lifecycle of a distribution major
> version (or at least, it makes things much more complicated in term of
> packaging, it multiply packages for each Python version...).
>
> .
> .
> .
>
> Don't worry (be happy!), I'm not writing this email to complain, but
> to propose a solution. Aha!
>
> I propose to modify the API step by step to add more functions to the
> "stable ABI" (PEP 384) to be able to compile more and more C
> extensions using the "stable API" (please try to follow, first it was
> B, now it's P... seriously, the difference between the ABI and the API
> is subtle, to simplify let's say that it's the same thing, ok? :-)).
>
> I wrote a draft PEP, but I never found time to update it after the two
> rounds of feedbacks (python-ideas and the sprint) to write a proper
> PEP. So I will only give a link to my draft, sorry!
>
>   https://github.com/vstinner/misc/blob/master/python/pep_c_api.rst
>
> In short, the plan is to add back the PyTuple_GET_ITEM() *API* to the
> "stable API" but change its implementation to a *function call* rather
> than the existing macro, so the compiled C extension will use a
> function call and so don't rely on the ABI anymore.
>
>
> My plan is to have two main milestones:
>
> (1) Python 3.7: Extend the *existing* opt-in "stable API" which
> requires to compile C extensions in a special mode. Add maybe an
> option in distutils to ease the compilation of a C extension with the
> "stable API"?
>
> (2) In Pyth

Re: [Python-Dev] Python initialization and embedded Python

[Nick Coghlan]

On 18 November 2017 at 10:01, Victor Stinner  wrote:
> I'm writing this email to ask if this change is an issue or not to
> embedded Python and the Python C API. Is it still possible to call
> "all" functions of the C API before calling Py_Initialize()?

It isn't technically permitted to call any of them, unless their
documentation specifically says that calling them before
`Py_Initialize` is permitted (and that permission is only given for a
select few configuration APIs in
https://docs.python.org/3/c-api/init.html).

While it's still PEP 432's intention to eventually expose a public
multi-phase start-up API, it's *also* the case that we're not actually
ready to do that yet - we're not sure we have the data model right,
and we don't want to commit to a supported API until that's resolved.

So for Python 3.7, I'd suggest pursuing one of the following options:

1. Add a variant of Py_DecodeLocale that accepts a memory allocation
function directly and reports back both the allocated pointer and its
size (allowing the calling program to manage that memory); or
2. Offer a new `Py_SetProgramNameFromString` API that accepts a `char
*` directly. That way, CPython can take care of lazily decoding it
after the decoding machinery has been fully set up, rather than
expecting the embedding application to always do it;

(While we could also make the promise that PyMem_RawMalloc and
Py_DecodeLocale will be callable before Py_Initialize, I don't think
we're far enough into the startup refactoring process to be making
those kinds of promises).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python initialization and embedded Python

[Serhiy Storchaka]

18.11.17 16:17, Nick Coghlan пише:

On 18 November 2017 at 10:01, Victor Stinner  wrote:

I'm writing this email to ask if this change is an issue or not to
embedded Python and the Python C API. Is it still possible to call
"all" functions of the C API before calling Py_Initialize()?


It isn't technically permitted to call any of them, unless their
documentation specifically says that calling them before
`Py_Initialize` is permitted (and that permission is only given for a
select few configuration APIs in
https://docs.python.org/3/c-api/init.html).


The Py_Initialize() is not complete. It mentions only 
Py_SetProgramName(), Py_SetPythonHome() and Py_SetPath(). But in other 
places it is documented that Py_SetStandardStreamEncoding(), 
PyImport_AppendInittab(), PyImport_ExtendInittab() should be called 
before Py_Initialize(). And the embedding examples call 
Py_DecodeLocale() before Py_Initialize(). PyMem_RawMalloc(), 
PyMem_RawFree() and PyInitFrozenExtensions() are called before 
Py_Initialize() in Py_FrozenMain(). Also these functions call 
_PyMem_RawStrdup().


Hence, the minimal set of functions that can be called before 
Py_Initialize() is:


* Py_SetProgramName()
* Py_SetPythonHome()
* Py_SetPath()
* Py_SetStandardStreamEncoding()
* PyImport_AppendInittab()
* PyImport_ExtendInittab()
* Py_DecodeLocale()
* PyMem_RawMalloc()
* PyMem_RawFree()
* PyInitFrozenExtensions()

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Show DeprecationWarning in debug mode?

[Brett Cannon]

+1 from me as well.

On Sat, Nov 18, 2017, 02:36 Serhiy Storchaka,  wrote:

> 18.11.17 03:22, Victor Stinner пише:
> > I noticed that Python not only hides DeprecationWarning, but also
> > PendingDeprecationWarning and ImportWarning by default. While I
> > understand why we decided to hide these warnings to users for a Python
> > compiled in release mode, why are they hidden in Python debug builds?
> >
> > I'm asking the question because in debug mode, Python shows
> > ResourceWarning warnings (whereas these warnings are hidden in release
> > mode). Why only displaying ResourceWarning, but not other warnings in
> > debug mode?
>
> +1 for showing all warning (except maybe PendingDeprecationWarning) in
> the debug build! I constantly forgot about this.
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Show DeprecationWarning in debug mode?

[Barry Warsaw]

On Nov 17, 2017, at 20:22, Victor Stinner  wrote:
> 
> Or maybe we should start adding new modes like -X
> all-warnings-except-PendingDeprecationWarning, -X
> I-really-really-love-warnings and -X warnings-hater, as Barry
> proposed?

Well, if I can’t convince you about a `-X the-flufls-gonna-gitcha` mode, +1 for 
turning on all those other warnings in debug mode.

-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Make the stable API-ABI usable

[Victor Stinner]

Le 18 nov. 2017 10:44, "Serhiy Storchaka"  a écrit :

The simplest way to do this:

#define PyTuple_GET_ITEM PyTuple_GetItem

This will not add new names to ABI. Such defines can be added in a separate
header file included for compatibility.


It is exactly what I am proposing :-)

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python initialization and embedded Python

[Nick Coghlan]

On 19 November 2017 at 01:45, Serhiy Storchaka  wrote:
> 18.11.17 16:17, Nick Coghlan пише:
>>
>> On 18 November 2017 at 10:01, Victor Stinner 
>> wrote:
>>>
>>> I'm writing this email to ask if this change is an issue or not to
>>> embedded Python and the Python C API. Is it still possible to call
>>> "all" functions of the C API before calling Py_Initialize()?
>>
>>
>> It isn't technically permitted to call any of them, unless their
>> documentation specifically says that calling them before
>> `Py_Initialize` is permitted (and that permission is only given for a
>> select few configuration APIs in
>> https://docs.python.org/3/c-api/init.html).
>
> The Py_Initialize() is not complete. It mentions only Py_SetProgramName(),
> Py_SetPythonHome() and Py_SetPath(). But in other places it is documented
> that Py_SetStandardStreamEncoding(), PyImport_AppendInittab(),
> PyImport_ExtendInittab() should be called before Py_Initialize(). And the
> embedding examples call Py_DecodeLocale() before Py_Initialize().
> PyMem_RawMalloc(), PyMem_RawFree() and PyInitFrozenExtensions() are called
> before Py_Initialize() in Py_FrozenMain(). Also these functions call
> _PyMem_RawStrdup().
>
> Hence, the minimal set of functions that can be called before
> Py_Initialize() is:
>
> * Py_SetProgramName()
> * Py_SetPythonHome()
> * Py_SetPath()
> * Py_SetStandardStreamEncoding()
> * PyImport_AppendInittab()
> * PyImport_ExtendInittab()
> * Py_DecodeLocale()
> * PyMem_RawMalloc()
> * PyMem_RawFree()
> * PyInitFrozenExtensions()

OK, in that case I think the answer to Victor's question is:

1. Breaking calling Py_DecodeLocale() before calling Py_Initialize()
is a compatibility break with the API implied by our own usage
examples, and we'll need to revert the breakage for 3.7, and ensure at
least one release's worth of DeprecationWarning before requiring
either the use of an alternative API (where the caller controls the
memory management), or else a new lower level pre-initialization API
(i.e. making `PyRuntime_Initialize` a public API)
2. We should provide a consolidated list of these functions in the C
API initialization docs
3. We should add more test cases to _testembed.c that ensure they all
work correctly prior to Py_Initialize (some of them are already tested
there, but definitely not all of them)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Show DeprecationWarning in debug mode?

[Nick Coghlan]

On 18 November 2017 at 11:22, Victor Stinner  wrote:
> Hi,
>
> I noticed that Python not only hides DeprecationWarning, but also
> PendingDeprecationWarning and ImportWarning by default. While I
> understand why we decided to hide these warnings to users for a Python
> compiled in release mode, why are they hidden in Python debug builds?
>
> I'm asking the question because in debug mode, Python shows
> ResourceWarning warnings (whereas these warnings are hidden in release
> mode). Why only displaying ResourceWarning, but not other warnings in
> debug mode?

I don't recall the exact reasoning (if I ever even knew it), but if I
had to guess, it would just be because ResourceWarning is newer than
the others.

BytesWarning has its own control flags and shouldn't be on by default
even in debug builds, but I think it would be reasonable to have at
least ImportWarning (which our own test suite should never be hitting)
and DeprecationWarning on by default in debug builds.

For PendingDeprecationWarning, I don't really mind either way, but
"everything except BytesWarning, because that has its own dedicated
command line flags" would be an easier guideline to remember.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 565: Show DeprecationWarning in __main__

[Nick Coghlan]

On 13 November 2017 at 01:45, Guido van Rossum  wrote:
> On Sun, Nov 12, 2017 at 1:24 AM, Nick Coghlan  wrote:
>>
>> In Python 2.7 and Python 3.2, the default warning filters were updated to
>> hide
>> DeprecationWarning by default, such that deprecation warnings in
>> development
>> tools that were themselves written in Python (e.g. linters, static
>> analysers,
>> test runners, code generators) wouldn't be visible to their users unless
>> they
>> explicitly opted in to seeing them.
>
> Looking at the official What's New entry for the change
> (https://docs.python.org/3/whatsnew/2.7.html#changes-to-the-handling-of-deprecation-warnings)
> it's not just about development tools. It's about any app (or tool, or
> utility, or program) written in Python whose users just treat it as "some
> app", not as something that's necessarily part of their Python environment.
> While in extreme cases such apps can *bundle* their own Python interpreter
> (like Dropbox does), many developers opt to assume or ensure that Python is
> avaiable, perhaps via the OS package management system. (Because my day job
> is software development I am having a hard time coming up with concrete
> examples that aren't development tools, but AFAIK at Dropbox the
> *deployment* of e.g. Go binaries is managed through utilities written in
> Python. The Go developers couldn't care less about that.)

It occurs to me that the original Dropbox client itself would be an
example of deprecation warnings in deployed code being unhelpful noise
- assuming that the runtime warnings were reported back to the
developers, the deprecation warnings would be entirely irrelevant to a
developer trying to debug a production issue with client/server
communication.

I've updated the PEP to try to make the explanation of the historical
rationale more accurate:
https://github.com/python/peps/commit/30daada7867dd7f0e008545c7fd98612282ec602

I still emphasise the developer tooling case, as I think that's the
easiest one for Python developers to follow when it comes to
appreciating the problems with the old defaults (I know it is for me),
but I've added some references to regular user facing applications as
well.

I've also added a new "Documentation Updates" section, which doesn't
technically need to be in the PEP, but I think is useful to include in
terms of giving the existing features more visibility than they might
otherwise receive (I learned quite a few new things myself about the
basics of warnings control researching and working on implementing
this PEP, so I assume plenty of end users are in a similar situation,
where we know the warnings module exists, but aren't necessarily
completely familiar with how to reconfigure it at runtime).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 565: Show DeprecationWarning in __main__

[Nick Coghlan]

On 19 November 2017 at 17:22, Nick Coghlan  wrote:
> I've updated the PEP to try to make the explanation of the historical
> rationale more accurate:
> https://github.com/python/peps/commit/30daada7867dd7f0e008545c7fd98612282ec602

With these changes, I think the version now live at
https://www.python.org/dev/peps/pep-0565/ is ready for pronouncement.

Are you happy to review and pronounce on that in this thread, or would
you prefer that I started a new one?

Cheers,
Nick.

P.S. The actual implementation still needs some work, due to details
of the dual Python/C warnings module implementation, and the
subtleties of how that interacts with the re module. However, I've
implemented a working version with a test case, and have a pretty
clear idea of the extra test cases that we need to close some of the
test coverage gaps that draft implementation has revealed.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python initialization and embedded Python

[Serhiy Storchaka]

19.11.17 04:17, Nick Coghlan пише:

1. Breaking calling Py_DecodeLocale() before calling Py_Initialize()
is a compatibility break with the API implied by our own usage
examples, and we'll need to revert the breakage for 3.7, and ensure at
least one release's worth of DeprecationWarning before requiring
either the use of an alternative API (where the caller controls the
memory management), or else a new lower level pre-initialization API
(i.e. making `PyRuntime_Initialize` a public API)


There is a way to to control the memory manager. The caller should just 
define their own PyMem_RawMalloc(), PyMem_RawFree(), etc. It seems to me 
that the reasons of introducing these functions were:


1. Get around the implementation detail when malloc(0) could return 
NULL. PyMem_RawMalloc() always should return an unique address (unless 
error).


2. Allow the caller to control the memory management by providing their 
own implementations.


Let use existing possibilities and not expand the API. I don't think the 
deprecation and breaking compatibility are needed here.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com