Re: [Python-Dev] Procedure for adding new public C API

[Masayuki YAMAMOTO]

Thanks Serhiy, I missed adding PyThread_tss_* to Doc/data/refcounts.dat. I
opened a PR to fix it.
https://github.com/python/cpython/pull/7038

Regards,
Masayuki

2018-05-21 21:41 GMT+09:00 Serhiy Storchaka :

> Please don't forgot to perform the following steps when add a  new public
> C API:
>
> * Document it in Doc/c-api/.
>
> * Add an entry in the What's New document.
>
> * Add it in Doc/data/refcounts.dat.
>
> * Add it in PC/python3.def.
>
> If you want to include it in the limited API, wrap its declaration with:
>
>#if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 >= 0x0308
>#endif
>
> (use the correct Python version of introducing a feature in the comparison)
>
> If you don't want to include it in the limited API, wrap its declaration
> with:
>
>#ifndef Py_LIMITED_API
>#endif
>
> You are free of adding private C API, but its name should start with _Py
> or _PY and its declaration should be wrapped with:
>
>#ifndef Py_LIMITED_API
>#endif
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/ma3yuki.
> 8mamo10%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 539 v3: A new C API for Thread-Local Storage in CPython

[Masayuki YAMAMOTO]

2017-09-09 2:09 GMT+09:00 Nick Coghlan :

> [...]
> No, we genuinely want to consolidate that state into a single shared
> location. However, the struct definition can be adjusted as needed as
> part of the PEP 539 implementation (and we'll get Eric Snow to be one
> of the PR reviewers).
>
I see. I do merge the field of the runtime structure with the replacement
code as is.

Thanks,
Masayuki
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 539 v3: A new C API for Thread-Local Storage in CPython

[Masayuki YAMAMOTO]

Awesome! Thank you for accepting the PEP :)
The only thing missing is a reference implementation, I'm going to complete.

BTW, one of TSS keys is going to move to the runtime struct by bpo-30860
"Consolidate stateful C globals under a single struct". Although that's PR
was merged, the issue status is still open yet. Is there a chance for
rollback?

Masayuki


2017-09-08 23:37 GMT+09:00 Nick Coghlan <ncogh...@gmail.com>:

> On 8 September 2017 at 00:30, Masayuki YAMAMOTO
> <ma3yuki.8mam...@gmail.com> wrote:
> > Hi folks,
> >
> > I submit PEP 539 third draft for the finish. Thank you for all the advice
> > and the help!
>
> Thank you Erik & Yamamoto-san for all of your work on this PEP!
>
> The updates look good, so I'm happy to say as BDFL-Delegate that this
> proposal is now accepted :)
>
> Regards,
> Nick.
>
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] PEP 539 v3: A new C API for Thread-Local Storage in CPython

[Masayuki YAMAMOTO]

Hi folks,

I submit PEP 539 third draft for the finish. Thank you for all the advice
and the help!

Summary of technical changes:
* Don't use `bool` types in the implementation, per discussion at
python/cpython#1362 [1]
* Update for removal of --without-threads

Best regards,
Masayuki


[1]: https://github.com/python/cpython/pull/1362#discussion_r136357583

First round:
https://mail.python.org/pipermail/python-ideas/2016-December/043983.html

Second round:
https://mail.python.org/pipermail/python-dev/2017-August/149091.html

Discussion for the issue:
https://bugs.python.org/issue25658

HTML version for PEP 539 draft:
https://www.python.org/dev/peps/pep-0539/

Diff from first round to version 3:
https://gist.github.com/ma8ma/624f9e4435ebdb26230130b11ce12d20/revisions

And the pull-request for reference implementation (work in progress):
https://github.com/python/cpython/pull/1362




PEP: 539
Title: A New C-API for Thread-Local Storage in CPython
Version: $Revision$
Last-Modified: $Date$
Author: Erik M. Bray, Masayuki Yamamoto
BDFL-Delegate: Nick Coghlan
Status: Draft
Type: Informational
Content-Type: text/x-rst
Created: 20-Dec-2016
Post-History: 16-Dec-2016


Abstract


The proposal is to add a new Thread Local Storage (TLS) API to CPython which
would supersede use of the existing TLS API within the CPython interpreter,
while deprecating the existing API.  The new API is named the "Thread
Specific Storage (TSS) API" (see `Rationale for Proposed Solution`_ for the
origin of the name).

Because the existing TLS API is only used internally (it is not mentioned in
the documentation, and the header that defines it, ``pythread.h``, is not
included in ``Python.h`` either directly or indirectly), this proposal
probably only affects CPython, but might also affect other interpreter
implementations (PyPy?) that implement parts of the CPython API.

This is motivated primarily by the fact that the old API uses ``int`` to
represent TLS keys across all platforms, which is neither POSIX-compliant,
nor portable in any practical sense [1]_.

.. note::

Throughout this document the acronym "TLS" refers to Thread Local
Storage and should not be confused with "Transportation Layer Security"
protocols.


Specification
=

The current API for TLS used inside the CPython interpreter consists of 6
functions::

PyAPI_FUNC(int) PyThread_create_key(void)
PyAPI_FUNC(void) PyThread_delete_key(int key)
PyAPI_FUNC(int) PyThread_set_key_value(int key, void *value)
PyAPI_FUNC(void *) PyThread_get_key_value(int key)
PyAPI_FUNC(void) PyThread_delete_key_value(int key)
PyAPI_FUNC(void) PyThread_ReInitTLS(void)

These would be superseded by a new set of analogous functions::

PyAPI_FUNC(int) PyThread_tss_create(Py_tss_t *key)
PyAPI_FUNC(void) PyThread_tss_delete(Py_tss_t *key)
PyAPI_FUNC(int) PyThread_tss_set(Py_tss_t *key, void *value)
PyAPI_FUNC(void *) PyThread_tss_get(Py_tss_t *key)

The specification also adds a few new features:

* A new type ``Py_tss_t``--an opaque type the definition of which may
  depend on the underlying TLS implementation.  It is defined::

  typedef struct {
  int _is_initialized;
  NATIVE_TSS_KEY_T _key;
  } Py_tss_t;

  where ``NATIVE_TSS_KEY_T`` is a macro whose value depends on the
  underlying native TLS implementation (e.g. ``pthread_key_t``).

* A constant default value for ``Py_tss_t`` variables,
  ``Py_tss_NEEDS_INIT``.

* Three new functions::

  PyAPI_FUNC(Py_tss_t *) PyThread_tss_alloc(void)
  PyAPI_FUNC(void) PyThread_tss_free(Py_tss_t *key)
  PyAPI_FUNC(int) PyThread_tss_is_created(Py_tss_t *key)

  The first two are needed for dynamic (de-)allocation of a ``Py_tss_t``,
  particularly in extension modules built with ``Py_LIMITED_API``, where
  static allocation of this type is not possible due to its implementation
  being opaque at build time.  A value returned by ``PyThread_tss_alloc`` is
  in the same state as a value initialized with ``Py_tss_NEEDS_INIT``, or
  ``NULL`` in the case of dynamic allocation failure.  The behavior of
  ``PyThread_tss_free`` involves calling ``PyThread_tss_delete``
  preventively, or is a no-op if the value pointed to by the ``key``
  argument is ``NULL``.  ``PyThread_tss_is_created`` returns non-zero if the
  given ``Py_tss_t`` has been initialized (i.e. by ``PyThread_tss_create``).

The new TSS API does not provide functions which correspond to
``PyThread_delete_key_value`` and ``PyThread_ReInitTLS``, because these
functions were needed only for CPython's now defunct built-in TLS
implementation; that is the existing behavior of these functions is treated
as follows: ``PyThread_delete_key_value(key)`` is equalivalent to
``PyThread_set_key_value(key, NULL)``, and ``PyThread_ReInitTLS()`` is a
no-op [8]_.

The new ``PyThread_tss_`` functions are almost exactly analogous to their
original

Re: [Python-Dev] PEP 539 (second round): A new C API for Thread-Local Storage in CPython

[Masayuki YAMAMOTO]

2017-08-31 19:40 GMT+09:00 Erik Bray :

> [...]
> the changes is nice.   I just have a few minor changes to suggest
> (typos and such) that I'll make in a pull request.
>
Steve Dower points out which avoids the use of bool in header declarations
[*]. I'd change PyThread_tss_is_created declaration's bool to replace with
int.

Erik, would you change the specification at present PEP PR? (I comment also
the PR)

Thanks,
Masayuki

[*]: https://github.com/python/cpython/pull/1362#pullrequestreview-59884901
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 539 (second round): A new C API for Thread-Local Storage in CPython

[Masayuki YAMAMOTO]

Because I couldn't go to there by myself, I'm really grateful to you and
your first draft.

Masayuki

2017-08-31 19:40 GMT+09:00 Erik Bray <erik.m.b...@gmail.com>:

> On Thu, Aug 31, 2017 at 10:16 AM, Masayuki YAMAMOTO
> <ma3yuki.8mam...@gmail.com> wrote:
> > Hi python-dev,
> >
> > Since Erik started the PEP 539 thread on python-ideas, I've collected
> > feedbacks in the discussion and pull-request, and tried improvement for
> the
> > API specification and reference implementation, as the result I think
> > resolved issues which pointed out by feedbacks.
>
> Thanks Masayuki for taking the lead on updating the PEP.  I've been
> off the ball with it for a while.  In particular the table summarizing
> the changes is nice.   I just have a few minor changes to suggest
> (typos and such) that I'll make in a pull request.
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 539 (second round): A new C API for Thread-Local Storage in CPython

[Masayuki YAMAMOTO]

2017-08-31 18:51 GMT+09:00 Nick Coghlan :

> [...]
> I think that's just a bug in the startup refactoring - we don't
> currently test the "Py_Initialize()/Py_Initialize()/Py_Finalize()"
> sequence anywhere, and I'd missed that it's explicitly documented as
> being permitted. I'd still want to keep the "multiple calls without an
> intervening finalize are prohibited" behaviour for the new more
> granular APIs (since it's simpler and easier to explain if it just
> always fails rather than attempting to check that the previous
> initialization supplied the same config settings), but the documented
> Py_Initialize() behaviour can be reinstated by restoring the early
> return in _Py_InitializeEx_Private.
>
I get the point of difference between document and current code, I don't
mind :)

As far as the PEP itself goes, this version looks good to me, so if
> there aren't any other significant comments between now and then, I'm
> likely to accept it at the core development sprint next week.
>
I took time, but I'm happy to near to finish. Thanks you for helping!

Masayuki
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] PEP 539 (second round): A new C API for Thread-Local Storage in CPython

[Masayuki YAMAMOTO]

Hi python-dev,

Since Erik started the PEP 539 thread on python-ideas, I've collected
feedbacks in the discussion and pull-request, and tried improvement for the
API specification and reference implementation, as the result I think
resolved issues which pointed out by feedbacks.

Well, it's probably not finish yet, there is one which bothers me.  I'm not
sure the CPython startup sequence design (PEP 432 Restructuring the CPython
startup sequence, it might be a conflict with the draft specification [1]),
please let me know what you think about the new API specification.  In any
case, I start a new thread of the updated draft.


Summary of technical changes:

- Two functions which correspond PyThread_delete_key_value and
PyThread_ReInitTLS are omitted, because these are for the removed CPython's
own TLS implementation.

- Add an internal field "_is_initialized" and a constant default value
"Py_tss_NEEDS_INIT" to Py_tss_t type to indicate the thread key's
initialization state independent of the underlying implementation.

- Then, define behaviors for functions which uses the "_is_initialized"
field.

- Change the key argument to pass a pointer, allow to use in the limited
API that does not know the key type size.

- Add three functions which dynamic (de-)allocation and the key's
initialization state checking, because handle opaque struct.

- Change platform support in the case of enabling thread support, all
platforms are required at least one of native thread implementations.

Also the draft has been added explanations and rationales for above
changes, moreover, additional annotations for information.


Regards,
Masayuki


[1]: The specifications of thread key creation and deletion refer how to
use in the API clients (Modules/_tracemalloc.c and Python/pystate.c).  One
of those, Py_Initialize function that is a caller's origin of
PyThread_tss_create is the flow "no-op when called for a second time" until
CPython 3.6 [2].  However, an internal function _Py_InitializeCore that has
been added newly in the current master branch is the flow "fatal error when
called for a second time" [3].

[2]: https://docs.python.org/3.6/c-api/init.html#c.Py_Initialize

[3]: https://github.com/python/cpython/blob/master/Python/pylifecycle.c#L508

First round for PEP 539:
https://mail.python.org/pipermail/python-ideas/2016-December/043983.html

Discussion for the issue:
https://bugs.python.org/issue25658

HTML version for PEP 539 draft:
https://www.python.org/dev/peps/pep-0539/

Diff between first round and second round:
https://gist.github.com/ma8ma/624f9e4435ebdb26230130b11ce12d20/revisions

And the pull-request for reference implementation (work in progress):
https://github.com/python/cpython/pull/1362




PEP: 539
Title: A New C-API for Thread-Local Storage in CPython
Version: $Revision$
Last-Modified: $Date$
Author: Erik M. Bray, Masayuki Yamamoto
BDFL-Delegate: Nick Coghlan
Status: Draft
Type: Informational
Content-Type: text/x-rst
Created: 20-Dec-2016
Post-History: 16-Dec-2016


Abstract


The proposal is to add a new Thread Local Storage (TLS) API to CPython which
would supersede use of the existing TLS API within the CPython interpreter,
while deprecating the existing API.  The new API is named "Thread Specific
Storage (TSS) API" (see `Rationale for Proposed Solution`_ for the origin of
the name).

Because the existing TLS API is only used internally (it is not mentioned in
the documentation, and the header that defines it, ``pythread.h``, is not
included in ``Python.h`` either directly or indirectly), this proposal
probably
only affects CPython, but might also affect other interpreter
implementations
(PyPy?) that implement parts of the CPython API.

This is motivated primarily by the fact that the old API uses ``int`` to
represent TLS keys across all platforms, which is neither POSIX-compliant,
nor portable in any practical sense [1]_.

.. note::

Throughout this document the acronym "TLS" refers to Thread Local
Storage and should not be confused with "Transportation Layer Security"
protocols.


Specification
=

The current API for TLS used inside the CPython interpreter consists of 6
functions::

PyAPI_FUNC(int) PyThread_create_key(void)
PyAPI_FUNC(void) PyThread_delete_key(int key)
PyAPI_FUNC(int) PyThread_set_key_value(int key, void *value)
PyAPI_FUNC(void *) PyThread_get_key_value(int key)
PyAPI_FUNC(void) PyThread_delete_key_value(int key)
PyAPI_FUNC(void) PyThread_ReInitTLS(void)

These would be superseded by a new set of analogous functions::

PyAPI_FUNC(int) PyThread_tss_create(Py_tss_t *key)
PyAPI_FUNC(void) PyThread_tss_delete(Py_tss_t *key)
PyAPI_FUNC(int) PyThread_tss_set(Py_tss_t *key, void *value)
PyAPI_FUNC(void *) PyThread_tss_get(Py_tss_t *key)

The specification also adds a few new features:

* A new type ``Py_ts

[Python-Dev] Remove own implementation for thread-local storage

[Masayuki YAMAMOTO]

Hi, python-dev.

I'd propose removing code which I think out-of-date.
CPython has provided the own implementation for thread-local storage (TLS)
on Python/thread.c, it's used in the case which a platform has not supplied
native TLS.  However, currently all supported platforms (NT and pthreads)
have provided native TLS and defined the Py_HAVE_NATIVE_TLS macro with
unconditional in any case.
If the code is removed, the new TLS API for PEP 539 won't have to care the
reinitialization of the thread keys managed by the interpreter (i.e.
PyThread_ReInitTLS function has been working for own implementation and
will be no longer necessary for new API).  Does anyone have a reason we
should keep it?

Regards,
Masayuki
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Removing Python/thread_foobar.h

[Masayuki YAMAMOTO]

Hi, python-dev.

I'm reading the part of threading feature to replace TLS API with TSS API
(PEP 539). Python/thread_foobar.h hasn't be included to Python/thread.c
(its include directive is comment out) and anywhere. This looks like dead
code and we want to delete it, does anyone have a reason we should keep it?

Regards,
Masayuki
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com