[issue33919] Expose _PyCoreConfig structure to Python

[STINNER Victor]

STINNER Victor  added the comment:

So far, there is no clear agreement to expose C PyConfig structure in Python, 
so I close the issue.

My PEP 587 has been accepted. I chose to not expose PyConfig in Python in the 
PEP. But I'm open to revisit this idea later, especially to move towards PEP 
432: implement multi-phase initialization (only partially supported in my PEP 
587).

But I would prefer to a different rationale than exposing hash_seed. For 
hash_seed alone, I don't think that it's worth it. Moreover, Christian wrote:

> hash_seed and use_hash_seed could be added to sys.hash_info. This would be 
> the first place I'd look for the information. After all I implemented it. :)

--
resolution:  -> rejected
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33919] Expose _PyCoreConfig structure to Python

[STINNER Victor]

STINNER Victor  added the comment:

Update. I implemented _testinternalcapi.get_configs() which exports *all* 
Python configuration used to initialize Python. It contains the hash seed for 
example. The function is only written for tests.

Moreover, I proposed the PEP 587 to expose the new _PyCoreConfig as a public C 
API.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33919] Expose _PyCoreConfig structure to Python

[Barry A. Warsaw]

Barry A. Warsaw  added the comment:

On Jun 23, 2018, at 20:21, Nick Coghlan  wrote:
> 
> Only exposing a `forced_hash_seed` (and hiding randomly generated ones as 
> `forced_hash_seed=None`) seems reasonable though, since those can already be 
> read from os.environ anyway.

Only mirroring $PYTHONHASHSEED probably makes the whole ask less useful.  Maybe 
I should abandon the PR, although it may still make sense to export the full 
_PyCoreConfig structure.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33919] Expose _PyCoreConfig structure to Python

[Nick Coghlan]

Nick Coghlan  added the comment:

Back on the original hash seed topic:

1. The exact size of the seed ranges from 128 bits (SIPHash) to 32-bits 
depending on exactly which hash algorithm you're talking about 
(https://www.python.org/dev/peps/pep-0456/#hash-secret)

2. While PEP 456 doesn't state it explicitly, my recollection is that omitting 
the exact hash seed value from the Python level API was a deliberate decision, 
since one of the *purposes* of PEP 456 was to protect against seed recovery 
attacks like https://131002.net/siphash/poc.py. Being able to read the seed 
directly from the sys modules would rather simplify the task of seed recovery :)

Only exposing a `forced_hash_seed` (and hiding randomly generated ones as 
`forced_hash_seed=None`) seems reasonable though, since those can already be 
read from os.environ anyway.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33919] Expose _PyCoreConfig structure to Python

[Nick Coghlan]

Nick Coghlan  added the comment:

I'm thoroughly open to co-author requests for PEP 432 - the "Let's implement it 
as a private API for Python 3.7 and see what we learn from the experience" plan 
worked beautifully, but it *also* means the PEP text is now woefully out of 
date with reality :)

The pieces that are missing are:

- bring it up to date with what we actually did for 3.7
- decide which of those pieces we want to make public as-is, and which we want 
to tweak before making them generally available (e.g. does the 
"ConfigureMainInterpreter" naming still make sense? Or should we go back to the 
earlier "BeginInitialization" and "EndInitialization" pair?)
- now that we store this state in a more coherent way, what do we want to make 
public at the Python layer, and where should we make it public to avoid causing 
too many problems for other implementations? 

However, while I'd definitely be able to make time to review a PR to the PEP, I 
can't make any promises as to when I'd be able to sit down and actually draft 
that update myself.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33919] Expose _PyCoreConfig structure to Python

[STINNER Victor]

STINNER Victor  added the comment:

> We could make the hash_seed 64 bits.

On my 64-bit Linux, _Py_HashSecret_t takes 24 bytes (192 bits).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33919] Expose _PyCoreConfig structure to Python

[Barry A. Warsaw]

Change by Barry A. Warsaw :


--
keywords: +patch
pull_requests: +7475
stage: needs patch -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33919] Expose _PyCoreConfig structure to Python

[Eric Snow]

Change by Eric Snow :


--
nosy: +emilyemorehouse, eric.snow

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33919] Expose _PyCoreConfig structure to Python

[Barry A. Warsaw]

Barry A. Warsaw  added the comment:

Although I guess that would require modifications to lcg_urandom().  I don't 
feel qualified to change that function.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33919] Expose _PyCoreConfig structure to Python

[Barry A. Warsaw]

Barry A. Warsaw  added the comment:

We could make the hash_seed 64 bits.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33919] Expose _PyCoreConfig structure to Python

[STINNER Victor]

STINNER Victor  added the comment:

Barry: generating a 32 bit seed gives less entropy and so makes Python
easier to crash. If you need reproducible Python: generate a seed and set
the env var before starting Python. Tox does that. Regrtest should do that.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33919] Expose _PyCoreConfig structure to Python

[Barry A. Warsaw]

Barry A. Warsaw  added the comment:

Nosying Nick.  I agree there's some overlap with Python startup restructuring, 
but it feels kind of orthogonal too.  I really am only exposing (some elements) 
of that structure to Python.

What might be interesting though would be if we want to expose the entire 
structure and not just the hash seeds, as I'm leaning toward here (given that 
we already have sys.hash_info).

--
nosy: +ncoghlan

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33919] Expose _PyCoreConfig structure to Python

[STINNER Victor]

STINNER Victor  added the comment:

Nick plans to finish his PEP 432 for Python 3.8 and make the API public.
See with him? The PEP should document these structures but I was ahead and
made changes which were not scheduled and the PEP is now outdated.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33919] Expose _PyCoreConfig structure to Python

[Barry A. Warsaw]

Barry A. Warsaw  added the comment:

I think there's another thing I'd like to change, and it seems like it's "just" 
an implementation detail.  In _Py_HashRandomization_Init(), if use_hash_seed is 
0, then we directly inject the random bits into the buffer, and then there's no 
hash_seed.  I'd like to change that so that if use_hash_seed is 0, then we 
create a random hash seed first, and then call lcg_urandom() for the hash 
secret.  That way, even if Python itself uses a random hash seed, we'll have a 
record of that in the runtime that can be used to reproduce the hashing.  In 
this case, I'd still leave use_hash_seed == 0, and that would tell you what 
combinations of env vars were used.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33919] Expose _PyCoreConfig structure to Python

[Barry A. Warsaw]

Barry A. Warsaw  added the comment:

Thanks for the hint!  I had a feeling there had to be an API to get at it, but 
I couldn’t find it.  Maybe we should start documenting the Python Secret 
Underscore API? :)

On Jun 22, 2018, at 00:04, STINNER Victor  wrote:
> 
> _PyCoreConfig *core_config = 
> &_PyGILState_GetInterpreterStateUnsafe()->core_config;
> ...
> PyStructSequence_SET_ITEM(seq, pos++, PyBool_FromLong(core_config->dev_mode));
> 
> The interpreter really owns the copy of these configs and they are kept until 
> the interpreter object is destroyed.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33919] Expose _PyCoreConfig structure to Python

[STINNER Victor]

STINNER Victor  added the comment:

> I think the basic implementation problem is that by the time you get to 
> get_hash_info() in sysmodule.c, you no longer have access to the 
> _PyCoreConfig object, nor the _PyMain object that it's generally attached to.

An interpreter now keeps a copy of _PyCoreConfig and _PyMainInterpreterConfig.

See for example make_flags() in sysmodule.c:

_PyCoreConfig *core_config = 
&_PyGILState_GetInterpreterStateUnsafe()->core_config;
...
PyStructSequence_SET_ITEM(seq, pos++, PyBool_FromLong(core_config->dev_mode));

The interpreter really owns the copy of these configs and they are kept until 
the interpreter object is destroyed.

Another example:

static PyObject *
import_find_and_load(PyObject *abs_name)
{
...
int import_time = interp->core_config.import_time;
...
}

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33919] Expose _PyCoreConfig structure to Python

[Barry A. Warsaw]

Barry A. Warsaw  added the comment:

I think the basic implementation problem is that by the time you get to 
get_hash_info() in sysmodule.c, you no longer have access to the _PyCoreConfig 
object, nor the _PyMain object that it's generally attached to.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33919] Expose _PyCoreConfig structure to Python

[STINNER Victor]

STINNER Victor  added the comment:

Is is still a secret seed if it's public? :)

--
nosy: +vstinner

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33919] Expose _PyCoreConfig structure to Python

[Barry A. Warsaw]

Barry A. Warsaw  added the comment:

On Jun 20, 2018, at 13:28, Christian Heimes  wrote:
> 
> Christian Heimes  added the comment:
> 
> hash_seed and use_hash_seed could be added to sys.hash_info. This would be 
> the first place I'd look for the information. After all I implemented it. :)

That was the first place I looked too :)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33919] Expose _PyCoreConfig structure to Python

[Christian Heimes]

Christian Heimes  added the comment:

hash_seed and use_hash_seed could be added to sys.hash_info. This would be the 
first place I'd look for the information. After all I implemented it. :)

--
nosy: +christian.heimes

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33919] Expose _PyCoreConfig structure to Python

[Barry A. Warsaw]

New submission from Barry A. Warsaw :

The _PyCoreConfig structure in pystate.h has some interesting fields that I 
don't think are exposed anywhere else to Python-land.  I was particularly 
interested recently in hash_seed and use_hash_seed.   I'm thinking that it may 
be useful to expose this structure in the sys module.

--
assignee: barry
components: Interpreter Core
messages: 320107
nosy: barry
priority: normal
severity: normal
stage: needs patch
status: open
title: Expose _PyCoreConfig structure to Python
type: enhancement
versions: Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com