[Python-Dev] Extracting python bytecode from a linux core dump?

[CompuTinker]

I hope this is the proper place for internals questions...

I have a core file (produced via the gcore command) of a linux python2.6 
process.  I need to extract the byte code and de-compile it.

I looked at https://wiki.python.org/moin/DebuggingWithGdb and related pages.  
However, these all seem to require either a running process and/or a binary 
with debugging symbols.

I'm thinking that the compiled bytecode is likely in an array or contiguous set 
of memory within the python executable's image and that there's probably a way 
to pull it out with gdb.  Unsurprisingly, the pyc 0xd1f20d0a magic number isn't 
kept in memory. So, how do I find the memory holding the compiled byte-code ?

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7

[Jim Baker]

On Wed, Jun 7, 2017 at 2:31 AM, Cory Benfield  wrote:

>
> On 6 Jun 2017, at 18:49, Jim Baker  wrote:
>
> With Nick's suggestion of _tls_bootstrap, this has certainly become more
> complicated. I'm still thinking of the ramifications for future Jython 2.7
> support, if Python dev goes down this path. It still seems easier to me to
> avoid exposing the SSLObject/MemoryBIO model to 2.7 and instead concentrate
> on just having native TLS support. Yes, this might require more work than a
> simple backport, but everyone is resource constrained, not just the
> Requests or Jython teams.
>
>
> Oh. This hadn’t occurred to me until just now, but doesn’t PyOpenSSL (or
> anything built on CFFI) just basically not run on Jython? Or am I mistaken?
>

Sorry, I wish this were true, but PyOpenSSL is not available on Jython,
because we do not yet support CFFI for Jython. CFFI support is something we
have looked at, but we have not implemented. (There is a related and far
more ambitious project to implement support for the C Extension API,
http://jyni.org/, which Stefan Richthofer is working on with me under the
Google Summer of Code.)

Having said that, bundling PyOpenSSL for use by pip is something that we
would *not* want to do for Jython itself. We want to use  the native Java
ecosystem where possible for built-in functionality, in part because we
already have native support already to the underlying system trust
store on *all
platforms*. (Java development did all the heavy lifting for us.)

Instead our current implementation of socket/ssl/select is built on Netty,
plus some additional work for managing SSLContext in such a way that is
compatible with Python. There is an out-of-date doc I prepared describing
what was done (but broad aspects are still current): https://github.com/
jimbaker/socket-reboot


>
> My concrete suggestion is that any work on PEP 543 will benefit from
> improving the testing currently found in test_ssl as part of its
> implementation. For a variety of reasons, test functions like ssl_io_loop (
> https://github.com/python/cpython/blob/master/Lib/test/test_ssl.py#L1691)
> avoid asserting more than it can properly manage the framing of
> wrapped/unwrapped data, so for example one doesn't want to state explicitly
> when SSL_ERROR_WANT_READ would be called (too much white box at that
> point). On the other hand, the lack of unit testing of, for example,
> SSLObject.read, but instead only doing at the functional test level, means
> that there's nothing explicit testing "will raise an SSLWantReadError if it
> needs more data than the incoming BIO has available" (per the docs), or
> similar use of signaling (SSLEOFError comes to mind).
>
>
> Yeah, PEP 543 just basically assumes the stdlib’s testing of TLS doesn’t
> exist and that it’ll have to manufacture its own. Unfortunately, because it
> is attempting to abstract across many different implementations the tests
> will need to be fairly high-level: for example, there is no consistent way
> to discuss the actual size of the buffers in the buffer-based TLS
> implementation, and they are allowed to be unbounded buffers, so tests
> cannot validate that TLSWantReadError and TLSWantWriteError are ever
> actually raised: all they can do is run tests that will handle them in the
> even that they are raised and confirm the data is transferred appropriately.
>

Right, so such improved testing, regardless of level, will still be an
improvement, and we look forward to seeing it for use by Jython as well.

I should mention that we sometimes see undocumented functionality leak out.
For example,
https://docs.python.org/3/library/socket.html#socket.socket.listen doesn't
say anything about backlog=0, but the requests test suite (last time I
looked on Jython) now depends on it. We assumed it was something like
http://pubs.opengroup.org/onlinepubs/009695399/functions/listen.html, but
as described in http://bugs.python.org/issue8498, it now means that "a
server application in python that accepts exactly 1 connection", by passing
to the underlying C. More tests, more docs, please, even though of course
that's a lot of dev effort.

- Jim
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7

[Nathaniel Smith]

On Jun 7, 2017 6:29 AM, "Victor Stinner"  wrote:

2017-06-07 10:56 GMT+02:00 Nathaniel Smith :
> Another testing challenge is that the stdlib ssl module has no way to
> trigger a renegotiation, and therefore there's no way to write tests
> to check that it properly handles a renegotiation, even though
> renegotiation is by far the trickiest part of the protocol to get
> right. (In particular, renegotiation is the only case where attempting
> to read can give WantWrite and vice-versa.)

Renegociation was the source of a vulnerability in SSL/TLS protocols,
so maybe it's a good thing that it's not implemented :-)
https://www.rapid7.com/db/vulnerabilities/tls-sess-renegotiation

Renegociation was removed from the new TLS 1.3 protocol:
https://tlswg.github.io/tls13-spec/
"TLS 1.3 forbids renegotiation"


Oh, sure, renegotiation is awful, no question. The HTTP/2 spec also forbids
it. But it does still get used and python totally implements it :-). If
python is talking to some peer and the peer says "hey, let's renegotiate
now" then it will. There just aren't any tests for what happens next.

Not that this has much to do with MemoryBIOs. Sorry for the tangent.

-n
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Support for RFC 6920: Naming Things with Hashes?

[Sylvain Bellemare]

Hi,

 I hope this is the right place to post this kind of question. If not I
apologize.

I was simply wondering if anyone had been looking into supporting RFC 6920 (
https://tools.ietf.org/html/rfc6920).

For a simple example of what this is about see
https://tools.ietf.org/html/rfc6920#section-8

Thanks!

Sylvain
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7

[Cory Benfield]

> On 7 Jun 2017, at 14:29, Victor Stinner  wrote:
> 
> 2017-06-07 10:56 GMT+02:00 Nathaniel Smith :
>> Another testing challenge is that the stdlib ssl module has no way to
>> trigger a renegotiation, and therefore there's no way to write tests
>> to check that it properly handles a renegotiation, even though
>> renegotiation is by far the trickiest part of the protocol to get
>> right. (In particular, renegotiation is the only case where attempting
>> to read can give WantWrite and vice-versa.)
> 
> Renegociation was the source of a vulnerability in SSL/TLS protocols,
> so maybe it's a good thing that it's not implemented :-)
> https://www.rapid7.com/db/vulnerabilities/tls-sess-renegotiation
> 
> Renegociation was removed from the new TLS 1.3 protocol:
> https://tlswg.github.io/tls13-spec/
> "TLS 1.3 forbids renegotiation"

Renegotiation remains extremely widely deployed with TLS client certificates in 
enterprise environments, sadly.

Cory
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7

[Victor Stinner]

2017-06-07 10:56 GMT+02:00 Nathaniel Smith :
> Another testing challenge is that the stdlib ssl module has no way to
> trigger a renegotiation, and therefore there's no way to write tests
> to check that it properly handles a renegotiation, even though
> renegotiation is by far the trickiest part of the protocol to get
> right. (In particular, renegotiation is the only case where attempting
> to read can give WantWrite and vice-versa.)

Renegociation was the source of a vulnerability in SSL/TLS protocols,
so maybe it's a good thing that it's not implemented :-)
https://www.rapid7.com/db/vulnerabilities/tls-sess-renegotiation

Renegociation was removed from the new TLS 1.3 protocol:
https://tlswg.github.io/tls13-spec/
"TLS 1.3 forbids renegotiation"

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7

[Nathaniel Smith]

On Tue, Jun 6, 2017 at 10:49 AM, Jim Baker  wrote:
> With Nick's suggestion of _tls_bootstrap, this has certainly become more
> complicated. I'm still thinking of the ramifications for future Jython 2.7
> support, if Python dev goes down this path. It still seems easier to me to
> avoid exposing the SSLObject/MemoryBIO model to 2.7 and instead concentrate
> on just having native TLS support. Yes, this might require more work than a
> simple backport, but everyone is resource constrained, not just the Requests
> or Jython teams.

Yeah, presumably the preferred way to do PEP 543 on Jython would be to
have a native backend based on javax.net.ssl.SSLEngine. Not sure how
that affects the MemoryBIO discussion – maybe it means that Jython
just doesn't need a MemoryBIO backport in the stdlib ssl module,
because everyone who might use it will find themselves using SSLEngine
instead.

> My concrete suggestion is that any work on PEP 543 will benefit from
> improving the testing currently found in test_ssl as part of its
> implementation. For a variety of reasons, test functions like ssl_io_loop
> (https://github.com/python/cpython/blob/master/Lib/test/test_ssl.py#L1691)
> avoid asserting more than it can properly manage the framing of
> wrapped/unwrapped data, so for example one doesn't want to state explicitly
> when SSL_ERROR_WANT_READ would be called (too much white box at that point).
> On the other hand, the lack of unit testing of, for example, SSLObject.read,
> but instead only doing at the functional test level, means that there's
> nothing explicit testing "will raise an SSLWantReadError if it needs more
> data than the incoming BIO has available" (per the docs), or similar use of
> signaling (SSLEOFError comes to mind).

Another testing challenge is that the stdlib ssl module has no way to
trigger a renegotiation, and therefore there's no way to write tests
to check that it properly handles a renegotiation, even though
renegotiation is by far the trickiest part of the protocol to get
right. (In particular, renegotiation is the only case where attempting
to read can give WantWrite and vice-versa.)

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7

[Cory Benfield]

> On 6 Jun 2017, at 18:49, Jim Baker  wrote:
> 
> With Nick's suggestion of _tls_bootstrap, this has certainly become more 
> complicated. I'm still thinking of the ramifications for future Jython 2.7 
> support, if Python dev goes down this path. It still seems easier to me to 
> avoid exposing the SSLObject/MemoryBIO model to 2.7 and instead concentrate 
> on just having native TLS support. Yes, this might require more work than a 
> simple backport, but everyone is resource constrained, not just the Requests 
> or Jython teams.

Oh. This hadn’t occurred to me until just now, but doesn’t PyOpenSSL (or 
anything built on CFFI) just basically not run on Jython? Or am I mistaken?

> My concrete suggestion is that any work on PEP 543 will benefit from 
> improving the testing currently found in test_ssl as part of its 
> implementation. For a variety of reasons, test functions like ssl_io_loop 
> (https://github.com/python/cpython/blob/master/Lib/test/test_ssl.py#L1691 
> ) 
> avoid asserting more than it can properly manage the framing of 
> wrapped/unwrapped data, so for example one doesn't want to state explicitly 
> when SSL_ERROR_WANT_READ would be called (too much white box at that point). 
> On the other hand, the lack of unit testing of, for example, SSLObject.read, 
> but instead only doing at the functional test level, means that there's 
> nothing explicit testing "will raise an SSLWantReadError if it needs more 
> data than the incoming BIO has available" (per the docs), or similar use of 
> signaling (SSLEOFError comes to mind).

Yeah, PEP 543 just basically assumes the stdlib’s testing of TLS doesn’t exist 
and that it’ll have to manufacture its own. Unfortunately, because it is 
attempting to abstract across many different implementations the tests will 
need to be fairly high-level: for example, there is no consistent way to 
discuss the actual size of the buffers in the buffer-based TLS implementation, 
and they are allowed to be unbounded buffers, so tests cannot validate that 
TLSWantReadError and TLSWantWriteError are ever actually raised: all they can 
do is run tests that will handle them in the even that they are raised and 
confirm the data is transferred appropriately.

Cory___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com