[Python-Dev] Add mimetypes.mimesniff

[Dong-hee Na]

Hi,

A few weeks ago, I suggested adding mimetypes.mimesniff on stdlib.
(https://bugs.python.org/issue40841,
https://github.com/python/cpython/pull/20720)

Detecting MIME types well is an important feature and we already have
mimetypes detection library but AFAIK it is not good enough.

Note that some of our stdlib module already use the sniffing algorithm
(e.g imghdr)

The question is how exactly the mime sniffing should be done in terms
of file formats and algorithm. Luckily, WHATWG published the standard
for mime sniffing, and I think we should follow it.
(https://mimesniff.spec.whatwg.org/)

So I created the issue on the bpo and implemented it.
I 'd like to listen to all your opinions :)

-- 
Software Development Engineer at Kakao corp.

Tel: +82 10-3353-9127
Email: donghee.n...@gmail.com | denn...@kakaocorp.com
Linkedin: https://www.linkedin.com/in/dong-hee-na-2b713b49/
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/633E4OMXY6D2SN4OIJPEYWJSW7DFKYKK/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 626: Precise line numbers for debugging and other tools.

[Guido van Rossum]

But on which line is the RETURN opcode if there is more than a docstring?
Doesn’t it make sense to have it attached to the last line of the body?
(Too bad about pytype, that kind of change happens — we had this kind of
thing for mypy too, when line numbers in the AST were fixed.)

On Wed, Jul 22, 2020 at 17:29 Gregory P. Smith  wrote:

>
>
> On Wed, Jul 22, 2020 at 5:19 AM Mark Shannon  wrote:
>
>>
>>
>> On 21/07/2020 9:46 pm, Gregory P. Smith wrote:
>> >
>> >
>> > On Fri, Jul 17, 2020 at 8:41 AM Ned Batchelder > > > wrote:
>> >
>> > https://www.python.org/dev/peps/pep-0626/ :)
>> >
>> > --Ned.
>> >
>> > On 7/17/20 10:48 AM, Mark Shannon wrote:
>> >  > Hi all,
>> >  >
>> >  > I'd like to announce a new PEP.
>> >  >
>> >  > It is mainly codifying that Python should do what you probably
>> > already
>> >  > thought it did :)
>> >  >
>> >  > Should be uncontroversial, but all comments are welcome.
>> >  >
>> >  > Cheers,
>> >  > Mark.
>> >
>> >
>> > """When a frame object is created, the f_lineno will be set to the line
>> > at which the function or class is defined. For modules it will be set
>> to
>> > zero."""
>> >
>> > Within this PEP it'd be good for us to be very pedantic.  f_lineno is a
>> > single number.  So which number is it given many class and function
>> > definition statements can span multiple lines.
>> >
>> > Is it the line containing the class or def keyword?  Or is it the line
>> > containing the trailing :?
>>
>> The line of the `def`/`class`. It wouldn't change for the current
>> behavior. I'll add that to the PEP.
>>
>> >
>> > Q: Why can't we have the information about the entire span of lines
>> > rather than consider a definition to be a "line"?
>>
>> Pretty much every profiler, coverage tool, and debugger ever expects
>> lines to be natural numbers, not ranges of numbers.
>> A lot of tooling would need to be changed.
>>
>> >
>> > I think that question applies to later sections as well.  Anywhere we
>> > refer to a "line", it could actually mean a span of lines. (especially
>> > when you consider \ continuation in situations you might not otherwise
>> > think could span lines)
>>
>> Let's take an example:
>> ```
>> x = (
>>  a,
>>  b,
>> )
>> ```
>>
>> You would want the BUILD_TUPLE instruction to have a of span lines 1 to
>> 4 (inclusive), rather just line 1?
>> If you wanted to break on the BUILD_TUPLE where you tell pdb to break?
>>
>> I don't see that it would add much value, but it would add a lot of
>> complexity.
>>
>
> We should have the data about the range at bytecode compilation time,
> correct?  So why not keep it?  sure, most existing tooling would just use
> the start of the range as the line number as it always has.  but some
> tooling could find the range useful (ex: semantic code indexing for use in
> display, search, editors, IDEs. Rendering lint errors more accurately
> instead of just claiming a single line or resorting to parsing hacks to
> come up with a range, etc.).  The downside is that we'd be storing a second
> number in bytecode making it slightly larger.  Though it could be stored
> efficiently as a prefixed delta so it'd likely average out as less than 2
> bytes per line number stored.  (i don't have a feeling for our current
> format to know if that is significant or not - if it is, maybe this idea
> just gets nixed)
>
> The reason the range concept was on my mind is due to something not quite
> related but involving a changed idea of a line number in our current system
> that we recently ran into with pytype during a Python upgrade.
>
> """in 3.7, if a function body is a plain docstring, the line number of the
> RETURN_VALUE opcode corresponds to the docstring, whereas in 3.6 it
> corresponds to the function definition.""" (Thanks, Martin & Rebecca!)
>
> ```python
> def no_op():
>   """docstring instead of pass."""
> ```
>
> so the location of what *was* originally an end of line `# pytype:
> disable=bad-return-type` comment (to work around an issue not relevant
> here) turned awkward and version dependent.  pytype is bytecode based, thus
> that is where its line numbers come from.  metadata comments in source can
> only be tied to bytecode via line numbers.  making end of line directives
> occasionally hard to match up.
>
> When there is no return statement, this opcode still exists.  what line
> number does it belong to?  3.6's answer made sense to me.  3.7's seems
> wrong - a docstring isn't responsible for a return opcode.  I didn't check
> what 3.8 and 3.9 do.  An alternate answer after this PEP is that it
> wouldn't have a line number when there is no return statement (pedantically
> correct, I approve! #win).
>
> -gps
>
>
>>
>> Cheers,
>> Mark.
>>
>> >
>> > -gps
>>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org

[Python-Dev] Re: PEP 626: Precise line numbers for debugging and other tools.

[Gregory P. Smith]

On Wed, Jul 22, 2020 at 5:19 AM Mark Shannon  wrote:

>
>
> On 21/07/2020 9:46 pm, Gregory P. Smith wrote:
> >
> >
> > On Fri, Jul 17, 2020 at 8:41 AM Ned Batchelder  > > wrote:
> >
> > https://www.python.org/dev/peps/pep-0626/ :)
> >
> > --Ned.
> >
> > On 7/17/20 10:48 AM, Mark Shannon wrote:
> >  > Hi all,
> >  >
> >  > I'd like to announce a new PEP.
> >  >
> >  > It is mainly codifying that Python should do what you probably
> > already
> >  > thought it did :)
> >  >
> >  > Should be uncontroversial, but all comments are welcome.
> >  >
> >  > Cheers,
> >  > Mark.
> >
> >
> > """When a frame object is created, the f_lineno will be set to the line
> > at which the function or class is defined. For modules it will be set to
> > zero."""
> >
> > Within this PEP it'd be good for us to be very pedantic.  f_lineno is a
> > single number.  So which number is it given many class and function
> > definition statements can span multiple lines.
> >
> > Is it the line containing the class or def keyword?  Or is it the line
> > containing the trailing :?
>
> The line of the `def`/`class`. It wouldn't change for the current
> behavior. I'll add that to the PEP.
>
> >
> > Q: Why can't we have the information about the entire span of lines
> > rather than consider a definition to be a "line"?
>
> Pretty much every profiler, coverage tool, and debugger ever expects
> lines to be natural numbers, not ranges of numbers.
> A lot of tooling would need to be changed.
>
> >
> > I think that question applies to later sections as well.  Anywhere we
> > refer to a "line", it could actually mean a span of lines. (especially
> > when you consider \ continuation in situations you might not otherwise
> > think could span lines)
>
> Let's take an example:
> ```
> x = (
>  a,
>  b,
> )
> ```
>
> You would want the BUILD_TUPLE instruction to have a of span lines 1 to
> 4 (inclusive), rather just line 1?
> If you wanted to break on the BUILD_TUPLE where you tell pdb to break?
>
> I don't see that it would add much value, but it would add a lot of
> complexity.
>

We should have the data about the range at bytecode compilation time,
correct?  So why not keep it?  sure, most existing tooling would just use
the start of the range as the line number as it always has.  but some
tooling could find the range useful (ex: semantic code indexing for use in
display, search, editors, IDEs. Rendering lint errors more accurately
instead of just claiming a single line or resorting to parsing hacks to
come up with a range, etc.).  The downside is that we'd be storing a second
number in bytecode making it slightly larger.  Though it could be stored
efficiently as a prefixed delta so it'd likely average out as less than 2
bytes per line number stored.  (i don't have a feeling for our current
format to know if that is significant or not - if it is, maybe this idea
just gets nixed)

The reason the range concept was on my mind is due to something not quite
related but involving a changed idea of a line number in our current system
that we recently ran into with pytype during a Python upgrade.

"""in 3.7, if a function body is a plain docstring, the line number of the
RETURN_VALUE opcode corresponds to the docstring, whereas in 3.6 it
corresponds to the function definition.""" (Thanks, Martin & Rebecca!)

```python
def no_op():
  """docstring instead of pass."""
```

so the location of what *was* originally an end of line `# pytype:
disable=bad-return-type` comment (to work around an issue not relevant
here) turned awkward and version dependent.  pytype is bytecode based, thus
that is where its line numbers come from.  metadata comments in source can
only be tied to bytecode via line numbers.  making end of line directives
occasionally hard to match up.

When there is no return statement, this opcode still exists.  what line
number does it belong to?  3.6's answer made sense to me.  3.7's seems
wrong - a docstring isn't responsible for a return opcode.  I didn't check
what 3.8 and 3.9 do.  An alternate answer after this PEP is that it
wouldn't have a line number when there is no return statement (pedantically
correct, I approve! #win).

-gps


>
> Cheers,
> Mark.
>
> >
> > -gps
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/H3YBK275SUSCR5EHWHYBTJBF655UK7JG/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: How to customize CPython to a minimal set

[Simon Cross]

I don't know if they suit your purposes, but both sqlite3 and a small array
implementation are already available:

https://github.com/micropython/micropython-lib/tree/master/sqlite3
https://github.com/v923z/micropython-ulab
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ITYDEI7RKIDYKULJWK7H5HW32GZZQS3D/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: How to customize CPython to a minimal set

[Steve Holden]

Sadly micropython is not intended to support numerical libraries and other
such complex modules: the support for the Python standard library is pretty
much non-existent.

Kind regards,
Steve


On Wed, Jul 22, 2020 at 3:10 PM Huang, Yang  wrote:

> Thank you for all your comments.
>
> I cannot agree any more. I did try but there were so many dependencies.
> One change all change.
>
>
>
> Micropython is a choice. But not sure if numpy and sqlite3 can be
> supported well. And what’s the compatibility of the libs in Pypi.
>
>
>
>
>
> *From:* Guido van Rossum 
> *Sent:* Tuesday, July 21, 2020 10:57 PM
> *To:* Huang, Yang 
> *Cc:* python-dev@python.org
> *Subject:* Re: [Python-Dev] How to customize CPython to a minimal set
>
>
>
> I expect it will be unfeasible to strip CPython. If you disagree, try it.
> ;-)
>
>
>
> On Mon, Jul 20, 2020 at 22:35 Huang, Yang  wrote:
>
> Hi, Guido
>
>
>
> Yes. Micropyhton is also in consideration.
>
> But sqlite3 is the first usage. There should be some additional features
> like numpy, scipy... Not sure if micropython supports well?
>
>
>
> Or is there a feasible way to strip CPython ?
>
>
>
> Thanks.
>
>
>
> *From:* Guido van Rossum 
> *Sent:* Monday, July 20, 2020 10:45 PM
> *To:* Huang, Yang 
> *Cc:* python-dev@python.org
> *Subject:* Re: [Python-Dev] How to customize CPython to a minimal set
>
>
>
> Have you considered starting with micropython? It’s made for embedded
> systems and fully supports Python 3 syntax. Adding sqlite3 support to it
> will be less work than stripping all the I/O from CPython.
>
>
>
> —Guido
>
>
>
> On Mon, Jul 20, 2020 at 06:48 Huang, Yang  wrote:
>
>
> Hi, all
>
> There is a request to run python in a Linux-based embedded resource
> constrained system with sqlite3 support.
>
> So many features are not required, like posixmodule, signalmodule,
> hashtable ...
> But seems there are some dependencies among the
> Modules/Parser/Python/Objects/Programs...
>
> Is there a way to tailor CPython 3 to a minimal set with sqlite3 (the less
> syscalls the better) ?
> Is it possible to do that?
>
> Thank you.
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/ECPLKXQ42VNLHD5DP3RG57L3QTJ77FUT/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
> --
>
> --Guido (mobile)
>
> --
>
> --Guido (mobile)
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/OMUGASSNBFJOT3W2ES4OIOQ4LOWMNHHG/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/JZDVYFODJRDYOLQOSV3OHMCLE4LZK6I4/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: How to customize CPython to a minimal set

[Huang, Yang]

Thank you for all your comments.
I cannot agree any more. I did try but there were so many dependencies. One 
change all change.

Micropython is a choice. But not sure if numpy and sqlite3 can be supported 
well. And what’s the compatibility of the libs in Pypi.


From: Guido van Rossum 
Sent: Tuesday, July 21, 2020 10:57 PM
To: Huang, Yang 
Cc: python-dev@python.org
Subject: Re: [Python-Dev] How to customize CPython to a minimal set

I expect it will be unfeasible to strip CPython. If you disagree, try it. ;-)

On Mon, Jul 20, 2020 at 22:35 Huang, Yang 
mailto:yang.hu...@intel.com>> wrote:
Hi, Guido

Yes. Micropyhton is also in consideration.
But sqlite3 is the first usage. There should be some additional features like 
numpy, scipy... Not sure if micropython supports well?

Or is there a feasible way to strip CPython ?

Thanks.

From: Guido van Rossum mailto:gu...@python.org>>
Sent: Monday, July 20, 2020 10:45 PM
To: Huang, Yang mailto:yang.hu...@intel.com>>
Cc: python-dev@python.org
Subject: Re: [Python-Dev] How to customize CPython to a minimal set

Have you considered starting with micropython? It’s made for embedded systems 
and fully supports Python 3 syntax. Adding sqlite3 support to it will be less 
work than stripping all the I/O from CPython.

—Guido

On Mon, Jul 20, 2020 at 06:48 Huang, Yang 
mailto:yang.hu...@intel.com>> wrote:

Hi, all

There is a request to run python in a Linux-based embedded resource constrained 
system with sqlite3 support.

So many features are not required, like posixmodule, signalmodule, hashtable ...
But seems there are some dependencies among the 
Modules/Parser/Python/Objects/Programs...

Is there a way to tailor CPython 3 to a minimal set with sqlite3 (the less 
syscalls the better) ?
Is it possible to do that?

Thank you.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to 
python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ECPLKXQ42VNLHD5DP3RG57L3QTJ77FUT/
Code of Conduct: http://python.org/psf/codeofconduct/
--
--Guido (mobile)
--
--Guido (mobile)
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/OMUGASSNBFJOT3W2ES4OIOQ4LOWMNHHG/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 626: Precise line numbers for debugging and other tools.

[Inada Naoki]

On Wed, Jul 22, 2020 at 10:53 PM Antoine Pitrou  wrote:
>
>
> Le 22/07/2020 à 15:48, Inada Naoki a écrit :
> > On Wed, Jul 22, 2020 at 8:51 PM Antoine Pitrou  wrote:
> >>
> >>>
> >>> I don't think all attempts are failed.  Note that current CPython includes
> >>> some optimization already.
> >>
> >> The set of compile-time optimizations has almost not changed since at
> >> least 15 years ago.
> >>
> >
> > Constant folding is rewritten and unused constants are removed from 
> > co_consts.
> > That's one of what Victor did his project.
>
> Constant folding is not a new optimization, so this does not contradict
> what I said.  Also, constant folding is not precluded by Mark's
> proposal, AFAIK.
>

Yes, this is to off topic. Please stop it.

-- 
Inada Naoki  
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ARPS3HD4AGATCX4ZXSO2QIRICE4FFE7O/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 626: Precise line numbers for debugging and other tools.

[Antoine Pitrou]

Le 22/07/2020 à 15:48, Inada Naoki a écrit :
> On Wed, Jul 22, 2020 at 8:51 PM Antoine Pitrou  wrote:
>>
>>>
>>> I don't think all attempts are failed.  Note that current CPython includes
>>> some optimization already.
>>
>> The set of compile-time optimizations has almost not changed since at
>> least 15 years ago.
>>
> 
> Constant folding is rewritten and unused constants are removed from co_consts.
> That's one of what Victor did his project.

Constant folding is not a new optimization, so this does not contradict
what I said.  Also, constant folding is not precluded by Mark's
proposal, AFAIK.

Regards

Antoine.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/LCHU3UC4Q4MKLEMSNSC5Z27E4JNYCMDS/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 626: Precise line numbers for debugging and other tools.

[Inada Naoki]

On Wed, Jul 22, 2020 at 8:51 PM Antoine Pitrou  wrote:
>
> >
> > I don't think all attempts are failed.  Note that current CPython includes
> > some optimization already.
>
> The set of compile-time optimizations has almost not changed since at
> least 15 years ago.
>

Constant folding is rewritten and unused constants are removed from co_consts.
That's one of what Victor did his project.

> > And I think there are some potential optimization if we can limit some
> > debugging/introspecting features, like some C variables are "optimzed away"
> > in gdb when
> > we use -O option.
>
> You can think it, but where's the proof?  Or at least the design
> document for these optimizations?  How do you explain that Victor's
> attempt at static optimization failed?
>

I have some opinion about it (especially, PHP 7.x achieved significant
performance improvement without JIT. I envy it.).
But I don't have time to prove it, and it is too off topic because it
is not related to precise line number. Please forget what I said about
blocking future optimization.

My idea was just merging code blocks, but it is not worth enough. And
it is not related to execution speed.

On the other hand, if we can not remove lnotab, it is still
considerable to avoid having two lnotabs in -O mode.
Memory overhead of lnotab is not negligible.

Regards,
-- 
Inada Naoki  
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/5T6ORCSTFS4EBQTYB7XLUJZ37C2WZECP/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 626: Precise line numbers for debugging and other tools.

[Mark Shannon]

On 22/07/2020 12:23 pm, Ned Batchelder wrote:

On 7/17/20 10:48 AM, Mark Shannon wrote:

Hi all,

I'd like to announce a new PEP.

It is mainly codifying that Python should do what you probably already 
thought it did :)


Should be uncontroversial, but all comments are welcome.


Thanks for thinking about these aspects of the interpreter, and for 
using the PEP process to work them out before implementation.


In the PEP, you mention, "some bytecodes will need to be marked as 
artificial, and not have a meaningful line number" (twice), but there's 
no example of what this means.  Can you elaborate?


Take the simple Python function:
```
def f(cond):
if cond:
   g()
else:
   h()
```
which compiles to the following bytecode:

 0 LOAD_FAST0 (cond)
 2 POP_JUMP_IF_FALSE   12

 4 LOAD_GLOBAL  0 (g)
 6 CALL_FUNCTION0
 8 POP_TOP
10 JUMP_FORWARD 6 (to 18)

12 LOAD_GLOBAL  1 (h)
14 CALL_FUNCTION0
16 POP_TOP
18 LOAD_CONST   0 (None)
20 RETURN_VALUE

Some of those instruction don't correspond to any line of code.

   Line number:
 0 LOAD_FAST   1
 2 POP_JUMP_IF_FALSE   1

 4 LOAD_GLOBAL 2
 6 CALL_FUNCTION   2
 8 POP_TOP 2
10 JUMP_FORWARD2 or artificial; it's debatable*

12 LOAD_GLOBAL 3
14 CALL_FUNCTION   3
16 POP_TOP 3
18 LOAD_CONST  Artificial; there is no `None` in the source.
20 RETURN_VALUEArtificial; there is no `return` statement.

*For practical reasons we would label this as line 2. It's faster and 
makes the line table more compact.


Cheers,
Mark.



--Ned.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/OTYRLXBR74IGQVEPEZIG272SMKZ4J2SZ/ 


Code of Conduct: http://python.org/psf/codeofconduct/

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/3VSYZ625CVTNN5UIAFYR6DIGHOQVVNY4/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 626: Precise line numbers for debugging and other tools.

[Mark Shannon]

On 21/07/2020 9:46 pm, Gregory P. Smith wrote:



On Fri, Jul 17, 2020 at 8:41 AM Ned Batchelder > wrote:


https://www.python.org/dev/peps/pep-0626/ :)

--Ned.

On 7/17/20 10:48 AM, Mark Shannon wrote:
 > Hi all,
 >
 > I'd like to announce a new PEP.
 >
 > It is mainly codifying that Python should do what you probably
already
 > thought it did :)
 >
 > Should be uncontroversial, but all comments are welcome.
 >
 > Cheers,
 > Mark.


"""When a frame object is created, the f_lineno will be set to the line 
at which the function or class is defined. For modules it will be set to 
zero."""


Within this PEP it'd be good for us to be very pedantic.  f_lineno is a 
single number.  So which number is it given many class and function 
definition statements can span multiple lines.


Is it the line containing the class or def keyword?  Or is it the line 
containing the trailing :?


The line of the `def`/`class`. It wouldn't change for the current 
behavior. I'll add that to the PEP.




Q: Why can't we have the information about the entire span of lines 
rather than consider a definition to be a "line"?


Pretty much every profiler, coverage tool, and debugger ever expects 
lines to be natural numbers, not ranges of numbers.

A lot of tooling would need to be changed.



I think that question applies to later sections as well.  Anywhere we 
refer to a "line", it could actually mean a span of lines. (especially 
when you consider \ continuation in situations you might not otherwise 
think could span lines)


Let's take an example:
```
x = (
a,
b,
)
```

You would want the BUILD_TUPLE instruction to have a of span lines 1 to 
4 (inclusive), rather just line 1?

If you wanted to break on the BUILD_TUPLE where you tell pdb to break?

I don't see that it would add much value, but it would add a lot of 
complexity.


Cheers,
Mark.



-gps

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/XKLAQ26BTLRQ3KRP5KEGYM4YSJJGDSQ7/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 626: Precise line numbers for debugging and other tools.

[Antoine Pitrou]

On Wed, 22 Jul 2020 19:42:30 +0900
Inada Naoki  wrote:
> On Wed, Jul 22, 2020 at 6:12 PM Antoine Pitrou  wrote:
> >  
> > >
> > > But if we merge two equal code blocks, we can not produce precise line
> > > numbers, can we?
> > > Is this inconsistent microoptimization that real optimization harder?
> > > This optimization must be prohibited in future Python?  
> >
> > All attempts to improve Python performance by compile-time
> > bytecode optimizations have more or less failed (the latter was
> > Victor's, AFAIR).  Is there still interest in pursuing that avenue?
> >
> > Regards
> >
> > Antoine.
> >  
> 
> I don't think all attempts are failed.  Note that current CPython includes
> some optimization already.

The set of compile-time optimizations has almost not changed since at
least 15 years ago.

> And I think there are some potential optimization if we can limit some
> debugging/introspecting features, like some C variables are "optimzed away"
> in gdb when
> we use -O option.

You can think it, but where's the proof?  Or at least the design
document for these optimizations?  How do you explain that Victor's
attempt at static optimization failed?

Regards

Antoine.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/46J4WU5ETYVIJ2UXYZY6CD6FLS3U33WE/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 626: Precise line numbers for debugging and other tools.

[Mark Shannon]

On 22/07/2020 11:42 am, Inada Naoki wrote:
On Wed, Jul 22, 2020 at 6:12 PM Antoine Pitrou > wrote:

 >
 > >
 > > But if we merge two equal code blocks, we can not produce precise line
 > > numbers, can we?
 > > Is this inconsistent microoptimization that real optimization harder?
 > > This optimization must be prohibited in future Python?
 >
 > All attempts to improve Python performance by compile-time
 > bytecode optimizations have more or less failed (the latter was
 > Victor's, AFAIR).  Is there still interest in pursuing that avenue?
 >
 > Regards
 >
 > Antoine.
 >

I don't think all attempts are failed.  Note that current CPython includes
some optimization already. If they are all failed, we must remove them
  to make compiler simple.

And I think there are some potential optimization if we can limit some
debugging/introspecting features, like some C variables are "optimzed 
away" in gdb when

we use -O option.


C is a pain to debug. Thankfully Python is not C.
Damaging people's ability to debug their code, to squeeze out 1% 
performance, is not worthwhile IMO.
Especially if that 1 or 2% costs us 10% later because it makes more 
sophisticated optimizations impractical.


Cheers,
Mark.



Regards,

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/BMWA4JES4FFFXMNZAVWCCRJD5NQCPMAK/
Code of Conduct: http://python.org/psf/codeofconduct/


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/4K3FMR6XO3YX3EBPBTNR6YRKPSWQGRU7/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 626: Precise line numbers for debugging and other tools.

[Ned Batchelder]

On 7/17/20 10:48 AM, Mark Shannon wrote:

Hi all,

I'd like to announce a new PEP.

It is mainly codifying that Python should do what you probably already 
thought it did :)


Should be uncontroversial, but all comments are welcome.


Thanks for thinking about these aspects of the interpreter, and for 
using the PEP process to work them out before implementation.


In the PEP, you mention, "some bytecodes will need to be marked as 
artificial, and not have a meaningful line number" (twice), but there's 
no example of what this means.  Can you elaborate?


--Ned.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/OTYRLXBR74IGQVEPEZIG272SMKZ4J2SZ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 626: Precise line numbers for debugging and other tools.

[Mark Shannon]

On 22/07/2020 10:07 am, Antoine Pitrou wrote:

On Wed, 22 Jul 2020 12:46:40 +0900
Inada Naoki  wrote:

On Wed, Jul 22, 2020 at 3:43 AM Mark Shannon  wrote:


On 18/07/2020 9:20 am, Inada Naoki wrote:

It seems great improvement, but I am worrying about performance.

Adding more attributes to the code object will increase memory usage
and importing time. Is there some estimation of the overhead?


Zero overhead (approximately).
We are just replacing one compressed table with another at the C level.
The other attributes are computed.
  


And I am worrying precise tracing blocks future advanced bytecode optimization.
Can we omit precise tracing and line number information when
optimization (`-O`) is enabled?


I don't think that is a good idea.
Performing any worthwhile performance optimization requires that we can
reason about the behavior of programs.
Consistent behavior makes that much easier.
Inconsistent "micro optimizations" make real optimizations harder.

Cheers,
Mark.
  


Tracing output is included in the program behavior?

For example, if two code block is completely equal:

if a == 1:
very very
long
code block
elif a == 2:
very very
long
code block

This code can be translated into like this (pseudo code):

if a == 1:
 goto block1
if a == 2:
 goto block1
block1:
 very very
 long
 code block

But if we merge two equal code blocks, we can not produce precise line
numbers, can we?
Is this inconsistent microoptimization that real optimization harder?
This optimization must be prohibited in future Python?


All attempts to improve Python performance by compile-time
bytecode optimizations have more or less failed (the latter was
Victor's, AFAIR).  Is there still interest in pursuing that avenue?


We are continually improving the bytecode, but there is probably only 
one or two percent speed up left possible from such improvements.

None of those improvements would prevent accurate line numbers.

Cheers,
Mark.



Regards

Antoine.

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/BDLZC3SSEKUJZNHN3CW6OIDDSDTTSUTH/
Code of Conduct: http://python.org/psf/codeofconduct/


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/K5JBMGURY7NG6O3I3WKSVDGVICTBLRWT/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 626: Precise line numbers for debugging and other tools.

[Ned Batchelder]

On 7/22/20 6:42 AM, Inada Naoki wrote:
On Wed, Jul 22, 2020 at 6:12 PM Antoine Pitrou > wrote:

>
> >
> > But if we merge two equal code blocks, we can not produce precise line
> > numbers, can we?
> > Is this inconsistent microoptimization that real optimization harder?
> > This optimization must be prohibited in future Python?
>
> All attempts to improve Python performance by compile-time
> bytecode optimizations have more or less failed (the latter was
> Victor's, AFAIR).  Is there still interest in pursuing that avenue?
>
> Regards
>
> Antoine.
>

I don't think all attempts are failed.  Note that current CPython includes
some optimization already. If they are all failed, we must remove them
 to make compiler simple.

And I think there are some potential optimization if we can limit some
debugging/introspecting features, like some C variables are "optimzed 
away" in gdb when

we use -O option.


We seem to like following the C model when it comes to implementing 
optimizations, and then skip the part where C developers can disable all 
optimizations when reasoning about code is more important than speed.  I 
am fine with any optimizations at all, as long as there is a simple and 
supported way to ask that they be disabled.


--Ned.

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/HBNCQPU7ECM5EE3O2MDMEQDKX5D4L6DW/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 626: Precise line numbers for debugging and other tools.

[Ned Batchelder]

On 7/21/20 5:04 PM, Gregory P. Smith wrote:
Given it is impossible for tools doing passive inspection of Python VM 
instances to execute code, co_linetable's exact format will be 
depended on just as co_lnotab was.  co_lnotab was only 
quasi-"officially" documented in the Python docs, it's spec lives in 
https://github.com/python/cpython/blob/master/Objects/lnotab_notes.txt (pointed 
to by a couple module's docs). The lnotab format "changed" once, in 
3.6, an unsigned delta was changed to signed (but I don't believe 
anything beyond some experiments ever actually used negatives?).


Negatives definitely happen.  When I comment out the line in coverage.py 
that deals with negative deltas, 34 of my tests fail.


For example:

   a = (
    1
   )

With 3.8 compiles to:

  2   0 LOAD_CONST   0 (1)

  1   2 STORE_NAME   0 (a)
  4 LOAD_CONST   1 (None)
  6 RETURN_VALUE

With an lnotab of "02 ff".

When executed, this produces these trace events:

   call on line 1
   line on line 2
   line on line 1
   return on line 1

--Ned.

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/CSMZXXEYISMTVDPTNOG4J65IZJ76OI2Y/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 626: Precise line numbers for debugging and other tools.

[Inada Naoki]

On Wed, Jul 22, 2020 at 6:12 PM Antoine Pitrou  wrote:
>
> >
> > But if we merge two equal code blocks, we can not produce precise line
> > numbers, can we?
> > Is this inconsistent microoptimization that real optimization harder?
> > This optimization must be prohibited in future Python?
>
> All attempts to improve Python performance by compile-time
> bytecode optimizations have more or less failed (the latter was
> Victor's, AFAIR).  Is there still interest in pursuing that avenue?
>
> Regards
>
> Antoine.
>

I don't think all attempts are failed.  Note that current CPython includes
some optimization already. If they are all failed, we must remove them
 to make compiler simple.

And I think there are some potential optimization if we can limit some
debugging/introspecting features, like some C variables are "optimzed away"
in gdb when
we use -O option.

Regards,
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/BMWA4JES4FFFXMNZAVWCCRJD5NQCPMAK/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 626: Precise line numbers for debugging and other tools.

[Antoine Pitrou]

On Wed, 22 Jul 2020 12:46:40 +0900
Inada Naoki  wrote:
> On Wed, Jul 22, 2020 at 3:43 AM Mark Shannon  wrote:
> >
> > On 18/07/2020 9:20 am, Inada Naoki wrote:  
> > > It seems great improvement, but I am worrying about performance.
> > >
> > > Adding more attributes to the code object will increase memory usage
> > > and importing time. Is there some estimation of the overhead?  
> >
> > Zero overhead (approximately).
> > We are just replacing one compressed table with another at the C level.
> > The other attributes are computed.
> >  
> > >
> > > And I am worrying precise tracing blocks future advanced bytecode 
> > > optimization.
> > > Can we omit precise tracing and line number information when
> > > optimization (`-O`) is enabled?  
> >
> > I don't think that is a good idea.
> > Performing any worthwhile performance optimization requires that we can
> > reason about the behavior of programs.
> > Consistent behavior makes that much easier.
> > Inconsistent "micro optimizations" make real optimizations harder.
> >
> > Cheers,
> > Mark.
> >  
> 
> Tracing output is included in the program behavior?
> 
> For example, if two code block is completely equal:
> 
> if a == 1:
>very very
>long
>code block
> elif a == 2:
>very very
>long
>code block
> 
> This code can be translated into like this (pseudo code):
> 
> if a == 1:
> goto block1
> if a == 2:
> goto block1
> block1:
> very very
> long
> code block
> 
> But if we merge two equal code blocks, we can not produce precise line
> numbers, can we?
> Is this inconsistent microoptimization that real optimization harder?
> This optimization must be prohibited in future Python?

All attempts to improve Python performance by compile-time
bytecode optimizations have more or less failed (the latter was
Victor's, AFAIR).  Is there still interest in pursuing that avenue?

Regards

Antoine.

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/BDLZC3SSEKUJZNHN3CW6OIDDSDTTSUTH/
Code of Conduct: http://python.org/psf/codeofconduct/