Re: [Python-Dev] Lost sight
Like everyone else, I am sorry to hear this news, and extend my sympathies. It's marvellous that you are still able to work on Python at all, still. Since you have to be careful not to do too much, you will just have to choose your battles carefully. Thank you for your many contributions to Python to date. Kind regards Steve On Sat, Jan 19, 2019 at 10:14 AM Serhiy Storchaka wrote: > I have virtually completely lost the sight of my right eye (and the loss > is quickly progresses) and the sight of my left eye is weak. That is why > my activity as a core developer was decreased significantly at recent > time. My apologies to those who are waiting for my review. I will do it > slowly. > > ___ > Python-Dev mailing list > [email protected] > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/steve%40holdenweb.com > ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Add more SyntaxWarnings?
https://bugs.python.org/issue15248 is about situations like the following: >>> [(1,2,3) (4,5,6)] Traceback (most recent call last): File "", line 2, in (4,5,6)] TypeError: 'tuple' object is not callable The original poster requested that the error message be augmented with something like "(missing preceding comma?)" Ezio Melotti suggested a FAQ entry like https://docs.python.org/3/faq/programming.html#why-am-i-getting-an-unboundlocalerror-when-the-variable-has-a-value (I think such entries below in a separate doc and will try to post on python-ideas when I have a prototype.) Serhiy Storchaka suggested a compiler SyntaxWarning and uploaded a proof-of-concept diff that handled the above and many similar cases. The diff is based on the idea that while we can only positively identify 'callables' at runtime, we *can* negatively identify many non-callables when compiling. Ditto for subscriptables and indexables. Serhiy concluded with "This patch was inspired by usability improvements in GCC 8. https://developers.redhat.com/blog/2018/03/15/gcc-8-usability-improvements/ I haven't created a pull request because I have doubts about that this should be in the compiler rather of a third-party linter. But if several other core developers will support this idea I'll continue working in this direction." I was impressed with how clear and readable the patch is and consider it a plausible enhancement. I would like other core developers to comment. -- Terry Jan Reedy ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Sub-interpreters: importing numpy causes hang
If your primary concern is module clashes between plugins, maybe you
can hack around that:
1) if the plugins are providing copies of any other modules, then you
can simply require them to put them in their own namespace — that is,
a plug-in is a single package, with however many sub modules as it may
need.
2) if plugins might require third party packages that need to be
isolated, then maybe you could use an import hook that
re-names/isolates the modules each plugin loads, so they are kept
separate.
I haven’t thought through how to do any of this, but in principle, you
can have the same module loaded twice if it has a different name.
Not that sub interpreters aren’t cool and useful, but you can probably
handle module clashes in a simpler way.
-CHB
Sent from my iPhone
> On Jan 23, 2019, at 11:41 AM, Stephan Reiter wrote:
>
> You all do make me feel very welcome in this community! Thank you very much!
> :-)
>
> And thank you for all the thought and time you put into your message,
> Eric. I do appreciate in particular all the alternatives you
> presented; you provide a good picture of my options.
> Not ruling out any of them, I'll stick with (single process + multiple
> subinterpreters + plugins can't keep state in Python + all my Python
> calls are performed on the main thread) for the time being. That's
> quite a limited environment, which I hope I can make work in the long
> run. And I think the concept of subinterpreters is nice and I'd like
> to spend some time on the challenge of improving the situation.
>
> So, I updated my changes and have the following on top of 3.6.1 at the moment:
> https://github.com/stephanreiter/cpython/commit/c1afa0c8cdfab862f409f1c7ff02b189f5191cbe
>
> I did what Henry suggested and ran the Python test suite. On Windows,
> with my changes I get as output:
>
> 357 tests OK.
>
> 2 tests failed:
> test_re test_subprocess
>
> 46 tests skipped:
> test_bz2 test_crypt test_curses test_dbm_gnu test_dbm_ndbm
> test_devpoll test_epoll test_fcntl test_fork1 test_gdb test_grp
> test_idle test_ioctl test_kqueue test_lzma test_nis test_openpty
> test_ossaudiodev test_pipes test_poll test_posix test_pty test_pwd
> test_readline test_resource test_smtpnet test_socketserver
> test_spwd test_sqlite test_ssl test_syslog test_tcl
> test_threadsignals test_timeout test_tix test_tk test_ttk_guionly
> test_ttk_textonly test_turtle test_urllib2net test_urllibnet
> test_wait3 test_wait4 test_winsound test_xmlrpc_net test_zipfile64
>
> Total duration: 6 min 20 sec
> Tests result: FAILURE
>
> I dropped my changes and ran the test suite again using vanilla Python
> and got the same result.
> So, it seems that the change doesn't break anything that is tested,
> but that probably doesn't mean a lot.
>
> Tomorrow, I'll investigate the following situation if I find time:
>
> If we create a fresh OS thread and make it call PyGILState_Ensure, it
> won't have a PyThreadState saved under autoTLSkey. That means it will
> create one using the main interpreter. I, as the developer embedding
> Python into my application and using multiple interpreters, have no
> control here. Maybe I know that under current conditions a certain
> other interpreter should be used.
>
> I'll try to provoke this situation and then introduce a callback from
> Python into my application that will allow me to specify which
> interpreter should be used, e.g. code as follows:
>
> PyInterpreter *pickAnInterpreter() {
> return activePlugin ? activePlugin->interpreter : nullptr; //
> nullptr maps to main interpreter
> }
>
> PyGILState_SetNewThreadInterpreterSelectionCallback(&pickAnInterpreter);
>
> Maybe rubbish. But I think a valuable experiment that will give me a
> better understanding.
>
> Stephan
>
> Am Mi., 23. Jan. 2019 um 18:11 Uhr schrieb Eric Snow
> :
>>
>> Hi Stephan,
>>
>>> On Tue, Jan 22, 2019 at 9:25 AM Stephan Reiter
>>> wrote:
>>> I am new to the list and arriving with a concrete problem that I'd
>>> like to fix myself.
>>
>> That is great! Statements like that are a good way to get folks
>> interested in your success. :)
>>
>>> I am embedding Python (3.6) into my C++ application and I would like
>>> to run Python scripts isolated from each other using sub-interpreters.
>>> I am not using threads; everything is supposed to run in the
>>> application's main thread.
>>
>> FYI, running multiple interpreters in the same (e.g. main) thread
>> isn't as well thought out as running them in separate threads. There
>> may be assumptions in the runtime that would cause crashes or
>> inconsistency in the runtime, so be vigilant. Is there a reason not
>> to run the subinterpreters in separate threads?
>>
>> Regarding isolation, keep in mind that there are some limitations. At
>> an intrinsic level subinterpreters are never truly isolated since they
>> run in the same process. This matters if you have concerns about
>> security (which you should always consider) and stability (if a
>> subinterpreter cras
Re: [Python-Dev] Add more SyntaxWarnings?
On 2019-01-24, Terry Reedy wrote: > Serhiy Storchaka suggested a compiler SyntaxWarning and uploaded a > proof-of-concept diff that handled the above and many similar cases. I believe that in general we should give better errors or warnings if we can do it without huge difficulty. Serhiy's patch is quite simple. The same check *could* be done by a linting tool. Putting it in CPython will make it more widely available. These checks could be helpful to beginners who probably won't have linting tools setup. I think we should not make it an error, otherwise we have changed Python "the language". We don't want to force other Python implementations to do the same check. It might be hard for them to implement. So, SyntaxWarning seems like a reasonable compromise. Regards, Neil ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Add more SyntaxWarnings?
In this case I agree that a SyntaxWarning is a good idea, per Serhiy's patch. I would be even more conservative, and only warn if the first object is a tuple -- the case of the missing comma in the original example appears likely enough, but I don't expect people to write e.g. `[[1, 2], (3, 4)]` very often, so leaving the comma out there would be very unlikely. Regarding the issue of when it's appropriate to issue a SyntaxWarning vs. when to leave it up to linters, again I would recommend great caution and only warn about code that is *definitely* going to fail when executed. (Or at least is *definitely* not going to please the programmer -- one of the first cases where we added a SyntaxWarning was actually `assert(condition, message)`, which "fails" by never failing. :-) But this is one of those cases. On Thu, Jan 24, 2019 at 2:10 PM Neil Schemenauer wrote: > On 2019-01-24, Terry Reedy wrote: > > Serhiy Storchaka suggested a compiler SyntaxWarning and uploaded a > > proof-of-concept diff that handled the above and many similar cases. > > I believe that in general we should give better errors or warnings > if we can do it without huge difficulty. Serhiy's patch is quite > simple. The same check *could* be done by a linting tool. Putting > it in CPython will make it more widely available. These checks > could be helpful to beginners who probably won't have linting tools > setup. > > I think we should not make it an error, otherwise we have changed > Python "the language". We don't want to force other Python > implementations to do the same check. It might be hard for them to > implement. So, SyntaxWarning seems like a reasonable compromise. > > Regards, > > Neil > ___ > Python-Dev mailing list > [email protected] > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Add more SyntaxWarnings?
On Thu, Jan 24, 2019 at 04:01:26PM -0600, Neil Schemenauer wrote:
> On 2019-01-24, Terry Reedy wrote:
> > Serhiy Storchaka suggested a compiler SyntaxWarning and uploaded a
> > proof-of-concept diff that handled the above and many similar cases.
>
> I believe that in general we should give better errors or warnings
> if we can do it without huge difficulty. Serhiy's patch is quite
> simple. The same check *could* be done by a linting tool. Putting
> it in CPython will make it more widely available. These checks
> could be helpful to beginners who probably won't have linting tools
> setup.
+1 to what Neil says here.
> I think we should not make it an error, otherwise we have changed
> Python "the language".
We're allowed to, we're the ones who say what the language is :-)
The remainder of my comments are more speculative.
> We don't want to force other Python
> implementations to do the same check. It might be hard for them to
> implement. So, SyntaxWarning seems like a reasonable compromise.
We could say that implementations are allowed to raise errors at compile
time instead of run time, but aren't required to. Then it becomes a
matter of "quality of implementation".
For literal ints, strings, None, etc we can tell at compile time that an
error will occur. All of these cannot fail to raise (short of changing
the interpreter, in which case you're not using Python anymore):
1 + "1" # raises TypeError
None[x] # TypeError
1.234(x) # TypeError
"spam".idnex("a") # AttributeError
In these specific cases, there is nothing wrong with the *syntax*, but a
compiler should be permitted to immediately raise the same exception
that would otherwise occur at run time. This is a form of peephole
optimization, I guess.
If people truly wanted to delay the exception, they could either turn
off the peephole optimizer, or work around it:
a = 1; a + "1"
Thoughts?
--
Steve
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Add more SyntaxWarnings?
On Fri, Jan 25, 2019 at 9:42 AM Steven D'Aprano wrote:
> We could say that implementations are allowed to raise errors at compile
> time instead of run time, but aren't required to. Then it becomes a
> matter of "quality of implementation".
>
> For literal ints, strings, None, etc we can tell at compile time that an
> error will occur. All of these cannot fail to raise (short of changing
> the interpreter, in which case you're not using Python anymore):
>
> 1 + "1" # raises TypeError
> None[x] # TypeError
> 1.234(x) # TypeError
> "spam".idnex("a") # AttributeError
>
> In these specific cases, there is nothing wrong with the *syntax*, but a
> compiler should be permitted to immediately raise the same exception
> that would otherwise occur at run time. This is a form of peephole
> optimization, I guess.
+1. If it's something that the peephole optimizer is already allowed
to change (eg "1"+"1" is constant-folded) and there is absolutely no
way that it can ever be changed at run time, then raising at compile
time can't hurt [1]. It'd be as implementation-dependent and
version-dependent as the peephole optimizer itself.
Does there need to be a new subclass of SyntaxError for "Technically
Valid But Not Meaningful" problems? Is there value in distinguishing
"InevitableTypeError" from "InevitableAttributeError"?
ChrisA
[1] Yes, I know about XKCD 1172, but if someone's saying "if
shouldnt_happen: None[None]" then that's their problem.
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Add more SyntaxWarnings?
On Thu, Jan 24, 2019 at 2:55 PM Chris Angelico wrote:
> On Fri, Jan 25, 2019 at 9:42 AM Steven D'Aprano
> wrote:
> > We could say that implementations are allowed to raise errors at compile
> > time instead of run time, but aren't required to. Then it becomes a
> > matter of "quality of implementation".
> >
> > For literal ints, strings, None, etc we can tell at compile time that an
> > error will occur. All of these cannot fail to raise (short of changing
> > the interpreter, in which case you're not using Python anymore):
> >
> > 1 + "1" # raises TypeError
> > None[x] # TypeError
> > 1.234(x) # TypeError
> > "spam".idnex("a") # AttributeError
> >
> > In these specific cases, there is nothing wrong with the *syntax*, but a
> > compiler should be permitted to immediately raise the same exception
> > that would otherwise occur at run time. This is a form of peephole
> > optimization, I guess.
>
> +1. If it's something that the peephole optimizer is already allowed
> to change (eg "1"+"1" is constant-folded) and there is absolutely no
> way that it can ever be changed at run time, then raising at compile
> time can't hurt [1]. It'd be as implementation-dependent and
> version-dependent as the peephole optimizer itself.
>
I'm -1 on all of these cases. There's nothing mysterious about e.g.
`TypeError: unsupported operand type(s) for +: 'int' and 'str'`, unlike the
case of the two concatenated tuples. (Surely people get errors about
int+str all the time, and they've never complained -- unlike the tuple
tuple case.)
> Does there need to be a new subclass of SyntaxError for "Technically
> Valid But Not Meaningful" problems? Is there value in distinguishing
> "InevitableTypeError" from "InevitableAttributeError"?
>
I don't think there's a *general* problem to be solved here.
> ChrisA
>
> [1] Yes, I know about XKCD 1172, but if someone's saying "if
> shouldnt_happen: None[None]" then that's their problem.
>
--
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Add more SyntaxWarnings?
>. There's nothing mysterious about e.g. `TypeError: unsupported operand type(s) for +: 'int' and 'str'`, unlike the case of the two concatenated tuples. (Surely people get errors about int+str all the time, and they've never complained -- unlike the tuple tuple case.) Well, yes, that particular example is pretty clear. But as a rule, there are a LOT of errors that can be pretty mysterious to newbies. I would love to see Python become generally more informative with errors. In this case, you’d probably get a similar error, but it’s still nice to get it sooner, and if the hooks are in place, We could have others that are really helpful. -CHB Is there value in distinguishing > "InevitableTypeError" from "InevitableAttributeError"? > I don’t think so — what we need are helpful error messages. If it will be raised at compile time, then it won’t generally be catchable in s try-except— so the actual exception type isn’t very important. -CHB ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Add more SyntaxWarnings?
On Thu, Jan 24, 2019 at 3:45 PM Chris Barker - NOAA Federal < [email protected]> wrote: > >. There's nothing mysterious about e.g. `TypeError: unsupported operand > type(s) for +: 'int' and 'str'`, unlike the case of the two concatenated > tuples. (Surely people get errors about int+str all the time, and they've > never complained -- unlike the tuple tuple case.) > > Well, yes, that particular example is pretty clear. But as a rule, there > are a LOT of errors that can be pretty mysterious to newbies. > > I would love to see Python become generally more informative with errors. > > In this case, you’d probably get a similar error, but it’s still nice to > get it sooner, and if the hooks are in place, We could have others that are > really helpful. > I doubt people are writing `42 + "abc"` by accident. They'll write `x + y` and by accident the types won't match. So better error messages at runtime would help. But I doubt we'll see much mileage out of the syntax checks. And making things work different based on whether it's a literal or a variable isn't very Pythonic. -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Add more SyntaxWarnings?
On 1/24/2019 6:16 PM, Eric V. Smith wrote: On 1/24/2019 5:52 PM, Chris Angelico wrote: +1. If it's something that the peephole optimizer is already allowed to change (eg "1"+"1" is constant-folded) and there is absolutely no way that it can ever be changed at run time, then raising at compile time can't hurt [1]. It'd be as implementation-dependent and version-dependent as the peephole optimizer itself. It would be a change if the code is never called. I'm not sure we care about code that's never called, but it is a change. Which Chris already pointed out, in some text that I inadvertently deleted. I didn't catch his meaning, which is my fault. Eric ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Add more SyntaxWarnings?
On 1/24/2019 5:52 PM, Chris Angelico wrote:
On Fri, Jan 25, 2019 at 9:42 AM Steven D'Aprano wrote:
We could say that implementations are allowed to raise errors at compile
time instead of run time, but aren't required to. Then it becomes a
matter of "quality of implementation".
For literal ints, strings, None, etc we can tell at compile time that an
error will occur. All of these cannot fail to raise (short of changing
the interpreter, in which case you're not using Python anymore):
1 + "1" # raises TypeError
None[x] # TypeError
1.234(x) # TypeError
"spam".idnex("a") # AttributeError
In these specific cases, there is nothing wrong with the *syntax*, but a
compiler should be permitted to immediately raise the same exception
that would otherwise occur at run time. This is a form of peephole
optimization, I guess.
+1. If it's something that the peephole optimizer is already allowed
to change (eg "1"+"1" is constant-folded) and there is absolutely no
way that it can ever be changed at run time, then raising at compile
time can't hurt [1]. It'd be as implementation-dependent and
version-dependent as the peephole optimizer itself.
It would be a change if the code is never called. I'm not sure we care
about code that's never called, but it is a change.
Eric
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Add more SyntaxWarnings?
At the cost of breaking threading a bit, I'm going to reply to a few different people in one post. On Fri, Jan 25, 2019 at 09:52:53AM +1100, Chris Angelico wrote: > On Fri, Jan 25, 2019 at 9:42 AM Steven D'Aprano wrote: > > We could say that implementations are allowed to raise errors at compile > > time instead of run time, but aren't required to. Then it becomes a > > matter of "quality of implementation". [...] > Does there need to be a new subclass of SyntaxError for "Technically > Valid But Not Meaningful" problems? Is there value in distinguishing > "InevitableTypeError" from "InevitableAttributeError"? If we did this (which is looking less and less likely...) I would argue for plain old TypeError and AttributeError. There should be no difference in the exception raised, only in the timing (compile time versus run time). On Thu, Jan 24, 2019 at 11:45:49PM +, Chris Barker wrote: > I would love to see Python become generally more informative with > errors. > > In this case, you’d probably get a similar error, but it’s still nice > to get it sooner That is precisely my thinking. On Thu, Jan 24, 2019 at 03:24:41PM -0800, Guido van Rossum wrote: > I'm -1 on all of these cases. There's nothing mysterious about e.g. > `TypeError: unsupported operand type(s) for +: 'int' and 'str'` Its not about the error being mysterious, its about raising the exception as early as possible. It shouldn't take 20 years to discover that this snippet buried deep in a large code base: if datetime.date.today().year == 2029: spam = 1 + '1' will fail :-) On Thu, Jan 24, 2019 at 03:58:50PM -0800, Guido van Rossum wrote: > But I doubt we'll see much mileage out of the syntax checks. Frankly, I agree, so I'm not going to argue hard for this. It was more speculation than a solid proposal. A project manager I used to work with would have dismissed this as "turd polishing", but I prefer to think of it as "craftsmanship" -- a bit like a carpenter ensuring that his screws don't protrude above the surface of the wood. Sure, its not going to hold the wood together any better, but its just a nicer end-user experience :-) I expect that the cost of implementing these checks would probably exceed the benefit of detecting the errors early, but I could be wrong. Maybe it would be easy to implement. But if there is strong opposition to it, I'm not going to champion the idea any further. -- Steve ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Add more SyntaxWarnings?
Chris Barker - NOAA Federal via Python-Dev writes: > Well, yes, that particular example is pretty clear. But as a rule, > there are a LOT of errors that can be pretty mysterious to newbies. Isn't that the very definition of "newbie"? That's half a joke, but I really don't think that programmers new to Python should be the standard. The problematic cases are those where even a relatively experienced Python programmer needs to be told why an error is raised, because it's too hard to figure out from background knowledge of the language, you need to know about implementation internals. > I would love to see Python become generally more informative with > errors. I would love to see all automated systems become more informative with errors! One thing I like about Python the development community is that people are doing something about it. Slowly but surely > I don’t think so — what we need are helpful error messages. If it > will be raised at compile time, then it won’t generally be > catchable in s try-except— so the actual exception type isn’t very > important. +1 In general depending much on exception type is a hit-or-miss affair. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
