[issue32309] Implement asyncio.run_in_executor shortcut

[Yury Selivanov]

Yury Selivanov  added the comment:

> 
> 
> 
> IMO, I think it would be a bit more clear to just explicitly call it
> "threads" or "max_threads", as that explains what it's effectively doing.
> While "concurrency" is still a perfectly correct way of describing the
> behavior, I think it might be a little too abstract for an argument name.
> 
> 
> 

> 
> 
> 
> 
> 
> 
> 

And that's why I like it. If we add ProcessPool it will have the same argument: 
concurrency.

max_workers isn't correct, as we want to spawn all threads and all processes 
when we start. Thus btw makes me think that initializing threads/processes in 
__init__ is a bad idea, as spawning can be asynchronous.

> 
> 
> 
> 
> 
> 
> 
>

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37903] IDLE Shell sidebar.

[Guido van Rossum]

Guido van Rossum  added the comment:

Strong +1. I should have done this 20 years ago.

--
nosy: +gvanrossum

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37607] segfault running code in jupyter on macOS 10.14.5 - crashed on child side of fork pre-exec

[wesinator]

wesinator <13hu...@gmail.com> added the comment:

Thanks ned - should I report issue to libcurl ?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37903] IDLE Shell sidebar.

[Terry J. Reedy]

Terry J. Reedy  added the comment:

My previous response was to comments up to msg355812.

To answer Raymond's subsequent questions: The user will enter python code the 
same as before.  Currently, the indents and resulting appearance of the code on 
Shell screen is different from (and worse, sometimes far worse, than) the 
appearance of the same code in the REPL.  After this patch *and* a required 
immediate follow-up to replace tab indents with, by default, 4-space indents, 
the code on the screen *and in saved sessions* will appear the same as in the 
REPL (when 4-space indents are used).

In Tal's screenshot, the appearance is better, but the indents are still tabs.  
Because of the regression being fixed by #38636, PR 17008, he could not 
manually switch with alt-T and alt-U.

(The current sidebar patch is also missing output markers.  Also, for my custom 
dark theme, I colored the sidebar dark blue on light bluish gray and the 
result, to me, is aesthetically much more pleasing than in the screenshot.)

The improvement is, to me, even more evident in the following pair of examples 
-- first REPL and improved IDLE, second current IDLE. 

>>> if True:
... print('true')
... else:
... print('false')

versus

>>> if True:
print('true')
else:
print('false')

#7676 was marked as a bugfix issue and I regard changing examples like the 
above to be a long-overdue and badly needed fix.  Any version that does not get 
this fix will also not get followup changes.  I just added msg355853 to #37892 
to list some I have thought of.

One of my intended followups is to add multiple ways of saving a shell session. 
 IDLE can juggle prompts/markers, code, and output to match 'templates' much 
easier than users can.  I had not thought about doctests, but saving shell 
interaction in the proper format should be an easy addition.

SyntaxErrors aside, output will be the same as it is in IDLE now, with 'Out', 
'Err', and maybe 'Inp' markers ('Imp' for user code input(prompt) lines) 
planned for the sidebar.

IDLE already handles syntax errors in the code being entered differently from 
the REPL by replacing the caret line with an error highlight, as in the editor. 
 I would like it follow the logic of not imitating dumb terminals by putting 
the cursor at the marked spot, as in the editor, and letting the user directly 
correct the problem.  Messing up the input with a usually useless traceback and 
a new prompt and requiring retrieval of the statement with the marker removed 
and the cursor in the wrong place is a nuisance.  I also don't see any reason 
to burden the session history with bad code that was never submitted for 
execution.  (Note that IDLE does not RESTART when code in the editor has syntax 
errors.)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32309] Implement asyncio.run_in_executor shortcut

[Kyle Stanley]

Kyle Stanley  added the comment:

> def __init__(self, concurrency=None):

Minor clarification: the default should probably be None, which would 
effectively set the default maximum number of threads to min(32, os.cpu_count() 
+ 4), once it's passed to ThreadPoolExecutor.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32309] Implement asyncio.run_in_executor shortcut

[Kyle Stanley]

Kyle Stanley  added the comment:

> Number of OS threads to spawn.

Ah I see, so this would correspond with the "max_workers" argument of 
ThreadPoolExecutor then, correct? If so, we could pass this in the __init__ for 
ThreadPool:

def __init__(self, concurrency):
...
self._executor = 
concurrent.futures.ThreadPoolExecutor(max_workers=concurrency)

IMO, I think it would be a bit more clear to just explicitly call it "threads" 
or "max_threads", as that explains what it's effectively doing. While 
"concurrency" is still a perfectly correct way of describing the behavior, I 
think it might be a little too abstract for an argument name.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38630] subprocess.Popen.send_signal() should poll the process

[Giampaolo Rodola']

Giampaolo Rodola'  added the comment:

-1 about the PR solution to suppress ProcessLookupError in case the process is 
gone. In psutil I solved the “pid reused problem” by using process creation 
time to identify a process uniquely (on start). 
A decorator can be used to protect the sensibile methods interacting with the 
PID/handle (communicate(), send_signal(), terminate(), kill()) and raise an 
exception (maybe ProcessLooKupError(“PID has been reused”)?).

--
nosy: +giampaolo.rodola

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32309] Implement asyncio.run_in_executor shortcut

[Yury Selivanov]

Yury Selivanov  added the comment:

>> async with asyncio.ThreadPool(concurrency=10) as pool:

> I'm definitely on board with the usage of an async context manager and the 
> functionality shown in the example, but I'm not sure that I entirely 
> understand what the "concurrency" kwarg in "concurrency=10" is supposed to 
> represent in this case. Could you elaborate on what that would do 
> functionally?

Number of OS threads to spawn.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32309] Implement asyncio.run_in_executor shortcut

[Kyle Stanley]

Kyle Stanley  added the comment:

> I believe behavior occurs within shutdown_default_executor(), correct? 
> Specifically, within for ThreadPoolExecutor when executor.shutdown(wait=True) 
> is called and all of the threads are joined without a timeout, it simply 
> waits for each thread to terminate gracefully.
 
Correction, I phrased this poorly:

I believe this behavior occurs within shutdown_default_executor(), correct? 
Specifically, when executor.shutdown(wait=True) is called within _do_shutdown() 
and ...

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37892] IDLE Shell: isolate user code input

[Terry J. Reedy]

Terry J. Reedy  added the comment:

One of the related changes is to add multiple options for saving the shell 
history.  For editing, one wants code without prompts and output either 
omitting or commented out.  For doctests, one wants code with '>>> ' and '... ' 
prompts and unmarked output.

A second would be to *not* print SyntaxError and a new prompt, but to otherwise 
indicate 'error' and put the cursor after the already highlighted error, as 
done in the editor.

A third would being able to submit multiline statements for execution with with 
F5 instead of having to go to the end of the last line and enter a blank line.  
This would be especially useful after correcting a syntax error or other typo.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32309] Implement asyncio.run_in_executor shortcut

[Kyle Stanley]

Kyle Stanley  added the comment:

> async with asyncio.ThreadPool(concurrency=10) as pool:

I'm definitely on board with the usage of an async context manager and the 
functionality shown in the example, but I'm not sure that I entirely understand 
what the "concurrency" kwarg in "concurrency=10" is supposed to represent in 
this case. Could you elaborate on what that would do functionally?

> It should be the "async with".

> We should not reuse the default executor.

> I think it's OK to initialize the thread pool in `ThreadPool.__init__`.  
> `ThreadPool.__aenter__` would simply return `self` then.

> I think we should aim for shipping a replacement for `loop.run_in_executor` 
> in 3.9.

Strong +1 on all of those points, I agree. 

> `await ThreadPool.aclose()` would close the loop gracefully (awaiting for all 
> submitted and still running callables) in cases when people use the 
> threadpool without 'async with'.

I believe behavior occurs within shutdown_default_executor(), correct? 
Specifically, within for ThreadPoolExecutor when executor.shutdown(wait=True) 
is called and all of the threads are joined without a timeout, it simply waits 
for each thread to terminate gracefully. 

So if we moved that functionality to a private coroutine method 
_shutdown_executor() as suggested in my above example, this could also be done 
for ThreadPool. Unless you want ThreadPool to be it's own entirely separate 
implementation that doesn't depend at all on ThreadPoolExecutor. 

Personally I think we can use ThreadPoolExecutor in the internal details; it 
seems that the main issue with it isn't the functionality, but rather the low 
level API offered with loop.run_in_executor().

Also another point to consider is if we should have users explicitly call 
pool.aclose() or if this should be done automatically when exiting the context 
manager through the __aexit__. I prefer the latter myself for similar reasons 
to what I previously mentioned, with a context manager initializing it's own 
resources on entry and finalizing them upon exit.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37903] IDLE Shell sidebar.

[Raymond Hettinger]

Raymond Hettinger  added the comment:

Thanks.  Can you also show a script running with F5 showing a restart, the 
output from the script and a subsequent interactive session that shows the 
inspecting variables.  Also show what an exception and SyntaxError looks like.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37903] IDLE Shell sidebar.

[Tal Einat]

Tal Einat  added the comment:

To help get a feel for what the current PR looks like, I'm attaching a 
screenshot (taken on Windows 10).

--
Added file: https://bugs.python.org/file48690/screenshot.png

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38500] Provide a way to get/set PyInterpreterState.frame_eval without needing to access interpreter internals

[Dino Viehland]

Dino Viehland  added the comment:

Adding the getter/setters seems perfectly reasonable to me, and I agree they 
should be underscore prefixed as well.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38631] Replace Py_FatalError() with regular Python exceptions

[STINNER Victor]

STINNER Victor  added the comment:

(I have a few more functions that I would like to patch, so I keep the issue 
open as place holder.)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37903] IDLE Shell sidebar.

[Terry J. Reedy]

Terry J. Reedy  added the comment:

As stated in my second opening sentence, this issue "implements the first phase 
of #37892", which gives the context and specific rationale.

There are 3 separate parts to a Python Shell.

1. Code entry, indentation, display, and copying.  This issue fixes Shell 
design flaws that have been recognized as such for at least 15 years.  In the 
process, it makes Shell *more similar* to the standard REPL in important 
respects.  In particular, by default, compound statements will look the same in 
Shell as the same code entered into REPL with 4-space indents.  It will also be 
possible to cut and paste into an editor have it look right.

Raymond: I do not plan to retain a buggy shell mode that in #7676 you called a 
"major PITA" (msg215288) and "a continual source of frustration for students in 
my Python courses" that you were "looking forward to it being fixed."

2. Code execution.  IDLE executes syntactically correct user code so that, to 
the extent possible and sensible, the result is the same as if executed 
directly with python.  Not an issue here.

3. Everything else.  Alternate shells are intentionally difference from and 
hopefully improve upon the interactive python.exe REPL.

Paul: The existence of two prompts, sys.ps1 and sys.ps2 in python.exe 
interactive mode is a kludge needed to interface nested multiline statement 
Python with *single text line entry* system terminals.  I believe that most 
beginners never fiddle with them.

IDLE's Shell is a GUI-based python-statement-aware interface.  The python.exe 
executing user code from shell or editor is never in interactive mode.  For 
statements entered and executed, '>>>' is quite sufficient to mark code.  The 
sidebar should be minimal and unobtrusive and 3 columns is sufficient.  So I 
reject the idea of expanding it.

#37892 mentions the possibility of later adding an expanded full-line prompt 
for the code entry area, such as "Enter below a complete Python statement."  
When we get to that, we can consider an option (for non-beginners) to customize 
it.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32309] Implement asyncio.run_in_executor shortcut

[Yury Selivanov]

Yury Selivanov  added the comment:

Few thoughts:

1. I like the idea of having a context manager to create a thread pool. It 
should be initialized in a top-level coroutine and the passed down to other 
code, as in:

  async def main():
async with asyncio.ThreadPool(concurrency=10) as pool:
  await something(pool)
  await something_else(pool)

  await pool.run(callable, *args)

  asyncio.run(main())

2. It should be the "async with".

3. We should not reuse the default executor. The default executor is used for 
things like DNS resolution.  We don't want network activity to pause just 
because some task offloaded some blocking computation to its pool.

4. I think it's OK to initialize the thread pool in `ThreadPool.__init__`.  
`ThreadPool.__aenter__` would simply return `self` then.

5. `await ThreadPool.aclose()` would close the loop gracefully (awaiting for 
all submitted and still running callables) in cases when people use the 
threadpool without 'async with'.

6. I think we should aim for shipping a replacement for `loop.run_in_executor` 
in 3.9.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38500] Provide a way to get/set PyInterpreterState.frame_eval without needing to access interpreter internals

[Eric Snow]

Eric Snow  added the comment:

It depends on how you look at the degree to which you are interacting with the 
runtime.  This is a fairly low-level hook into the runtime.  So arguably if you 
are using this API then you should specify being a "core" extension.  That 
said, getting that clever about it is a bit too much.  The authors or PEP 523 
can correct me if I'm wrong, but it seems like there isn't a good reason to 
restrict access.

So basically, I agree with you. :)

How about one of the following?

* _PyInterpreterState_SetEvalFrame(_PyFrameEvalFunction eval_frame)
* _PyInterpreterState_SetFrameEval(_PyFrameEvalFunction eval_frame)

The underscore basically says "don't use this unless you know what you are 
doing".  Or perhaps that is overkill too?  "_PyFrameEvalFunction" has an 
underscore, so perhaps not.

Also, it would make sense to have a matching getter.

--
nosy: +brett.cannon, dino.viehland

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37903] IDLE Shell sidebar.

[Raymond Hettinger]

Raymond Hettinger  added the comment:

> By the way, how popular are doctests these days?

Arguably, Sphinx has made them more popular than ever.  For me, they are 
essential.  And as long as doctest remains a non-deprecated module, we have to 
retain support.

For the record, I'm opposed to backporting this to 3.8 and 3.7.  If this PR 
ends-up killing our teaching workflow, we need something to fallback to.  Also, 
you don't want to immediately invalidate every video or written tutorial that 
uses IDLE for demonstrations.

> Raymond, with the current PR, this replaces the inline prompts. 
> Making it toggle-able could be very awkward. Is this a 
> deal-breaker in your opinion?

I can't tell for sure because I'm having a hard time building with Tkinter 
support, so I can't easily try out the PR.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38430] Memory leak in ThreadPoolExecutor + run_in_executor

[Yury Selivanov]

Yury Selivanov  added the comment:

> It is a more complex solution but definitely 100% backward compatible; plus 
> the solution we can prepare people for removing the deprecated code 
> eventually.

Yeah.  Do you think it's worth it bothering with this old low-level API instead 
of making a new high-level one?  I don't, but if you do the feel free to change 
it.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32309] Implement asyncio.run_in_executor shortcut

[Kyle Stanley]

Kyle Stanley  added the comment:

> Also in this case run awaits and returns the result. Yury suggested earlier 
> just to return the future and not await.

Yeah that's roughly what my initial version was doing. I'm personally leaning a 
bit more towards returning the future rather than the result, but I'm okay with 
either option. What are your thoughts on this Yury and Andrew?

> I agree that shutdown_default_executor and _do_shutdown should be changed to 
> accept an executor argument so that any executor can be shutdown 
> asynchronously

We could potentially add an internal method _shutdown_executor, but this would 
also require modification of _do_shutdown (so that it shuts down the executor 
passed, rather than the default one). I mentioned this in an earlier example, 
but this one shows all three together and changes _shutdown_executor to a 
method of BaseEventLoop:

async def shutdown_default_executor(self):
"""Schedule the shutdown of the default executor."""
self._executor_shutdown_called = True
executor = self._default_executor
await self._shutdown_executor(executor)

async def _shutdown_executor(self, executor):
if executor is None:
return
future = self.create_future()
thread = threading.Thread(target=self._do_shutdown, 
args=(executor,future))
thread.start()
try:
await future
finally:
thread.join()

def _do_shutdown(self, executor, future):
try:
executor.shutdown(wait=True)
self.call_soon_threadsafe(future.set_result, None)
except Exception as ex:
self.call_soon_threadsafe(future.set_exception, ex)

Functionally, it works the same for shutdown_default_executor(), but allows 
_shutdown_executor to be used for asyncio.ThreadPool as well. Since GH-16360 
(adding timeout param) also makes changes to shutdown_default_executor(), it 
will be blocking this issue.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38656] mimetypes for python 3.7.5 fails to detect matroska video

[toonn]

toonn  added the comment:

The result is the same for 3.7.4, on my mac.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37903] IDLE Shell sidebar.

[Tal Einat]

Tal Einat  added the comment:

> This is marked for 3.7 through 3.9.  Do you intend to backport this PR?

IDLE generally has a different backporting policy: We generally backport all 
IDLE development to all versions of Python that aren't in security-only or 
unmaintained status, except for 2.7. Currently, that means 3.7 and above.

There is always room for exceptions, of course.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37903] IDLE Shell sidebar.

[Tal Einat]

Tal Einat  added the comment:

Stephen (Zero), thanks for the clarification.

FWIW I'm -1 on the ability to customize the prompts in the sidebar. It would 
complicate the implementation while reducing the straightforwardness and 
simplicity of IDLE.

Also, anyone who *really* wants to customize them could just change the 
hard-coded values in the code in their own copy of IDLE. I don't think it's 
worth the effort to make it more easily configurable.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37903] IDLE Shell sidebar.

[Tal Einat]

Tal Einat  added the comment:

> After the PR, can someone easily  reproduce the interactive prompt sessions 
> in the tutorial and maindocs.

Yes, except with prompts (both >>> and ...) in the sidebar rather than inline.

> Will it look the same?

No, see above.

> Would they be able to build doctest examples in the IDLE shell?

That's a good thought. It should be rather simple to add a context menu option 
to "copy for doctest" which would add the prompts to the copied text.

(By the way, how popular are doctests these days? My impression is that they've 
completely fallen out of style, but I'm sure that could vary between different 
areas and industries.)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37903] IDLE Shell sidebar.

[Raymond Hettinger]

Raymond Hettinger  added the comment:

After the PR, can someone easily  reproduce the interactive prompt sessions in 
the tutorial and maindocs.  Will it look the same?  Would they be able to build 
doctest examples in the IDLE shell?

This is marked for 3.7 through 3.9.  Do you intend to backport this PR?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38430] Memory leak in ThreadPoolExecutor + run_in_executor

[Andrew Svetlov]

Andrew Svetlov  added the comment:

I thought about returning a special subclass.
Future has too rich API: get_loop(), done(), result() etc.

What's about returning the proxy object with future instance embedded; the 
object raises DeprecationWarning for everythin except __repr__, __del__ and 
__await__, __getattr__ redirects to getattr(self._fut, name) for all other 
attributes access. 

It is a more complex solution but definitely 100% backward compatible; plus the 
solution we can prepare people for removing the deprecated code eventually.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38430] Memory leak in ThreadPoolExecutor + run_in_executor

[Yury Selivanov]

Yury Selivanov  added the comment:

> The change is slightly not backward compatible but

Yeah, that's my main problem with converting `loop.run_in_executor()` to a 
coroutine. When I attempted doing that I discovered that there's code that 
expects the method to return a Future, and so expects it have the `cancel()` 
method.

If we convert it to a coroutine a lot of code will break, which might be OK if 
it's really necessary. Is it though? Can we return a special Future subclass 
that complains if it's not awaited?  Would that fix the problem?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38657] String format for hexadecimal notation breaks padding with alternative form

[Vedran Čačić]

Vedran Čačić  added the comment:

The width doesn't mean "the number of bits", it means "the width of the field". 
In every other case too:

* when we format negative numbers, width includes the minus sign
* when we format decimal numbers, width includes decimal point (or comma)
* when we format strings with !r, width includes the quotes

So, not only would it break too much code, but it would actually be 
inconsistent to formatting all other types currently.

--
nosy: +veky

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38664] await execution order leads to throw or bad syntax

[Andrew Svetlov]

Andrew Svetlov  added the comment:

What patch are you talking about?
For me, patch means something special; e.g. working fork of CPython where the 
proposed idea is implemented.

I read await coro()[key] as 
1. Function named coro() returns a dict-like object
2. An element under key is selected from dict, which is expected to be an 
awaitable.
3. The awaitable is awaited.

Pretty clear and unambiguous.

Changing await priority depending on the presence or absence of brackets on the 
right makes Python syntax very complex if possible at all.

Sorry, the proposal has zero chance to be accepted.

--
resolution:  -> not a bug
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37527] Timestamp conversion on windows fails with timestamps close to EPOCH

[Paul Ganssle]

Paul Ganssle  added the comment:

Ah, my mistake. The examples all use `datetime.fromtimestamp`, so I didn't 
notice that it was failing only on the `timestamp` side. Re-opening, thanks!

--
resolution: duplicate -> 
status: closed -> open
superseder: [Windows] datetime.fromtimestamp(t) when 0 <= t <= 86399 fails on 
Python 3.6 -> 

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38657] String format for hexadecimal notation breaks padding with alternative form

[Mark Dickinson]

Change by Mark Dickinson :


--
nosy: +mark.dickinson

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36439] Inconsistencies with datetime.fromtimestamp(t) when t < 0

[Steve Dower]

Steve Dower  added the comment:

I've emailed some colleagues to see what they can add here.

Ultimately, the Windows CRT is just doing arithmetic, since POSIX time formats 
do not resemble Windows time formats at all, so it's all emulation. So if we 
replaced it with our own calculation I don't think that would be the worst 
possible outcome.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38665] Crash when running SQLAlchemy with pyodbc

[Dave Johansen]

New submission from Dave Johansen :

We're using SQLAlchemy 1.3.10 with pyodbc 4.0.27 in the python:3.7.5-alpine 
docker image to connect to a MySQL 13.0.5026.0 database and it's crashing with 
the following error:
python: malloc.c:2406: sysmalloc: Assertion `(old_top == initial_top (av) && 
old_size == 0) || ((unsigned long) (old_size) >= MINSIZE && prev_inuse 
(old_top) && ((unsigned long) old_end & (pagesize - 1)) == 0)' failed.

--
messages: 355830
nosy: Dave Johansen
priority: normal
severity: normal
status: open
title: Crash when running SQLAlchemy with pyodbc
type: crash
versions: Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38664] await execution order leads to throw or bad syntax

[rmlibre]

New submission from rmlibre :

If a coroutine returns a sequence, a user cannot write this clean code:

>>> await coro()[index]
Or,
>>> await coro()[slice]
Or,
>>> await coro()[key]

This raises a TypeError("'coroutine' object is not subscriptable"). To
solve this on the user side, one must add parentheses, leading to a 
convention of ugly syntax:

>>> (await coro())[...]

Or, hiding the parentesis away in a function

>>> async def index_coroutine(coro=None, index=None):
>>> return (await coro)[index]
>>> await index_coroutine(coro(), slice(1, 5))

This is gross. It's a bug since it unnecessarily requires unpythonic 
code and throws pythonic code. My suggested patch would be to move the 
evaluation of the await statement higher up on the execution order so 
the former snippets would be valid python syntax. The await statement 
should imply parentheses when square brackets are used, since there is 
no other use case for indexing or subscripting a coroutine. This will
lead to more beautiful code.

--
components: asyncio
messages: 355829
nosy: asvetlov, rmlibre, yselivanov
priority: normal
severity: normal
status: open
title: await execution order leads to throw or bad syntax
type: crash
versions: Python 3.5, Python 3.6, Python 3.7, Python 3.8, Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38662] Decouple ensurepip from pip's internals using runpy

[Pradyun Gedam]

Pradyun Gedam  added the comment:

Note that the patch uses the fact that:


```
def foo():
try:
raise Exception("returned")
except Exception as e:
print("except")
return e.args[0]
finally:
print("finally")


print(foo())
```

will print:

```
except
finally
returned
```

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38663] Untokenize does not round-trip ws before bs-nl

[Edward K Ream]

New submission from Edward K Ream :

Tested on 3.6.

tokenize.untokenize does not round-trip whitespace before backslash-newlines 
outside of strings:

from io import BytesIO
import tokenize

# Round tripping fails on the second string.
table = (
r'''
print\
("abc")
''',
r'''
print \
("abc")
''',
)
for s in table:
tokens = list(tokenize.tokenize(
BytesIO(s.encode('utf-8')).readline))
result = g.toUnicode(tokenize.untokenize(tokens))
print(result==s)

I have an important use case that would benefit from a proper untokenize. After 
considerable study, I have not found a proper fix for tokenize.add_whitespace.

I would be happy to work with anyone to rewrite tokenize.untokenize so that 
unit tests pass without fudges in TestRoundtrip.check_roundtrip.

--
messages: 355827
nosy: edreamleo
priority: normal
severity: normal
status: open
title: Untokenize does not round-trip ws before bs-nl
type: behavior
versions: Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38662] Decouple ensurepip from pip's internals using runpy

[Pradyun Gedam]

Change by Pradyun Gedam :


--
keywords: +patch
pull_requests: +16543
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/17029

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38662] Decouple ensurepip from pip's internals using runpy

[Pradyun Gedam]

New submission from Pradyun Gedam :

Inspired by https://github.com/python/cpython/pull/16782#discussion_r335285656

This changes ensurepip's current assumption that pip would expose a "main" 
function, which can be invoked to run pip.

I've selected all actively supported Python versions for this ticket, since 
this patch should be backported to all Python versions, to make it robust to 
internal changes within pip.

--
messages: 355826
nosy: Marcus.Smith, dstufft, ncoghlan, paul.moore, pradyunsg
priority: normal
severity: normal
status: open
title: Decouple ensurepip from pip's internals using runpy
versions: Python 2.7, Python 3.5, Python 3.6, Python 3.7, Python 3.8, Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38657] String format for hexadecimal notation breaks padding with alternative form

[Eric V. Smith]

Eric V. Smith  added the comment:

Now that I re-read this, maybe it was a documentation request, not a functional 
change? I'd be okay with documenting the existing behavior, so I'll re-open 
this and change the type. Patches welcome.

--
components: +Documentation
resolution: rejected -> 
stage: resolved -> needs patch
status: closed -> open
versions: +Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37527] Timestamp conversion on windows fails with timestamps close to EPOCH

[Ma Lin]

Ma Lin  added the comment:

issue29097 fixed bug in `datetime.fromtimestamp()`.
But this issue is about `datetime.timestamp()`, not fixed yet.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38159] PyState_AddModule docs should say that it's not necessary to call it.

[miss-islington]

Change by miss-islington :


--
pull_requests: +16542
pull_request: https://github.com/python/cpython/pull/17027

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38159] PyState_AddModule docs should say that it's not necessary to call it.

[miss-islington]

Change by miss-islington :


--
pull_requests: +16541
pull_request: https://github.com/python/cpython/pull/17026

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38159] PyState_AddModule docs should say that it's not necessary to call it.

[Petr Viktorin]

Petr Viktorin  added the comment:


New changeset 9bc94eca0c69a551f928692364a99e9b67c4a45b by Petr Viktorin in 
branch 'master':
bpo-38159: Clarify documentation of PyState_AddModule (GH-16101)
https://github.com/python/cpython/commit/9bc94eca0c69a551f928692364a99e9b67c4a45b


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38661] Changes to tkinter result in (unexpected) widget map call return changes wrt. 3.7

[Serhiy Storchaka]

Change by Serhiy Storchaka :


--
nosy: +serhiy.storchaka

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36439] Inconsistencies with datetime.fromtimestamp(t) when t < 0

[Paul Ganssle]

Paul Ganssle  added the comment:

This has been coming up in a few different contexts lately, so I think it would 
be really good if we could get some sort of fix for it.

One option is to implement our own versions of these APIs for use in Windows, 
but a thought occurred to me recently: we have not investigated the possibility 
of seeing if Microsoft would be willing to either add support for negative 
timestamps in their localtime() or gmtime() implementations or add a new API 
that *does* support negative timestamps. It would also be good to rule out the 
possibility that such APIs already exist but we just don't know about them 
(preliminary googling doesn't show anything, though possibly something can be 
done with the Win32 APIs? Not sure how or if those work in C and how big a lift 
it would be to maintain compatibility if can switch: 
https://docs.microsoft.com/en-us/windows/win32/sysinfo/time-functions?redirectedfrom=MSDN
 ).

Adding Steve Dower to the nosy list in case he can shed some light onto the 
possibility of native support.

--
nosy: +steve.dower

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38631] Replace Py_FatalError() with regular Python exceptions

[Eric Snow]

Eric Snow  added the comment:

FWIW, I agree. :)

--
nosy: +eric.snow

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38233] datetime.datetime.fromtimestamp have different behaviour on windows and mac

[Paul Ganssle]

Paul Ganssle  added the comment:

Changing the superceder here as I think #36439 matches better than #37527.

--
nosy: +p-ganssle
resolution: duplicate -> 
status: closed -> open
superseder: Timestamp conversion on windows fails with timestamps close to 
EPOCH -> Inconsistencies with datetime.fromtimestamp(t) when t < 0

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37527] Timestamp conversion on windows fails with timestamps close to EPOCH

[Paul Ganssle]

Paul Ganssle  added the comment:

This indeed seems to be a duplicate of 29097, which is fixed in Python 3.7, so 
we can close this bug. Thank you for your report Dschoni, and thank you for 
finding the duplicate Ma Lin!

--
resolution:  -> duplicate
stage:  -> resolved
status: open -> closed
superseder:  -> [Windows] datetime.fromtimestamp(t) when 0 <= t <= 86399 fails 
on Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38661] Changes to tkinter result in (unexpected) widget map call return changes wrt. 3.7

[Jose Salvatierra]

New submission from Jose Salvatierra :

Hello!

I've encountered what might be a bug.

Up till now we had some working code that did this:

```
maps = self.style.map('TCombobox')
if maps:
self.style.map('DateEntry', **maps)
```

Modifying a custom style to mimic the map of another. This has worked fine 
until Python 3.7, because the return value of `.map` is something that you can 
pass to `.map` as kw and it'll process it fine.

The return value of `.map` in Python 3.7 is something like this, for the 
`TCombobox`:

```
: {'focusfill': [('readonly', 'focus', 'SystemHighlight')], 
'foreground': [('disabled', 'SystemGrayText'), ('readonly', 'focus', 
'SystemHighlightText')], 'selectforeground': [('!focus', 'SystemWindowText')], 
'selectbackground': [('!focus', 'SystemWindow')]}
```

Which is as you'd expect (and the docs say): a dictionary of properties to 
lists, where each list can contain multiple tuples describing the required 
state and final value of the property.

However in Python 3.8, the value returned by `.map` is this:

```
: {'focusfill': ['readonly focus', 'SystemHighlight'], 
'foreground': ['disabled', 'SystemGrayText', 'readonly focus', 
'SystemHighlightText'], 'selectforeground': ['!focus', 'SystemWindowText'], 
'selectbackground': ['!focus', 'SystemWindow']}
```

The tuples are missing. This then causes a number of problems downstream, such 
as the final property values being split into the constituent letters instead 
of the values within each tuple.

--
components: Tkinter, Windows
messages: 355818
nosy: jslvtr, paul.moore, steve.dower, tim.golden, zach.ware
priority: normal
severity: normal
status: open
title: Changes to tkinter result in (unexpected) widget map call return changes 
wrt. 3.7
type: behavior
versions: Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38654] `urllib.request.Request` uses mutable value as default value

[Steven D'Aprano]

Steven D'Aprano  added the comment:

> Also, if some new Python coders saw `[]` or `{}` being used as default 
> values in the standard library, they might think “I’ll do it too since 
> the standard library does it”.

Great! Having Python coders learn good progamming skills from the 
standard library is a *good* thing.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24159] Misleading TypeError when pickling bytes to a file opened as text

[Борис Верховский]

Борис Верховский  added the comment:

As I said in issue38226, the error message you get when you try to 
pickle.load() a file opened in "r" mode instead of "rb" mode,

UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 0: invalid 
start byte

is also confusing and can be improved.

--
nosy: +boris

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38226] pickle.dump and load error message when file isn't opened in binary mode are not useful

[Борис Верховский]

Change by Борис Верховский :


--
resolution:  -> duplicate
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38659] enum classes cause slow startup time

[Ethan Furman]

Ethan Furman  added the comment:

I was just looking at this problem, and creating a bare-bones, no safety belts 
version for use in the stdlib (no patch yet) which decreases Enum creation from 
14x slower to only 6x slower.  (Comparing to a class with simple attributes.)

Not sure if that's enough improvement, though.

If it needs to be even faster, a C version of that simplified Enum shouldn't be 
too hard.  Anyone that uses the _simple_enum, though, should have a test that 
uses the full Enum and compares the two to make sure nothing got lost in 
translation.

--
assignee:  -> ethan.furman

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38657] String format for hexadecimal notation breaks padding with alternative form

[Eric V. Smith]

Eric V. Smith  added the comment:

int.__format__ inherits this from %-formatting, which inherits it from C's 
printf.

There's no way we're going to change this at this point: the breakage would be 
too great. So, I'm going to reject this.

--
assignee:  -> eric.smith
nosy: +eric.smith
resolution:  -> rejected
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38226] pickle.dump and load error message when file isn't opened in binary mode are not useful

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

This looks like a duplicate of issue24159.

--
nosy: +serhiy.storchaka

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37903] IDLE Shell sidebar.

[Stephen Paul Chappell]

Stephen Paul Chappell  added the comment:

Zero: "not to have them added as text as is usual in a terminal window"
taleinat: "removing prompts from the shell window's text widget"

Zero: "print the values of ps1 and ps2 in the proposed ShellIO subclass"
taleinat: "separate sidebar showing where prompts and user input were"

We appear to be in agreement.

terry.reedy: "Labels, such as '>>>', 'Out', 'Inp', and 'Err' would be used"
Zero:"Having IDLE react to sys.ps1 and sys.ps2"

My suggestion is to take those labels terry.reedy talks about from the values 
of ps1 and ps2 since they are already documented and standard for "the 
interpreter ... in interactive mode." If psX needs to introduced for other 
prompts that may be needed ("Maybe use 'Con', maybe use Dbg and Wrn."), it 
would provide a sensible way to customize those labels as well.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32309] Implement asyncio.run_in_executor shortcut

[Paul Martin]

Paul Martin  added the comment:

Good points. I made a mistake in run

Should be:

async def run(self, func, *args, **kwargs):
call = functools.partial(func, *args, **kwargs)
return await self._loop.run_in_executor(self._executor, call)

Also in this case run awaits and returns the result. Yury suggested earlier 
just to return the future and not await. I have no strong opinion either way. 
The above example does seem more higher level but Yury's example is more 
flexible.

I agree that shutdown_default_executor and _do_shutdown should be changed to 
accept an executor argument so that any executor can be shutdown 
asynchronously. So the loop API would have a shutdown_executor method. 
shutdown_default_executor would just call shutdown_executor with the default 
executor as argument. That would be a good first step.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30385] Segfault on OSX with 3.6.1

[ThePokestarFan]

ThePokestarFan  added the comment:

A pythonic way to do it is `os.environ[“no_proxy”]=“*”`

--
nosy: +ThePokestarFan

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32309] Implement asyncio.run_in_executor shortcut

[Kyle Stanley]

Kyle Stanley  added the comment:

Actually, I think it would be better to move the functionality of 
loop.shutdown_default_executor() to a new private method 
loop._shutdown_executor() that takes an executor argument rather than shutting 
down the default one. This could be used in both 
loop.shutdown_default_executor() and ThreadPool. There's no need to move it to 
function instead of being a method of BaseEventLoop though, that doesn't make 
sense now that I think about it more. 

Sorry if my thoughts are a bit disorganized, I think I need to get some sleep. 
(:

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13829] exception error in _scproxy.so when called after fork

[ThePokestarFan]

ThePokestarFan  added the comment:

Still present in python 3.8 and issue 38658. Another workaround is in 
https://bugs.python.org/issue30385#msg293958

--
nosy: +ThePokestarFan
versions: +Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32309] Implement asyncio.run_in_executor shortcut

[Kyle Stanley]

Kyle Stanley  added the comment:

> thread = threading.Thread(target=loop._do_shutdown, args=(executor,future))

Correction:

> thread = threading.Thread(target=_do_shutdown, args=(loop, executor,future))

Also, it might make more sense to rename _do_shutdown() to 
_do_executor_shutdown() to give the function's name more context; renaming 
shouldn't be an issue since it's private. Plus, it was just added recently in 
3.9, so there's even less backwards compatibility to be concerned with.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38658] Python Program crashes when running in fore and back ground

[ThePokestarFan]

ThePokestarFan  added the comment:

Thank you. Closing this issue.

--
resolution:  -> duplicate
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32309] Implement asyncio.run_in_executor shortcut

[Kyle Stanley]

Kyle Stanley  added the comment:

Also, I agree with Paul's idea of initializing the ThreadPoolExecutor in the 
__init__ instead of __aenter__, that makes more sense now that I think about it.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32309] Implement asyncio.run_in_executor shortcut

[Kyle Stanley]

Kyle Stanley  added the comment:

> I don't think changing the default executor is a good approach. What happens, 
> if two or more thread pools are running at the same time? In that case they 
> will use the same default executor anyway, so creating a new executor each 
> time seems like a waste. 

I agree that it would be better to have ThreadPool use an internal executor 
rather than relying on the event loop's default executor. The main reason I 
hadn't was because we hadn't implemented an asynchronous executor shutdown 
outside of loop.shutdown_default_executor(), but we could potentially move the 
functionality to a private function (in Lib/asyncio/base_events.py) so it's 
reusable for ThreadPool. It could be something like this:

async def _shutdown_executor(executor, loop):
future = loop.create_future()
thread = threading.Thread(target=loop._do_shutdown, 
args=(executor,future))
thread.start()
try:
await future
finally:
thread.join()

def _do_shutdown(loop, executor, future):
try:
executor.shutdown(wait=True)
loop.call_soon_threadsafe(future.set_result, None)
except Exception as ex:
loop.call_soon_threadsafe(future.set_exception, ex)

Note that the above would be for internal use only, for the existing 
loop.shutdown_default_executor() and the new asyncio.ThreadPool. For it to 
support both, it would have to accept an explicit loop argument. It also does 
not need a robust API, since it's private. 

> Shutting down the default executor seems unnecessary and could impact lower 
> level code which is using it. The default executor is shutdown at the end of 
> asyncio.run anyway.

I agree with your point regarding the shutdown of the default executor one. But 
I think we should shutdown the internal executor for the ThreadPool, as a main 
point context managers is to start and clean up their own resources. Also, I'm 
aware that asyncio.run() shuts down the default executor, I implemented that 
fairly recently in GH-15735. ;)

Another substantial concern is in the case of a coroutine that contains 
asyncio.ThreadPool being executed without asyncio.run(). There are still use 
cases for using the lower level loop.run_until_complete() for more complex 
asyncio programs. I don't think we should make asyncio.ThreadPool dependent on 
the coroutine being executed with asyncio.run(). Thus, it makes sense that 
ThreadPool should create a new instance of ThreadPoolExecutor upon entry of the 
context manager and then shut it down upon exit. 

> I'm not sure if a new ThreadPoolExecutor really needs to be created for each 
> ThreadPool though.

IMO, a context manager should create and then finalize it's own resources, 
rather than sharing the same executor across contexts. Sharing the same one 
seems to defeat the purpose of using a context manager in the first place, no?

> Run method should be:

async def run(self, func, *args, **kwargs):
call = functools.partial(func, *args, **kwargs)
return await self._loop.run_in_executor(None, call)

Correction: if we're using an internal executor now, this should instead be 
self._loop.run_in_executor(self._executor, call). With `None`, it will simply 
use the event loop's default executor rather the context manager's 
ThreadPoolExecutor.

> `run` should be awaitable method, see #38430

Agreed, good point. 

> Paul's version looks better.

I think he had some good points, particularly around using an internal executor 
instead the event loop's default executor; but there's some parts that I 
disagree with, see above reasons.

> 1. get_running_loop() should be used instead of get_event_loop()

Note: If get_running_loop() is used instead, it has to set self._loop within a 
coroutine (since get_running_loop() can only be used within coroutines), that's 
why in my version it was within __aenter__. I think this would make the most 
sense. 

> 2. There is no `await executer.shutdown()` API, the method is synchronous.

That's why in my version I was using the existing event loop API, since we had 
already implemented an asynchronous loop.shutdown_default_executor() method 
that calls executor.shutdown(). However, if we added a private 
_shutdown_executor() and _do_shutdown() as I mentioned above, that wouldn't be 
an issue. 

Thanks for the feedback on the prototype Paul and Andrew, both of you brought 
up some good points. I'll start working on a PR.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37903] IDLE Shell sidebar.

[Tal Einat]

Tal Einat  added the comment:

Stephen, perhaps what you're suggesting is out of context for this issue? This 
issue is about removing prompts from the shell window's text widget, replacing 
them with a separate sidebar showing where prompts and user input were. This 
will make IDLE's shell less similar to the basic command line shell, not more 
similar.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32309] Implement asyncio.run_in_executor shortcut

[Andrew Svetlov]

Andrew Svetlov  added the comment:

Paul's version looks better.

Two notes: 
1. get_running_loop() should be used instead of get_event_loop()
2. There is no `await executer.shutdown()` API, the method is synchronous.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9495] argparse unittest tracebacks are confusing if an error is raised when not expected

[alclarks]

alclarks  added the comment:

I'm a newcomer looking to contribute - would it be a good idea for me to update 
one of the attached diffs and raise a pull request?

--
nosy: +alclarks

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32309] Implement asyncio.run_in_executor shortcut

[Paul Martin]

Paul Martin  added the comment:

Run method should be:

async def run(self, func, *args, **kwargs):
call = functools.partial(func, *args, **kwargs)
return await self._loop.run_in_executor(None, call)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32309] Implement asyncio.run_in_executor shortcut

[Paul Martin]

Paul Martin  added the comment:

I don't think changing the default executor is a good approach. What happens, 
if two or more thread pools are running at the same time? In that case they 
will use the same default executor anyway, so creating a new executor each time 
seems like a waste. 

Shutting down the default executor seems unnecessary and could impact lower 
level code which is using it. The default executor is shutdown at the end of 
asyncio.run anyway.

I also think it would be good to have a synchronous entry point, and not 
require a context manager. Having a ThreadPool per class instance would be a 
common pattern.


class ThreadPool:
def __init__(self, timeout=None):
self.timeout = timeout
self._loop = asyncio.get_event_loop()
self._executor = concurrent.futures.ThreadPoolExecutor()

async def close(self): 
await self._executor.shutdown(timeout=self.timeout)  

async def __aenter__(self):
return self

async def __aexit__(self, *args):
await self.close()

def run(self, func, *args, **kwargs):
call = functools.partial(func, *args, **kwargs)
return self._loop.run_in_executor(self._executor, call)


I'm not sure if a new ThreadPoolExecutor really needs to be created for each 
ThreadPool though.

--
nosy: +primal

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32309] Implement asyncio.run_in_executor shortcut

[Andrew Svetlov]

Andrew Svetlov  added the comment:

`run` should be awaitable method, see #38430

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38660] Checking if two regexes are equal should test if they are functionally equivalent

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

Python's regular expressions are not actually regular. Lookarounds and groups 
make things more complex. Even it it is possible to build an ambiguous graph, 
its size and the time of building can be too large for practical use. Dealing 
with unicode is only minor part of the problem.

I suggest you first to implement this feature as a third-party module on PyPI. 
After this we can consider including it in the stdlib.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32309] Implement asyncio.run_in_executor shortcut

[Kyle Stanley]

Kyle Stanley  added the comment:

So, here's a prototype implementation of asyncio.ThreadPool that would function 
exactly as Yury described, but I'm not convinced about the design. Usually, it 
seems preferred to separate the context manager from the rest of the class (as 
was done with asyncio.Lock), but I think this one might be simple enough to be 
a single class:

class ThreadPool:
def __init__(self, timeout=None):
self.timeout = timeout
self._loop = None

async def __aenter__(self):
self._loop = asyncio.get_running_loop()
# Ensure that ThreadPoolExecutor is being used
self._loop.default_executor = concurrent.futures.ThreadPoolExecutor()
return self

async def __aexit__(self, *args):
await self._loop.shutdown_default_executor(timeout=self.timeout)

def run(self, func, *args, **kwargs):
call = functools.partial(func, *args, **kwargs)
return self._loop.run_in_executor(None, call)

It utilizes the existing lower level event loop API to provide an asynchronous 
context manager. I'd say the main advantage is that it's significantly more 
user friendly, as there's no loop or executor argument to be specified, and 
users can freely pass kwargs to run() thanks to functools.partial().

Additionally, I also included a timeout parameter for shutting down the 
ThreadPoolExecutor, which will depend upon GH-16360. This can be used as such:

async def main():
async with asyncio.ThreadPool(timeout=600) as pool:
fut1 = pool.run(func)
fut2 = pool.run(func, arg1, arg2)
fut2 = pool.run(func, arg1, kwarg1=True)
print(await asyncio.gather(fut1, fut2, fut3))

asyncio.run(main())

I don't expect that this would be the final design, but I think it's a decent 
prototype to demonstrate the functionality. Thoughts?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38226] pickle.dump and load error message when file isn't opened in binary mode are not useful

[Борис Верховский]

Change by Борис Верховский :


--
type:  -> enhancement

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38660] Checking if two regexes are equal should test if they are functionally equivalent

[Борис Верховский]

Борис Верховский  added the comment:

I saw two Python regexes, one derived from a regex in the standard library. 
There was a comment saying that they're interchangeable and I wanted to check 
if they were actually the same without having to read what all the regex 
symbols mean. Plus a true comparison would be more correct.

Besides that I don't have a real use case. There's a few people asking about 
doing this if you Google for "check if two regexes are equivalent". Some for 
Python specifically.

@serhiy I read what \w meant from the first Google result and got it wrong. 
Sounds like a good argument for why Python should be able to do this for me :)

Regexes are not Turing complete, so not quite. If I understand it correctly, 
comparing them is somewhere between comparing two graphs and comparing two 
programs (which is generally impossible). Theoretically it's a decidable 
problem, but the extra logic that Python's implementation has (for dealing with 
unicode and whatever) makes it hard, but it should still be theoretically 
possible, unless Python's regexes are somehow Turing complete.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38657] String format for hexadecimal notation breaks padding with alternative form

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

Yes, the width is the width of the formatted value, not the number of digits.

What is your proposition?

--
nosy: +serhiy.storchaka

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38660] Checking if two regexes are equal should test if they are functionally equivalent

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

These two regexes are not the same.

>>> re.compile('([-_.a-zA-Z0-9]+)').match('ä')
>>> re.compile(r'([-\w.]+)').match('ä')


As Ammar said checking that two regexes always matches the same is very 
difficult problem. It is the problem of determining if two programs are the 
same.

--
nosy: +serhiy.storchaka

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38660] Checking if two regexes are equal should test if they are functionally equivalent

[Ammar Askar]

Ammar Askar  added the comment:

The notion of equivalent regular expressions does exist but is way more 
complicated than the simple example you described.

For example:

r"a|b" is the same as r"[ab]",
r"^aa*$" is the same as r"^a+$"

Implementing this properly would probably require a significant amount of 
effort, and just implementing simple equivalence for character classes would be 
really surprising.

Could you explain the use case and motivation behind this request?

--
nosy: +ammar2

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33187] Document ElementInclude (XInclude) support in ElementTree

[Stefan Behnel]

Change by Stefan Behnel :


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38660] Checking if two regexes are equal should test if they are functionally equivalent

[Борис Верховский]

New submission from Борис Верховский :

re.compile('([-_.a-zA-Z0-9]+)') == re.compile(r'([-\w.]+)') 

should return True because those are the same regex (\w is a-z, A-Z, 0-9 and 
the underscore). 

If you want to check if two regexes are identical you would compare the 
original strings, before re.compile.

--
components: Regular Expressions
messages: 355791
nosy: boris, ezio.melotti, mrabarnett
priority: normal
severity: normal
status: open
title: Checking if two regexes are equal should test if they are functionally 
equivalent
type: enhancement
versions: Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com