[Python-ideas] Re: [Python-Dev] Re: Re: Amend PEP-8 to require clear, understandable comments instead of Strunk & White Standard English comments

[Giampaolo Rodola']

On Mon, Jun 29, 2020 at 12:34 PM Nathaniel Smith  wrote:

> On Mon, Jun 29, 2020 at 2:31 AM Steve Holden  wrote:
> > The commit message used, however, reveals implementation details of the
> change which are irrelevant to the stated aim, which is making the
> documentation clear and concise. Use of such language is certainly
> regrettable, since it carries with it the implication that the Python
> developer community has somehow been wilfully sanctioning "relics of white
> supremacy" up until the change was made.
> >
> > There certainly is a place in tech for politics, as I have argued many
> times, and I am sure nobody wishes to continue to use language that might
> be offensive to readers. But I would suggest that the politics can safely
> be omitted from commit messages, since they can only properly be fully
> addressed in the conversation about the PR in advance. The wording of the
> commit message has the appearance (probably specious) of wanting to rub
> former misdeeds in the face of a largely innocent community, and that is
> the principal reason I found it distasteful and unnecessary.
>
> I just re-read the commit message, and I think you're being
> oversensitive and imagining things that aren't there. The actual
> commit message is written in a straightforward and factual way, and
> spends special effort on *absolving* the community of this kind of
> guilt.
>

"The community" has nothing to be absolved of, "Strunk & White" has nothing
to do with white supremacy and there is no guilt. If you feel guilty
because you're white then that's your problem. I don't feel guilty for
being white, the same way a black person should not feel guilty for being
black. And I have literally ZERO excuses to make to you or anybody else in
here because I'm white. Assuming guilt based on the color of your skin and
constantly attacking that specific group because of that is racist. It's
that simple. I find it astonishing how some people here don't seem to
realize that (or pretend not to).

And what's the goal anyway? Make us all feel guilty, create yet another
heated discussion, widen divisions, wait for the occasional folks who dare
to speak up against this vitriol and kick them out? And then what? What is
the plan here exactly? Don't you folks realize this is a technical forum?
Don't you understand how inappropriate it is to constantly bring up these
kinds of messages up here, and force people to either witness them silently
for fear of repercussions, or to engage in the discussion and risk paying
the consequences in terms of work / hiring / career / status / reputation
etc.? Because that's what happens, and we all know it. This is a very
public forum and we can all be traced back to here. There are professionals
here, people who go to conferences and/or make a living out of Python, who
pay the rent and support their family with it, and that don't want to be
put in this position.

It does not scale. It will never scale. Because whether we like it or not
we have to coexist together in this virtual space, including with people we
don't like. And this is why it is such a good idea to leave politics out of
the door and only stay focused on Python. We will still have different
opinions and occasional clashes, but as long as they are technical they
will be genuine, prolific and everything will be fine as it was before
"this" started (I've been reading this list for 12 years now). Discussing
politics, on the other hand, will only keep bringing conflict over and over
again. There's tons of proof of this already, and I can't envision a
different outcome in the long run. Because most of us are not OK with being
put against a wall and being blamed for "supremacy", "guilt", "privilege"
or whatever term you have in your jargon. I certainly am not. Furthermore,
that jargon makes no sense outside of the US and it's just ridiculous. I'm
European, am split between living here and in Asia, and I can guarantee you
that much.

Please, stop this.

-
Giampaolo - gmpy.dev 
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/TMLKHLDLDFE3JC3M6HOCUW4RIPQEXEIP/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Amend PEP-8 to require clear, understandable comments instead of Strunk & White Standard English comments

[Giampaolo Rodola']

From:
https://github.com/python/peps/commit/0c6427dcec1e98ca0bd46a876a7219ee4a9347f4

> Instead of requiring that comments be written in Strunk & White Standard
English, require instead that English-language comments be clear and easily
understandable by other English speakers. This accomplishes the same goal
without upholding relics of white supremacy. Many native English speakers
do not use Standard English as their native dialect, so requiring
conformation to Standard English centers whiteness in an inappropriate and
unnecessary way, and can alienate and put up barriers for people of color
and those whose native dialect of English is not Standard English. This
change is a simple way to correct that while maintaining the original
intent of the requirement.

This has nothing to do with making the wording "clear and understandable"
(I agree on that). It's about, once again, bringing race-based politics
into Python, and spreading hate towards a specific group of people: whites.
Whether you're aware of it or not, there is a term for this: it's racism.
I want to remind everyone that most of us here simply want to contribute
code. We do it for free, and don't want to be involved in "this", because
frankly it's disgusting. Doing something out of passion and for free, and
at the same time seeing these sorts of things happening on a regular basis,
looks and feels like an insult, and will only lead to people leaving this
place.

On Fri, Jun 26, 2020 at 11:27 PM Keara Berlin  wrote:

> Hi all, this is a very small change, but I thought I would field it here
> to see if anyone has suggestions or ideas. Instead of requiring that
> comments be written in Strunk & White Standard English, PEP-8 should
> require instead that English-language comments be clear and easily
> understandable by other English speakers. This accomplishes the same goal
> without alienating or putting up barriers for people (especially people of
> color) whose native dialect of English is not Standard English. This change
> is a simple way to correct that while maintaining the original intent of
> the requirement. This change may even make the requirement more clear to
> people who are not familiar with Strunk & White, since for programmers, the
> main relevant aspect of that standard is "be clear and concise;" simply
> saying that instead of referencing Strunk & White may communicate this more
> effectively.
> Here is the current line in PEP-8: "When writing English, follow Strunk
> and White."
> I propose changing this line to "When writing English, ensure that your
> comments are clear and easily understandable to other English speakers."
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-ideas@python.org/message/AE2M7KOIQR37K3XSQW7FSV5KO4LMYHWX/
> Code of Conduct: http://python.org/psf/codeofconduct/
>


-- 
Giampaolo - gmpy.dev 
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/J2LC553NWBTANBRIPI52CQW53JXCCUTH/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Provide additional debug info for OSError and WindowsError

[Giampaolo Rodola']

When dealing with C extensions from Python there are circumstances
where a function is called and we get an OSError(errno) exception
without knowing what exactly went wrong internally. This is especially
not obvious on Windows, where multiple MSDN APIs may be invoked within
the same C function and we're not sure which one of them failed.

There are other times where the underlying C syscall is obvious, but
it would still be useful to append some additional information. One
example is socket.bind():
https://github.com/python/cpython/blob/56065d4c8ac03042cb7e29ffda9b1ac544a37b4d/Lib/asyncio/base_events.py#L940-L949

In order to work around that in psutil (on Windows) I stored the debug
msg string in OSError.filename attribute:
https://github.com/giampaolo/psutil/pull/1428/
As such I was thinking that perhaps it would be nice to provide 2 new
cPython APIs:

PyErr_SetFromErrnoWithMsg(PyObject *type, const char *msg)
PyErr_SetFromWindowsErrWithMsg(int ierr, const char *msg)
PyErr_SetExcFromWindowsErrWithMsg(PyObject *type, int ierr, const char *msg)

With this in place also OSError and WindowsError would probably have
to host a new "extramsg" attribute or something (but not necessarily).

Thoughts?

-- 
Giampaolo - http://grodola.blogspot.com
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Improve os.times() resolution

[Giampaolo Rodola']

On Sun, Mar 24, 2019 at 2:29 PM Anders Hovmöller  wrote:
>
> Have you checked how much overhead the two functions have? That seems like an 
> obvious way this proposal could go south.

Without patch:

$ ./python -m timeit -s "import os" "os.times()"
50 loops, best of 5: 546 nsec per loop

With patch:

$ ./python -m timeit -s "import os" "os.times()"
20 loops, best of 5: 1.23 usec per loop

The patch:

diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c
index 3f76018357..ad91ed702a 100644
--- a/Modules/posixmodule.c
+++ b/Modules/posixmodule.c
@@ -8035,6 +8035,14 @@ os_times_impl(PyObject *module)
 #else /* MS_WINDOWS */
 {

+#if defined(HAVE_SYS_RESOURCE_H)
+struct rusage ruself;
+struct rusage ruchildren;
+if (getrusage(RUSAGE_SELF, ) == -1)
+return posix_error();
+if (getrusage(RUSAGE_CHILDREN, ) == -1)
+return posix_error();
+#endif

 struct tms t;
 clock_t c;
@@ -8043,10 +8051,18 @@ os_times_impl(PyObject *module)
 if (c == (clock_t) -1)
 return posix_error();
 return build_times_result(
+
+#if defined(HAVE_SYS_RESOURCE_H)
+ doubletime(ruself.ru_utime),
+ doubletime(ruself.ru_stime),
+ doubletime(ruchildren.ru_utime),
+ doubletime(ruchildren.ru_stime),
+#else
  (double)t.tms_utime / ticks_per_second,
  (double)t.tms_stime / ticks_per_second,
  (double)t.tms_cutime / ticks_per_second,
  (double)t.tms_cstime / ticks_per_second,
+#endif
  (double)c / ticks_per_second);
 }
 #endif /* MS_WINDOWS */

-- 
Giampaolo - http://grodola.blogspot.com
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Improve os.times() resolution

[Giampaolo Rodola']

It turns out we could use resource.getrusage() which provides micro
seconds (tested on Linux and macOS):

import os, resource
for x in range(1000):  # warm up
pass
for x in range(5):
a = os.times()
b = resource.getrusage(resource.RUSAGE_SELF)
print(a.user, a.system)
print(b.ru_utime, b.ru_stime)

...it prints:

0.39 0.01
0.394841 0.011963
0.39 0.01
0.394899 0.011966
0.39 0.01
0.394908 0.011966
0.39 0.01
0.394936 0.011967
0.39 0.01
0.394963 0.011968

getrusage(RUSAGE_CHILDREN) can be used to calculate "children_user"
and "children_system". I see 2 possibilities here:

1) doc fix, mentioning that resource.getrusage provides a better resolution
2) if available (it should always be as it's a POSIX standard), just
use getrusage in Modules/posixmodule.c. It seems we can check
availability by reusing HAVE_SYS_RESOURCE_H and HAVE_SYS_TIME_H
definitions which are already in place.

I'm not sure what's best to do as os.* functions usually expose the
original C function with the same name, but given that "elapsed" field
is not part of times(2) struct and that on Windows "elapsed",
"children_user" and "children_system" are set to 0 it appears there
may be some space for flexibility here.

Thoughts?

-- 
Giampaolo - http://grodola.blogspot.com
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Add subprocess.Popen suspend() and resume()

[Giampaolo Rodola']

On Wed, Mar 20, 2019 at 11:19 PM eryk sun  wrote:
>
> On 3/18/19, Giampaolo Rodola'  wrote:
> >
> > I've been having these 2 implemented in psutil for a long time. On
> > POSIX these are convenience functions using os.kill() + SIGSTOP /
> > SIGCONT (the same as CTRL+Z  / "fg"). On Windows they use
> > undocumented NtSuspendProcess and NtResumeProcess Windows
> > APIs available since XP.
>
> Currently, Windows Python only calls documented C runtime-library and
> Windows API functions. It doesn't directly call NT runtime-library and
> system functions. Maybe it could in the case of documented functions,
> but calling undocumented functions in the standard library should be
> avoided. Unfortunately, without NtSuspendProcess and NtResumeProcess,
> I don't see a way to reliably implement this feature for Windows. I'm
> CC'ing Steve Dower. He might say it's okay in this case, or know of
> another approach.
>
> DebugActiveProcess, the other simple approach mentioned in the linked
> SO answer [1], is unreliable and has the wrong semantics.  A process
> only has a single debug port, so DebugActiveProcess will fail the PID
> as an invalid parameter if another debugger is already attached to the
> process. (The underlying NT call, DbgUiDebugActiveProcess, fails with
> STATUS_PORT_ALREADY_SET.) Additionally, the semantics that I expect
> here, at least for Windows, is that each call to suspend() will
> require a corresponding call to resume(), since it's incrementing the
> suspend count on the threads; however, a debugger can't reattach to
> the same process. Also, if the Python process exits while it's
> attached as a debugger, the system will terminate the debugee as well,
> unless we call DebugSetProcessKillOnExit(0), but that interferes with
> the Python process acting as a debugger normally, as does this entire
> wonky idea. Also, the debugging system creates a thread in the debugee
> that calls NT DbgUiRemoteBreakin, which executes a breakpoint. This
> thread is waiting, but it's not suspended, so the process will never
> actually appear as suspended in Task Manager or Process Explorer.
>
> That leaves enumerating threads in a snapshot and calling OpenThread
> and SuspendThread on each thread that's associated with the process.
> In comparison, let's take an abridged look at the guts of
> NtSuspendProcess.
>
> nt!NtSuspendProcess:
> ...
> mov r8,qword ptr [nt!PsProcessType]
> ...
> callnt!ObpReferenceObjectByHandleWithTag
> ...
> callnt!PsSuspendProcess
> ...
> mov ebx,eax
> callnt!ObfDereferenceObjectWithTag
> mov eax,ebx
> ...
> ret
>
> nt!PsSuspendProcess:
> ...
> callnt!ExAcquireRundownProtection
> cmp al,1
> jne nt!PsSuspendProcess+0x74
> ...
> callnt!PsGetNextProcessThread
> xor ebx,ebx
> jmp nt!PsSuspendProcess+0x62
>
> nt!PsSuspendProcess+0x4d:
> ...
> callnt!PsSuspendThread
> ...
> callnt!PsGetNextProcessThread
>
> nt!PsSuspendProcess+0x62:
> ...
> testrax,rax
> jne nt!PsSuspendProcess+0x4d
> ...
> callnt!ExReleaseRundownProtection
> jmp nt!PsSuspendProcess+0x79
>
> nt!PsSuspendProcess+0x74:
> mov ebx,0C10Ah (STATUS_PROCESS_IS_TERMINATING)
>
> nt!PsSuspendProcess+0x79:
> ...
> mov eax,ebx
> ...
> ret

Thanks for chiming in with useful info as usual. I agree with your
rationale after all. I've been dealing with undocumented Windows APIs
in psutil for a long time and they have always been in a sort of grey
area where despite they stayed "stable" since forever, the lack of an
official stand from Microsoft probably makes this addition
inappropriate for the stdlib.

> This code repeatedly calls PsGetNextProcessThread to walk the
> non-terminated threads of the process in creation order (based on a
> linked list in the process object) and suspends each thread via
> PsSuspendThread. In contrast, a Tool-Help thread snapshot is
> unreliable since it won't include threads created after the snapshot
> is created. The alternative is to use a different undocumented system
> call, NtGetNextThread [2], which is implemented via
> PsGetNextProcessThread. But that's slightly worse than calling
> NtSuspendProcess.
>
> [1]: https://stackoverflow.com/a/11010508
> [2]: 
> https://github.com/processhacker/processhacker/blob/v2.39/phnt/include/ntpsapi.h#L848

FWIW older psutil versions relied on Thread32

[Python-ideas] Add subprocess.Popen suspend() and resume()

[Giampaolo Rodola']

Hello,
I've been having these 2 implemented in psutil for a long time. On
POSIX these are convenience functions using os.kill() + SIGSTOP /
SIGCONT (the same as CTRL+Z  / "fg"). On Windows they use undocumented
NtSuspendProcess and NtResumeProcess Windows APIs available since XP.
The same approach is used by ProcessHacker and - I suppose -
pssupend.exe, both from Sysinternals team. It must be noted that there
are 3 different ways to do this on Windows
(https://stackoverflow.com/a/11010508/376587) but
NtSuspend/ResumeProcess appears to be the best choice. Possible use
case:

<>

Thoughts?

-- 
Giampaolo - http://grodola.blogspot.com
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Add "default" kwarg to list.pop()

[Giampaolo Rodola']

Sorry in advance if this has been proposed in the past but I couldn't find
anything on python-ideas:

>>> l = []
>>> l.pop(default=1)
1

FWIW my use case consists in reading entries from /proc/diskstats where
lines can have a variable number of fields depending on the kernel version:
https://github.com/giampaolo/psutil/issues/1354#issuecomment-434495870
https://github.com/giampaolo/psutil/blob/d8b05151e65f9348aff9b58da977abd8cacb2127/psutil/_pslinux.py#L1068
As such it would be convenient to set missing fields to 0 as "reads =
fields.pop(default=0)" instead of catching IndexError every time. Extra:
for consistency set.pop() should probably have the same.

Thoughts?

-- 
Giampaolo - http://grodola.blogspot.com
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Retire or reword the "Beautiful is better than ugly" Zen clause

[Giampaolo Rodola']

On Thu, Sep 13, 2018 at 12:51 PM Oleg Broytman  wrote:
> 2. I was sure it was trolling on the trail of
>https://bugs.python.org/issue34605

Wow! I find it a bit excessive that #34605 was not discussed first and got
checked in so quickly. I hope there won't be similar initiatives about
terms such as killing, abortion, daemon, termination, disabled, etc. If
somebody gets offended about these terms being used in computer science
it's entirely their problem.  Trying to make such individuals happy is
useless and a waste of python-dev time.

--
Giampaolo - http://grodola.blogspot.com


-- 
Giampaolo - http://grodola.blogspot.com
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Retire or reword the "Beautiful is better than ugly" Zen clause

[Giampaolo Rodola']

On Thu, Sep 13, 2018 at 10:38 AM Samantha Quan  wrote:
>
> First, I'd like to express how grateful I am to see more and more technical 
> communities embrace diversity and inclusivity, particularly big tech 
> communities like Python, Redis, and Django.
>
> In the spirit of the big recent terminology change, I propose retiring or 
> rewording the "Beautiful is better than ugly" Zen clause for perpetuating 
> beauty bias and containing lookist slur. I realize that Zen is old, but you 
> can't argue that the word "ugly" is harmless, now that society condemns body 
> shaming, and instead promotes body acceptance and self-love. One alternative 
> to that clause I could think of is "Clean is better than dirty", but please 
> do speak up if you have better ideas.
>
> I ask you to give this change serious consideration, even if it seems 
> over-the-top to you now, because times change, and this will be of great help 
> in the battle for the more tolerant and less judgemental society.
>
> I understand that this topic may seem controversial to some, so please be 
> open-minded and take extra care to respect the PSF Code Of Conduct when 
> replying.
>
> Thank you!
>
>   - Sam
>
> Some references:
>
> https://www.urbandictionary.com/define.php?term=Lookism
> https://en.m.wikipedia.org/wiki/Lookism
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/

This is simply ridiculous. I'm not sure if this is political
correctness pushed to its limits or just trolling.

-- 
Giampaolo - http://grodola.blogspot.com
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] A better (simpler) approach to PEP 505

[Giampaolo Rodola']

On Tue, Jul 24, 2018 at 1:57 AM Stefan Behnel  wrote:
>
> David Mertz schrieb am 23.07.2018 um 16:12:
> > The need addressed by PEP 505 is real; it's also MUCH more niche and
> > uncommon than something that would merit new syntax.  Moreover, the actual
> > legitimate purpose served by the PEP 505 syntax is easily served by
> > existing Python simply by using a wrapper class.
>
> The discussion so far made it clear to me that
>
> a) there is a use case for this feature, although I never needed it myself
> b) throwing new syntax at it is not the right solution
>
> Especially since there seem to be slightly diverging ideas about the exact
> details in behaviour. Since this can be done in form of a library, people
> should just drop it into a couple of competing libraries and let users
> choose what they like better in their specific situation. And since we
> already have a PEP now, let's continue to use it as a basis for discussion
> about how these libraries should best behave in general and what mistakes
> they should avoid.
>
> Stefan

+1
There is still no proof that such a programming pattern would be
widely used or is desirable in practice.
A library on PYPI could help clarifying that (and the PEP should mention it).

-- 
Giampaolo - http://grodola.blogspot.com
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP 505: None-aware operators

[Giampaolo Rodola']

On Tue, Jul 24, 2018 at 11:50 AM Steven D'Aprano  wrote:
>
> On Tue, Jul 24, 2018 at 12:05:14AM +0200, Giampaolo Rodola' wrote:
>
> > This:
> >
> > v = a?.b
> >
> > ...*implicitly* checks if value is not None [and continues execution].
>
> Do you agree that:
>
> obj.attribute
> x + 1
> func(arg)
>
> explicitly looks up an attribute on obj, explicitly adds 1 to x, and
> explicitly calls func with a single argument? I don't think that we have
> to write COBOL-esque code to be explicit:
>
> GET ATTRIBUTE "attribute" FROM obj
> ADD 1 TO x
> CALL FUNCTION func WITH ARGUMENT arg
>
> I don't accept that the use of punctuation makes something implicit. But
> if you want to argue that anything with punctuation is "implicit", then
> okay, Python has lots of implicit punctuation.
> By definition, ?. checks for None before doing the attribute lookup.
> That is completely explicit, regardless of how it is spelled:
>
> obj?.attribute

You totally missed my point about explicitness. Nevermind.

> > This
> >
> > v = a
> > if a.b is not None:
> > v = a.b
> >
> > ...*explicitly* checks if value is not None and continues execution.
>
> If you are trying to match the behaviour of a?.b above, it is also
> completely buggy and doesn't do what is intended.
>
> # Equivalent of a?.b
> v = a
> if v is not None:
> v = v.b
>
>
> > If for some reason '?'[ is also going to swallow LookupError
>
> What makes you think that ?[...] will swallow LookupError?
>
> Please don't argue against misfeatures that the PEP doesn't propose.
> Nothing in PEP 505 swallows any exceptions. Swallowing exceptions is
> explicitly rejected, and swallowing LookupError isn't part of the
> proposal.

I know it's not in the PEP. I merely mentioned that as PEP author was
questioning that possibility in previous messages. It was an example
(for you) on how *that* would make things even less explicit.

> [...]
> > One may argue that silently returning None instead of raising
> > AttributeError is also less explicit.
>
> And again, you are arguing against a misfeature which PEP 505 does not
> propose. The ?. operator will not suppress AttributeErrors.
>
> # Wrong! No! This is not what the PEP proposes!
> obj = 1.234
> assert obj?.hexx is None

That is not what I meant at all!
I seriously question whether you really don't understand or you're
just pretending.
What I meat in here was 'a?.b?.c?' returning None in case 'b' is None.

> [...]
> > > It isn't a first. Many existing operators use two adjacent symbols not
> > > interrupted by a space:
> > >
> > > e.g.  ==  <=  >=  !=  **  //  << >> +=  -=  *= etc.
> >
> > You say 'a == b'. You can't say 'a ?. b' (not that it matters, it
> > would be less intuitive anyway). You can't because '.?' is the only
> > couple of contiguous symbols requiring "something" before and after
> > with no spaces in between, and that's a first in the language.
>
> Why do you think spaces aren't allowed? The PEP explicitly says that
> the new operators can be used wherever the regular operators can be
> used.
> [...]
> That tells me that ?. will be legal anywhere . is legal, so if x . y is
> legal (and it is) so will x ?. y be legal.

Yes, I forgot 'a . b' was legal - my bad.

> [...]
> > The difference is that 'a.b.c.d' will result in AttributeError as soon
> > as something is None while 'a?.b?.c?.d' will return None instead.
>
> Correct. Because sometimes you want an AttributeError, and sometimes you
> want None.
>
> You are criticising the operator for doing what it is designed and
> intended to do. You might as well criticise getattr(obj, 'spam', None)
> for returning None.
>
> If you want an AttributeError, then don't use ?. and use ordinary .
> instead.

Again, you missed my point.

> > > Likewise the logical operators "or" and "and" are designed to
> > > short-circuit. If ?? and friends are a mistake because they
> > > short-circuit, why aren't "or" and "and" mistakes?
> > > I'm not asking this as a rhetorical question. If you think there is a
> > > reason why it is okay for or/and to short-circuit, but it is bad for ??
> > > and friends to short-circuit, then please explain why they are
> > > different. I will be very happy to listen to your arguments.
> >
> > The argument about this is that '?.' short-circuits execution
> > *silently*.
>
> Other short-circuit operators also short-circuit execution silently.

Re: [Python-ideas] PEP 505: None-aware operators

[Giampaolo Rodola']

On Tue, Jul 24, 2018 at 2:22 AM MRAB  wrote:

> >> > It
> >> > does so by introducing a brand new operator ("?") which can be spelled
> >> > in two forms ("a?.b" and "a?[b]") by using two adjacent symbols not
> >> > interrupted by any space, which is an absolute first in the Python
> >> > syntax
> >>
> >> It isn't a first. Many existing operators use two adjacent symbols not
> >> interrupted by a space:
> >>
> >> e.g.  ==  <=  >=  !=  **  //  << >> +=  -=  *= etc.
> >
> > You say 'a == b'. You can't say 'a ?. b' (not that it matters, it
> > would be less intuitive anyway). You can't because '.?' is the only
> > couple of contiguous symbols requiring "something" before and after
> > with no spaces in between, and that's a first in the language.
>
> You _can_ say 'a ?. b', just as you _can_ say 'a . b'.

You're right. It's so uncommon I forgot this style was valid. Anyway,
as I said 'a ?. b' would be even worse the same way 'a . b' is worse
than 'a.b'. The recommended and broadly used spelling would be 'a?.b'.

-- 
Giampaolo - http://grodola.blogspot.com
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP 505: None-aware operators

[Giampaolo Rodola']

On Mon, Jul 23, 2018 at 6:53 PM Steven D'Aprano  wrote:
>
> On Mon, Jul 23, 2018 at 02:04:17PM +0200, Giampaolo Rodola' wrote:
> > "a?.b" does two and that's a fundamental difference (explicitness).
>
> How is "two things" less explicit than "one thing"?
> Comments like the above is why I think that "explicit" and "implicit"
> are used to mean "I like it" and "I don't like it" rather than being
> objective arguments, or indeed having anything to do with explicitness
> or implicitness.

This:

v = a?.b

...*implicitly* checks if value is not None [and continues execution]. This:

v = a
if a.b is not None:
v = a.b

...*explicitly* checks if value is not None and continues execution.
If for some reason '?'[ is also going to swallow LookupError then
*that* would further decrease explicitness, because LookupError would
be nowhere in sight, the same way "if", "is", "not", "None", ":", "new
line" are nowhere in sight in the 'a?.b' example. Some argued "A ?? B"
is less explicit than "A if A is not None else B" for the same reason.
One may argue that silently returning None instead of raising
AttributeError is also less explicit.
This - and this only - is my argument about explicitness. It doesn't
have to do with how many things are hidden behind an import statement
or what happens on sorted() (that's comparing apples and oranges).
I hope it's clear now.

> > It
> > does so by introducing a brand new operator ("?") which can be spelled
> > in two forms ("a?.b" and "a?[b]") by using two adjacent symbols not
> > interrupted by any space, which is an absolute first in the Python
> > syntax
>
> It isn't a first. Many existing operators use two adjacent symbols not
> interrupted by a space:
>
> e.g.  ==  <=  >=  !=  **  //  << >> +=  -=  *= etc.

You say 'a == b'. You can't say 'a ?. b' (not that it matters, it
would be less intuitive anyway). You can't because '.?' is the only
couple of contiguous symbols requiring "something" before and after
with no spaces in between, and that's a first in the language. The
argument about this is that it's ugly and less readable. My additional
argument at the beginning of this thread was that if you add PEP-572
to the mix you dangerously enter into Perl territory:

foo(x=(x := a?.b?[c] ?? d))

> > and that's the second and fundamental difference. I cannot move
> > the same criticism to the "a.b" form: it's simpler, it does one thing
> > and it uses one symbol.
>
> You criticised ?. because it can interupt left-to-right execution:
>
> a?.b?.c?.d
>
> True. But so can a single dot:
>
> a.b.c.d
>
> is no more guaranteed to execute all the way to the right.

The difference is that 'a.b.c.d' will result in AttributeError as soon
as something is None while 'a?.b?.c?.d' will return None instead.

> Likewise the logical operators "or" and "and" are designed to
> short-circuit. If ?? and friends are a mistake because they
> short-circuit, why aren't "or" and "and" mistakes?
> I'm not asking this as a rhetorical question. If you think there is a
> reason why it is okay for or/and to short-circuit, but it is bad for ??
> and friends to short-circuit, then please explain why they are
> different. I will be very happy to listen to your arguments.

The argument about this is that '?.' short-circuits execution
*silently*. Instead of AttributeError you get None. You may chain ?.
in order to lazily traverse a long tree, inadvertently assign None to
a variable, continue code execution and fail later rather than sooner:

 email = request?.context?.user?.email  # None
 ...
 sendmail(subject, body, email)

Some (Antoine) rightly argued this may even have security implications
(replace 'email' with 'password').

-- 
Giampaolo - http://grodola.blogspot.com
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP 505: None-aware operators

[Giampaolo Rodola']

On Mon, Jul 23, 2018 at 11:52 AM Steve Dower  wrote:

> I'm borderline on ?[] right now. Honestly, I think it works best if it
> also silently handles LookupError (e.g. for traversing a loaded JSON
> dict), but then it's inconsistent with ?. which I think works best if it
> handles None but allows AttributeError.

That would easily make typos pass unnoticed:

request.context.user.usernme  # raises AttributeError
request?.context?.user?.usernme  # return None

Same goes for LookupError: if a key or index is missing on 'a?[b]' I
do want an exception. If I don't, which should be the exception rather
than the rule, I will simply take the risk myself and do:

default = ''
try:
name = d['user']['details']['name'] or default
except KeyError:
name = default

But certainly there should be no native syntax encouraging me to do
any of that. Talking about arbitrarily swallowing exceptions is the
worst direction this proposal can take as it breaks yet another
fundamental Python Zen: "errors should never pass silently". IMO this
shows how fundamentally detached from the base philosophy of the
language this whole idea is.

--
Giampaolo - http://grodola.blogspot.com
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP 505: None-aware operators

[Giampaolo Rodola']

On Mon, Jul 23, 2018 at 3:12 AM Steven D'Aprano  wrote:
> > ? has no spaces, it's literally "variable names interrupted by
> > question marks" and evaluation can stop at any time while scanning the
> > line from left to right.
>
> Just like ordinary attribute access.
>
> This is the point I was making earlier: you accept existing punctuation
> doing these things:
>
> try:
> obj.spam.egsg.tomato.cheese  # oops a typo
> except AttributeError:
> # evaluation can stop at any time
> ...
>
> while demanding a higher standard for new punctuation.
>
> All of your criticisms of ? punctuation applies to . as well.

I don't think they do. For once, "a.b" does one and one thing only,
"a?.b" does two and that's a fundamental difference (explicitness). It
does so by introducing a brand new operator ("?") which can be spelled
in two forms ("a?.b" and "a?[b]") by using two adjacent symbols not
interrupted by any space, which is an absolute first in the Python
syntax and that's the second and fundamental difference. I cannot move
the same criticism to the "a.b" form: it's simpler, it does one thing
and it uses one symbol.

Your argument is basically a revisitation of "it's just another
symbol" and "it's like a + b" which you have being pulling different
times in this thread already. You want to imply that since symbols are
already used in the grammar (and "." in particular) then it's
perfectly fine to also have "?" and its spell variants (which are 4 in
total). I don't think that's how changes of such importance should be
discussed.

> > Multiple "?" can live on the same line so
> > that's incentive to write one-liners, really, and to me one-liners are
> > always less explicit than the same logic split on multiple lines.
>
> Explicit is not always better.
>
> import this
>
> is much better than:
>
> for location in sys.path:
> try:
> for file in os.listdir(location):
> if os.splitext(file) in ('.pyc', '.py', '.so'):
> ...

I honestly don't see how this example is related with anything discussed so far.

> If punctuation is unreadable and Perlish, so is "." and ":" punctuation.
> If ?? is bad because it is "implicit", then so is import or sorted.

I'm not even sure how to reply to this.

-- 
Giampaolo - http://grodola.blogspot.com
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP 505: None-aware operators

[Giampaolo Rodola']

On Mon, Jul 23, 2018 at 12:08 AM Chris Angelico  wrote:
>
> On Mon, Jul 23, 2018 at 7:51 AM, Giampaolo Rodola'  wrote:
> > On Sun, Jul 22, 2018 at 10:55 PM Chris Angelico  wrote:
> >>
> >> On Mon, Jul 23, 2018 at 6:43 AM, Giampaolo Rodola'  
> >> wrote:
> >> > On Sun, Jul 22, 2018 at 10:01 PM Chris Angelico  wrote:
> >> >>
> >> >> On Mon, Jul 23, 2018 at 1:09 AM, Giampaolo Rodola'  
> >> >> wrote:
> >> >> > On Sun, Jul 22, 2018 at 3:38 PM Chris Angelico  
> >> >> > wrote:
> >> >> > I find it less explicit mainly because it does 3 things at once: check
> >> >> > if attribute is None, use it if it's not None and continue the
> >> >> > evaluation from left to right. I find that logic to be more explicit
> >> >> > when living on different lines or is clearly delimited by keywords and
> >> >> > spaces. ? has no spaces, it's literally "variable names interrupted by
> >> >> > question marks" and evaluation can stop at any time while scanning the
> >> >> > line from left to right. Multiple "?" can live on the same line so
> >> >> > that's incentive to write one-liners, really, and to me one-liners are
> >> >> > always less explicit than the same logic split on multiple lines.
> >> >>
> >> >> Ah, I see what you mean. Well, think about what actually happens when
> >> >> you write "lst.sort()". In terms of "hidden behaviour", there is far
> >> >> FAR more of it in existing syntax than in the new proposals.
> >> >
> >> > I am not sure I'm following you (what does lst.sort() have to do with 
> >> > "?"?).
> >>
> >> The "." in "lst.sort" is an operator. How much hidden behaviour is
> >> there in that? Do you actually even know every possible thing that can
> >> happen? Don't feel bad if you don't - it's not an indictment of your
> >> quality as a programmer, but an acknowledgement that Python's
> >> attribute access is incredibly complicated.
> >
> > I'm [not] going to engage into a discussion about the analogy between "?"
> > and "." because simply there is none. It doesn't prove anything except
> > that you're not really interested in having a serious discussion about
> > the pros and cons of this PEP: you just want it to happen no matter
> > what.
>
> That's because the dot already exists in the language, and you have
> become so accustomed to it that you don't see it any more. You've just
> proven my point.

You're back at "since we have X that justifies the addition of Y" [1]
and AFAICT that's the only argument you have provided so far in a 100+
messages discussion.

[1] https://mail.python.org/pipermail/python-ideas/2018-July/052068.html

-- 
Giampaolo - http://grodola.blogspot.com
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP 505: None-aware operators

[Giampaolo Rodola']

On Sun, Jul 22, 2018 at 10:55 PM Chris Angelico  wrote:
>
> On Mon, Jul 23, 2018 at 6:43 AM, Giampaolo Rodola'  wrote:
> > On Sun, Jul 22, 2018 at 10:01 PM Chris Angelico  wrote:
> >>
> >> On Mon, Jul 23, 2018 at 1:09 AM, Giampaolo Rodola'  
> >> wrote:
> >> > On Sun, Jul 22, 2018 at 3:38 PM Chris Angelico  wrote:
> >> > I find it less explicit mainly because it does 3 things at once: check
> >> > if attribute is None, use it if it's not None and continue the
> >> > evaluation from left to right. I find that logic to be more explicit
> >> > when living on different lines or is clearly delimited by keywords and
> >> > spaces. ? has no spaces, it's literally "variable names interrupted by
> >> > question marks" and evaluation can stop at any time while scanning the
> >> > line from left to right. Multiple "?" can live on the same line so
> >> > that's incentive to write one-liners, really, and to me one-liners are
> >> > always less explicit than the same logic split on multiple lines.
> >>
> >> Ah, I see what you mean. Well, think about what actually happens when
> >> you write "lst.sort()". In terms of "hidden behaviour", there is far
> >> FAR more of it in existing syntax than in the new proposals.
> >
> > I am not sure I'm following you (what does lst.sort() have to do with "?"?).
>
> The "." in "lst.sort" is an operator. How much hidden behaviour is
> there in that? Do you actually even know every possible thing that can
> happen? Don't feel bad if you don't - it's not an indictment of your
> quality as a programmer, but an acknowledgement that Python's
> attribute access is incredibly complicated.

I'm going to engage into a discussion about the analogy between "?"
and "." because simply there is none. It doesn't prove anything except
that you're not really interested in having a serious discussion about
the pros and cons of this PEP: you just want it to happen no matter
what.

> Imagine if we were talking about people, rather than features in a
> language; imagine if, to join the Warriors Guild, you had to first
> slay a red dragon with nothing but a rusty dagger, despite none of the
> existing members having done so. Is that reasonable to ask? Can you
> say "well, the guild is mature now, so yeah, it's a good thing"?

Ditto.

-- 
Giampaolo - http://grodola.blogspot.com
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP 505: None-aware operators

[Giampaolo Rodola']

On Sun, Jul 22, 2018 at 11:51 PM Giampaolo Rodola'  wrote:
>
> On Sun, Jul 22, 2018 at 10:55 PM Chris Angelico  wrote:
> >
> > On Mon, Jul 23, 2018 at 6:43 AM, Giampaolo Rodola'  
> > wrote:
> > > On Sun, Jul 22, 2018 at 10:01 PM Chris Angelico  wrote:
> > >>
> > >> On Mon, Jul 23, 2018 at 1:09 AM, Giampaolo Rodola'  
> > >> wrote:
> > >> > On Sun, Jul 22, 2018 at 3:38 PM Chris Angelico  
> > >> > wrote:
> > >> > I find it less explicit mainly because it does 3 things at once: check
> > >> > if attribute is None, use it if it's not None and continue the
> > >> > evaluation from left to right. I find that logic to be more explicit
> > >> > when living on different lines or is clearly delimited by keywords and
> > >> > spaces. ? has no spaces, it's literally "variable names interrupted by
> > >> > question marks" and evaluation can stop at any time while scanning the
> > >> > line from left to right. Multiple "?" can live on the same line so
> > >> > that's incentive to write one-liners, really, and to me one-liners are
> > >> > always less explicit than the same logic split on multiple lines.
> > >>
> > >> Ah, I see what you mean. Well, think about what actually happens when
> > >> you write "lst.sort()". In terms of "hidden behaviour", there is far
> > >> FAR more of it in existing syntax than in the new proposals.
> > >
> > > I am not sure I'm following you (what does lst.sort() have to do with 
> > > "?"?).
> >
> > The "." in "lst.sort" is an operator. How much hidden behaviour is
> > there in that? Do you actually even know every possible thing that can
> > happen? Don't feel bad if you don't - it's not an indictment of your
> > quality as a programmer, but an acknowledgement that Python's
> > attribute access is incredibly complicated.
>
> I'm going to engage into a discussion [...]

s/I'm going/I'm not going

-- 
Giampaolo - http://grodola.blogspot.com
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP 505: None-aware operators

[Giampaolo Rodola']

On Sun, Jul 22, 2018 at 10:01 PM Chris Angelico  wrote:
>
> On Mon, Jul 23, 2018 at 1:09 AM, Giampaolo Rodola'  wrote:
> > On Sun, Jul 22, 2018 at 3:38 PM Chris Angelico  wrote:
> > I find it less explicit mainly because it does 3 things at once: check
> > if attribute is None, use it if it's not None and continue the
> > evaluation from left to right. I find that logic to be more explicit
> > when living on different lines or is clearly delimited by keywords and
> > spaces. ? has no spaces, it's literally "variable names interrupted by
> > question marks" and evaluation can stop at any time while scanning the
> > line from left to right. Multiple "?" can live on the same line so
> > that's incentive to write one-liners, really, and to me one-liners are
> > always less explicit than the same logic split on multiple lines.
>
> Ah, I see what you mean. Well, think about what actually happens when
> you write "lst.sort()". In terms of "hidden behaviour", there is far
> FAR more of it in existing syntax than in the new proposals.

I am not sure I'm following you (what does lst.sort() have to do with "?"?).

> Which is back to what Steven said: people demand such a high
> bar for new syntax that few existing pieces of syntax would pass it.

Probably. That's what happens when a language is mature. Personally I
don't think that's a bad thing.

-- 
Giampaolo - http://grodola.blogspot.com
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP 505: None-aware operators

[Giampaolo Rodola']

On Sun, Jul 22, 2018 at 3:38 PM Chris Angelico  wrote:
>
> On Sun, Jul 22, 2018 at 11:35 PM, Giampaolo Rodola'  
> wrote:
> > On Sun, Jul 22, 2018 at 2:10 PM Steven D'Aprano  wrote:
> >>
> >> On Sun, Jul 22, 2018 at 12:13:04PM +0200, Giampaolo Rodola' wrote:
> >> > On Sun, Jul 22, 2018 at 3:55 AM Steven D'Aprano  
> >> > wrote:
> >> [...]
> >> > > I don't think that "+" is harder to read than
> >> > > "standard_mathematics_operators_numeric_addition"
> >> >
> >> >
> >> > Please let's drop the argument that + - * / = and ? are the same.
> >> [...]
> >> But if we insist that every symbol we use is instantly recognisable and
> >> intuitively obvious to every programmer, we're putting the bar for
> >> acceptance impossibly high.
> >
> > I personally don't find "a ?? b" too bad (let's say I'm -0 about it)
> > but idioms such as "a?.b", "a ??= b" and "a?[3] ?? 4" look too
> > Perl-ish to me, non pythonic and overall not explicit, no matter what
> > the chosen symbol is gonna be.
>
> Please explain what is not explicit about it. "a?.b" is very simple
> and perfectly explicit: it means "None if a is None else a.b". What
> does "not explicit" mean, other than "I don't like this code"?

I find it less explicit mainly because it does 3 things at once: check
if attribute is None, use it if it's not None and continue the
evaluation from left to right. I find that logic to be more explicit
when living on different lines or is clearly delimited by keywords and
spaces. ? has no spaces, it's literally "variable names interrupted by
question marks" and evaluation can stop at any time while scanning the
line from left to right. Multiple "?" can live on the same line so
that's incentive to write one-liners, really, and to me one-liners are
always less explicit than the same logic split on multiple lines.

-- 
Giampaolo - http://grodola.blogspot.com
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP 505: None-aware operators

[Giampaolo Rodola']

On Sun, Jul 22, 2018 at 2:10 PM Steven D'Aprano  wrote:
>
> On Sun, Jul 22, 2018 at 12:13:04PM +0200, Giampaolo Rodola' wrote:
> > On Sun, Jul 22, 2018 at 3:55 AM Steven D'Aprano  wrote:
> [...]
> > > I don't think that "+" is harder to read than
> > > "standard_mathematics_operators_numeric_addition"
> >
> >
> > Please let's drop the argument that + - * / = and ? are the same.
> [...]
> But if we insist that every symbol we use is instantly recognisable and
> intuitively obvious to every programmer, we're putting the bar for
> acceptance impossibly high.

I personally don't find "a ?? b" too bad (let's say I'm -0 about it)
but idioms such as "a?.b", "a ??= b" and "a?[3] ?? 4" look too
Perl-ish to me, non pythonic and overall not explicit, no matter what
the chosen symbol is gonna be. It looks like they want to do too much
for the sole reason of allowing people to write more compact code and
save a few lines. Compact code is not necessarily a good thing,
especially when it comes at the expense of readability and
explicitness, as I think is this case.

> All the obvious operators are already in use. Anything we add now is
> going to be a little bit niche, a little bit unusual.

That's basically my point. And I know I'll sound very conservative
here but to me that is a valid enough reason to not take action or be
extremely careful at the very least. Not to state the obvious but it's
not that we *have to* use the remaining unused symbols just because
they're there.

-- 
Giampaolo - http://grodola.blogspot.com
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP 505: None-aware operators

[Giampaolo Rodola']

On Sun, Jul 22, 2018 at 12:26 PM Paul Moore  wrote:

> On 22 July 2018 at 11:13, Giampaolo Rodola'  wrote:
> > - "a?[2] ?? 3" means "index 2 of list a is picked up if a is not None,
> else
> > use 3"
>
> Actually, doesn't it mean
>
> if a is not None, pick up index 2 of the list.
> If a is None, OR IF a[2] IS NONE, then use 3.
> If a is None but a[2] is not None, use a[2].
>
> ?
>
> Which is subtly different, and probably at least as prone to
> "accidental" errors as some of the constructs the None-aware operators
> are intended to replace.
>
> Paul
>

Yes, I think you're right.

-- 
Giampaolo - http://grodola.blogspot.com
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP 505: None-aware operators

[Giampaolo Rodola']

On Sun, Jul 22, 2018 at 3:55 AM Steven D'Aprano  wrote:

> Indeed we do. But we also say:
>
> - we say "+" instead of "add"
> - we say "//" instead of "floor division"
> - we say "**" instead of "exponentiation"
> - we say "&" instead of "bitwise AND"
> - we say "f( ... )" instead of "call f with arguments ..."

[...]

I don't think that "+" is harder to read than

"standard_mathematics_operators_numeric_addition"


Please let's drop the argument that + - * / = and ? are the same. They
clearly are not. Anybody learned those symbols at elementary schools, all
programming languages have them and using math in programming is common
enough to justify a symbol over a keyword. "a + b" is literally just an
addition and nothing else. The "?" variants have multiple meanings,
spellings and implications:

- "a ?? b" means "b is chosen over a if a is None"

- "a ??= b" means "a is set to b if a is None"

- "a?.b" means "a.b is executed but only if a is not None"

- "a?[2] ?? 3" means "index 2 of list a is picked up if a is not None, else
use 3"

"a?.b"and "a?[2]" in particular go way beyond the mere "it's not pretty"
argument which, I concur, can be subjective, as you don't know where
evaluation stops. Even "a ??= b" goes beyond that as it introduces yet
another assignment operator (the third, as we now have = and :=). So again,
I don't think it's fair to dismiss the whole thing as "it's just another
symbol" or "it's like a + b".

As for bitwise operators: they are kinda obscure and low-levelish and when
I bump into them I still have to pause to reason what's going on. The
difference with ? though is that you basically have no other way to do the
same thing. Also they are much more rare and also are present in many other
languages since... forever.

-- 
Giampaolo - http://grodola.blogspot.com
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP 505: None-aware operators

[Giampaolo Rodola']

On Thu, Jul 19, 2018 at 3:39 PM Steven D'Aprano  wrote:
> Tens of thousands of non-English speakers have had to learn the meaning
> of what might as well be meaningless, random sets of symbols (to them)
> like "class", "import", "while" and "True". If they can do so, perhaps
> we English-speakers should stop complaining about how hard it is to
> memorise the meaning of a couple of symbols like ??.

"class", "import", "while" and "True" are keywords, not symbols. A
symbol is more cryptic than a keyword hence it comes at a higher cost
in terms of readability. Which is the reason why conditionals use
keywords instead symbols:

- we say "and" instead of "&&"
- we say "or" instead of "||"
- we say "not" instead of "!"
- we say "is" instead of "==="

AFAICT "?" would be the first one breaking this rule for conditionals
[1], and not only for them since it can also arbitrarily appear in
non-conditionals (x.?y).

[1] I know, we say "==" instead of "equal" and "!=" instead of
"different" but the equal sign implies an equality. "?" does not imply
"None-ness": it's arbitrary. Furthermore "==" and "!=" are much more
common than testing for "None", hence they have more reason to exist.

> Surely its no more
> difficult than learning the various meanings of ** and [] which we've
> already done.

It is more difficult if you consider that ? has different spellings
(?, ??, a?.b, ??=, ...), each spelling with a different meaning.

> *Its just spelling*. If it is a useful and well-defined feature, we'll
> get used to the spelling soon enough.

Such an argument may apply to other languages but not Python. The
philosophy of the language is all about readability and beauty since
day 1, and the main reason why Python got so successful despite being
slow. That's why we have mandatory indentation, to say one. We don't
*have to* get used to idioms which can decrease readability. When we
do there must be a good reason and the spelling should matter (I
remember the debates about decorators' syntax).

--
Giampaolo - http://grodola.blogspot.com
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP 505: None-aware operators

[Giampaolo Rodola']

On Thu, Jul 19, 2018 at 11:22 AM Antoine Pitrou  wrote:
>
> On Thu, 19 Jul 2018 19:11:33 +1000
> Chris Angelico  wrote:
>
> > On Thu, Jul 19, 2018 at 4:06 PM, Greg Ewing  
> > wrote:
> > > Chris Angelico wrote:
> > >>
> > >> I'd love to hear an explanation of WHY this doesn't look like Python
> > >> any more. For instance, is the + operator somehow wrong for Python,
> > >> and it should have been the word "add"?
> > >
> > >
> > > There's a very long tradition of using the symbol "+" to
> > > represent addition, so it's something most people are
> > > familiar with. There's no such tradition for the new
> > > operators being proposed.
> >
> > Okay. What about bitwise operators, then? They don't have centuries of
> > mathematical backing to support them, yet it isn't considered
> > "unpythonic" to have &|^~ peppering our code.
>
> They have decades of widespread presence in other programming
> languages, though.
>
> > Coalescing None to a value is _at least_ as common as
> > performing bit manipulations in integers.
>
> Certainly, but spelling that as a "?*" operator is a syntactical
> novelty.
>
> Consider that for the ternary operator, Python chose "B if A else C"
> over "A ? B : C", even though the latter had precedent in several
> languages.
>
> Regards
>
> Antoine.
>
>
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/

^
Agreed. To me even bitwise operators "feel" a bit weird when using
them but fortunately they are rare. On the other hand "?" has the
potential to be used (and abused) much more than bitwise operators.
Also I don't consider "since we have X then let's add Y" a valid
enough reasoning.

-- 
Giampaolo - http://grodola.blogspot.com
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP 505: None-aware operators

[Giampaolo Rodola']

On Thu, Jul 19, 2018 at 2:06 AM Chris Angelico  wrote:
> > With all due respect (and I am sorry for being “vocal” about a PEP once
> > again) I find this simply ugly. To me this basically doesn’t look like
> > python anymore, so a strong -1 from me.
>
> I'd love to hear an explanation of WHY this doesn't look like Python
> any more.

Because it looks like Perl.

> For instance, is the + operator somehow wrong for Python,
> and it should have been the word "add"?

The meaning of "+" is obvious to anybody, including non programmers.
"?" is arbitrary so you cannot guess what it does or means, especially
when it can be spelled in so many different forms (?, ??, a?.b, ??=,
...), each form requiring a different mental replacement. Use this and
:= on the same line and the reader will practically be reading another
language.

>  [ please trim quotes, you just quoted the entire PEP in your post ]

Ouch! Sorry about that.

-- 
Giampaolo - http://grodola.blogspot.com
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP 505: None-aware operators

[Giampaolo Rodola']

On Wed, Jul 18, 2018 at 7:46 PM Steve Dower  wrote:
>
> Possibly this is exactly the wrong time to propose the next big syntax
> change, since we currently have nobody to declare on it, but since we're
> likely to argue for a while anyway it probably can't hurt (and maybe
> this will become the test PEP for whoever takes the reins?).
>
> FWIW, Guido had previously indicated that he was generally favourable
> towards most of this proposal, provided we could figure out coherent
> semantics. Last time we tried, that didn't happen, so this time I've
> made the semantics much more precise, have implemented and verified
> them, and made much stronger statements about why we are proposing these.
>
> Additional thanks to Mark Haase for writing most of the PEP. All the
> fair and balanced parts are his - all the overly strong opinions are mine.
>
> Also thanks to Nick Coghlan for writing PEPs 531 and 532 last time we
> went through this - if you're unhappy with "None" being treated as a
> special kind of value, I recommend reading those before you start
> repeating them.
>
> There is a formatted version of this PEP at
> https://www.python.org/dev/peps/pep-0505/
>
> My current implementation is at
> https://github.com/zooba/cpython/tree/pep-505 (though I'm considering
> removing some of the new opcodes I added and just generating more
> complex code - in any case, let's get hung up on the proposal rather
> than the implementation :) )
>
> Let the discussions begin!
>
> ---
>
> PEP: 505
> Title: None-aware operators
> Version: $Revision$
> Last-Modified: $Date$
> Author: Mark E. Haase , Steve Dower
> 
> Status: Draft
> Type: Standards Track
> Content-Type: text/x-rst
> Created: 18-Sep-2015
> Python-Version: 3.8
>
> Abstract
> 
>
> Several modern programming languages have so-called "``null``-coalescing"
or
> "``null``- aware" operators, including C# [1]_, Dart [2]_, Perl, Swift,
> and PHP
> (starting in version 7). These operators provide syntactic sugar for
common
> patterns involving null references.
>
> * The "``null``-coalescing" operator is a binary operator that returns
> its left
>operand if it is not ``null``. Otherwise it returns its right operand.
> * The "``null``-aware member access" operator accesses an instance
> member only
>if that instance is non-``null``. Otherwise it returns ``null``.
> (This is also
>called a "safe navigation" operator.)
> * The "``null``-aware index access" operator accesses an element of a
> collection
>only if that collection is non-``null``. Otherwise it returns
> ``null``. (This
>is another type of "safe navigation" operator.)
>
> This PEP proposes three ``None``-aware operators for Python, based on the
> definitions and other language's implementations of those above.
> Specifically:
>
> * The "``None`` coalescing`` binary operator ``??`` returns the left
> hand side
>if it evaluates to a value that is not ``None``, or else it evaluates
and
>returns the right hand side. A coalescing ``??=`` augmented assignment
>operator is included.
> * The "``None``-aware attribute access" operator ``?.`` evaluates the
> complete
>expression if the left hand side evaluates to a value that is not
> ``None``
> * The "``None``-aware indexing" operator ``?[]`` evaluates the complete
>expression if the left hand site evaluates to a value that is not
> ``None``
>
> Syntax and Semantics
> 
>
> Specialness of ``None``
> ---
>
> The ``None`` object denotes the lack of a value. For the purposes of these
> operators, the lack of a value indicates that the remainder of the
> expression
> also lacks a value and should not be evaluated.
>
> A rejected proposal was to treat any value that evaluates to false in a
> Boolean context as not having a value. However, the purpose of these
> operators
> is to propagate the "lack of value" state, rather that the "false" state.
>
> Some argue that this makes ``None`` special. We contend that ``None`` is
> already special, and that using it as both the test and the result of
these
> operators does not change the existing semantics in any way.
>
> See the `Rejected Ideas`_ section for discussion on the rejected
approaches.
>
> Grammar changes
> ---
>
> The following rules of the Python grammar are updated to read::
>
>  augassign: ('+=' | '-=' | '*=' | '@=' | '/=' | '%=' | '&=' | '|=' |
> '^=' |
>  '<<=' | '>>=' | '**=' | '//=' | '??=')
>
>  power: coalesce ['**' factor]
>  coalesce: atom_expr ['??' factor]
>  atom_expr: ['await'] atom trailer*
>  trailer: ('(' [arglist] ')' |
>'[' subscriptlist ']' |
>'?[' subscriptlist ']' |
>'.' NAME |
>'?.' NAME)
>
> Inserting the ``coalesce`` rule in this location ensures that expressions
> resulting in ``None`` are natuarlly coalesced before they are used in
> operations that would typically raise ``TypeError``. Like ``and`` and

Re: [Python-ideas] Add new `Symbol` type

[Giampaolo Rodola']

Historically this has always been achieved by using:

_default = object()

def fun(arg=_default):
 if arg is not _default:
  

...which does its job just fine. If you need something like this you're
typically a medium/advanced Python user so you either already know about it
or you'll find the solution on Google pretty quickly. A dedicated
sentinel()/Sentinel/... thingy would just be more "official", but at the
end of the day it would achieve exactly the same thing by adding more
complexity to the language for no practical benefit (and I seriously think
Python is getting too big).

-1


On Thu, Jul 5, 2018 at 9:39 PM Flavio Curella 
wrote:

> More than once I've found myself wanting to create a 'sentinel' value. The
> most common use case is to differentiate between an argument that has not
> been provided, and an argument provided with the value `None`.
>
> This would be solvable by implementing something similar to what
> JavaScript calls [`Symbol`](
> https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Symbol
> ).
>
> This could be implemented as a 3rd-party library, but there won't be a way
> to have ['Global' Symbols](
> https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Symbol/for
> )
>
> Furthermore, without a common implementation in the std library, various
> Python libraries had to write their own implementations, which all differs
> in functionality and behavior.
>
> Is this something that the Python community is interested in? I'm willing
> to write the PEP
>
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>


-- 
Giampaolo - http://grodola.blogspot.com
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] datetime.timedelta literals

[Giampaolo Rodola']

On Sat, Jun 2, 2018 at 2:21 PM, Pål Grønås Drange 
wrote:

> Elevator pitch:
>
> (2.5h - 14min + 9300ms).total_seconds()
> # 8169.3
>
> from datetime import datetime as dt
> start = dt.now()
> end = dt.now()
> (end-start) < 5s
> # True
>
>
> chrono::duration:
>
> In C++ 14 the std::chrono::duration was introduced which corresponds
> somewhat to
> datetime.timedelta.
>
> C++ 14 introduced so-called chrono literals[1], which are literals
> specified as
> [number][h|min|s|ms|us|ns], e.g.
>
> * 2.5h
> * 14min
> * 9300ms
>
> These literals should correspond to
>
> * datetime.timedelta(0, 9000)  # 2.5h = 2.5*3600 = 9000 seconds
> * datetime.timedelta(0, 840)   # 14min = 14*60 = 840 seconds
> * datetime.timedelta(0, 9, 30)  # 9300ms = 9 seconds + 3*10^5
> microseconds
>
>
> If a literal was interpreted as a datetime.timedelta, the following would
> work
> out of the box:
>
> 2.5h - 14min + 9300ms * 2
>
> which would correspond to
>
> from datetime import timedelta as D
>
> D(hours=2.5) - D(minutes=14) + D(milliseconds=9300) * 2
> # datetime.timedelta(0, 8178, 60)  # (*2 precedes, so that's to be
> expected)
>
>
> (D(hours=2.5) - D(minutes=14) + D(milliseconds=9300)) * 2
> # datetime.timedelta(0, 16338, 60)
>
>
>
> Notes:
>
> * C++ uses `min` instead of `m`.  `min` is a keyword in Python.
> * In C++, `1d` means the first day of a month [2].
> * In C++, `1990y` means the year 1990 (in the Proleptic Gregorian
> calendar) [3].
> * C++ have the types signed integers and not floats, so 2.5h would not be
> valid.
> * My apologies if this has been discussed before; my search-fu gave me
> nothing.
>
>
>
> References:
>
>
> [1] std::literals::chrono_literals::operator""min
> http://en.cppreference.com/w/cpp/chrono/operator%22%22min
>
> [2] http://en.cppreference.com/w/cpp/chrono/day
>
> [3] http://en.cppreference.com/w/cpp/chrono/year
>
>
> Best regards,
> Pål Grønås Drange
>

IMO datetimes are not common enough to deserve their own literals. It would
make the language more complex and harder to learn for a relatively little
benefit. This would probably make more sense as a third party lib:

>>> import datetimeutils
>>> datetimeutils.interpretstr("2.5h - 14min + 9300ms")
datetime(...)

Both the string and the possibility to specify function arguments would
give you way more expressiveness than language literals.


-- 
Giampaolo - http://grodola.blogspot.com
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] shutil zero-copy and exotic filesystems

[Giampaolo Rodola']

Whops, I hit "send" too soon. Sorry about the messed up message.

On Tue, May 29, 2018 at 10:56 AM, Giampaolo Rodola' 
wrote:

> Hello,
> I've been working on a patch which speeds up shutil.copy* operations for
> all 3 major platforms (Linux, Windows, OSX):
> https://bugs.python.org/issue33671
> Since the speedup is quite consistent I'd love to see this merged in, but
> considering shutil.copy* is quite crucial I wanted to hear other folk's
> opinion first. Attached patch attempts to use platform-specific zero-copy
> syscalls [1] by default and fallbacks on using plain read() / write()
> variant in case of immediate failure. In theory this should work fine, in
> practice I haven't tested it on exotic (e.g. network) filesystems. In order
> to diminish risks of breakage I think if it would make sense to:
> - add a global shutil.NO_ZEROCOPY variable defaulting to False
> - add a "no_zerocopy" argument to all functions involving a copy
> (copyfile(). copy(), copy2(), copytree(), move())
>
> Thoughts?
>
> [1] sendfile() (Linux), fcopyfile() (OSX), CopyFileW (Windows)
>
>
>
>
> since the matter is a bit sensitive in terms of potential breakage on
> exotic / untested (e.g. network) filesystems I want to raise some attention
> about:
> https://bugs.python.org/issue33671
> Attached patch attempts to use platform-specific zero-copy syscalls by
> default and fallbacks on using plain read() / write() copy in case of
> immediate failure.  In order to diminish risks I think it would make sense
> to:
> - add a global shutil.NO_ZEROCOPY variable defaulting to False
> - add a "no_zerocopy" argument to all functions involving a copy
> (copyfile(). copy(), copy2(), copytree(), move())
>
> Thoughts?
>
> --
> Giampaolo - http://grodola.blogspot.com
>
>
>


-- 
Giampaolo - http://grodola.blogspot.com
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] shutil zero-copy and exotic filesystems

[Giampaolo Rodola']

Hello,
I've been working on a patch which speeds up shutil.copy* operations for
all 3 major platforms (Linux, Windows, OSX):
https://bugs.python.org/issue33671
Since the speedup is quite consistent I'd love to see this merged in, but
considering shutil.copy* is quite crucial I wanted to hear other folk's
opinion first. Attached patch attempts to use platform-specific zero-copy
syscalls [1] by default and fallbacks on using plain read() / write()
variant in case of immediate failure. In theory this should work fine, in
practice I haven't tested it on exotic (e.g. network) filesystems. In order
to diminish risks of breakage I think if it would make sense to:
- add a global shutil.NO_ZEROCOPY variable defaulting to False
- add a "no_zerocopy" argument to all functions involving a copy
(copyfile(). copy(), copy2(), copytree(), move())

Thoughts?

[1] sendfile() (Linux), fcopyfile() (OSX), CopyFileW (Windows)




since the matter is a bit sensitive in terms of potential breakage on
exotic / untested (e.g. network) filesystems I want to raise some attention
about:
https://bugs.python.org/issue33671
Attached patch attempts to use platform-specific zero-copy syscalls by
default and fallbacks on using plain read() / write() copy in case of
immediate failure.  In order to diminish risks I think it would make sense
to:
- add a global shutil.NO_ZEROCOPY variable defaulting to False
- add a "no_zerocopy" argument to all functions involving a copy
(copyfile(). copy(), copy2(), copytree(), move())

Thoughts?

-- 
Giampaolo - http://grodola.blogspot.com
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Add shutil.chown(..., recursive=False)

[Giampaolo Rodola']

...as in (not tested):

def _rchown(dir, user, group):
for root, dirs, files in os.walk(dir, topdown=False):
for name in files:
chown(os.path.join(root, name), user, group)

def chown(path, user=None, group=None, recursive=False):
if recursive and os.path.isdir(path):
_rchown(dir, user, group)
...

It appears like a common enough use case to me ("chown -R path").
Thoughts?

-- 
Giampaolo - http://grodola.blogspot.com
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Please consider skipping hidden directories in os.walk, os.fwalk, etc.

[Giampaolo Rodola']

On Tue, May 8, 2018 at 2:00 PM, David Mertz  wrote:

> I like the idea. I think an argument to os.walk() is the simplest option
> for most users. But per some comments, "hidden" is actually more subtle
> than the filesystem bit sometimes. I.e. dot-files, ~ suffix, maybe .bak,
> etc.
>
> I'd suggest meeting the ideas slightly and making the new argument
> 'filter' or 'skip' that takes a callable. Default to None, but provide an
> os.is_hidden that users don't need to figure out how to implement. E.g.
>
> os.walk(PATH, skip=os.is_hidden)
>
> os.walk(PATH, skip=lambda entry: entry.name.endswith(('~', '.bak',
> '.tmp')))
>

I think this would be a good addition because it gives direct access to the
underlying os.scandir() objects which are currently inaccessible and
discarded (if os.walk() were to be written today it'd probably yield (root,
os.DirEntry) instead of (root, dirs, files)). As such one can implement
advanced filtering logic without having to call os.stat() for each path
string yielded by os.walk() (faster).

IMO the callback should accept a (root, os.DirEntry) pair though, because the
"root" path can also be part of the filtering logic.

-- 

Giampaolo - http://grodola.blogspot.com
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Move optional data out of pyc files

[Giampaolo Rodola']

On Fri, 13 Apr 2018 at 03:47, M.-A. Lemburg  wrote:

> I think moving data out of pyc files is going in a wrong direction:
> more stat calls means slower import and slower startup time.
>
> Trying to make pycs smaller also isn't really worth it (they
> compress quite well).
>
> Saving memory could be done by disabling reading objects lazily
> from the file - without removing anything from the pyc file.
> Whether the few 100kB RAM this saves is worth the effort depends
> on the application space.
>
> This leaves the proposal to restructure pyc files into a sectioned
> file and possibly indexed file to make access to (lazily) loaded
> parts faster.


+1. With this in place -O and -OO cmdline options would become even less
useful (which is good).

-- 
Giampaolo - http://grodola.blogspot.com
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Generalized version of contextlib.close

[Giampaolo Rodola']

On Mon, Mar 26, 2018 at 10:35 AM, Roberto Martínez <
robertomartin...@gmail.com> wrote:

> Hi,
>
> sometimes I need to use contextlib.close but over methods with a different
> name, for example stop(), halt(), etc. For those cases I have to write my
> own contextlib.close specialized version with a hard-coded method name.
>
> I think adding a "method" argument to contextlib.close can be very useful:
>
> @contextmanager
> def closing(thing, method="close"):
> try:
> yield thing
> finally:
> getattr(thing, method)()
>
> Or maybe something even more generic:
>
> @contextmanager
> def calling(fn, *args, **kwargs):
> try:
> yield
> finally:
> fn(*args, **kwargs)
>
>
> Best regards,
> Roberto
>
>
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
>
+1

-- 
Giampaolo - http://grodola.blogspot.com
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Official site-packages/test directory

[Giampaolo Rodola']

On Fri, Jan 19, 2018 at 5:23 PM, Paul Moore  wrote:

> Another common approach is to not ship tests as part of your (runtime)
> package at all - they are in the sdist but not the wheels nor are they
> deployed with "setup.py install". In my experience, this is the usual
> approach projects take if they don't have the tests in the package
> directory. (I don't think I've *ever* seen a project try to install
> tests except by including them in the package directory...)


I personally include them in psutil distribution so that users can test the
installation with "python -m psutil.test". I have even this documented as I
think it's an added value.


-- 
Giampaolo - http://grodola.blogspot.com
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Memory limits [was Re: Membership of infinite iterators]

[Giampaolo Rodola']

On Thu, Oct 19, 2017 at 10:05 AM, Stephan Houben 
wrote:

> Hi Steve,
>
> 2017-10-19 1:59 GMT+02:00 Steven D'Aprano :
>
>> On Wed, Oct 18, 2017 at 02:51:37PM +0200, Stefan Krah wrote:
>>
>> > $ softlimit -m 10 python3
>> [...]
>> > MemoryError
>> >
>> >
>> > People who are worried could make a python3 alias or use Ctrl-\.
>>
>> I just tried that on two different Linux computers I have, and neither
>> have softlimit.
>>
>>
> Yeah, not sure what "softlimit" is either.
> I'd suggest sticking to POSIX-standard ulimit or just stick
> something like this in the .pythonrc.py:
>
> import resource
> resource.setrlimit(resource.RLIMIT_DATA, (2 * 1024**3, 2 * 1024**3))
>
> Nor (presumably) would this help Windows users.
>>
>
> I (quickly) tried to get something to work using the win32 package,
> in particular the win32job functions.
> However, it seems setting
> "ProcessMemoryLimit" using win32job.SetInformationJobObject
> had no effect
> (i.e.  a subsequent win32job.QueryInformationJobObject
> still showed the limit as 0)?
>
> People with stronger Windows-fu may be aware what is going on here...
>
> Stephan
>

I wasn't aware Windows was capable of setting such limits in a per-process
fashion. You gave me a good idea for psutil:
https://github.com/giampaolo/psutil/issues/1149
According to this cmdline tool:
https://cr.yp.to/daemontools/softlimit.html
the limit should kick in only when the system memory is full, whatever
that means:
<<-r n: Limit the resident set size to n bytes. This limit is not enforced
unless physical memory is full.>>
...so that would explain why it had no effect.

-- 
Giampaolo - http://grodola.blogspot.com
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Add os.usable_cpu_count()

[Giampaolo Rodola']

On Tue, Sep 5, 2017 at 12:59 PM, Nathaniel Smith <n...@pobox.com> wrote:

> On Mon, Sep 4, 2017 at 8:59 PM, Giampaolo Rodola' <g.rod...@gmail.com>
> wrote:
> > Recently os.cpu_count() on Windows has been fixed in order to take
> process
> > groups into account and return the number of all available CPUs:
> > http://bugs.python.org/issue30581
> > This made me realize that os.cpu_count() does not return the number of
> > *usable* CPUs, which could possibly represent a better default value for
> > things like multiprocessing and process pools.
> > It is currently possible to retrieve this info on UNIX with
> > len(os.sched_getaffinity(0)) which takes CPU affinity and (I think) Linux
> > cgroups into account, but it's not possible to do the same on Windows
> which
> > provides this value via  GetActiveProcessorCount() API.
> > As such I am planning to implement this in psutil but would like to know
> how
> > python-ideas feels about adding a new os.usable_cpu_count() function (or
> > having os.cpu_count(usable=True)).
>
> This was discussed in https://bugs.python.org/issue23530
>
> It looks like the resolution at that time was:
>
> - os.cpu_count() should *not* report the number of CPUs accessible to
> the current process, but rather continue to report the number of CPUs
> that exist in the system (whatever that means in these days of
> virtualization... e.g. if you use KVM to set up a virtual machine with
> limited CPUs, then that changes os.cpu_count, but if you do it with
> docker then that doesn't).
>
> - multiprocessing and similar should continue to treat os.cpu_count()
> as if it returned the number of CPUs accessible to the current
> process.
>
> Possibly some lines got crossed there...
>
> -n
>
> --
> Nathaniel J. Smith -- https://vorpus.org
>

I agree current os.cpu_count() behavior should remain unchanged. Point is
multiprocessing & similar are currently not taking CPU affinity and Linux
cgroups into account, spawning more processes than necessary.

-- 
Giampaolo - http://grodola.blogspot.com
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Add os.usable_cpu_count()

[Giampaolo Rodola']

Recently os.cpu_count() on Windows has been fixed in order to take process
groups into account and return the number of all available CPUs:
http://bugs.python.org/issue30581
This made me realize that os.cpu_count() does not return the number of
*usable* CPUs, which could possibly represent a better default value for
things like multiprocessing and process pools.
It is currently possible to retrieve this info on UNIX with
len(os.sched_getaffinity(0)) which takes CPU affinity and (I think) Linux
cgroups into account, but it's not possible to do the same on Windows which
provides this value via  GetActiveProcessorCount() API.
As such I am planning to implement this in psutil but would like to know
how python-ideas feels about adding a new os.usable_cpu_count() function
(or having os.cpu_count(usable=True)).

Thoughts?

-- 
Giampaolo - http://grodola.blogspot.com
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] namedtuple literals [Was: RE a new namedtuple]

[Giampaolo Rodola']

On Thu, Jul 20, 2017 at 3:35 AM, Alexander Belopolsky <
alexander.belopol...@gmail.com> wrote:

> On Wed, Jul 19, 2017 at 9:08 PM, Guido van Rossum 
> wrote:
> > The proposal in your email seems incomplete
>
> The proposal does not say anything about type((x=1, y=2)).  I assume
> it will be the same as the type currently returned by namedtuple(?, 'x
> y'), but will these types be cached?


I suppose that the type should be immutable at least as long as field names
are the same, and the cache will occur on creation, in order to retain the
0 memory footprint.

Will type((x=1, y=2)) is type((x=3, y=4)) be True?.


Yes.


> Maybe type((x=1, y=2))(values) will work?
>

It's supposed to behave like a tuple or any other primitive type (list,
set, etc.), so yes.

> Regarding that spec, I think there's something missing: given a list (or
> tuple!) of values, how do you turn it into an 'ntuple'?

As already suggested, it probably makes sense to just reuse the dict syntax:

>>> dict([('a', 1), ('b', 2)])
{'a': 1, 'b': 2}
>>> ntuple([('a', 1), ('b', 2)])
(a=1, b=2)
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] namedtuple literals [Was: RE a new namedtuple]

[Giampaolo Rodola']

On Tue, Jul 25, 2017 at 9:30 PM, MRAB <pyt...@mrabarnett.plus.com> wrote:

> On 2017-07-25 19:48, Giampaolo Rodola' wrote:
>
>>
>> On Tue, Jul 25, 2017 at 7:49 PM, MRAB <pyt...@mrabarnett.plus.com
>> <mailto:pyt...@mrabarnett.plus.com>> wrote:
>>
>> On 2017-07-25 02:57, Nick Coghlan wrote:
>>
>> On 25 July 2017 at 02:46, Michel Desmoulin
>> <desmoulinmic...@gmail.com <mailto:desmoulinmic...@gmail.com>>
>> wrote:
>>
>> Le 24/07/2017 à 16:12, Nick Coghlan a écrit :
>>
>> On 22 July 2017 at 01:18, Guido van Rossum
>> <gu...@python.org <mailto:gu...@python.org>> wrote:
>>
>> Honestly I would like to declare the bare (x=1,
>> y=0) proposal dead. Let's
>> encourage the use of objects rather than tuples
>> (named or otherwise) for
>> most data exchanges. I know of a large codebase
>> that uses dicts instead of
>> objects, and it's a mess. I expect the bare ntuple
>> to encourage the same
>> chaos.
>>
>>
>> This is the people working on big code base talking.
>>
>>
>> Dedicated syntax:
>>
>>  (x=1, y=0)
>>
>> New builtin:
>>
>>  ntuple(x=1, y=0)
>>
>> So the only thing being ruled out is the dedicated syntax option,
>> since it doesn't let us do anything that a new builtin can't
>> do, it's
>> harder to find help on (as compared to "help(ntuple)" or searching
>> online for "python ntuple"), and it can't be readily backported to
>> Python 3.6 as part of a third party library (you can't easily
>> backport
>> it any further than that regardless, since you'd be missing the
>> order-preservation guarantee for the keyword arguments passed
>> to the
>> builtin).
>>
>> [snip]
>>
>> I think it's a little like function arguments.
>>
>> Arguments can be all positional, but you have to decide in what
>> order they are listed. Named arguments are clearer than positional
>> arguments when calling functions.
>>
>> So an ntuple would be like a tuple, but with names (attributes)
>> instead of positions.
>>
>> I don't see how they could be compatible with tuples because the
>> positions aren't fixed. You would need a NamedTuple where the type
>> specifies the order.
>>
>> I think...
>>
>>
>> Most likely ntuple() will require keyword args only, whereas for
>> collections.namedtuple they are mandatory only during declaration. The
>> order is the same as kwargs, so:
>>
>> >>> nt = ntuple(x=1, y=2)
>> >>> nt[0]
>> 1
>> >>> nt[1]
>> 2
>>
>> What's less clear is how isinstance() should behave. Perhaps:
>>
>> >>> t = (1, 2)
>> >>> nt = ntuple(x=1, y=2)
>> >>> isinstance(nt, tuple)
>> True
>> >>> isinstance(t, ntuple)
>> False
>>
>
> Given:
>
> >>> nt = ntuple(x=1, y=2)
>
> you have nt[0] == 1 because that's the order of the args.
>
> But what about:
>
> >>> nt2 = ntuple(y=2, x=1)
>
> ? Does that mean that nt[0] == 2? Presumably, yes.


> Does nt == nt2?
>
> If it's False, then you've lost some of the advantage of using names
> instead of positions.
>
> It's a little like saying that functions can be called with keyword
> arguments, but the order of those arguments still matters!


Mmmm excellent point. I would expect "nt == nt2" to be True because
collections.namedtuple() final instance works like that (compares pure
values), because at the end of the day it's a tuple subclass and so should
be ntuple() (meaning I expect "isinstance(ntuple(x=1, y=2), tuple)" to be
True).

On the other hand it's also legitimate to expect "nt == nt2" to be False
because field names are different. That would be made clear in the doc, but
the fact that people will have to look it up means it's not obvious.

-- 
Giampaolo - http://grodola.blogspot.com
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] namedtuple literals [Was: RE a new namedtuple]

[Giampaolo Rodola']

On Thu, Jul 20, 2017 at 2:14 AM, Giampaolo Rodola' <g.rod...@gmail.com>
wrote
>
> In case of one module scripts it's not uncommon to add a leading
> underscore which makes __repr__ uglier.
>

Actually forget about this: __repr__ is dictated by the first argument. =)

-- 
Giampaolo - http://grodola.blogspot.com
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] namedtuple literals [Was: RE a new namedtuple]

[Giampaolo Rodola']

On Tue, Jul 18, 2017 at 6:31 AM, Guido van Rossum  wrote:

> On Mon, Jul 17, 2017 at 6:25 PM, Eric Snow 
>  wrote:
>
>> On Mon, Jul 17, 2017 at 6:01 PM, Ethan Furman  wrote:
>> > Guido has decreed that namedtuple shall be reimplemented with speed in
>> mind.
>>
>> FWIW, I'm sure that any changes to namedtuple will be kept as minimal
>> as possible.  Changes would be limited to the underlying
>> implementation, and would not include the namedtuple() signature, or
>> using metaclasses, etc.  However, I don't presume to speak for Guido
>> or Raymond. :)
>>
>
> Indeed. I referred people here for discussion of ideas like this:
>
> >>> a = (x=1, y=0)
>

Thanks for bringing this up, I'm gonna summarize my idea in form of a
PEP-like draft, hoping to collect some feedback.

Proposal


Introduction of a new syntax and builtin function to create lightweight
namedtuples "on the fly" as in:

>>> (x=10, y=20)
(x=10, y=20)

>>> ntuple(x=10, y=20)
(x=10, y=20)


Motivations
===

Avoid declaration
-

Other than the startup time cost:
https://mail.python.org/pipermail/python-dev/2017-July/148592.html
...the fact that namedtuples need to be declared upfront implies they
mostly end up being used only in public, end-user APIs / functions. For
generic functions returning more than 1 argument it would be nice to just
do:

def get_coordinates():
return (x=10, y=20)

...instead of:

from collections import namedtuple

Coordinates = namedtuple('coordinates', ['x', 'y'])

def get_coordinates():
return Coordinates(10, 20)

Declaration also has the drawback of unnecessarily polluting the module API
with an object (Coordinates) which is rarely needed. AFAIU namedtuple was
designed this way for efficiency of the pure-python implementation
currently in place and for serialization purposes (e.g. pickle), but I may
be missing something else. Generally namedtuples are declared in a private
module, imported from elsewhere and they are never exposed in the main
namespace, which is kind of annoying. In case of one module scripts it's
not uncommon to add a leading underscore which makes __repr__ uglier. To
me, this suggests that the factory function should have been a first-class
function instead.

Speed
--

Other than the startup declaration overhead, a namedtuple is slower than a
tuple or a C structseq in almost any aspect:

- Declaration (50x slower than cnamedtuple):

$ python3.7 -m timeit -s "from collections import namedtuple" \
"namedtuple('Point', ('x', 'y'))"
1000 loops, best of 5: 264 usec per loop

$ python3.7 -m timeit -s "from cnamedtuple import namedtuple" \
"namedtuple('Point', ('x', 'y'))"
5 loops, best of 5: 5.27 usec per loop

- Instantiation (3.5x slower than tuple):

$ python3.7 -m timeit -s "import collections; Point =
collections.namedtuple('Point', ('x', 'y')); x = [1, 2]" "Point(*x)"
100 loops, best of 5: 310 nsec per loop

$ python3.7 -m timeit -s "x = [1, 2]" "tuple(x)"
500 loops, best of 5: 88 nsec per loop

- Unpacking (2.8x slower than tuple):

$ python3.7 -m timeit -s "import collections; p =
collections.namedtuple( \
'Point', ('x', 'y'))(5, 11)" "x, y = p"
500 loops, best of 5: 41.9 nsec per loop

$ python3.7 -m timeit -s "p = (5, 11)" "x, y = p"
2000 loops, best of 5: 14.8 nsec per loop

- Field access by name (1.9x slower than structseq and cnamedtuple):

$ python3.7 -m timeit -s "from collections import namedtuple as nt; \
p = nt('Point', ('x', 'y'))(5, 11)" "p.x"
500 loops, best of 5: 42.7 nsec per loop

$ python3.7 -m timeit -s "from cnamedtuple import namedtuple as nt; \
p = nt('Point', ('x', 'y'))(5, 11)" "p.x"
1000 loops, best of 5: 22.5 nsec per loop

$ python3.7 -m timeit -s "import os; p = os.times()" "p.user"
1000 loops, best of 5: 22.6 nsec per loop

- Field access by index is the same as tuple:

$ python3.7 -m timeit -s "from collections import namedtuple as nt; \
p = nt('Point', ('x', 'y'))(5, 11)" "p[0]"
1000 loops, best of 5: 20.3 nsec per loop

$ python3.7 -m timeit -s "p = (5, 11)" "p[0]"
1000 loops, best of 5: 20.5 nsec per loop

It is being suggested that most of these complaints about speed aren't an
issue but in certain circumstances such as busy loops, getattr() being 1.9x
slower could make a difference, e.g.:
https://github.com/python/cpython/blob/3e2ad8ec61a322370a6fbdfb2209cf74546f5e08/Lib/asyncio/selector_events.py#L523
Same goes for values unpacking.

isinstance()


Probably a minor complaint, I just bring this up because I recently had to
do this in psutil's unit tests. Anyway, checking a namedtuple instance
isn't exactly straightforward:
https://stackoverflow.com/a/2166841

Backward compatibility
==

This is probably the biggest 

Re: [Python-ideas] namedtuple with ordereddict

[Giampaolo Rodola']

On Wed, Jul 19, 2017 at 3:27 AM, Steven D'Aprano 
wrote:

> On Tue, Jul 18, 2017 at 06:16:26PM -0400, Jim J. Jewett wrote:
> > Then constructing a specific instance from the arguments used to
> > create it could be as simple as keeping a reference to the temporary
> > created to pass those arguments...
>
> The bottleneck isn't creating the instances themselves, the expensive
> part is calling namedtuple() to generate the named tuple CLASS itself.
> Creating the instances themselves should be fast, they're just tuples.
>

Still much slower (-4.3x) than plain tuples though:

$ python3.7 -m timeit -s "import collections; Point =
collections.namedtuple('Point', ('x', 'y'));" "Point(5, 11)"
100 loops, best of 5: 313 nsec per loop

$ python3.7 -m timeit "tuple((5, 11))"
500 loops, best of 5: 71.4 nsec per loop

Giampaolo - http://grodola.blogspot.com
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] π = math.pi

[Giampaolo Rodola']

On Thu, Jun 1, 2017 at 8:47 AM, Serhiy Storchaka 
wrote:

> What you are think about adding Unicode aliases for some mathematic names
> in the math module? ;-)
>
> math.π = math.pi
> math.τ = math.tau
> math.Γ = math.gamma
> math.ℯ = math.e
>
> Unfortunately we can't use ∞, ∑ and √ as identifiers. :-(
>
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>

* it duplicates functionality

* I have no idea how to write those chars on Linux; if I did, I'm not sure
it'd be the same on OSX and Windows (probably not)

* duplicated aliases might make sense if they add readability; in this case
they don't unless (maybe) you have a mathematical background. I can infer
what "math.gamma" stands for but not being a mathematician math.Γ makes
absolutely zero sense to me.

* if you really want to do that you can simply do "from math import gamma
 as Γ" but it's something I wouldn't like if I were to read your code

* I generally dislike any non-ASCII API; the fact that Python 3 allows you
to do that should not be an incentive to promote such habit in the stdlib
or anywhere else except in the end-user code, and it's something I still
wouldn't like it except if in comments or docstrings

-1

-- 
Giampaolo - http://grodola.blogspot.com
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Add socket utilities for IPv4/6 dual-stack servers

[Giampaolo Rodola']

Some years ago I started working on a patch for the socket module which
added a couple of utility functions for being able to easily create a
server socket, with the addition of being able to accept both IPv4 and IPv6
connections (as a single socket):
https://bugs.python.org/issue17561

Given that not all OSes (Windows, many UNIXes) support this natively, I
later submitted a recipe adding a "multiple socket listener" class.
https://code.activestate.com/recipes/578504-server-supporting-ipv4-and-ipv6/

>From the user perspective, the whole thing can be summarized as follows:

>>> sock = create_server_sock(("", 8000))
>>> if not has_dual_stack(sock):
... sock.close()
... sock = MultipleSocketsListener([("0.0.0.0", 8000), ("::", 8000)])
>>>

Part of this work ended up being included into Tulip internals:
https://github.com/python/cpython/blob/70d28a184c42d107cc8c69a95aa52a4469e7929c/Lib/asyncio/base_events.py#L966-L1067
...and after that I basically forgot about the original patch. The other
day I bumped into a case where I needed exactly this (on Windows), so here
I am, trying to revamp the original proposal.

To be clear, the proposal is to add 3 new APIs in order to avoid the
low-level cruft needed when creating a server socket
(SO_REUSEADDR, getaddrinfo() etc.)
and being able to support IPv4/6 dual-stack socket servers in a
cross-platform fashion:

- socket.has_dual_stack()
- socket.create_server_sock()
- socket.MultipleSocketsListener

Whereas the first two functions are relatively straightforward,
MultipleSocketsListener is more debatable because, for instance, it's not
clear what methods like getsockname() should return (because there are 2
sockets involved). One possible solution is to *not* expose such (all
get*?) methods and simply expose the underlying socket objects as in:

>>> socket.MultipleSocketsListener(...).socks[0].getsockname()

On the other hand, all set* / write methods (setblocking(), setsockopt(),
shutdown(), ...) can be exposed and internally they can simply operate
against both sockets.

Thoughts?

-- 
Giampaolo - http://grodola.blogspot.com
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Context manager to temporarily set signal handlers

[Giampaolo Rodola']

I don't know if this is related (regarding functions registered in C) but
one problem I often have is to always execute exit functions. I have come
up with this:
http://grodola.blogspot.com/2016/02/how-to-always-execute-exit-functions-in-py.html


On Fri, Jan 20, 2017 at 1:46 PM, Thomas Kluyver 
wrote:

> Not uncommonly, I want to do something like this in code:
>
> import signal
>
> # Install my own signal handler
> prev_hup = signal.signal(signal.SIGHUP, my_handler)
> prev_term = signal.signal(signal.SIGTERM, my_handler)
> try:
> do_something_else()
> finally:
> # Restore previous signal handlers
> signal.signal(signal.SIGHUP, prev_hup)
> signal.signal(signal.SIGTERM, prev_term)
>
> This works if the existing signal handler is a Python function, or the
> special values SIG_IGN (ignore) or SIG_DFL (default). However, it breaks
> if code has set a signal handler in C: this is not returned, and there
> is no way in Python to reinstate a C-level signal handler once we've
> replaced it from Python.
>
> I propose two possible solutions:
>
> 1. The high-level approach: a context manager which can temporarily set
> one or more signal handlers. If this was implemented in C, it could
> restore C-level as well as Python-level signal handlers.
>
> 2. A lower level approach: signal() and getsignal() would gain the
> ability to return an opaque object which refers to a C-level signal
> handler. The only use for this would be to pass it back to
> signal.signal() to set it as a signal handler again. The context manager
> from (1) could then be implemented in Python.
>
> Crosslinking http://bugs.python.org/issue13285
>
> Thomas
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>



-- 
Giampaolo - http://grodola.blogspot.com
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] api suggestions for the cProfile module

[Giampaolo Rodola']

On Wed, Dec 21, 2016 at 1:50 AM, Thane Brimhall 
wrote:

> I use cProfile a lot, and would like to suggest three backwards-compatible
> improvements to the API.
>
> 1: When using cProfile on a specific piece of code I often use the
> enable() and disable() methods. It occurred to me that this would be an
> obvious place to use a context manager.
>

I think this makes sense.
I did that in https://bugs.python.org/issue9285 but unfortunately I got
stuck and the issue remained stagnant.
Signaling it here just in case somebody has some insights on how to proceed.



> 2: Enhance the `print_stats` method on Profile to accept more options
> currently available only through the pstats.Stats class. For example,
> strip_dirs could be a boolean argument, and limit could accept an int. This
> would reduce the number of cases you'd need to use the more complex API.
>

I'm not sure about this. I agree the current API is not the nicest one. I
use a wrapper on top of cProfile which does this:

stats = pstats.Stats(file.name)
if strip_dirs:
stats.strip_dirs()
if isinstance(sort, (tuple, list)):
stats.sort_stats(*sort)
else:
stats.sort_stats(sort)
stats.print_stats(lines)

With your proposal we would have 2 ways of doing the same thing and I'm not
entirely sure that is good.



3: I often forget which string keys are available for sorting. It would be
> nice to add an enum for these so a user could have their linter and IDE
> check that value pre-runtime. Since it would subclass `str` and `Enum` it
> would still work with all currently existing code.
>
> The current documentation contains the following code:
>
> import cProfile, pstats, io
> pr = cProfile.Profile()
> pr.enable()
> # ... do something ...
> pr.disable()
> s = io.StringIO()
> sortby = 'cumulative'
> ps = pstats.Stats(pr, stream=s).sort_stats(sortby)
> ps.print_stats()
> print(s.getvalue())
>
> While the code below doesn't exactly match the functionality above (eg.
> not using StringIO), I envision the context manager working like this,
> along with some adjustments on how to get the stats from the profiler:
>
> import cProfile, pstats
> with cProfile.Profile() as pr:
> # ... do something ...
> pr.print_stats(sort=pstats.Sort.cumulative, limit=10, strip_dirs=True)
>
> As you can see, the code is shorter and somewhat more self-documenting.
> The best thing about these suggestions is that as far as I can tell they
> would be backwards-compatible API additions.
>
> What do you think? Thank you in advance for your time!
>
> /Thane
>
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>



-- 
Giampaolo - http://grodola.blogspot.com
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Small improvements to the profile/cProfile API

[Giampaolo Rodola']

Long ago I posted a patch for this (decorator + context manager) but I
bumped into a weird error I wasn't able to fix (see last comment):
http://bugs.python.org/issue9285

On Wed, Nov 2, 2016 at 10:45 PM, Tim Mitchell 
wrote:

> Hi Ben,
>
> Mostly I just print to stdout, I imagine more flexibility would be needed
> in general.
>
> This is for python 2.7 - don't know if it works for 3.
>
>
> def profile(sort='time', restriction=(), callers=None, callees=None, 
> filename=None):
> def _profileDecorator(func):
> "print profile stats for decorated function"
> def wrapper(*args, **kwargs):
> print 'Profile for:', func.__name__
>
> prof = cProfile.Profile()
> result = prof.runcall(func, *args, **kwargs)
> _, statsFileName = tempfile.mkstemp()
> prof.dump_stats(statsFileName)
> if filename is None:
> stats = pstats.Stats(statsFileName)
> else:
> stats = pstats.Stats(statsFileName, stream=open(filename, 
> 'w'))
> if isinstance(sort, basestring):
> stats.sort_stats(sort)
> else:
> stats.sort_stats(*sort)
> if isinstance(restriction, (tuple, list)):
> stats.print_stats(*restriction)
> else:
> stats.print_stats(restriction)
> if callers is not None:
> if isinstance(callers, basestring):
> stats.print_callers(callers)
> else:
> stats.print_callers(*callers)
> if callees is not None:
> if isinstance(callees, basestring):
> stats.print_callees(callees)
> else:
> stats.print_callees(*callees)
> return result
> return wrapper
> return _profileDecorator
>
> Cheers
>
> Tim
>
>
> On 3 November 2016 at 09:58, Ben Hoyt  wrote:
>
>> Okay, got it, that sounds fair enough. With your @profile decorator how
>> do you tell it when and where to print the output? Can you post the source
>> for your decorator?
>>
>> On Wed, Nov 2, 2016 at 4:52 PM, Tim Mitchell > > wrote:
>>
>>> I use an @profile() decorator for almost all my profiling.  If you want
>>> to profile function foo you just decorate it and re-run the program.
>>> With a with block you have to find the places where foo is called and
>>> put with statements around the calls.
>>> I think both approaches are equally valid and useful.
>>>
 conduct/ 

>>>
>>>
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>



-- 
Giampaolo - http://grodola.blogspot.com
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Flagging blocking functions not to be used with asyncio

[Giampaolo Rodola']

On Fri, Oct 7, 2016 at 6:52 PM, Yury Selivanov 
wrote:

> On 2016-10-07 11:16 AM, Guido van Rossum wrote:
>
> Maybe a simpler approach would be to write a linter that checks for a
>> known list of common blocking functions, and anything that calls those
>> automatically gets the same property?
>>
>
> What if somebody uses logging module and logs to a file?  I think this is
> something that linters can't infer (how logging is configured).
>
> One way to solve this would be to monkeypatch the io and os modules
> (gevent does that, so it's possible) to issue a warning when it's used in
> an asyncio context.  This can be done as a module on PyPI.
>
> Another way would be to add some kind of IO tracing hooks to CPython.


How about something like this?
http://www.tornadoweb.org/en/stable/ioloop.html#tornado.ioloop.IOLoop.set_blocking_signal_threshold



-- 
Giampaolo - http://grodola.blogspot.com
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] async objects

[Giampaolo Rodola']

Independently from what the proposed solution is, I think you raised a very
valid concern: the DRY principle.
Right now the stdlib has tons of client network libraries which do not
support the new async model.
As such, library vendors will have to rewrite them by using the new syntax
and provide an "aftplib", "ahttplib" etc. and release them as third-party
libs hosted on PYPI. This trend is already happening as we speak:
https://github.com/python/asyncio/wiki/ThirdParty#clients
It would be awesome if somehow the Python stdlib itself would provide some
mechanism to make the existent "batteries" able to run asynchronously so
that, say, ftplib or httplib can be used with asyncio as the base IO loop
and at the same time maintain the same existent API.
Gevent tried to do the same thing with
http://www.gevent.org/gevent.monkey.html
As for *how* to do that, I'm sorry to say that I really have no idea. It's
a complicated issue, but I think it's good that this has been raised.


On Sun, Oct 2, 2016 at 3:26 PM, Rene Nejsum  wrote:

> Having followed Yury Selivanov yselivanov.ml at gmail.com proposal to add
> async/await to Python  (PEP 492 Coroutines with async and await syntax and
> (PEP 525  Asynchronous Generators) and and especially the discussion about
> PEP 530: Asynchronous Comprehensions  I would like to add some concerns
> about the direction Python is taking on this.
>
> As Sven R. Kunze srkunze at mail.de mentions the is a risk of having to
> double a lot of methods/functions to have an Async implementation. Just
> look at the mess in .NET  when Microsoft introduced async/await in their
> library, a huge number of functions had to be implemented with a Async
> version of each member. Definitely not the DRY principle.
>
> While I think parallelism and concurrency are very important features in a
> language, I feel the direction Python is taking right now is getting to
> complicated, being difficult to understand and implement correct.
>
> I thought it might be worth to look at using async at a higher level.
> Instead of making methods, generators and lists async, why not make the
> object itself async? Meaning that the method call (message to object) is
> async
>
> Example:
>
> class SomeClass(object):
>def some_method(self):
>return 42
>
> o = async SomeClass()  # Indicating that the user want’s an async version
> of the object
> r = o.some_method()# Will implicit be a async/await “wrapped” method
> no matter impl.
> # Here other code could execute, until the result (r) is referenced
> print r
>
> I think above code is easier to implement, use and understand, while it
> handles some of the use cases handled by defining a lot of methods as
> async/await.
>
> I have made a small implementation called PYWORKS (
> https://github.com/pylots/pyworks), somewhat based on the idea above.
> PYWORKS has been used in several real world implementation and seams to be
> fairly easy for developers to understand and use.
>
> br
> /Rene
>
> PS. This is my first post to python-ideas, please be gentle :-)
>
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>



-- 
Giampaolo - http://grodola.blogspot.com
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/