[Python-Dev] Re: Should set objects maintain insertion order too?

[Paddy McCarthy]

Hi Larry, sets (and mappings, ie dicts), have a common functionality among
many languages and libraries that does Not include an ordering. Already, in
CPython, there is a need to somehow indicate that insertion ordering is
being used in dicts or use OrderedDict. I am quite happy with keeping sets
as they are and coding any extra functionality required on top of this.
This aids me as I work in many languages that have their own sets and
mappings, usually without any insertion ordering.

Cheers, Paddy.


On Sun, Dec 29, 2019, 12:07 AM Larry Hastings  wrote:

>
> On 12/27/19 7:44 PM, Tim Peters wrote:
>
> [Nick Coghlan  ]
>
> I took Larry's request a slightly different way: he has a use case where
> he wants order preservation (so built in sets aren't good), but combined
> with low cost duplicate identification and elimination and removal of
> arbitrary elements (so lists and collections.deque aren't good). Organising
> a work queue that way seems common enough ...
>
>
> Is it?  I confess I haven't thought of a plausible use case.  Larry
> didn't really explain his problem, just suggested that an ordered set
> would be "a solution" to it.
>
>
> Here is the original description of my problem, from the original email in
> this thread.  I considered this an adequate explanation of my problem at
> the time.
>
>
> On 12/15/19 6:48 PM, Larry Hastings wrote:
>
> I do have a use case for this. In one project I maintain a "ready" list of
> jobs; I need to iterate over it, but I also want fast lookup because I
> soemtimes remove jobs when they're subsequently marked "not ready".  The
> existing set object would work fine here, except that it doesn't maintain
> insertion order.  That means multiple runs of the program with the same
> inputs may result in running jobs in different orders, and this instability
> makes debugging more difficult.  I've therefore switched from a set to a
> dict with all keys mapped to None, which provides all the set-like
> functionality I need.
>
>
> In subsequent emails I've clarified that my workloads are small
> enough--and computers are fast enough--that almost any data structure would
> work fine for me here, e.g. a list.  I don't think my needs should drive
> the decision making process regardless.  I only described my problem to get
> the conversation rolling.
>
> I opine that anybody who iterates over set objects and has bugs in their
> code would appreciate set objects maintaining insertion order.  I suspect
> it would make their debugging easier, because given identical inputs their
> set iteration would behave identically, thus making their bugs that much
> more stable.  That's as much use case as I have for the feature.
>
>
> */arry*
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/XWTP7OD4UMGL7DLVABM3S2CB26LN5RBP/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/AETTJXQYXYKFZOWT2CMGC6N6DAXWUCGM/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Please be more precise when commenting on PEP 611.

[Paddy McCarthy]

Mark wrote, in reply to me:

> On 09/12/2019 3:01 pm, Paddy McCarthy wrote:
> > "Bear in mind that the costs of higher limits are paid by everyone, but
> > the benefits are gained by few."
> >
> > Is there some evidence for the above statement? One of the issues with C
> > programming is the need for limits Its hard to choose the right limits
> > leading to so many failures. You propose limiting Python for performance
> > gains but loose in ease of use and security.
>
> How does this make anything less secure?
>
Many basic exploits have components taking advantage of limits known as
buffer overflows

Suppose we can improve the interpreter in such a way that it takes
> advantage of the one million bytecode per code object and speeds it up
> by 0.1% (I can do a *lot* better than that).
> What would the performance gain save globally? $100k, $1M p.a.?
> What does it cost? A few developer hours.


You are looking to unproven execution gains. Adding an arbitrary limit
might make it 0.1% faster to get the *wrong *result .
Optimisation isn't just speedThe less mind-clutter spent on such* micro
optimisations* might leave one able to think of a better algorithm
or a better data structure that could give thousands of % speedups.
You should only quote C or C++ compiler limitations as a warning of what *not
*to do. Python wins by being different; by being more
orthoganal; by having *less *arbitrary restrictions.

Case in point:
http://paddy3118.blogspot.com/2019/11/quasi-random-sobol-sequence.html
Itook decade old C++ library, converted it
to straight python, Python allowed me to see the opportunity for
improvement leading to a Python version that is half the speed of
the C++ code and generates points on demand. That's how Python wins big
time. Having to worry about stifling limits is the wrong
direction for the language to take.

On Mon, 9 Dec 2019 at 21:26, David Mertz  wrote:

> I think a much more sensible approach than mandating a limit because "who
> knows, it might speed something up" would be finding the speedup first.
>
> Probably that means one limit at a time too. E.g. maybe some patch imposes
> the 1 million LOC limit and demonstrates a repeatable benchmark improvement
> because of some coffee change that allows. That cold be interesting.
>
> Even them, I wouldn't want some arbitrary round number just for its own
> sake. For example, if taking 10 bits away from a word that holds a LOC
> index speeds something up, make the LOC limit 4,194,304 (2**22)... Maybe.
> If you only need 9 bits for that use, make the limit twice as much.
>
> On Mon, Dec 9, 2019, 12:53 PM Mark Shannon  wrote:
>
>>
>> On 09/12/2019 2:15 pm, Chris Angelico wrote:
>> > On Tue, Dec 10, 2019 at 1:09 AM Mark Shannon  wrote:
>> >> Bear in mind that the costs of higher limits are paid by everyone, but
>> >> the benefits are gained by few.
>> >
>> > Can we get some stats on what the costs of higher limits (or having no
>> > limit at all) is? Or, putting it the other way: Since CPython
>> > currently does not enforce limits, what are the benefits of placing
>> > those limits? Merely having a limit doesn't in itself give a benefit
>> > (and is a (minor) cost); it would help the discussion if we knew
>> > exactly WHAT the costs of the higher limits were.
>>
>> Given there is an infinite number of potential optimizations that it
>> would enable, it is a bit hard to put a number on it :)
>>
>> It is also impossible to put precise numbers on the speedups of a
>> particular optimizations unless it is implemented. I suspect no one is
>> going to do that unless paid to do so, or are guaranteed that the work
>> won't be thrown away because the PEP is rejected.
>>
>> Cheers,
>> Mark.
>>
>> >
>> > ChrisA
>> > ___
>> > Python-Dev mailing list -- python-dev@python.org
>> > To unsubscribe send an email to python-dev-le...@python.org
>> > https://mail.python.org/mailman3/lists/python-dev.python.org/
>> > Message archived at
>> https://mail.python.org/archives/list/python-dev@python.org/message/4IKBEMNGC5DZXOAR555ZUGNJGWSFV3QI/
>> > Code of Conduct: http://python.org/psf/codeofconduct/
>> >
>> ___
>> Python-Dev mailing list -- python-dev@python.org
>> To unsubscribe send an email to python-dev-le...@python.org
>> https://mail.python.org/mailman3/lists/python-dev.python.org/
>> Message archived at
>> https://mail.python.org/archives/list/python-dev@python.org/message/BWTOEHKYZ3NES4XPZA7QA57UHTVGRMQZ/
>> Code of Conduct: http://pyt

[Python-Dev] Re: PEP proposal to limit various aspects of a Python program to one million.

[Paddy McCarthy]

One aspect of scripting is being able to throw something together to create
a correct solution to an immediate problem. If the proprietary software
that you script around takes over 300 Gigs to lay out a CPU and delays are
hugely expensive, then I don't want to waste time on optimisations to get
around arbitrary limits.
It sounds analogous to wanting to move to X bit integers to save a little
here and there.
If you are thinking of making limits then you might think of what limit
say, 4 terabytes of Ram would impose on the smallest object and propose
that.
Remember, 2020 means trials of 2 chip, 256 thread AMD servers with 500Gigs
of ram. Now if I can get Python sub interpreters working on that! Bliss. 

On Fri, Dec 6, 2019, 9:37 AM Steven D'Aprano  wrote:

> Although I am cautiously and tentatively in favour of setting limits
> if the benefits Mark suggests are correct, I have thought of at least
> one case where a million classes may not be enough.
>
> I've seen people write code like this:
>
> for attributes in list_of_attributes:
> obj = namedtuple("Spam", "fe fi fo fum")(*attributes)
> values.append(obj)
>
>
> not realising that every obj is a singleton instance of a unique class.
> They might end up with a million dynamically created classes, each with
> a single instance, when what they wanted was a single class with a
> million instances.
>
> Could there be people doing this deliberately? If so, it must be nice
> to have so much RAM that we can afford to waste it so prodigiously: a
> namedtuple with ten items uses 64 bytes, but the associated class uses
> 444 bytes, plus the sizes of the methods etc. But I suppose there could
> be a justification for such a design.
>
> (Quoted sizes on my system running 3.5; YMMV.)
>
>
>
> --
> Steven
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/VIK7QKORCYRJOF5EQZGYBNE6L62J5M6L/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/Q4OSA2RR45KK2C2KFDZWOLZZ7FZXMZHK/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP proposal to limit various aspects of a Python program to one million.

[Paddy McCarthy]

I would not have such a small limit.
I can envisage generating code from a log then evaluating that code. 1
million lines could be small, given the speed of the interpreter on modern
machines.
One might want to generate data as a Python file rather than a pile and
load that as a module. There might well be alternate methods, but if it
works and is quickly debugged at small scale, then you put a possible extra
barrier in the way.
Python is still a scripting language, why limit a "quick hack".

On Tue, Dec 3, 2019, 4:22 PM Mark Shannon  wrote:

> Hi Everyone,
>
> I am proposing a new PEP, still in draft form, to impose a limit of one
> million on various aspects of Python programs, such as the lines of code
> per module.
>
> Any thoughts or feedback?
>
> The PEP:
> https://github.com/markshannon/peps/blob/one-million/pep-100.rst
>
> Cheers,
> Mark.
>
>
> Full text
> *
>
> PEP: 100
> Title: The one million limit
> Author: Mark Shannon 
> Status: Active
> Type: Enhancement
> Content-Type: text/x-rst
> Created: 03-Dec-2019
> Post-History:
>
>
>
> Abstract
> 
> This PR proposes a limit of one million (1 000 000) for various aspects
> of Python code and its implementation.
>
> The Python language does not specify limits for many of its features.
> Not having any limit to these values seems to enhance programmer freedom,
> at least superficially, but in practice the CPython VM and other Python
> virtual
> machines have implicit limits or are forced to assume that the limits are
> astronomical, which is expensive.
>
> This PR lists a number of features which are to have a limit of one
> million.
> If a language feature is not listed but appears unlimited and must be
> finite,
> for physical reasons if no other, then a limit of one million should be
> assumed.
>
>
> Motivation
> ==
>
> There are many values that need to be represented in a virtual machine.
> If no limit is specified for these values, then the representation must
> either be inefficient or vulnerable to overflow.
> The CPython virtual machine represents values like line numbers,
> stack offsets and instruction offsets by 32 bit values. This is
> inefficient, and potentially unsafe.
>
> It is inefficient as actual values rarely need more than a dozen or so
> bits to represent them.
>
> It is unsafe as malicious or poorly generated code could cause values to
> exceed 2\ :sup:`32`.
>
> For example, line numbers are represented by 32 bit values internally.
> This is inefficient, given that modules almost never exceed a few
> thousand lines.
> Despite being inefficent, is is still vulnerable to overflow as
> it is easy for an attacker to created a module with billions of newline
> characters.
>
> Memory access is usually a limiting factor in the performance of modern
> CPUs.
> Better packing of data structures enhances locality and reduces memory
> bandwith,
> at a modest increase in ALU usage (for shifting and masking).
> Being able to safely store important values in 20 bits would allow
> memory savings
> in several data structures including, but not limited to:
>
> * Frame objects
> * Object headers
> * Code objects
>
> There is also the potential for a more efficient instruction format,
> speeding up interpreter dispatch.
>
> Rationale
> =
>
> Imposing a limit on values such as lines of code in a module, and the
> number of local variables,
> has significant advantages for ease of implementation and efficiency of
> virtual machines.
> If the limit is sufficiently large, there is no adverse effect on users
> of the language.
>
> By selecting a fixed but large limit for these values,
> it is possible to have both safety and efficiency whilst causing no
> inconvience to human programmers
> and only very rare problems for code generators.
>
> One million
> ---
>
> The Java Virtual Machine (JVM) [1]_ specifies a limit of 2\ :sup:`16`-1
> (65535) for many program
> elements similar to those covered here.
> This limit enables limited values to fit in 16 bits, which is a very
> efficient machine representation.
> However, this limit is quite easily exceeded in practice by code
> generators and
> the author is aware of existing Python code that already exceeds 2\
> :sup:`16` lines of code.
>
> A limit of one million fits into 20 bits which, although not as
> convenient for machine representation,
> is still reasonably compact. Three signed valuses in the range -1000_000
> to +1000_000 can fit into a 64 bit word.
> A limit of one million is small enough for efficiency advantages (only
> 20 bits),
> but large enough not to impact users (no one has ever written a module
> of one million lines).
>
> The value "one million" is very easy to remember.
>
> Isn't this "640K ought to be enough for anybody" again?
> ---
>
> The infamous 640K memory limit was a limit on machine usable resources.
> The proposed one million limit is a limit on human generated code.
>
> While 

Re: [Python-Dev] Call for prudence about PEP-572

[Paddy McCarthy]

On Sat, 7 Jul 2018 at 16:50, Guido van Rossum  wrote:

> This seems more suitable for a style guide.
>

Which would be good enough, if the automatic checkers were improved  to
flag such poor use of ':='..

Would that be possible?
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 575 (Unifying function/method classes) update: limit?

[Paddy McCarthy]

On Tue, 8 May 2018, 16:33 Jeroen Demeyer,  wrote:

> On 2018-05-06 09:35, Nick Coghlan wrote:
> > Thanks for this update Jeroen! If it doesn't come up otherwise, I'll try
> > to claim one of the lightning talk slots at the Language Summit to
> > discuss this with folks in person :)
>
> Sounds great! I'd love to hear what people think.
>
> As an example of how the new functionality of PEP 575 can be used, I
> changed functools.lru_cache to implement the _lru_cache_wrapper class as
> subclass of base_function. I added this to the reference implementation
> https://github.com/jdemeyer/cpython/tree/pep575
> ___
>
It is not so much the use, but the abuse that I am worried about. If their
was a way to limit the complexity of expression allowed around the use of
the name assigned to by :=, then that would be a start.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Dealing with tone in an email

[Paddy McCarthy]

> > Whatever anyone may think of this, it worked.


I help on other forums and have two practises that I work at:

When asking a question I try to be polite. It may be more challenging to be
ultra polite but sometimes it's worthwhile. Being told I am polite online
is a much rarer accolade and can lift my day :-)

On those being brusk/abusive in asking for help then I don't like to reward
that behaviour and someone will tell them it's not necessary. If repeated
then the best one can do is ignore them leaving them knowing why.

Hopefully it leads to a better community. People have to understand **"Help
on the helpers terms".**
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 572: A backward step in readability

[Paddy McCarthy]

On 30 April 2018 at 17:37, Steven D'Aprano <st...@pearwood.info> wrote:
>
> On Mon, Apr 30, 2018 at 08:09:35AM +0100, Paddy McCarthy wrote:
> [...]
> > A PEP that can detract from readability; *readability*, a central
> > tenet of Python, should
> > be rejected, (on principle!), when such objections are treated so 
> > dismissively.

> Unless you have an objective measurement of readability, that objection
> is mere subjective personal preference, and not one that everyone agrees
> with.

True, as is the dismissal from the PEP. It is the PEP, looking to
force change to
the language, to prove its point rather than dismiss statements of its
detractors.

>
> The "not readable" objection has been made, extremely vehemently,
> against nearly all major syntax changes to Python:

I don't count myself as usually against change. I applaud the move to Python 3,
I use all of the language features you mention at times; many in my Rosetta Code
task examples; but this change opens the door to a class of bug that will
take care to avoid and  which I remember cutting my C coding to get away from.

The PEP fails to adequately address the concerns of "us naysayers"

For example; if someone were to find out that "assignment expressions were
faster", then you would be hard pressed to stop their over-use.
As soon as someone assigns to a name and uses that same name in the one
expression, you need a better grasp of the order of expression evaluation to
read it. hat is best avoided, in my subjective, personal, view.


> --
> Steve

With respect, but in disagreement - Paddy.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] PEP 572: A backward step in readability

[Paddy McCarthy]

The PEP s section on Frequently raised objections includes:

(https://www.python.org/dev/peps/pep-0572/#this-could-be-used-to-create-ugly-code)

> This could be used to create ugly code!
>
> So can anything else. This is a tool, and it is up to the programmer to use 
> it where
> it makes sense, and not use it where superior constructs can be used.

The above is so dismissive of a very real problem.

* Python makes syntax decisions to make it easier to read.
* One of the first, and enduring ones was "No  assignment within expressions".
* I had both seen, and debugged problems in the C-language caused by assignment
  expressions; and was glad to be rid of them in Python.

Assignment within expressions was, I think,  difficult to *teach* in
C. And will become
something error-prone to *learn* in Python.

The Python community has grown over the decades, and we have attracted community
members who want to help  and are motivated, but Python is not a
tick-list of features.

A PEP that can detract from readability; *readability*, a central
tenet of Python, should
be rejected, (on principle!), when such objections are treated so dismissively.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com