Re: [Python-ideas] Why is design-by-contracts not widely adopted?
On Wed, Sep 26, 2018 at 2:47 PM Marko Ristin-Kaufmann wrote: > > Hi Chris, > >> An extraordinary claim is like "DbC can improve *every single project* >> on PyPI". That requires a TON of proof. Obviously we won't quibble if >> you can only demonstrate that 99.95% of them can be improved, but you >> have to at least show that the bulk of them can. > > > I tried to give the "proof" (not a formal one, though) in my previous message. (Formal proof isn't necessary here; we say "extraordinary proof", but it'd be more accurate to say "extraordinary evidence".) > The assumptions are that: > * There are always contracts, they can be either implicit or explicit. You > need always to figure them out before you call a function or use its result. Not all code has such contracts. You could argue that code which does not is inferior to code which does, but not everything follows a strictly-definable pattern. > * Figuring out contracts by trial-and-error and reading the code (the > implementation or the test code) is time consuming and hard. Agreed. > * The are tools for formal contracts. That's the exact point you're trying to make, so it isn't evidence for itself. Tools for formal contracts exist as third party in Python, and if that were good enough for you, we wouldn't be discussing this. There are no such tools in the standard library or language that make formal contracts easy. > * The contracts written in documentation as human text inevitably rot and > they are much harder to maintain than automatically verified formal contracts. Agreed. > * The reader is familiar with formal statements, and hence reading formal > statements is faster than reading the code or trial-and-error. Disagreed. I would most certainly NOT assume that every reader knows any particular syntax for such contracts. However, this is a weaker point. So I'll give you two and two halves for that. Good enough to make do. > I then went on to show why I think, under these assumptions, that formal > contracts are superior as a documentation tool and hence beneficial. Do you > think that any of these assumptions are wrong? Is there a hole in my logical > reasoning presented in my previous message? I would be very grateful for any > pointers! > > If these assumptions hold and there is no mistake in my reasoning, wouldn't > that qualify as a proof? > It certainly qualifies as proof that SOME code MAY benefit from contracts. It does not reach the much higher bar to support the claim that "there are X projects on PyPI and every single one of them would benefit". For instance, would youtube-dl benefit from DbC? To most people, it's an application, not a library. Even if you're invoking it from within a Python script, it's usually easiest to use the main entrypoint rather than delve into its internals. Case in point: https://github.com/Rosuav/MegaClip/blob/master/megaclip.py#L71 It's actually easier to shell out to a subprocess than to call on youtube-dl as a library. Would DbC benefit youtube-dl's internal functions? Maybe, but that's for the youtube-dl people to decide; it wouldn't in the slightest benefit my app. You might argue that a large proportion of PyPI projects will be "library-style" packages, where the main purpose is to export a bunch of functions. But even then, I'm not certain that they'd all benefit from DbC. Some would, and you've definitely made the case for that; but I'm still -0.5 on adding anything of the sort to the stdlib, as I don't yet see that *enough* projects would actually benefit. People have said the same thing about type checking, too. Would *every* project on PyPI benefit from MyPy's type checks? No. Syntax for them was added, not because EVERYONE should use them, but because SOME will use them, and it's worth having some language support. You would probably do better to argue along those lines than to try to claim that every single project ought to be using contracts. ChrisA ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Why is design-by-contracts not widely adopted?
Hi Chris, It's easy to show beautiful examples that may actually depend on other > things. Whether that's representative of all contracts is another > question. I agree. There are also many contracts which are simply too hard to write down formally. But many are also easily captured in formal manner in my experience. The question is, of course, how many and you make a fair point there. @Chris and others requesting data: my time is way too limited to provide a large-scale code analysis of many pypi packages (family obligations with a toddler, 9-6 job). I'm not doing research, and such a study would require substantial time resources. Is there an alternative request that you think that I (and other volunteers?) could accomplish in a reasonable (free) time? Maybe you could compile a list of 100-200 (or even less) functions from representative modules and I try to annotate them with contracts and we see if that's convincing? It's up to you to pick representative functions and up to me to annotate them with contracts. That would diffuse the argument that I intentionally picked the functions whose contracts are easily and nice to annotate. Cheers, Marko On Wed, 26 Sep 2018 at 01:20, Chris Angelico wrote: > On Wed, Sep 26, 2018 at 7:59 AM Kyle Lahnakoski > wrote: > > I use DbC occasionally to clarify my thoughts during a refactoring, and > then only in the places that continue to make mistakes. In general, I am > not in a domain that benefits from DbC. > > > > Contracts are code: More code means more bugs. > > Contracts are executable documentation. If you can lift them directly > into user-readable documentation (and by "user" here I mean the user > of a library), they can save you the work of keeping your > documentation accurate. > > > This contract does not help me: > > > > What is_absolute()? is "file:///" absolute? > > I'd have to assume that is_absolute() is defined elsewhere. Which > means that the value of this contract depends entirely on having other > functions, probably ALSO contractually-defined, to explain it. > > > How does this code fail? > > What does a permission access problem look like? > > Probably an exception. This is Python code, and I would generally > assume that problems are reported as exceptions. > > > Can initial_paths can be None? > > This can be answered from the type declaration. It doesn't say > Optional, so no, it can't be None. > > > Can initial_paths be files? directories? > > Presumably not a question you'd get if you were actually using it; the > point of the function is to "[r]esolve the initial paths of the > dependency graph by recursively adding *.py files beneath given > directories", so you'd call it because you have directories and want > files back. > > > What are the side effects? > > Hopefully none, other than the normal implications of hitting the file > system. > > It's easy to show beautiful examples that may actually depend on other > things. Whether that's representative of all contracts is another > question. > > ChrisA > ___ > Python-ideas mailing list > Python-ideas@python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Why is design-by-contracts not widely adopted?
Hi Kyle, 6) The name of the method > 7) How the method is called throughout the codebase > 10) relying on convention inside, and outside, the application > Sorry, by formulating 2) as "docstring" I excluded names of the methods as well as variables. Please assume that 2) actually entails those as well. They are human text and hence not automatically verifiable, hence qualify as 2). 8) observing input and output values during debugging > 9) observing input and output values in production > Sorry, again I implicitly subsumed 8-9 under 4), reading the implementation code (including the trial-and-error). My assumption was that it is incomparably more costly to apply trial-and-error than read the contracts given that contracts can be formulated. Of course, not all contracts can be formulated all the time. 11) Don't communicate - Sometimes / is too high; > code is not repaired, only replaced. > I don't see this as an option for any publicly available, high-quality module on pypi or in any organization. As I already noted in my message to Hugh, the argument in favor of* undocumented and/or untested code* are not the arguments. I assume we want a *maintainable* and *usable* modules. I've never talked about undocumented throw-away exploratory code. Most of the Python features become futile in that case (type annotations and static type checking with mypy, to name only the few). Does it work on Windows? > This is probably impossible to write as a contract, but needs to be tested (though maybe there is a way to check it and encapsulate the check in a separate function and put it into the contract). What is_absolute()? is "file:///" absolute? > Since the type is pathlib.Path (as written in the type annotation), it's pathlib.Path.is_absolute() method. Please see https://docs.python.org/3/library/pathlib.html#pathlib.PurePath.is_absolute At a high level, I can see the allure of DbC: Programming can be a craft, > and a person can derive deep personal satisfaction from perfecting the code > they work on. DbC provides you with more decoration, more elaboration, more > ornamentation, more control. This is not bad, but I see all your arguments > as personal ascetic sense. DbC is only appealing under certain accounting > rules. Please consider the possibility that "the best code" is: low $$$, > buggy, full of tangles, and mostly gets the job done. :) Actually, this goes totally contrary to most of my experience. Bad code is unmaintainable and ends up being much more costly down the line. It's also what we were taught in software engineering lectures in the university (some 10-15 years ago) and I always assumed that the studies presented there were correct. Saying that writing down contracts is costly is a straw-man. It is costly if you need to examine the function and write them down. If you *are writing *the function and just keep adding the contracts as-you-go, it's basically very little overhead cost. You make an assumption of the input, and instead of just coding on, you scroll up, write it down formally, and go back where you stopped and continue the implementation. Or you think for a minute what contracts your function needs to expect/satisfy before you start writing it (or during the design). I don't see how this can be less efficient than trial-and-error and making possibly wrong assumptions based on the output that you see without any documentation by running the code of the module. Cheers, Marko On Tue, 25 Sep 2018 at 23:59, Kyle Lahnakoski wrote: > > I use DbC occasionally to clarify my thoughts during a refactoring, and > then only in the places that continue to make mistakes. In general, I am > not in a domain that benefits from DbC. > > Contracts are code: More code means more bugs. Declarative contracts are > succinct, but difficult to debug when wrong; I believe this because the > debugger support for contracts is poor; There is no way to step through the > logic and see the intermediate reasoning in complex contracts. A contract > is an incomplete duplication of what the code already does: at some level > of complexity I prefer to use a duplicate independent implementation and > compare inputs/outputs. > Writing contracts cost time and money; and that cost should be weighed > against the number and flexibility of the customers that use the code. A > one-time script, a webapp for you team, an Android app for your startup, > fraud software, and Facebook make different accounting decisions. I > contend most code projects can not justify DbC. > > > On 2018-09-24 03:46, Marko Ristin-Kaufmann wrote: > > When you are documenting a method you have the following options: > 1) Write preconditions and postconditions formally and include them > automatically in the documentation (*e.g., *by using icontract library). > 2) Write precondtions and postconditions in docstring of the method as > human text. > 3) Write doctests in the docstring of the method. > 4) Expect the user to read the actual
Re: [Python-ideas] Why is design-by-contracts not widely adopted?
Hi Chris, An extraordinary claim is like "DbC can improve *every single project* > on PyPI". That requires a TON of proof. Obviously we won't quibble if > you can only demonstrate that 99.95% of them can be improved, but you > have to at least show that the bulk of them can. I tried to give the "proof" (not a formal one, though) in my previous message. The assumptions are that: * There are always contracts, they can be either implicit or explicit. You need always to figure them out before you call a function or use its result. * Figuring out contracts by trial-and-error and reading the code (the implementation or the test code) is time consuming and hard. * The are tools for formal contracts. * The contracts written in documentation as human text inevitably rot and they are much harder to maintain than automatically verified formal contracts. * The reader is familiar with formal statements, and hence reading formal statements is faster than reading the code or trial-and-error. I then went on to show why I think, under these assumptions, that formal contracts are superior as a documentation tool and hence beneficial. Do you think that any of these assumptions are wrong? Is there a hole in my logical reasoning presented in my previous message? I would be very grateful for any pointers! If these assumptions hold and there is no mistake in my reasoning, wouldn't that qualify as a proof? Cheers, Marko On Tue, 25 Sep 2018 at 21:43, Chris Angelico wrote: > On Wed, Sep 26, 2018 at 3:19 AM Marko Ristin-Kaufmann > wrote: > >> Claiming that DbC annotations will improve the documentation of every > >> single library on PyPI is an extraordinary claim, and such claims > >> require extraordinary proof. > > > > > > I don't know what you mean by "extraordinary" claim and "extraordinary" > proof, respectively. I tried to show that DbC is a great tool and far > superior to any other tools currently used to document contracts in a > library, please see my message > https://groups.google.com/d/msg/python-ideas/dmXz_7LH4GI/5A9jbpQ8CAAJ. > Let me re-use the enumeration I used in the message and give you a short > summary. > > > > An ordinary claim is like "DbC can be used to improve code and/or > documentation", and requires about as much evidence as you can stuff > into a single email. Simple claim, low burden of proof. > > An extraordinary claim is like "DbC can improve *every single project* > on PyPI". That requires a TON of proof. Obviously we won't quibble if > you can only demonstrate that 99.95% of them can be improved, but you > have to at least show that the bulk of them can. > > > There are 150K projects on pypi.org. Each one of them would benefit if > annotated with the contracts. > > This is the extraordinary claim. To justify it, you have to show that > virtually ANY project would benefit from contracts. So far, I haven't > seen any such proof. > > ChrisA > ___ > Python-ideas mailing list > Python-ideas@python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] "old" values in postconditions
Hi,
Franklin wrote:
> The name "before" is a confusing name. It's not just something that
> happens before. It's really a pre-`let`, adding names to the scope of
> things after it, but with values taken before the function call. Based
> on that description, other possible names are `prelet`, `letbefore`,
> `predef`, `defpre`, `beforescope`. Better a name that is clearly
> confusing than one that is obvious but misleading.
James wrote:
> I suggest that instead of “@before” it’s “@snapshot” and instead of “old”
> it’s “snapshot”.
I like "snapshot", it's a bit clearer than prefixing/postfixing verbs with
"pre" which might be misread (*e.g., *"prelet" has a meaning in Slavic
languages and could be subconsciously misread, "predef" implies to me a pre-
*definition* rather than prior-to-definition , "beforescope" is very clear
for me, but it might be confusing for others as to what it actually refers
to ). What about "@capture" (7 letters for captures *versus *8 for
snapshot)? I suppose "@let" would be playing with fire if Python with
conflicting new keywords since I assume "let" to be one of the candidates.
Actually, I think there is probably no way around a decorator that
captures/snapshots the data before the function call with a lambda (or even
a separate function). "Old" construct, if we are to parse it somehow from
the condition function, would limit us only to shallow copies (and be
complex to implement as soon as we are capturing out-of-argument values
such as globals *etc.)*. Moreove, what if we don't need shallow copies? I
could imagine a dozen of cases where shallow copy is not what the
programmer wants: for example, s/he might need to make deep copies, hash or
otherwise transform the input data to hold only part of it instead of
copying (*e.g., *so as to allow equality check without a double copy of the
data, or capture only the value of certain property transformed in some
way).
I'd still go with the dictionary to allow for this extra freedom. We could
have a convention: "a" denotes to the current arguments, and "b" denotes
the captured values. It might make an interesting hint that we put "b"
before "a" in the condition. You could also interpret "b" as "before" and
"a" as "after", but also "a" as "arguments".
@capture(lambda a: {"some_identifier": some_func(a.some_argument.some_attr)})
@post(lambda b, a, result: b.some_identifier > result +
a.another_argument.another_attr)
def some_func(some_argument: SomeClass, another_argument:
AnotherClass) -> SomeResult:
...
"b" can be omitted if it is not used. Under the hub, all the arguments to
the condition would be passed by keywords.
In case of inheritance, captures would be inherited as well. Hence the
library would check at run-time that the returned dictionary with captured
values has no identifier that has been already captured, and the linter
checks that statically, before running the code. Reading values captured in
the parent at the code of the child class might be a bit hard -- but that
is case with any inherited methods/properties. In documentation, I'd list
all the captures of both ancestor and the current class.
I'm looking forward to reading your opinion on this and alternative
suggestions :)
Marko
On Tue, 25 Sep 2018 at 18:12, Franklin? Lee
wrote:
> On Sun, Sep 23, 2018 at 2:05 AM Marko Ristin-Kaufmann
> wrote:
> >
> > Hi,
> >
> > (I'd like to fork from a previous thread, "Pre-conditions and
> post-conditions", since it got long and we started discussing a couple of
> different things. Let's discuss in this thread the implementation of a
> library for design-by-contract and how to push it forward to hopefully add
> it to the standard library one day.)
> >
> > For those unfamiliar with contracts and current state of the discussion
> in the previous thread, here's a short summary. The discussion started by
> me inquiring about the possibility to add design-by-contract concepts into
> the core language. The idea was rejected by the participants mainly because
> they thought that the merit of the feature does not merit its costs. This
> is quite debatable and seems to reflect many a discussion about
> design-by-contract in general. Please see the other thread, "Why is
> design-by-contract not widely adopted?" if you are interested in that
> debate.
> >
> > We (a colleague of mine and I) decided to implement a library to bring
> design-by-contract to Python since we don't believe that the concept will
> make it into the core language anytime soon and we needed badly a tool to
> facilitate our work with a growing code base.
> >
> > The library is available at http://github.com/Parquery/icontract. The
> hope is to polish it so that the wider community could use it and once the
> quality is high enough, make a proposal to add it to the standard Python
> libraries. We do need a standard library for contracts, otherwise projects
> with conflicting contract libraries can not integrate (e.g., the contracts
> can not be inherited
Re: [Python-ideas] "while:" for the loop
On Wed, Sep 26, 2018 at 1:29 PM Mikhail V wrote: > > On Wed, Sep 26, 2018 at 5:38 AM Chris Angelico wrote: > > > > > > I like saying while "something": where the string describes the loop's > > real condition. For instance, while "moar data": if reading from a > > socket, or while "not KeyboardInterrupt": if the loop is meant to be > > halted by SIGINT. > > > > ChrisA > > if doing so, would not it be more practical > to write is as an in-line comment then? > with new syntax it could be like this: > """ > while: # not KeyboardInterrupt > asd asd asd > asd asd asd > asd asd asd > """ > Similar effect, but I would find it better at least because it would > be highlighted as a comment and not as a string, + no quotes noise. A comment is not better than an inline condition, no. I *want* it to be highlighted as part of the code, not as a comment. Because it isn't a comment - it's a loop condition. ChrisA ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] "while:" for the loop
On Wed, Sep 26, 2018 at 5:38 AM Chris Angelico wrote: > > > I like saying while "something": where the string describes the loop's > real condition. For instance, while "moar data": if reading from a > socket, or while "not KeyboardInterrupt": if the loop is meant to be > halted by SIGINT. > > ChrisA if doing so, would not it be more practical to write is as an in-line comment then? with new syntax it could be like this: """ while: # not KeyboardInterrupt asd asd asd asd asd asd asd asd asd """ Similar effect, but I would find it better at least because it would be highlighted as a comment and not as a string, + no quotes noise. ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] "while:" for the loop
On Wed, Sep 26, 2018 at 12:35 PM Michael Selik wrote: > > On Tue, Sep 25, 2018 at 8:46 PM Mikhail V wrote: > > I suggest allowing "while:" syntax for the infinite loop. > > I.e. instead of "while 1:" and "while True:" notations. > > > > My opinion: > > 1. I think it'd definitely improve clarity. > > I prefer the explicit phrase, ``while True:``. Saying "while" without > a condition is strange, like a sentence fragment. The ``while 1:`` > pattern is a carryover from Python 2, when ``True`` was not yet a > keyword. I like saying while "something": where the string describes the loop's real condition. For instance, while "moar data": if reading from a socket, or while "not KeyboardInterrupt": if the loop is meant to be halted by SIGINT. ChrisA ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] "while:" for the loop
On Tue, Sep 25, 2018 at 8:46 PM Mikhail V wrote: > I suggest allowing "while:" syntax for the infinite loop. > I.e. instead of "while 1:" and "while True:" notations. > > My opinion: > 1. I think it'd definitely improve clarity. I prefer the explicit phrase, ``while True:``. Saying "while" without a condition is strange, like a sentence fragment. The ``while 1:`` pattern is a carryover from Python 2, when ``True`` was not yet a keyword. ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] "while:" for the loop
I suggest allowing "while:" syntax for the infinite loop. I.e. instead of "while 1:" and "while True:" notations. IIRC, in the past this was mentioned in python-list discussions as alternative for the "while True:"/"while 1:" syntax. I even had impression that there was nothing rational against this (apart from the traditional "don't change anything" principle) My opinion: 1. I think it'd definitely improve clarity. Although it's not extremely frequent statement, it still appears in algorithms, where additional noise interfers the reader's concentration. 2. This should become the answer to the "how should I denote an infinte loop?" question. 3. In schools/unis they teach algorithms with Python syntax so it will be easier to remember and to write. Adoption of this spelling is natural and straightforward. 4. It does seem to be a rare case of a very easy to implement syntax change (an expert note needed) Also I have personal sympathy for this because I like to use explicit "break" in the loop body, even though I could use "while expression:" syntax, but I prefer this: while 1: ... if i == N : break instead of this: while i < N: ... It helps me to concentrate by reading, especially in nested loops and those situations with multiple break points. So by me this syntax would definitely achieve an extra +. There were alternative suggestions, e.g. introducing a new keyword "loop", but obviously a new keyword is much harder to do now. I don't know what to add to this actually, I think the idea is understood. I see, it is not the most important problem, but if there is nothing serious against this, I think it's worth it and would be quite a positive (small) improvement and a nice gift to those involved in algorithms. As for statistics - IIRC someone gave statistics once, but the only thing I can remember - "while 1/True" is used quite a lot in the std lib, so the numbers exceeded my expectation (because I expected that it's used mostly in algorithms). Mikhail ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Why is design-by-contracts not widely adopted?
On Wed, Sep 26, 2018 at 7:59 AM Kyle Lahnakoski wrote: > I use DbC occasionally to clarify my thoughts during a refactoring, and then > only in the places that continue to make mistakes. In general, I am not in a > domain that benefits from DbC. > > Contracts are code: More code means more bugs. Contracts are executable documentation. If you can lift them directly into user-readable documentation (and by "user" here I mean the user of a library), they can save you the work of keeping your documentation accurate. > This contract does not help me: > > What is_absolute()? is "file:///" absolute? I'd have to assume that is_absolute() is defined elsewhere. Which means that the value of this contract depends entirely on having other functions, probably ALSO contractually-defined, to explain it. > How does this code fail? > What does a permission access problem look like? Probably an exception. This is Python code, and I would generally assume that problems are reported as exceptions. > Can initial_paths can be None? This can be answered from the type declaration. It doesn't say Optional, so no, it can't be None. > Can initial_paths be files? directories? Presumably not a question you'd get if you were actually using it; the point of the function is to "[r]esolve the initial paths of the dependency graph by recursively adding *.py files beneath given directories", so you'd call it because you have directories and want files back. > What are the side effects? Hopefully none, other than the normal implications of hitting the file system. It's easy to show beautiful examples that may actually depend on other things. Whether that's representative of all contracts is another question. ChrisA ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Why is design-by-contracts not widely adopted?
I use DbC occasionally to clarify my thoughts during a refactoring, and then only in the places that continue to make mistakes. In general, I am not in a domain that benefits from DbC. Contracts are code: More code means more bugs. Declarative contracts are succinct, but difficult to debug when wrong; I believe this because the debugger support for contracts is poor; There is no way to step through the logic and see the intermediate reasoning in complex contracts. A contract is an incomplete duplication of what the code already does: at some level of complexity I prefer to use a duplicate independent implementation and compare inputs/outputs. Writing contracts cost time and money; and that cost should be weighed against the number and flexibility of the customers that use the code. A one-time script, a webapp for you team, an Android app for your startup, fraud software, and Facebook make different accounting decisions. I contend most code projects can not justify DbC. On 2018-09-24 03:46, Marko Ristin-Kaufmann wrote: > When you are documenting a method you have the following options: > 1) Write preconditions and postconditions formally and include them > automatically in the documentation (/e.g., /by using icontract library). > 2) Write precondtions and postconditions in docstring of the method as > human text. > 3) Write doctests in the docstring of the method. > 4) Expect the user to read the actual implementation. > 5) Expect the user to read the testing code. > There are other ways to communicate how a method works. 6) The name of the method 7) How the method is called throughout the codebase 8) observing input and output values during debugging 9) observing input and output values in production 10) relying on convention inside, and outside, the application 11) Don't communicate - Sometimes / is too high; code is not repaired, only replaced. > This is again something that eludes me and I would be really thankful > if you could clarify. Please consider for an example, pypackagery > (https://pypackagery.readthedocs.io/en/latest/packagery.html) and the > documentation of its function resolve_initial_paths: > > |packagery.||resolve_initial_paths|(/initial_paths/) > > Resolve the initial paths of the dependency graph by recursively > adding |*.py| files beneath given directories. > > Parameters: > > *initial_paths* (|List|[|Path|]) – initial paths as absolute paths > > Return type: > > |List|[|Path|] > > Returns: > > list of initial files (/i.e./ no directories) > > Requires: > > * |all(pth.is_absolute() for pth in initial_paths)| > > Ensures: > > * |len(result) >= len(initial_paths) if initial_paths else > result == []| > * |all(pth.is_absolute() for pth in result)| > * |all(pth.is_file() for pth in result)| > > > How is this difficult to read,[...]? This contract does not help me: Does it work on Windows? What is_absolute()? is "file:///" absolute? How does this code fail? What does a permission access problem look like? Can initial_paths can be None? Can initial_paths be files? directories? What are the side effects? resolve_initial_path() is a piece code is better understood by looking at the callers (#7), or not exposing it publicly (#11). You can also use a different set of abstractions, to make the code easier to read: UNION(file for p in initial_paths for file in p.leaves() if file.extension=="py") At a high level, I can see the allure of DbC: Programming can be a craft, and a person can derive deep personal satisfaction from perfecting the code they work on. DbC provides you with more decoration, more elaboration, more ornamentation, more control. This is not bad, but I see all your arguments as personal ascetic sense. DbC is only appealing under certain accounting rules. Please consider the possibility that "the best code" is: low $$$, buggy, full of tangles, and mostly gets the job done. :) ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Why is design-by-contracts not widely adopted?
On Wed, Sep 26, 2018 at 6:09 AM Lee Braiden wrote: > > Eh. It's too easy to cry "show me the facts" in any argument. To do that too > often is to reduce all discussion to pendantry. > > That verifying data against the contract a function makes code more reliable > should be self evident to anyone with even the most rudimentary understanding > of a function call, let alone a library or large application. It's the > reason why type checking exists, and why bounds checking exists, and why unit > checking exists too. > It's easy, but it's also often correct. >From my reading of this thread, there HAS been evidence given that DbC can be beneficial in some cases. I do not believe there has been evidence enough to cite the number of projects on PyPI as "this is how many projects would benefit". Part of the trouble is finding a concise syntax for the contracts that is still sufficiently expressive. ChrisA ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Why is design-by-contracts not widely adopted?
Eh. It's too easy to cry "show me the facts" in any argument. To do that too often is to reduce all discussion to pendantry. That verifying data against the contract a function makes code more reliable should be self evident to anyone with even the most rudimentary understanding of a function call, let alone a library or large application. It's the reason why type checking exists, and why bounds checking exists, and why unit checking exists too. On Tue, 25 Sep 2018, 20:43 Chris Angelico, wrote: > On Wed, Sep 26, 2018 at 3:19 AM Marko Ristin-Kaufmann > wrote: > >> Claiming that DbC annotations will improve the documentation of every > >> single library on PyPI is an extraordinary claim, and such claims > >> require extraordinary proof. > > > > > > I don't know what you mean by "extraordinary" claim and "extraordinary" > proof, respectively. I tried to show that DbC is a great tool and far > superior to any other tools currently used to document contracts in a > library, please see my message > https://groups.google.com/d/msg/python-ideas/dmXz_7LH4GI/5A9jbpQ8CAAJ. > Let me re-use the enumeration I used in the message and give you a short > summary. > > > > An ordinary claim is like "DbC can be used to improve code and/or > documentation", and requires about as much evidence as you can stuff > into a single email. Simple claim, low burden of proof. > > An extraordinary claim is like "DbC can improve *every single project* > on PyPI". That requires a TON of proof. Obviously we won't quibble if > you can only demonstrate that 99.95% of them can be improved, but you > have to at least show that the bulk of them can. > > > There are 150K projects on pypi.org. Each one of them would benefit if > annotated with the contracts. > > This is the extraordinary claim. To justify it, you have to show that > virtually ANY project would benefit from contracts. So far, I haven't > seen any such proof. > > ChrisA > ___ > Python-ideas mailing list > Python-ideas@python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Why is design-by-contracts not widely adopted?
On Wed, Sep 26, 2018 at 3:19 AM Marko Ristin-Kaufmann wrote: >> Claiming that DbC annotations will improve the documentation of every >> single library on PyPI is an extraordinary claim, and such claims >> require extraordinary proof. > > > I don't know what you mean by "extraordinary" claim and "extraordinary" > proof, respectively. I tried to show that DbC is a great tool and far > superior to any other tools currently used to document contracts in a > library, please see my message > https://groups.google.com/d/msg/python-ideas/dmXz_7LH4GI/5A9jbpQ8CAAJ. Let me > re-use the enumeration I used in the message and give you a short summary. > An ordinary claim is like "DbC can be used to improve code and/or documentation", and requires about as much evidence as you can stuff into a single email. Simple claim, low burden of proof. An extraordinary claim is like "DbC can improve *every single project* on PyPI". That requires a TON of proof. Obviously we won't quibble if you can only demonstrate that 99.95% of them can be improved, but you have to at least show that the bulk of them can. > There are 150K projects on pypi.org. Each one of them would benefit if > annotated with the contracts. This is the extraordinary claim. To justify it, you have to show that virtually ANY project would benefit from contracts. So far, I haven't seen any such proof. ChrisA ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] "old" values in postconditions
Ah. It wasn't clear to me from the thread that James was using `inspect`. As it happens, not only does getsource give more than it should, it also gives less than it should. The following bug still exists in 3.6.1. It was closed as a wontfix bug back in Python 2 because, I presume, fixing it would require changing code objects. https://bugs.python.org/issue17631 Regardless, I think it'd be better not to rely on such magic for the proposal. While it's a fun puzzle, whether it's possible or not, it still modifies the syntax and/or scoping and/or evaluation order rules of Python. On Tue, Sep 25, 2018 at 1:54 PM Marko Ristin-Kaufmann wrote: > > Hi James and Franklin, > > getsource() definitely does not work. I tried for a long, long time to make > it work and finally gave up. I parse in icontract the whole file where the > lambda function resides and use asttokens to locate the node of the lambda > (along some tree traversing upwards and making assumptions where the > condition lambda lives). > > Have a look at: > > https://github.com/Parquery/icontract/blob/391d43005287831892b19dfdcbcfd3d48662c638/icontract/represent.py#L309 > > and > https://github.com/Parquery/icontract/blob/391d43005287831892b19dfdcbcfd3d48662c638/icontract/represent.py#L157 > > > On Tue, 25 Sep 2018 at 19:48, James Lu wrote: >> >> > I'm surprised you haven't found >> >inspect.getsource(func) >> >> I did. That’s exactly what I was describing in the paragraph. It wouldn’t >> work in interactive mode and it includes everything on the same line of the >> lambda definition. >> ___ >> Python-ideas mailing list >> Python-ideas@python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Why is design-by-contracts not widely adopted?
Hi Hugh, > As soon as you need to document your code, and > > this is what most modules have to do in teams of more than one person > > (especially so if you are developing a library for a wider audience), you > > need to write down the contracts. Please see above where I tried to > > explained that 2-5) are inferior approaches to documenting contracts > > compared to 1). > > You left off option 6), plain text. Comments. Docstrings. > That was actually the option 2): > 2) Write precondtions and postconditions in docstring of the method as > human text. The problem with text is that it is not verifiable and hence starts to "rot". Noticing that text is wrong involves much more developer time & attention than automatically verifying the formal contracts. In Python we can write something like: def foo(x): x.bar(y) What's the type of x? What's the type of y? What is the contract of bar? Don't know, don't care. x, or y, can be an instance, a class, a module, a proxy for a remote web service. The only "contract" is that object x will respond to message bar that takes one argument. Object x, do whatever you want with it. I still don't see how this is connected to contracts or how contracts play a role there? If foo can accept any x and return any result then there is *no *contract. But hardly any function is like that. Most exercise a certain behavior on a subset of possible input values. The outputs also satisfy certain contracts, *i.e.* they also live in a certain subset of possible outputs. (Please mind that I don't mean strictly numerical ranges here -- it can be any subset of structured data.) As I already mentioned, the contracts have nothing to do with typing. You can use them for runtime type checks -- but that's a reduction of the concept to a very particular use case. Usually contracts read like this (from the numpy example linked in another message, https://www.numpy.org/devdocs/reference/generated/numpy.ndarray.transpose.html#numpy.ndarray.transpose ): ndarray.transpose(**axes*) Returns a view of the array with axes transposed. *For a 1-D array, this has no effect. (To change between column and row vectors, first cast the 1-D array into a matrix object.) For a 2-D array, this is the usual matrix transpose. For an n-D array, if axes are given, their order indicates how the axes are permuted (see Examples). If axes are not provided and a.shape = (i[0], i[1], ... i[n-2], i[n-1]), then a.transpose().shape = (i[n-1], i[n-2], ... i[1], i[0]).* (emphasis mine) Mind the three postconditions (case 1D array, case 2D array, case N-D array). As for 4) reading the code, why not? "Use the source, Luke" is now a > programming cliche because it works. It's particularly appropriate for > Python packages which are usually distributed in source form and, as > you yourself noted, easy to read. Because it is hard and costs a lot of time. The point of encapsulating a function is that I as a user don't have to know its details of implementation and its wider dependencies in the implementation. Looking at the code is the tool of last resort to figure out the contracts. Imagine if you had to look at the implementation of numpy.transpose() to figure out what happens when transposing a N-D array. Cheers, Marko On Tue, 25 Sep 2018 at 13:13, Hugh Fisher wrote: > > Date: Mon, 24 Sep 2018 09:46:16 +0200 > > From: Marko Ristin-Kaufmann > > To: Python-Ideas > > Subject: Re: [Python-ideas] Why is design-by-contracts not widely > > adopted? > > Message-ID: > > i7eq2gr8kddofoneab-qvt8lz...@mail.gmail.com> > > Content-Type: text/plain; charset="utf-8" > > [munch] > > > Their users would hugely benefit from a more mature > > and standardized contracts library with informative violation messages. > > Will respond in another message, because it's a big topic. > > > I really don't see how DbC has to do with duck typing (unless you reduce > it > > to mere isinstance conditions, which would simply be a straw-man > argument) > > -- could you please clarify? > > I argue that Design by Contract doesn't make sense for Python and other > dynamically typed, duck typed languages because it's contrary to how the > language, and the programmer, expects to work. > > In Python we can write something like: > > def foo(x): > x.bar(y) > > What's the type of x? What's the type of y? What is the contract of bar? > Don't know, don't care. x, or y, can be an instance, a class, a module, a > proxy for a remote web service. The only "contract" is that object x will > respond to message bar that takes one argument. Object x, do whatever > you want with it. > > And that's a feature, not a bug, not bad design. It follows Postel's Law > for Internet protocols of being liberal in what you accept. It follows the > Agile principle of valuing working software over comprehensive doco. > It allows software components to be glued together quickly and easily. > > It's a style of programming that has been successful for many
Re: [Python-ideas] "old" values in postconditions
Hi James and Franklin, getsource() definitely does not work. I tried for a long, long time to make it work and finally gave up. I parse in icontract the whole file where the lambda function resides and use asttokens to locate the node of the lambda (along some tree traversing upwards and making assumptions where the condition lambda lives). Have a look at: https://github.com/Parquery/icontract/blob/391d43005287831892b19dfdcbcfd3d48662c638/icontract/represent.py#L309 and https://github.com/Parquery/icontract/blob/391d43005287831892b19dfdcbcfd3d48662c638/icontract/represent.py#L157 On Tue, 25 Sep 2018 at 19:48, James Lu wrote: > > I'm surprised you haven't found > >inspect.getsource(func) > > I did. That’s exactly what I was describing in the paragraph. It wouldn’t > work in interactive mode and it includes everything on the same line of the > lambda definition. > ___ > Python-ideas mailing list > Python-ideas@python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Why is design-by-contracts not widely
Hi Hugh, Software projects, in any language, never have enough time to do everything. > By your own example, the Python developers of numpy, OpenCV, nlk, and > sklearn; *who most certainly weren't writing contracts;* produced better > quality > software than the Eiffel equivalent developers who (I assume) did use DbC. > Shouldn't the Eiffel developers be changing their development method, not > the Python developers? (emphasis mine) This is *absolutely* *not true* as you can notice if you multiply any two matrices of wrong dimensions in numpy or opencv (or use them as weights in sklearn). For example, have a look at OpenCV functions. *Most of them include preconditions and postconditions *(*e.g., * https://docs.opencv.org/3.4.3/dc/d8c/namespacecvflann.html#a57191110b01f200e478c658f3b7a362d). I would even go as far to claim that OpenCV would be unusable without the contracts. Imagine if you had to figure out the dimensions of the matrix after each operation if it were lacking in the documentation. That would make the development sluggish as a snail. Numpy provides contracts in text, *e.g. *see https://www.numpy.org/devdocs/reference/generated/numpy.ndarray.transpose.html#numpy.ndarray.transpose : ndarray.transpose(**axes*) Returns a view of the array with axes transposed. For a 1-D array, this has no effect. (To change between column and row vectors, first cast the 1-D array into a matrix object.) For a 2-D array, this is the usual matrix transpose. For an n-D array, if axes are given, their order indicates how the axes are permuted (see Examples). If axes are not provided and a.shape = (i[0], i[1], ... i[n-2], i[n-1]), then a.transpose().shape = (i[n-1], i[n-2], ... i[1], i[0]). As you can see, there are three contracts: 1) no effect on 1D array, 2) if a 2D array, it equals the matrix transpose, 3) if n-D array, the order of axes indicates the permutation. The contract 3) is written out formally. It might not be very clear or precise what is meant in 2) where formalizing it (at least to a certain extent) would remove many doubts. It is obvious to me that supplementing or replacing these contracts *in text* with *formal *contracts (1 and 2, since 3 is already formal) is extremely beneficial since: a) many developers use numpy and an improvement in documentation (such as higher precision and clarity) has a large impact on the users and b) enforcing the contracts automatically (be it only during testing or in production) prevents bugs related to contract violation in numpy such that the users can effectively rely on the contracts. The argument b) is important since now I just rely that these statements are true whenever I use numpy. If there is an error in numpy it takes a long time to figure out since I doubt the last that there is an error in numpy and especially it takes even longer that I suspect numpy of not satisfying its written contracts. Please mind that contracts can be toggled on/off whenever the performance is important so that slow execution is not an argument against the formal contracts. Cheers, Marko On Tue, 25 Sep 2018 at 14:01, Hugh Fisher wrote: > > Date: Mon, 24 Sep 2018 09:46:16 +0200 > > From: Marko Ristin-Kaufmann > > To: Python-Ideas > > Subject: Re: [Python-ideas] Why is design-by-contracts not widely > > adopted? > > [munch] > > > Python is easier to write and read, and there are no libraries which are > > close in quality in Eiffel space (notably, Numpy, OpenCV, nltk and > > sklearn). I really don't see how the quality of these libraries have > > anything to do with lack (or presence) of the contracts. OpenCV and Numpy > > have contracts all over their code (written as assertions and not > > documented), albeit with very non-informative violation messages. And > they > > are great libraries. Their users would hugely benefit from a more mature > > and standardized contracts library with informative violation messages. > > I would say the most likely outcome of adding Design by Contract would > be no change in the quality or usefulness of these libraries, with a small > but not insignificant chance of a decline in quality. > > Fred Brooks in his "No Silver Bullet" paper distinguished between essential > complexity, which is the problem we try to solve with software, and > accidental > complexity, solving the problems caused by your tools and/or process that > get in the way of solving the actual problem. "Yak shaving" is a similar, > less > formal term for accidental complexity, when you have to do something before > you can do something before you can actually do some useful work. > > Adding new syntax or semantics to a programming language very often adds > accidental complexity. > > C and Python (currently) are known as simple languages. When starting a > programming project in C or Python, there's maybe a brief discussion about > C99 or C11, or Python 3.5 or 3.6, but that's it. There's one way to do it. > > On the other hand C++ is notorious for having
Re: [Python-ideas] "old" values in postconditions
> I'm surprised you haven't found >inspect.getsource(func) I did. That’s exactly what I was describing in the paragraph. It wouldn’t work in interactive mode and it includes everything on the same line of the lambda definition. ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] "old" values in postconditions
Ew, magic. `{{foo}}` is already valid syntax (though it will always
fail). I don't like this path. If the proposal requires new syntax or
magic, it will be less likely to get accepted or even pip'd. Remember
also that PyPy, IronPython, and Jython are still alive, and the latter
two are still aiming for Python 3 (PyPy 3 is already available).
On Tue, Sep 25, 2018 at 12:52 PM James Lu wrote:
>
> Hmm, I was wrong: there is no reliable way to get the code of a lambda
> function.
You may be looking for inspect.signature(func).
> Or maybe we could send a PEP to add the .func_code attribute to lambdas as
> well as normal functions.
The new name is `.__code__`. There is some documentation of code
objects in the `inspect` documentation (names beginning with `co_`).
More documentation:
https://docs.python.org/3.7/reference/datamodel.html#index-55
> There’s also a technique online where they find the lambda’s source by
> locating the file the function was defined in and then removing the
> irrelevant parts, but that just doesn’t sound practical to me.
I'm surprised you haven't found
inspect.getsource(func)
Doesn't always work if the source is not associated to the definition
of func, such as with functions defined in C. That shouldn't be a
problem with lambdas. You might need to worry about file permission. I
don't know if other interpreters will find it difficult to support
`inspect`.
You may be looking for `ast.parse(inspect.getsource(func))`.
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Why is design-by-contracts not widely adopted?
Hi Steve, I'll give it a shot and implement a proof-of-concept icontrac-macro library based on macropy and see if that works. I'll keep you posted. Cheers, Marko On Tue, 25 Sep 2018 at 12:08, Stephen J. Turnbull < turnbull.stephen...@u.tsukuba.ac.jp> wrote: > Marko Ristin-Kaufmann writes: > > > Thanks a lot for pointing us to macropy -- I was not aware of the > library, > > it looks very interesting! > > > > Do you have any experience how macropy fit > > Sorry, no. I was speaking as someone who is familiar with macros from > Lisp but doesn't miss them in Python, and who also has been watching > python-dev and python-ideas for about two decades now, so I've heard > of things like MacroPy and know how the core developers think to a > great extent. > > > I'm also a bit worried how macropy would work out in the libraries > > published to pypi -- imagine if many people start using contracts. > > Suddenly, all these libraries would not only depend on a contract > library > > but on a macro library as well. > > That's right. > > > Is that something we should care about? > > Yes. Most Pythonistas (at least at present) don't much like macros. > They fear turning every program into its own domain-specific language. > I can't claim much experience with dependency hell, but I think that's > much less important from your point of view (see below). > > My point is mainly that, as you probably are becoming painfully aware, > getting syntax changes into Python is a fairly drawnout process. For > an example of the kind of presentation that motivates people to change > their mind from the default state of "if it isn't in Python yet, > YAGNI" to "yes, let's do *this* one", see > https://www.python.org/dev/peps/pep-0572/#appendix-a-tim-peters-s-findings > > Warning: Tim Peters is legendary, though still active occasionally. > All he has to do is post to get people to take notice. But this > Appendix is an example of why he gets that kind of R-E-S-P-E-C-T.[1] > > So the whole thing is a secret plot ;-) to present the most beautiful > syntax possible in your PEP (which will *not* be about DbC, but rather > about a small set of enabling syntax changes, hopefully a singleton), > along with an extended example, or a representative sample, of usage. > Because you have a working implementation using MacroPy (or the less > pretty[2] but fewer dependencies version based on condition strings > and eval) people can actually try it on their own code and (you hope, > they don't :-) they find a nestful of bugs by using it. > > > Potential dependency hell? (I already have a bad feeling about > > making icontract depend on asttokens and considerin-lining > > asttokens into icontract particularly for that reason). > > I don't think so. First, inlining an existing library is almost > always a bad idea. As for the main point, if the user sticks to one > major revision, and only upgrades to compatible bugfixes in the > Python+stdlib distribution, I don't see why two or three libraries > would be a major issue for a feature that the developer/project uses > extremely frequently. I've rarely experienced dependency hell, and in > both cases it was web frameworks (Django and Zope, to be specific, and > the dependencies involved were more or less internal to those > frameworks). If you or people you trust have other experience, forget > what I just said. :-) > > Of course it depends on the library, but as long as the library is > pretty strict about backward compatibility, you can upgrade it and get > new functionality for other callers in your code base (which are > likely to appear, you know -- human beings cannot stand to leave a > tool unused once they install it!) > > > > Note that this means *you cannot use macros in a file that is run > > > directly*, as it will not be passed through the import hooks. > > > > That would make contracts unusable in any stand-alone script, > > right? > > Yes, but really, no: > > # The run.py described by the MacroPy docs assumes a script that > # runs by just importing it. I don't have time to work out > # whether that makes more sense. This idiom of importing just a > # couple of libraries, and then invoking a function with a > # conventional name such as "run" or "process" is quite common. > # If you have docutils install, check out rstpep2html.py. > > import macropy.activate > from my_contractful_library import main > main() > > and away you go. 5 years from now that script will be a badge of > honor among Pythonic DbCers, and you won't be willing to give it up! > Just kidding, of course -- the ideal outcome is that the use case is > sufficiently persuasive to justify a syntax change so you don't need > MacroPy, or, perhaps some genius will come along and provide some > obscure construct that is already legal syntax! > > HTH > > Footnotes: > [1] R.I.P. Aretha! > > [2] Urk, I just realized there's another weakness to strings: you get > no
Re: [Python-ideas] Why is design-by-contracts not widely adopted?
Hi Robert, You'll lose folks attention very quickly when you try to tell folk > what they do and don't understand. I apologize if I sounded offending, that was definitely not my intention. I appreciate that you addressed that. I suppose it's cultural/language issue and the wording was probably inappropriate. Please let me clarify what I meant: there was a misconception as DbC was reduced to a tool for testing, and, in a separate message, reduced to type-checks at runtime. These are clearly misconceptions, as DbC (as origianally proposed by Hoare and later popularized by Meyer) include other relevant aspects which are essential and hence can not be overseen or simply ignored. If we are arguing about DbC without these aspects then we are simply falling pray to a straw-man fallacy. Claiming that DbC annotations will improve the documentation of every > single library on PyPI is an extraordinary claim, and such claims > require extraordinary proof. I don't know what you mean by "extraordinary" claim and "extraordinary" proof, respectively. I tried to show that DbC is a great tool and far superior to any other tools currently used to document contracts in a library, please see my message https://groups.google.com/d/msg/python-ideas/dmXz_7LH4GI/5A9jbpQ8CAAJ. Let me re-use the enumeration I used in the message and give you a short summary. The implicit or explicit contracts are there willy-nilly. When you use a module, either you need to figure them out using trial-and-error or looking at the implementation (4), looking at the test cases and hoping that they generalize (5), write them as doctests (3) or write them in docstrings as human text (2); or you write them formally as explicit contracts (1). I could not identify any other methods that can help you with expectations when you call a function or use a class (apart from formal methods and proofs, which I omitted as they seem too esoteric for the current discussion). *Given that: * * There is no other method for representing contracts, * people are trained and can read formal statements and * there is tooling available to write, maintain and represent contracts in a nice way I see formal contracts (1) as a superior tool. The deficiencies of other approaches are: 2) Comments and docstrings inevitably rot and get disconnected from the implementation in my and many other people's experience and studies. 3) Doctests are much longer and hence more tedious to read and maintain, they need extra text to signal the intent (is it a simple test or an example how boundary conditions are handled or ...). In any non-trivial case, they need to include even the contract itself. 4) Looking at other people's code to figure out the contracts is tedious and usually difficult for any non-trivial function. 5) Test cases can be difficult to read since they include much broader testing logic (mocking, set up). Most libraries do not ship with the test code. Identifying test cases which demonstrate the contracts can be difficult. *Any* function that is used by multiple developers which operates on the restricted range of input values and gives out structured output values benefits from contracts (1) since the user of the function needs to figure them out to properly call the function and handle its results correctly. I assume that every package on pypi is published to be used by wider audience, and not the developer herself. Hence every package on pypi would benefit from formal contracts. Some predicates are hard to formulate, and we will never be able to formally write down *all* the contracts. But that doesn't imply for me to *not use contracts at all* (analogously, some functionality is untestable, but that doesn't mean that we don't test what we can). I would be very grateful if you could point me where this exposition is wrong (maybe referring to my original message, https://groups.google.com/d/msg/python-ideas/dmXz_7LH4GI/5A9jbpQ8CAAJ, which I spent more thought on formulating). So far, I was not confronted against nor read on the internet a plausible argument against formal contracts (the only two exceptions being lack of tools and less-skilled programmers have a hard time reading formal statements as soon as they include boolean logic and quantifiers). I'm actively working on the former, and hope that the latter would improve with time as education in computer sciences improves. Another argument, which I did read often on internet, but don't really count is that quality software is not a priority and most projects hence dispense of documentation or testing. This should, hopefully, not apply to public pypi packages and is highly impractical for any medium-size project with multiple developers (and very costly in the long run). I can think of many libraries where necessary pre and post conditions > (such as 'self is still locked') are going to be noisy, and at risk of > reducing comprehension if the DbC checks are used to enhance/extended > documentation. It
Re: [Python-ideas] Why is design-by-contracts not widely
Those arguments are rules of thumb, which may or may not apply to DbC, and speculation, based on why DbC isn't more popular, to explain why DbC isn't more popular. They are general arguments for features in general, whereas Marko has been giving arguments for why DbC in particular is good or why it isn't more popular. The general arguments don't address the specific arguments. I don't use DbC, but I do use Numpy. Numpy is a very mathematical library, with many pure functions. It has lots of similarities between its functions and methods. I can easily see how design-by-contract can help Numpy users read the documentation and compare functions. Text is often less structured, so it is less likely to come out consistent. After all, isn't that why we keep adding structure to it, such as with Javadocs and Sphinx? Those examples add more syntax, while Marko's proposal doesn't necessarily require more syntax. On Tue, Sep 25, 2018 at 8:00 AM Hugh Fisher wrote: > > > Date: Mon, 24 Sep 2018 09:46:16 +0200 > > From: Marko Ristin-Kaufmann > > To: Python-Ideas > > Subject: Re: [Python-ideas] Why is design-by-contracts not widely > > adopted? > > [munch] > > > Python is easier to write and read, and there are no libraries which are > > close in quality in Eiffel space (notably, Numpy, OpenCV, nltk and > > sklearn). I really don't see how the quality of these libraries have > > anything to do with lack (or presence) of the contracts. OpenCV and Numpy > > have contracts all over their code (written as assertions and not > > documented), albeit with very non-informative violation messages. And they > > are great libraries. Their users would hugely benefit from a more mature > > and standardized contracts library with informative violation messages. > > I would say the most likely outcome of adding Design by Contract would > be no change in the quality or usefulness of these libraries, with a small > but not insignificant chance of a decline in quality. > > Fred Brooks in his "No Silver Bullet" paper distinguished between essential > complexity, which is the problem we try to solve with software, and accidental > complexity, solving the problems caused by your tools and/or process that > get in the way of solving the actual problem. "Yak shaving" is a similar, less > formal term for accidental complexity, when you have to do something before > you can do something before you can actually do some useful work. > > Adding new syntax or semantics to a programming language very often adds > accidental complexity. > > C and Python (currently) are known as simple languages. When starting a > programming project in C or Python, there's maybe a brief discussion about > C99 or C11, or Python 3.5 or 3.6, but that's it. There's one way to do it. > > On the other hand C++ is notorious for having been designed with a shovel > rather than a chisel. The people adding all the "features" were well > intentioned, > but it's still a mess. C++ programming projects often start by > specifying exactly > which bits of the language the programming team will be allowed to use. I've > seen these reach hundreds of pages in length, consuming God knows how > many hours to create, without actually creating a single line of useful > software. > > I think a major reason that Design by Contract hasn't been widely adopted > in the three decades since its introduction is because, mostly, it creates > more accidental complexity than it reduces essential complexity, so the > costs outweigh any benefits. > > Software projects, in any language, never have enough time to do everything. > By your own example, the Python developers of numpy, OpenCV, nlk, and > sklearn; who most certainly weren't writing contracts; produced better quality > software than the Eiffel equivalent developers who (I assume) did use DbC. > Shouldn't the Eiffel developers be changing their development method, not > the Python developers? > > Maybe in a world with infinite resources contracts could be added to those > Python packages, or everything in PyPi, and it would be an improvement. > But we don't. So I'd like to see the developers of numpy etc keep doing > whatever it is that they're doing now. > > -- > > cheers, > Hugh Fisher > ___ > Python-ideas mailing list > Python-ideas@python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] "old" values in postconditions
Hmm, I was wrong: there is no reliable way to get the code of a lambda
function.
If it was possible to execute all code paths of the function, we could monkey
patch the builtins so { } used our own custom set class.
Alternatively, the decorator could also accept a string.
Or maybe we could send a PEP to add the .func_code attribute to lambdas as well
as normal functions.
There’s also a technique online where they find the lambda’s source by locating
the file the function was defined in and then removing the irrelevant parts,
but that just doesn’t sound practical to me.
There’s also MacroPy.
I think the best solution would be to mock the old object and record the
operations done to the object, like the other replier gave a PoC of. Proposed
syntax
from icontract import post, old
@post(lambda: ...,
key=old.self.key(),
)
Sent from my iPhone
> On Sep 25, 2018, at 1:49 AM, Marko Ristin-Kaufmann
> wrote:
>
> Hi James,
> Thanks for the feedback!
>
> I also thought about decompiling the condition to find its AST and figure out
> what old values are needed. However, this is not so easily done at the moment
> as all the decompilation libraries I looked at (meta, ucompyle6) are simply
> struggling to keep up with the development of the python bytecode. In other
> words, the decompiler needs to be updated with every new version of python
> which is kind of a loosing race (unless the python devs themselves don't
> provide this functionality which is not the case as far as I know).
>
> There is macropy (https://github.com/lihaoyi/macropy) which was suggested on
> the other thread
> (https://groups.google.com/forum/#!topic/python-ideas/dmXz_7LH4GI) that I'm
> currently looking at.
>
> Cheers,
> Marko
>
>
>> On Tue, 25 Sep 2018 at 00:35, James Lu wrote:
>> You could disassemble (import dis) the lambda to biew the names of the
>> lambdas.
>>
>> @before(lambda self, key, _, length, get: self.length(), self.get(key))
>>
>> Perhaps you could disassemble the function code and look at all operations
>> or accesses that are done to “old.” and evaluate those expressions before
>> the function runs. Then you could “replace” the expression.
>> @post(lambda self, key, old: old.get is None and old.length + 1 ==
>> self.length())
>>
>> Either the system would grab old.get and old.length or be greedy and grab
>> old.get is None and old.length + 1. It would then replace the old.get and
>> old.length with injects that only respond to is None and +1.
>>
>> Or, a syntax like this
>> @post(lambda self, key, old: [old.get(old.key)] is None and
>> [old.self.length() + 1] ==
>> self.length())
>>
>> Where the stuff inside the brackets is evaluated before the decorated
>> function runs. It would be useful for networking functions or functions that
>> do something ephemeral, where data related to the value being accessed
>> needed for the expression no longer exists after the function.
>>
>> This does conflict with list syntax forever, so maybe either force people to
>> do list((expr,)) or use an alternate syntax like one item set syntax { } or
>> double set syntax {{ }} or double list syntax [[ ]]. Ditto with having to
>> avoid the literals for the normal meaning.
>>
>> You could modify Python to accept any expression for the lambda function and
>> propose that as a PEP. (Right now it’s hardcoded as a dotted name and
>> optionally a single argument list surrounded by parentheses.)
>>
>> I suggest that instead of “@before” it’s “@snapshot” and instead of “old”
>> it’s “snapshot”.
>>
>> Python does have unary plus/minus syntax as well as stream operators (<<,
>> >>) and list slicing syntax and the @ operator and operators & and | if you
>> want to play with syntax. There’s also the line continuation character for
>> crazy lambdas.
>>
>> Personally I prefer
>> @post(lambda self, key, old: {{old.self.get(old.key)}} and
>> {{old.self.length() + 1}} ==
>> self.length())
>>
>> because it’s explicit about what it does (evaluate the expressions within {{
>> }} before the function runs. I also find it elegant.
>>
>> Alternatively, inside the {{ }} could be a special scope where locals() is
>> all the arguments @pre could’ve received as a dictionary. For either option
>> you can remove the old parameter from the lambda. Example:
>> @post(lambda self, key: {{self.get(key)}} and {{self.length() + 1}} ==
>> self.length())
>>
>> Perhaps the convention should be to write {{ expr }} (with the spaces in
>> between).
>>
>> You’d probably have to use the ast module to inspect it instead of the dis
>> modul. Then find some way to reconstruct the expressions inside the double
>> brackets- perhaps by reconstructing the AST and compiling it to a code
>> object, or perhaps by finding the part of the string the expression is
>> located. dis can give you the code as a string and you can run a carefully
>> crafted regex on it.
>>
Re: [Python-ideas] "old" values in postconditions
On Sun, Sep 23, 2018 at 2:05 AM Marko Ristin-Kaufmann
wrote:
>
> Hi,
>
> (I'd like to fork from a previous thread, "Pre-conditions and
> post-conditions", since it got long and we started discussing a couple of
> different things. Let's discuss in this thread the implementation of a
> library for design-by-contract and how to push it forward to hopefully add it
> to the standard library one day.)
>
> For those unfamiliar with contracts and current state of the discussion in
> the previous thread, here's a short summary. The discussion started by me
> inquiring about the possibility to add design-by-contract concepts into the
> core language. The idea was rejected by the participants mainly because they
> thought that the merit of the feature does not merit its costs. This is quite
> debatable and seems to reflect many a discussion about design-by-contract in
> general. Please see the other thread, "Why is design-by-contract not widely
> adopted?" if you are interested in that debate.
>
> We (a colleague of mine and I) decided to implement a library to bring
> design-by-contract to Python since we don't believe that the concept will
> make it into the core language anytime soon and we needed badly a tool to
> facilitate our work with a growing code base.
>
> The library is available at http://github.com/Parquery/icontract. The hope is
> to polish it so that the wider community could use it and once the quality is
> high enough, make a proposal to add it to the standard Python libraries. We
> do need a standard library for contracts, otherwise projects with conflicting
> contract libraries can not integrate (e.g., the contracts can not be
> inherited between two different contract libraries).
>
> So far, the most important bits have been implemented in icontract:
>
> Preconditions, postconditions, class invariants
> Inheritance of the contracts (including strengthening and weakening of the
> inherited contracts)
> Informative violation messages (including information about the values
> involved in the contract condition)
> Sphinx extension to include contracts in the automatically generated
> documentation (sphinx-icontract)
> Linter to statically check that the arguments of the conditions are correct
> (pyicontract-lint)
>
> We are successfully using it in our code base and have been quite happy about
> the implementation so far.
>
> There is one bit still missing: accessing "old" values in the postcondition
> (i.e., shallow copies of the values prior to the execution of the function).
> This feature is necessary in order to allow us to verify state transitions.
>
> For example, consider a new dictionary class that has "get" and "put" methods:
>
> from typing import Optional
>
> from icontract import post
>
> class NovelDict:
> def length(self)->int:
> ...
>
> def get(self, key: str) -> Optional[str]:
> ...
>
> @post(lambda self, key, value: self.get(key) == value)
> @post(lambda self, key: old(self.get(key)) is None and old(self.length())
> + 1 == self.length(),
> "length increased with a new key")
> @post(lambda self, key: old(self.get(key)) is not None and
> old(self.length()) == self.length(),
> "length stable with an existing key")
> def put(self, key: str, value: str) -> None:
> ...
>
> How could we possible implement this "old" function?
>
> Here is my suggestion. I'd introduce a decorator "before" that would allow
> you to store whatever values in a dictionary object "old" (i.e. an object
> whose properties correspond to the key/value pairs). The "old" is then passed
> to the condition. Here is it in code:
>
> # omitted contracts for brevity
> class NovelDict:
> def length(self)->int:
> ...
>
> # omitted contracts for brevity
> def get(self, key: str) -> Optional[str]:
> ...
>
> @before(lambda self, key: {"length": self.length(), "get": self.get(key)})
> @post(lambda self, key, value: self.get(key) == value)
> @post(lambda self, key, old: old.get is None and old.length + 1 ==
> self.length(),
> "length increased with a new key")
> @post(lambda self, key, old: old.get is not None and old.length ==
> self.length(),
> "length stable with an existing key")
> def put(self, key: str, value: str) -> None:
> ...
>
> The linter would statically check that all attributes accessed in "old" have
> to be defined in the decorator "before" so that attribute errors would be
> caught early. The current implementation of the linter is fast enough to be
> run at save time so such errors should usually not happen with a properly set
> IDE.
>
> "before" decorator would also have "enabled" property, so that you can turn
> it off (e.g., if you only want to run a postcondition in testing). The
> "before" decorators can be stacked so that you can also have a more
> fine-grained control when each one of them is running (some during test, some
Re: [Python-ideas] Fwd: Keyword only argument on function call
On Tue, Sep 25, 2018 at 8:32 AM Anders Hovmöller wrote: > > I'm still not sure why all this focus on new syntax or convoluted IDE > enhancements. I presented a very simple utility function that accomplishes > exactly the started goal of DRY in keyword arguments. > > And I’ve already stated my reasons for rejecting this specific solution, > but I’ll repeat them for onlookers: > > 1. Huge performance penalty > Huh? Have you actually benchmarked this is some way?! A couple lookups into the namespace are really not pricey operations. The cost is definitely more than zero, but for any function that does anything even slightly costly, the lookups would be barely in the noise. > 2. Rather verbose, so somewhat fails on the stated goal of improving > readability > The "verbose" idea I propose is 3-4 characters more, per function call, than your `fun(a, b, *, this, that)` proposal. It will actually be shorter than your newer `fun(a, b, =this, =that)` proposal once you use 4 or more keyword arguments. > 3. Tooling* falls down very hard on this > It's true that tooling doesn't currently support my hypothetical function. It also does not support your hypothetical syntax. It would be *somewhat easier* to add special support for a function with a special name like `use()` than for new syntax. But obviously that varies by which tool and what purpose it is accomplishing. Of course, PyCharm and MyPy and PyLint aren't going to bother special casing a `use()` function unless or until it is widely used and/or part of the builtins or standard library. I don't actually advocate for such inclusion, but I wouldn't be stridently against that since it's just another function name, nothing really special. -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Why is design-by-contracts not widely
On 25/09/18 12:59, Hugh Fisher wrote: Thank you for a very well thought out post, Hugh. I completely agree. I just wanted to pull out one comment: Adding new syntax or semantics to a programming language very often adds accidental complexity. This is, in my view, the main reason why the bar for adding new syntax to Python is and should be so high. People advocating new syntax often remark that programmers can choose not to use it; they don't have to write their Python using the new syntax. That is true as far as it goes. However, programmers do have to *read* Python using the new syntax, so it does impact on them. The additional accidental complexity isn't something you can just dismiss because not everyone will have to use it. -- Rhodri James *-* Kynesim Ltd ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Fwd: Keyword only argument on function call
> I'm still not sure why all this focus on new syntax or convoluted IDE > enhancements. I presented a very simple utility function that accomplishes > exactly the started goal of DRY in keyword arguments. And I’ve already stated my reasons for rejecting this specific solution, but I’ll repeat them for onlookers: 1. Huge performance penalty 2. Rather verbose, so somewhat fails on the stated goal of improving readability 3. Tooling* falls down very hard on this My macropy implementation that I linked to solves 1, improves 2 somewhat (but not much), and handled half of 3 by resulting in code that tooling can validate that the passed variables exists but fails in that tooling won’t correctly validate that the arguments actually correspond to existing parameters. * by tooling I mean editors like PyCharm and static analysis tools like mypy / Anders ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Why is design-by-contracts not widely
> Date: Mon, 24 Sep 2018 09:46:16 +0200 > From: Marko Ristin-Kaufmann > To: Python-Ideas > Subject: Re: [Python-ideas] Why is design-by-contracts not widely > adopted? [munch] > Python is easier to write and read, and there are no libraries which are > close in quality in Eiffel space (notably, Numpy, OpenCV, nltk and > sklearn). I really don't see how the quality of these libraries have > anything to do with lack (or presence) of the contracts. OpenCV and Numpy > have contracts all over their code (written as assertions and not > documented), albeit with very non-informative violation messages. And they > are great libraries. Their users would hugely benefit from a more mature > and standardized contracts library with informative violation messages. I would say the most likely outcome of adding Design by Contract would be no change in the quality or usefulness of these libraries, with a small but not insignificant chance of a decline in quality. Fred Brooks in his "No Silver Bullet" paper distinguished between essential complexity, which is the problem we try to solve with software, and accidental complexity, solving the problems caused by your tools and/or process that get in the way of solving the actual problem. "Yak shaving" is a similar, less formal term for accidental complexity, when you have to do something before you can do something before you can actually do some useful work. Adding new syntax or semantics to a programming language very often adds accidental complexity. C and Python (currently) are known as simple languages. When starting a programming project in C or Python, there's maybe a brief discussion about C99 or C11, or Python 3.5 or 3.6, but that's it. There's one way to do it. On the other hand C++ is notorious for having been designed with a shovel rather than a chisel. The people adding all the "features" were well intentioned, but it's still a mess. C++ programming projects often start by specifying exactly which bits of the language the programming team will be allowed to use. I've seen these reach hundreds of pages in length, consuming God knows how many hours to create, without actually creating a single line of useful software. I think a major reason that Design by Contract hasn't been widely adopted in the three decades since its introduction is because, mostly, it creates more accidental complexity than it reduces essential complexity, so the costs outweigh any benefits. Software projects, in any language, never have enough time to do everything. By your own example, the Python developers of numpy, OpenCV, nlk, and sklearn; who most certainly weren't writing contracts; produced better quality software than the Eiffel equivalent developers who (I assume) did use DbC. Shouldn't the Eiffel developers be changing their development method, not the Python developers? Maybe in a world with infinite resources contracts could be added to those Python packages, or everything in PyPi, and it would be an improvement. But we don't. So I'd like to see the developers of numpy etc keep doing whatever it is that they're doing now. -- cheers, Hugh Fisher ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Fwd: Keyword only argument on function call
I'm still not sure why all this focus on new syntax or convoluted IDE
enhancements. I presented a very simple utility function that accomplishes
exactly the started goal of DRY in keyword arguments.
Yes, I wrote a first version that was incomplete. And perhaps these 8-9
lines miss some corner case. But the basic goal is really, really easy to
accomplish with existing Python.
>>> import inspect
>>> def reach(name):
... for f in inspect.stack():
... if name in f[0].f_locals:
... return f[0].f_locals[name]
... return None
...
>>> def use(names):
... kws = {}
... for name in names.split():
... kws[name] = reach(name)
... return kws
...
>>> def function(a=11, b=22, c=33, d=44):
... print(a, b, c, d)
...
>>> function(a=77, **use('b d'))
77 None 33 None
>>> def foo():
... a, b, c = 1, 2, 3
... function(a=77, **use('b d'))
...
>>> foo()
77 2 33 None
On Tue, Sep 25, 2018, 6:31 AM Anders Hovmöller wrote:
> Hi,
>
> I'd like to reopen this discussion if anyone is interested. Some things
> have changed since I wrote my original proposal so I'll first summarize:
>
> 1. People seem to prefer the syntax `foo(=a)` over the syntax I suggested.
> I believe this is even more trivial to implement in CPython than my
> original proposal anyway...
> 2. I have updated my analysis tool:
> https://gist.github.com/boxed/610b2ba73066c96e9781aed7c0c0b25c It will
> now also give you statistics on the number of arguments function calls
> have. I would love to see some statistics for other closed source programs
> you might be working on and how big those code bases are.
> 3. I have made a sort-of implementation with MacroPy:
> https://github.com/boxed/macro-kwargs/blob/master/test.py I think this is
> a dead end, but it was easy to implement and fun to try!
> 4. I have also recently had the idea that a foo=foo type pattern could be
> handled in for example PyCharm as a code folding feature (and maybe as a
> completion feature).
>
> I still think that changing Pythons syntax is the right way to go in the
> long run but with point 4 above one could experience what this feature
> would feel like without running a custom version of Python and without
> changing your code. I admit to a lot of trepidation about wading into
> PyCharms code though, I have tried to do this once before and I gave up.
>
> Any thoughts?
>
> / Anders
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Why is design-by-contracts not widely adopted?
> Date: Mon, 24 Sep 2018 09:46:16 +0200 > From: Marko Ristin-Kaufmann > To: Python-Ideas > Subject: Re: [Python-ideas] Why is design-by-contracts not widely > adopted? > Message-ID: > > Content-Type: text/plain; charset="utf-8" [munch] > Their users would hugely benefit from a more mature > and standardized contracts library with informative violation messages. Will respond in another message, because it's a big topic. > I really don't see how DbC has to do with duck typing (unless you reduce it > to mere isinstance conditions, which would simply be a straw-man argument) > -- could you please clarify? I argue that Design by Contract doesn't make sense for Python and other dynamically typed, duck typed languages because it's contrary to how the language, and the programmer, expects to work. In Python we can write something like: def foo(x): x.bar(y) What's the type of x? What's the type of y? What is the contract of bar? Don't know, don't care. x, or y, can be an instance, a class, a module, a proxy for a remote web service. The only "contract" is that object x will respond to message bar that takes one argument. Object x, do whatever you want with it. And that's a feature, not a bug, not bad design. It follows Postel's Law for Internet protocols of being liberal in what you accept. It follows the Agile principle of valuing working software over comprehensive doco. It allows software components to be glued together quickly and easily. It's a style of programming that has been successful for many years, not just in Python but also in Lisp and Smalltalk and Perl and JavaScript. It works. Not for everything. If I were writing the avionics control routines for a helicopter gas turbine, I'd use formal notation and static type checking and preconditions and whatnot. But I wouldn't be using Python either. > As soon as you need to document your code, and > this is what most modules have to do in teams of more than one person > (especially so if you are developing a library for a wider audience), you > need to write down the contracts. Please see above where I tried to > explained that 2-5) are inferior approaches to documenting contracts > compared to 1). You left off option 6), plain text. Comments. Docstrings. README files. Web pages. Books. In my experience, this is what most people consider documentation. A good book, a good blog post, can explain more about how a library works and what the implementation requirements and restrictions are than formal contract notation. In particular, contracts in Eiffel don't explain *why* they're there. As for 4) reading the code, why not? "Use the source, Luke" is now a programming cliche because it works. It's particularly appropriate for Python packages which are usually distributed in source form and, as you yourself noted, easy to read. -- cheers, Hugh Fisher ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] "old" values in postconditions
Have you looked at the built-in AST module, ast?
https://docs.python.org/3/library/ast.html
I don’t see anything preventing you from walking the AST Python itself can give
you- you’d look for two Set AST nodes if we were to do {{ }}.
There’s also the parser built-in module. You can use it if you first use
dis.code_info to get the source then re-parse it. It helps with parse trees.
Parse trees are generated before the AST I think. You’d use the parser module’s
ST objects with the token module’s constants, for example token.LBRACE or
token.RBRACE.
Have you looked at the built-in dis module? You can use dis.code_info(obj) to
get the string of the function. Then you could look for your specified syntax
with regex and recompile that with the ast module.
Sent from my iPhone
> On Sep 25, 2018, at 1:49 AM, Marko Ristin-Kaufmann
> wrote:
>
> Hi James,
> Thanks for the feedback!
>
> I also thought about decompiling the condition to find its AST and figure out
> what old values are needed. However, this is not so easily done at the moment
> as all the decompilation libraries I looked at (meta, ucompyle6) are simply
> struggling to keep up with the development of the python bytecode. In other
> words, the decompiler needs to be updated with every new version of python
> which is kind of a loosing race (unless the python devs themselves don't
> provide this functionality which is not the case as far as I know).
>
> There is macropy (https://github.com/lihaoyi/macropy) which was suggested on
> the other thread
> (https://groups.google.com/forum/#!topic/python-ideas/dmXz_7LH4GI) that I'm
> currently looking at.
>
> Cheers,
> Marko
>
>
>> On Tue, 25 Sep 2018 at 00:35, James Lu wrote:
>> You could disassemble (import dis) the lambda to biew the names of the
>> lambdas.
>>
>> @before(lambda self, key, _, length, get: self.length(), self.get(key))
>>
>> Perhaps you could disassemble the function code and look at all operations
>> or accesses that are done to “old.” and evaluate those expressions before
>> the function runs. Then you could “replace” the expression.
>> @post(lambda self, key, old: old.get is None and old.length + 1 ==
>> self.length())
>>
>> Either the system would grab old.get and old.length or be greedy and grab
>> old.get is None and old.length + 1. It would then replace the old.get and
>> old.length with injects that only respond to is None and +1.
>>
>> Or, a syntax like this
>> @post(lambda self, key, old: [old.get(old.key)] is None and
>> [old.self.length() + 1] ==
>> self.length())
>>
>> Where the stuff inside the brackets is evaluated before the decorated
>> function runs. It would be useful for networking functions or functions that
>> do something ephemeral, where data related to the value being accessed
>> needed for the expression no longer exists after the function.
>>
>> This does conflict with list syntax forever, so maybe either force people to
>> do list((expr,)) or use an alternate syntax like one item set syntax { } or
>> double set syntax {{ }} or double list syntax [[ ]]. Ditto with having to
>> avoid the literals for the normal meaning.
>>
>> You could modify Python to accept any expression for the lambda function and
>> propose that as a PEP. (Right now it’s hardcoded as a dotted name and
>> optionally a single argument list surrounded by parentheses.)
>>
>> I suggest that instead of “@before” it’s “@snapshot” and instead of “old”
>> it’s “snapshot”.
>>
>> Python does have unary plus/minus syntax as well as stream operators (<<,
>> >>) and list slicing syntax and the @ operator and operators & and | if you
>> want to play with syntax. There’s also the line continuation character for
>> crazy lambdas.
>>
>> Personally I prefer
>> @post(lambda self, key, old: {{old.self.get(old.key)}} and
>> {{old.self.length() + 1}} ==
>> self.length())
>>
>> because it’s explicit about what it does (evaluate the expressions within {{
>> }} before the function runs. I also find it elegant.
>>
>> Alternatively, inside the {{ }} could be a special scope where locals() is
>> all the arguments @pre could’ve received as a dictionary. For either option
>> you can remove the old parameter from the lambda. Example:
>> @post(lambda self, key: {{self.get(key)}} and {{self.length() + 1}} ==
>> self.length())
>>
>> Perhaps the convention should be to write {{ expr }} (with the spaces in
>> between).
>>
>> You’d probably have to use the ast module to inspect it instead of the dis
>> modul. Then find some way to reconstruct the expressions inside the double
>> brackets- perhaps by reconstructing the AST and compiling it to a code
>> object, or perhaps by finding the part of the string the expression is
>> located. dis can give you the code as a string and you can run a carefully
>> crafted regex on it.
>> ___
>> Python-ideas mailing list
>> Python-ideas@python.org
>>
Re: [Python-ideas] Keyword only argument on function call
Hi, I'd like to reopen this discussion if anyone is interested. Some things have changed since I wrote my original proposal so I'll first summarize: 1. People seem to prefer the syntax `foo(=a)` over the syntax I suggested. I believe this is even more trivial to implement in CPython than my original proposal anyway... 2. I have updated my analysis tool: https://gist.github.com/boxed/610b2ba73066c96e9781aed7c0c0b25c It will now also give you statistics on the number of arguments function calls have. I would love to see some statistics for other closed source programs you might be working on and how big those code bases are. 3. I have made a sort-of implementation with MacroPy: https://github.com/boxed/macro-kwargs/blob/master/test.py I think this is a dead end, but it was easy to implement and fun to try! 4. I have also recently had the idea that a foo=foo type pattern could be handled in for example PyCharm as a code folding feature (and maybe as a completion feature). I still think that changing Pythons syntax is the right way to go in the long run but with point 4 above one could experience what this feature would feel like without running a custom version of Python and without changing your code. I admit to a lot of trepidation about wading into PyCharms code though, I have tried to do this once before and I gave up. Any thoughts? / Anders ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Why is design-by-contracts not widely adopted?
Angus Hollands writes: > yes I'd pass in some kind of 'old' object as a proxy to the old object > state. Mostly you shouldn't need to do this, you can copy the state: def method(self, args): import copy old = copy.deepcopy(self) This is easy but verbose to do with a decorator, and I imagine a bunch of issues about the 'old' object with multiple decorators, so I omit it here. You might want a variety of such decorators. Ie, using copy.copy vs copy.deepcopy vs a special-case copy for a particular class because there are large objects that are actually constant that you don't want to copy (an "is" test would be enough, so the copy would actually implement part of the contract). Or the copy function could be an argument to the decorator or a method on the object. ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Why is design-by-contracts not widely adopted?
Marko Ristin-Kaufmann writes: > Thanks a lot for pointing us to macropy -- I was not aware of the library, > it looks very interesting! > > Do you have any experience how macropy fit Sorry, no. I was speaking as someone who is familiar with macros from Lisp but doesn't miss them in Python, and who also has been watching python-dev and python-ideas for about two decades now, so I've heard of things like MacroPy and know how the core developers think to a great extent. > I'm also a bit worried how macropy would work out in the libraries > published to pypi -- imagine if many people start using contracts. > Suddenly, all these libraries would not only depend on a contract library > but on a macro library as well. That's right. > Is that something we should care about? Yes. Most Pythonistas (at least at present) don't much like macros. They fear turning every program into its own domain-specific language. I can't claim much experience with dependency hell, but I think that's much less important from your point of view (see below). My point is mainly that, as you probably are becoming painfully aware, getting syntax changes into Python is a fairly drawnout process. For an example of the kind of presentation that motivates people to change their mind from the default state of "if it isn't in Python yet, YAGNI" to "yes, let's do *this* one", see https://www.python.org/dev/peps/pep-0572/#appendix-a-tim-peters-s-findings Warning: Tim Peters is legendary, though still active occasionally. All he has to do is post to get people to take notice. But this Appendix is an example of why he gets that kind of R-E-S-P-E-C-T.[1] So the whole thing is a secret plot ;-) to present the most beautiful syntax possible in your PEP (which will *not* be about DbC, but rather about a small set of enabling syntax changes, hopefully a singleton), along with an extended example, or a representative sample, of usage. Because you have a working implementation using MacroPy (or the less pretty[2] but fewer dependencies version based on condition strings and eval) people can actually try it on their own code and (you hope, they don't :-) they find a nestful of bugs by using it. > Potential dependency hell? (I already have a bad feeling about > making icontract depend on asttokens and considerin-lining > asttokens into icontract particularly for that reason). I don't think so. First, inlining an existing library is almost always a bad idea. As for the main point, if the user sticks to one major revision, and only upgrades to compatible bugfixes in the Python+stdlib distribution, I don't see why two or three libraries would be a major issue for a feature that the developer/project uses extremely frequently. I've rarely experienced dependency hell, and in both cases it was web frameworks (Django and Zope, to be specific, and the dependencies involved were more or less internal to those frameworks). If you or people you trust have other experience, forget what I just said. :-) Of course it depends on the library, but as long as the library is pretty strict about backward compatibility, you can upgrade it and get new functionality for other callers in your code base (which are likely to appear, you know -- human beings cannot stand to leave a tool unused once they install it!) > > Note that this means *you cannot use macros in a file that is run > > directly*, as it will not be passed through the import hooks. > > That would make contracts unusable in any stand-alone script, > right? Yes, but really, no: # The run.py described by the MacroPy docs assumes a script that # runs by just importing it. I don't have time to work out # whether that makes more sense. This idiom of importing just a # couple of libraries, and then invoking a function with a # conventional name such as "run" or "process" is quite common. # If you have docutils install, check out rstpep2html.py. import macropy.activate from my_contractful_library import main main() and away you go. 5 years from now that script will be a badge of honor among Pythonic DbCers, and you won't be willing to give it up! Just kidding, of course -- the ideal outcome is that the use case is sufficiently persuasive to justify a syntax change so you don't need MacroPy, or, perhaps some genius will come along and provide some obscure construct that is already legal syntax! HTH Footnotes: [1] R.I.P. Aretha! [2] Urk, I just realized there's another weakness to strings: you get no help on checking their syntax from the compiler. For a proof-of- concept that's OK, but if you end up using the DbC library in your codebase for a couple years while the needed syntax change gathers support, that would be really bad. ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Why is design-by-contracts not widely adopted?
On Mon, 24 Sep 2018 at 19:47, Marko Ristin-Kaufmann wrote: > > Hi, > > Thank you for your replies, Hugh and David! Please let me address the points > in serial. > > Obvious benefits > You both seem to misconceive the contracts. The goal of the > design-by-contract is not reduced to testing the correctness of the code, as > I reiterated already a couple of times in the previous thread. The contracts > document formally what the caller and the callee expect and need to satisfy > when using a method, a function or a class. This is meant for a module that > is used by multiple people which are not necessarily familiar with the code. > They are not a niche. There are 150K projects on pypi.org. Each one of them > would benefit if annotated with the contracts. You'll lose folks attention very quickly when you try to tell folk what they do and don't understand. Claiming that DbC annotations will improve the documentation of every single library on PyPI is an extraordinary claim, and such claims require extraordinary proof. I can think of many libraries where necessary pre and post conditions (such as 'self is still locked') are going to be noisy, and at risk of reducing comprehension if the DbC checks are used to enhance/extended documentation. Some of the examples you've been giving would be better expressed with a more capable type system in my view (e.g. Rust's), but I have no good idea about adding that into Python :/. Anyhow, the thing I value most about python is its pithyness: its extremely compact, allowing great developer efficiency, but the cost of testing is indeed excessive if the tests are not structured well. That said, its possible to run test suites with 10's of thousands of tests in only a few seconds, so there's plenty of headroom for most projects. -Rob ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Why is design-by-contracts not widely adopted?
Hi Mario, yes I'd pass in some kind of 'old' object as a proxy to the old object state. My demo can handle function calls, unless they themselves ultimately call something which can't be proxies e.g is instance (which delegates to the test class, not the instance), or boolean evaluation of some expression (e.g an if block). I don't think that this is awful - contracts should probably be fairly concise while expressive - but definitely non-ideal. I haven't really time to work on this at the moment; I admit, it was a specific problem of interest, rather than a domain I have much experience with. In fact, it was probably an excuse to overload all of the operators on an object! Kind regards, Angus On Mon, 24 Sep 2018, 20:09 Marko Ristin-Kaufmann, wrote: > Hi Barry, > I think the main issue with pyffel is that it can not support function > calls in general. If I understood it right, and Angus please correct me, > you would need to wrap every function that you would call from within the > contract. > > But the syntax is much nicer than icontract or dpcontracts (see these > packages on pypi). What if we renamed "args" argument and "old" argument in > those libraries to just "a" and "o", respectively? Maybe that gives > readable code without too much noise: > > @requires(lambda self, a, o: self.sum == o.sum - a.amount) > def withdraw(amount: int) -> None: > ... > > There is this lambda keyword in front, but it's not too bad? > > I'll try to contact dpcontracts maintainers. Maybe it's possible to at > least merge a couple of libraries into one and make it a de facto standard. > @Agnus, would you also like to join the effort? > > Cheers, > Marko > > > > > > Le lun. 24 sept. 2018 à 19:57, Barry Scott a > écrit : > >> >> >> On 23 Sep 2018, at 11:13, Angus Hollands wrote: >> >> Hi Marko, >> >> I think there are several ways to approach this problem, though am not >> weighing in on whether DbC is a good thing in Python. I wrote a simple >> implementation of DbC which is currently a run-time checker. You could, >> with the appropriate tooling, validate statically too (as with all >> approaches). In my approach, I use a “proxy” object to allow the contract >> code to be defined at function definition time. It does mean that some >> things are not as pretty as one would like - anything that cannot be hooked >> into with magic methods i.e isinstance, but I think this is acceptable >> as it makes features like old easier. Also, one hopes that it encourages >> simpler contract checks as a side-effect. Feel free to take a look - >> https://github.com/agoose77/pyffel >> It is by no means well written, but a fun PoC nonetheless. >> >> This is an interesting PoC, nice work! I like that its easy to read the >> tests. >> >> Given a library like this the need to build DbC into python seems >> unnecessary. >> >> What do other people think? >> >> Barry >> >> >> >> Regards, >> Angus >> >> ___ >> Python-ideas mailing list >> Python-ideas@python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> >> >> ___ >> Python-ideas mailing list >> Python-ideas@python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Why is design-by-contracts not widely adopted?
Hi Steve and others,
After some thinking, I'm coming to a conclusion that it might be wrong to
focus too much about how the contracts are written -- as long as they are
formal, easily transformable to another representation and fairly
maintainable.
Whether it's with a lambda, without, with "args" or "a", with "old" or "o"
-- it does not matter that much as long as it is pragmatic and not
something crazy complex. This would also mean that we should not add
complexity (*e.g., *by adding macros) and limit the magic as much as
possible.
It is actually much more important in which form they are presented to the
end-user. I already made an example with sphinx-icontract in a message
before -- an improved version might use mathematical symbols (*e.g., *replace
all() with ∀, replace len() with |.|, nicely use subscripts for ranges, use
case distinction with curly bracket "{" instead of if.. else ..., etc.).
This would make them even shorter and easier to parse. Let me iterate the
example I already pasted in the thread before to highlight what I have in
mind:
packagery.resolve_initial_paths(*initial_paths*)
Resolve the initial paths of the dependency graph by recursively adding *.py
files beneath given directories.
Parameters:
*initial_paths* (List[Path]) – initial paths as absolute paths
Return type:
List[Path]
Returns:
list of initial files (*i.e.* no directories)
Requires:
- all(pth.is_absolute() for pth in initial_paths)
Ensures:
- all(pth in result for pth in initial_paths if pth.is_file()) (Initial
files also in result)
- len(result) >= len(initial_paths) if initial_paths else result == []
- all(pth.is_absolute() for pth in result)
- all(pth.is_file() for pth in result)
The contracts need to extend __doc__ of the function accordingly (and the
contracts in __doc__ also need to reflect the inheritance of the
contracts!), so that we can use help().
There should be also a plugin for Pycharm, Pydev, vim and emacs to show the
contracts in an abbreviated and more readable form in the code and only
show them in raw form when we want to edit them (*i.e., *when we move
cursor over them). I suppose inheritance of contracts needs to be reflected
in quick-inspection windows, but not in the code view.
Diffs and github/bitbucket/... code reviews might be a bit cumbersome since
they enforce the raw form of the contracts, but as long as syntax is
pragmatic, I don't expect this to be a blocker.
Is this a sane focus?
Cheers,
Marko
On Tue, 25 Sep 2018 at 08:18, Marko Ristin-Kaufmann
wrote:
> Hi Steve,
> Thanks a lot for pointing us to macropy -- I was not aware of the library,
> it looks very interesting!
>
> Do you have any experience how macropy fit with current IDEs and static
> linters (pylint, mypy)? I fired up pylint and mypy on the sample code from
> their web site, played a bit with it and it seems that they go along well.
>
> I'm also a bit worried how macropy would work out in the libraries
> published to pypi -- imagine if many people start using contracts.
> Suddenly, all these libraries would not only depend on a contract library
> but on a macro library as well. Is that something we should care about?
> Potential dependency hell? (I already have a bad feeling about making
> icontract depend on asttokens and considerin-lining asttokens into
> icontract particularly for that reason).
>
> I'm also worried about this one (from
> https://macropy3.readthedocs.io/en/latest/overview.html):
>
>> Note that this means *you cannot use macros in a file that is run
>> directly*, as it will not be passed through the import hooks.
>
>
> That would make contracts unusable in any stand-alone script, right?
>
> Cheers,
> Marko
>
> On Tue, 25 Sep 2018 at 06:56, Stephen J. Turnbull <
> turnbull.stephen...@u.tsukuba.ac.jp> wrote:
>
>> Barry Scott writes:
>>
>> > > @requires(lambda self, a, o: self.sum == o.sum - a.amount)
>> > > def withdraw(amount: int) -> None:
>> > > ...
>> > >
>> > > There is this lambda keyword in front, but it's not too bad?
>> >
>> > The lambda smells of internals that I should not have to care about
>> > being exposed.
>> > So -1 on lambda being required.
>>
>> If you want to get rid of the lambda you can use strings and then
>> 'eval' them in the condition. Adds overhead.
>>
>> If you want to avoid the extra runtime overhead of parsing
>> expressions, it might be nice to prototype with MacroPy. This should
>> also allow eliminating the lambda by folding it into the macro (I
>> haven't used MacroPy but it got really good reviews by fans of that
>> kind of thing). It would be possible to avoid decorator syntax if you
>> want to with this implementation.
>>
>> I'm not sure that DbC is enough of a fit for Python that it's worth
>> changing syntax to enable nice syntax natively, but detailed reports
>> on a whole library (as long as it's not tiny) using DbC with a nice
>> syntax (MacroPy would be cleaner, but I think it would be easy to "see
>> through" the quoted
Re: [Python-ideas] Why is design-by-contracts not widely adopted?
Hi Steve, Thanks a lot for pointing us to macropy -- I was not aware of the library, it looks very interesting! Do you have any experience how macropy fit with current IDEs and static linters (pylint, mypy)? I fired up pylint and mypy on the sample code from their web site, played a bit with it and it seems that they go along well. I'm also a bit worried how macropy would work out in the libraries published to pypi -- imagine if many people start using contracts. Suddenly, all these libraries would not only depend on a contract library but on a macro library as well. Is that something we should care about? Potential dependency hell? (I already have a bad feeling about making icontract depend on asttokens and considerin-lining asttokens into icontract particularly for that reason). I'm also worried about this one (from https://macropy3.readthedocs.io/en/latest/overview.html): > Note that this means *you cannot use macros in a file that is run > directly*, as it will not be passed through the import hooks. That would make contracts unusable in any stand-alone script, right? Cheers, Marko On Tue, 25 Sep 2018 at 06:56, Stephen J. Turnbull < turnbull.stephen...@u.tsukuba.ac.jp> wrote: > Barry Scott writes: > > > > @requires(lambda self, a, o: self.sum == o.sum - a.amount) > > > def withdraw(amount: int) -> None: > > > ... > > > > > > There is this lambda keyword in front, but it's not too bad? > > > > The lambda smells of internals that I should not have to care about > > being exposed. > > So -1 on lambda being required. > > If you want to get rid of the lambda you can use strings and then > 'eval' them in the condition. Adds overhead. > > If you want to avoid the extra runtime overhead of parsing > expressions, it might be nice to prototype with MacroPy. This should > also allow eliminating the lambda by folding it into the macro (I > haven't used MacroPy but it got really good reviews by fans of that > kind of thing). It would be possible to avoid decorator syntax if you > want to with this implementation. > > I'm not sure that DbC is enough of a fit for Python that it's worth > changing syntax to enable nice syntax natively, but detailed reports > on a whole library (as long as it's not tiny) using DbC with a nice > syntax (MacroPy would be cleaner, but I think it would be easy to "see > through" the quoted conditions in an eval-based implementation) would > go a long way to making me sit up and take notice. (I'm not > influential enough to care about, but I suspect some committers would > be impressed too. YMMV) > > Steve > ___ > Python-ideas mailing list > Python-ideas@python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
