Re: WedWonder: Scripts and Modules
On 9/12/19 5:11 PM, DL Neil via Python-list wrote: > On 12/09/19 8:22 PM, Barry Scott wrote: > >> In large projects with many modules import with side-effect can make >> for a maintenance >> burden. > > > You seem to have unmasked an assumption under which I operate. (well > done - you know what 'they' say about assumptions!) > > As you say, (non-designed) side-effects are undesirable. > > My 'rule' is that modules only contain definitions, eg classes and > functions. Thus, *nothing* executes upon import. > > During a code review, some eagle-eye, noticed (and questioned) I had > re-factored some 'constants' which control a "switch" structure from > out of the module-space, and into the class where they 'belonged', > without having any other 'good reason'. > > > I have no recollection of the history or rationale for this > policy/practice, nor can I remember whether it 'belongs to Python' or > some other language from which I've imported it (hah, punny joke!) > Perhaps it comes from my preference to from...import... rather than > import...as... which largely eschews the module namespace? > (or perhaps I'm just a closet control-freak?) > > That said, I'm sure there must be modules of my code which break this > 'rule', somewhere (certainly in the 'quick and dirty' department). > Well technically, def and class are statements that are executed when the module is imported, you need to execute them to add the definitions into the appropriate namespace. Perhaps the 'rule' needs to be somewhat restated to a form where the only thing that should execute when importing a module are the statements needed to create the definitions for the module. -- Richard Damon -- https://mail.python.org/mailman/listinfo/python-list
Re: WedWonder: Scripts and Modules
On 13Sep2019 08:40, DL Neil wrote: On 12/09/19 10:59 AM, Cameron Simpson wrote: On 12Sep2019 08:24, DL Neil wrote: In this day-and-age do you have a script in live/production-use, which is also a module? What is the justification/use case? Many. Many many. 1: Many of my modules run their unit tests if invoked as the main programme. I used to do this, but now prefer to keep tests in separate modules - and separate directories. My tests are also in a separate module, but I can invoke them from the primary module. Just convenience. 2: Several modules are their own utility programme. I've got a heap of these - anything that provides useful access to something can often be usefully used from the command line. This is an very interesting idea - you're way ahead of me on this! Would it be fair to describe these as more Python for systems programming/system administration, than an application for (simple) users? Perhaps. I'm not sure this is a meaningful distinction, except in so far as simple users generally expecting a GUI, or programmers expecting an API instead of a command line. Having a command line, particularly one doing the common pattern of "command op args..." with various "op"s for common simple tasks makes it easier to use the module in a script, such as a non-Python script such as the shell. That said, half these "op"s tend to be admin tasks or simple integration-like tests. So "make a new app user record in the database" => "command new-user ...". But also "test this application function" ==> "command do-thing ..." where "do-thing" is something I'm debugging or testing which is normally painfully buried in the module or at the end of some tedious user interaction. And of course "command selftest" to run the unit tests :-) Gives me pause for thought: perhaps I lean too heavily on putting 'stuff' in the test routines (and view the application from that 'direction' too often). My weakness lies in the other direction - not enough test driven development. IMO, my test writing ability is weak. So I've got this chunk of code and now (==later) I want to test some part of it. Things I am finding useful recently include: - doctests (which have the advantage that I can also use them as example code in the docstrings) which are nice for laying out common use and also specific corner cases - the icontract Python module, which lets me put @require and @ensure decorators on functions to express preconditions and postconditions; really great for catching misuse, especially if you also put some insinstance type checks in there for getting parameters misordered Both of these lack the cumbersomeness of a unit test suite. OTOH, they do not lend themselves to complex tests. Multiple entry-point systems seem relatively unusual these days - perhaps a natural flow-on effect of the rise of gui- and menu-based systems? Unsure. Quite possibly. With one client, over the years, we've developed a number of (basically) statistical analyses. Each analysis was born from a separate project. Each lives in its own directory (tree). There are some common modules held in a separate 'utility' library/package/directory. Whilst it has been suggested, the idea of an "overarching" menu-type top-level controller/distributor has never risen up the ranks of the "backlog" (sensible criticism: the money would be better spent on other priorities) - they (or at least the older ones) are quite happy to 'drop' to the command line and type the required command+args. I think that's the closest I come to what you have described. (out of interest) Would you characterise it as a common project structure? It certainly seems reasonable. And as you suggest, there's you've got a bunch of main programmes associated with distinct modules/packages. Cheers, Cameron Simpson -- https://mail.python.org/mailman/listinfo/python-list
Re: WedWonder: Scripts and Modules
On 12/09/19 8:22 PM, Barry Scott wrote: On 11 Sep 2019, at 21:24, DL Neil via Python-list wrote: In this day-and-age do you have a script in live/production-use, which is also a module? What is the justification/use case? (discounting distutils and similar installation tools, or unit testing methodology) There are over 500 questions on StackOverflow which refer to Python's if __name__ == __main__: construct. Even more if you include the idea of a main() multiple entry-point. This construct enables code to distinguish between being "run" as a "script", and being imported as a "module". (which is not in question) In my mind this is a question about what side-effects of importing a module are desireable and which are not. Precise description! A trivia script does not need the __name__ == '__main__' as its all about its side effects. As scripts become more complex having it run on import might make debugging harder and prevents reuse. Is this an informal distinction: that modules are written for re-use but main-lines to be unique/single-purpose? For example I will import a script at the REPL and examine it and call function in it to help me understand and fix problems. Having a __name__ == '__main__' is important to allow this. Why? If we import sys (from the PSL, purely as an example), we don't expect/need any execution phase, can immediately follow (in the REPL) with help(sys) and similar, and can debug/explore from there: import sys help(sys) sys.path ['', '/usr/lib64/python37.zip', '/usr/lib64/python3.7', '/usr/lib64/python3.7/lib-dynload', '/usr/lib64/python3.7/site-packages', '/usr/lib/python3.7/site-packages'] I often have modules that are part of a larger program that have their own main() functions to unittest or give access to parsers etc. and the main() is the sole content of the if __name__ etc structure? In large projects with many modules import with side-effect can make for a maintenance burden. You seem to have unmasked an assumption under which I operate. (well done - you know what 'they' say about assumptions!) As you say, (non-designed) side-effects are undesirable. My 'rule' is that modules only contain definitions, eg classes and functions. Thus, *nothing* executes upon import. During a code review, some eagle-eye, noticed (and questioned) I had re-factored some 'constants' which control a "switch" structure from out of the module-space, and into the class where they 'belonged', without having any other 'good reason'. I have no recollection of the history or rationale for this policy/practice, nor can I remember whether it 'belongs to Python' or some other language from which I've imported it (hah, punny joke!) Perhaps it comes from my preference to from...import... rather than import...as... which largely eschews the module namespace? (or perhaps I'm just a closet control-freak?) That said, I'm sure there must be modules of my code which break this 'rule', somewhere (certainly in the 'quick and dirty' department). -- Regards =dn -- https://mail.python.org/mailman/listinfo/python-list
Re: WedWonder: Scripts and Modules
On 12/09/19 10:59 AM, Cameron Simpson wrote: On 12Sep2019 08:24, DL Neil wrote: In this day-and-age do you have a script in live/production-use, which is also a module? What is the justification/use case? Many. Many many. 1: Many of my modules run their unit tests if invoked as the main programme. I used to do this, but now prefer to keep tests in separate modules - and separate directories. 2: Several modules are their own utility programme. I've got a heap of these - anything that provides useful access to something can often be usefully used from the command line. This is an very interesting idea - you're way ahead of me on this! Would it be fair to describe these as more Python for systems programming/system administration, than an application for (simple) users? (what other kind is there???) Consider: if you write a package, would it have a __main__.py? Well, if the answer is ever "yes" then the same applies to ordinary modules, simple enough to not be worth splitting onto a package. May not properly appreciate this point... So, yes, for me this is really really common. Even for my current client project, which is largely a package, several of the individual modules within the package have their own main programmes for testing and for various utility tasks dealing solely with that particular subsystem. There's an overarching shell script to set up the environment and then do various things from the command line, and it directly invokes particular modules for some operations that act only on one subsystem. Interesting! Gives me pause for thought: perhaps I lean too heavily on putting 'stuff' in the test routines (and view the application from that 'direction' too often). Multiple entry-point systems seem relatively unusual these days - perhaps a natural flow-on effect of the rise of gui- and menu-based systems? With one client, over the years, we've developed a number of (basically) statistical analyses. Each analysis was born from a separate project. Each lives in its own directory (tree). There are some common modules held in a separate 'utility' library/package/directory. Whilst it has been suggested, the idea of an "overarching" menu-type top-level controller/distributor has never risen up the ranks of the "backlog" (sensible criticism: the money would be better spent on other priorities) - they (or at least the older ones) are quite happy to 'drop' to the command line and type the required command+args. I think that's the closest I come to what you have described. (out of interest) Would you characterise it as a common project structure? -- Regards =dn -- https://mail.python.org/mailman/listinfo/python-list
Re: WedWonder: Scripts and Modules
On Thu, Sep 12, 2019 at 6:57 PM Barry Scott wrote: > As scripts become more complex having it run on import might make debugging > harder > and prevents reuse. > > For example I will import a script at the REPL and examine it and call > function in it to > help me understand and fix problems. Having a __name__ == '__main__' is > important > to allow this. If the script is fairly simple, and hasn't been built to operate as a module, it's often effective enough to just run "python3 -i scriptname.py" and have it do its stuff and immediately drop to REPL. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: WedWonder: Scripts and Modules
> On 11 Sep 2019, at 21:24, DL Neil via Python-list > wrote: > > In this day-and-age do you have a script in live/production-use, which is > also a module? What is the justification/use case? > > (discounting distutils and similar installation tools, or unit testing > methodology) > > > There are over 500 questions on StackOverflow which refer to Python's > > if __name__ == __main__: > > construct. Even more if you include the idea of a main() multiple entry-point. > > This construct enables code to distinguish between being "run" as a "script", > and being imported as a "module". (which is not in question) > In my mind this is a question about what side-effects of importing a module are desireable and which are not. A trivia script does not need the __name__ == '__main__' as its all about its side effects. As scripts become more complex having it run on import might make debugging harder and prevents reuse. For example I will import a script at the REPL and examine it and call function in it to help me understand and fix problems. Having a __name__ == '__main__' is important to allow this. I often have modules that are part of a larger program that have their own main() functions to unittest or give access to parsers etc. In large projects with many modules import with side-effect can make for a maintenance burden. Barry > -- > Regards, > =dn > -- > https://mail.python.org/mailman/listinfo/python-list > -- https://mail.python.org/mailman/listinfo/python-list
Re: WedWonder: Scripts and Modules
On 12/09/19 10:37 AM, Alan Bawden wrote: DL Neil writes: ... However, reversing the question in my mind led me to ask (myself): how many scripts do I have (in "production use") which are ever used (also) as a module by some other script? I think the answer is/was: "none"! Accordingly, (spoiler alert: this statement may be heresy) I stopped using the "if". I found that the problem with doing this is that `pydoc' now _runs_ my scripts, when all I wanted was a quick look at the constants and procedures defined in them. I haven't experienced this. However, I'll make a note to test... -- Regards =dn -- https://mail.python.org/mailman/listinfo/python-list
Re: WedWonder: Scripts and Modules
On 12Sep2019 08:24, DL Neil wrote:
In this day-and-age do you have a script in live/production-use,
which is also a module? What is the justification/use case?
Many. Many many.
1: Many of my modules run their unit tests if invoked as the main
programme.
2: Several modules are their own utility programme. I've got a heap of
these - anything that provides useful access to something can often be
usefully used from the command line.
A quick grep from my bin directory shows stuff like this:
[~/hg/css/bin(hg:default)]fleet*> grep ' -m cs.' *
beyonwiz:exec python3 -m cs.app.beyonwiz ${1+"$@"}
calibre:exec py3 -m cs.app.calibre ${1+"$@"}
haproxy-tool:exec python2 -m cs.app.haproxy ${1+"$@"}
iphoto:exec python3 -m cs.app.osx.iphoto ${1+"$@"}
maildb:exec python3 -m cs.app.maildb ${1+"$@"}
mailfiler:exec python3 -m cs.app.mailfiler ${1+"$@"}
mklinks:exec python -m cs.app.mklinks ${1+"$@"}
myke:exec python3 -m cs.app.myke ${1+"$@"}
nodedb:exec python -m cs.nodedb.__init__ "$CS_NODEDB_URL" "$op" ${1+"$@"}
pilfer:exec python3 -m cs.app.pilfer ${1+"$@"}
portfwd:exec python3 -m cs.app.portfwd ${1+"$@"}
s3-clone-website: set -- python3 -m cs.app.aws s3 "$s3bucket" sync-up -D -% .
svcd:exec python3 -m cs.app.svcd ${1+"$@"}
vbox:exec python -m cs.app.virtualbox ${1+"$@"}
vt:exec python3 -m cs.vt ${1+"$@"}
wol:exec python -m cs.wol ${1+"$@"}
Consider: if you write a package, would it have a __main__.py?
Well, if the answer is ever "yes" then the same applies to ordinary
modules, simple enough to not be worth splitting onto a package.
So, yes, for me this is really really common.
Even for my current client project, which is largely a package,
several of the individual modules within the package have their own
main programmes for testing and for various utility tasks dealing
solely with that particular subsystem. There's an overarching shell
script to set up the environment and then do various things from the
command line, and it directly invokes particular modules for some
operations that act only on one subsystem.
Cheers,
Cameron Simpson
--
https://mail.python.org/mailman/listinfo/python-list
Re: WedWonder: Scripts and Modules
DL Neil writes: > ... However, reversing the question in my mind led me to ask (myself): > how many scripts do I have (in "production use") which are ever used > (also) as a module by some other script? I think the answer is/was: > "none"! Accordingly, (spoiler alert: this statement may be heresy) I > stopped using the "if". I found that the problem with doing this is that `pydoc' now _runs_ my scripts, when all I wanted was a quick look at the constants and procedures defined in them. It's pretty handy sometimes to fire up `pydoc' in http server mode in a directory full of random Python code (some scripts and some modules), and use it to browse around to figure out what everything does. Scripts that aren't protected with the usual `if __name__' are more likely to crash in that situation, often crashing `pydoc' in the process. In fact, I have been known to add `if __name__' to my colleagues' Python scripts, just so that I can safely `pydoc' them. -- Alan Bawden -- https://mail.python.org/mailman/listinfo/python-list
Re: WedWonder: Scripts and Modules
On Thu, Sep 12, 2019 at 7:34 AM DL Neil via Python-list wrote: > > On 12/09/19 8:43 AM, Chris Angelico wrote: > > Yes, absolutely. It's the easiest way to share code between two > > scripts. Here's an example that I created recently: > > > > https://github.com/Rosuav/shed/blob/master/BL1_find_items.py > > https://github.com/Rosuav/shed/blob/master/BL2_find_items.py > > Interestingly, we appear to have opposite approaches to this situation - > as soon as I think the word "common" it morphs immediately into "module". I think we actually both agree here. The only difference is that in my view, a module that's needed by only one other script can remain as the script it is, whereas you instantly make it *exclusively* an importable module. BL1_find_items *is* a module as well as a script, and it has to be implemented with that in mind. > I have also needed to scan a directory/tree recently, but took the > course of refactoring a previous implementation's 'directory-walk' code > into its own (generator) function, and thus the mainline of the newer > application calls the utility function and chooses whether or not to > deal with each 'found' file/dir in-turn. The previous script using the > code was similarly refactored. > (and there are no doubt ?many more scripts 'lurking' in my code-base > which could be similarly 'improved' - such frequent commonality leading > to my preference). > > My bias (I'm not criticising/complaining about/decrying the choices you > have illustrated) probably comes out of "separation of concerns". An > issue which has 'bitten' me, more than once... > > For example both BL1 and BL2 feature: > > def money(savefile): savefile.money[0] += 500 # Add more dollars > > - a minor issue, agreed (and 'picking on it' purely to make a point). Actually no they don't, and that's part of the subtlety of working with the two distinctly different file formats. In BL2, money is an array (dollars, Eridium, Seraph tokens, etc), but in BL1, it's a single value. So the BL2 version is as you posted, but the BL1 one is slightly different: def money(savefile): savefile.money += 500 Both files follow the same structure of synthesizers (hence the common code in the FunctionArg class), so they will look similar, but they're not identical enough to actually share. :( > The "s-o-c" is, that one day it will be possible to decide that the unit > of addition should change, but only (remember to) amend the code in one > of the two scripts! If I cared that the amount added be the same, then I could put that into a constant, I guess; if anything, what I might do is parameterize it, but it's a fairly arbitrary figure and doesn't really matter much. > Which brings me back to the preference for 'encapsulation' (just to > prove I can speak "OOP"), and why I would have written quite separate > 'main-lines' and extracted money() (and presumably a lot more) into a > module which could then be called by both. When I started the BL1 project, I thought about sharing heaps of code with BL2, but it just didn't work out. > Aside2: > Meantime, thanks for the opportunity to review your code. I was thinking > of considering "data classes" (new in v3.7, IIRC) in relation to an SQL > interface on today's ToDo list - constructing the application's queries > and the data transfers 'in' and 'out', without the 'weight' of > SQLAlchemy. Perhaps? > Data classes are awesome. Also, this is a great demonstration of why annotations shouldn't be restricted to any particular definition of "type" - it's wonderfully elegant to define a data structure with constants for signatures, types for most values, and then use range(256) or range(65536) to mean 1-byte or 2-byte integers. Within the context of the parser/encoder here, these things ARE types, but in a general Python context, they aren't :) Incidentally, data classes can be used pre-3.7 if you pip install the backport module. So it doesn't have to be a hard requirement. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: WedWonder: Scripts and Modules
On 12/09/19 8:43 AM, Chris Angelico wrote: On Thu, Sep 12, 2019 at 6:34 AM DL Neil via Python-list wrote: In this day-and-age do you have a script in live/production-use, which is also a module? What is the justification/use case? Yes, absolutely. It's the easiest way to share code between two scripts. Here's an example that I created recently: https://github.com/Rosuav/shed/blob/master/BL1_find_items.py https://github.com/Rosuav/shed/blob/master/BL2_find_items.py These programs do similar jobs on very different formats of file, so there's a small amount of common code and a large amount that isn't common. One of the common sections is the FunctionArg class, which ties in with argparse; it's implemented in BL1_find_items, and then imported into BL2_find_items. Of course I could break this out into its own dedicated file... but why bother? It's not significant enough to warrant its own module, and I don't see any value in an importable file of "all the stuff that I might feel like importing"; it's just these two files that will need this. Basically, the script/module distinction is a convenient way to simplify a common situation that doesn't need the overhead of anything else. If the code starts getting used in lots more places, it'll eventually get promoted to actual module. Thanks for such a rapid response! Interestingly, we appear to have opposite approaches to this situation - as soon as I think the word "common" it morphs immediately into "module". I have also needed to scan a directory/tree recently, but took the course of refactoring a previous implementation's 'directory-walk' code into its own (generator) function, and thus the mainline of the newer application calls the utility function and chooses whether or not to deal with each 'found' file/dir in-turn. The previous script using the code was similarly refactored. (and there are no doubt ?many more scripts 'lurking' in my code-base which could be similarly 'improved' - such frequent commonality leading to my preference). My bias (I'm not criticising/complaining about/decrying the choices you have illustrated) probably comes out of "separation of concerns". An issue which has 'bitten' me, more than once... For example both BL1 and BL2 feature: def money(savefile): savefile.money[0] += 500 # Add more dollars - a minor issue, agreed (and 'picking on it' purely to make a point). The "s-o-c" is, that one day it will be possible to decide that the unit of addition should change, but only (remember to) amend the code in one of the two scripts! (when it comes to trusting my memory, I'm an 'old git' with history, but not an old git repo with "history"!) Which brings me back to the preference for 'encapsulation' (just to prove I can speak "OOP"), and why I would have written quite separate 'main-lines' and extracted money() (and presumably a lot more) into a module which could then be called by both. Aside1: However, you've started me thinking about a related consideration/philosophy (Danger Will Robinson!) - but please let me cogitate on that for a day or so... Aside2: Meantime, thanks for the opportunity to review your code. I was thinking of considering "data classes" (new in v3.7, IIRC) in relation to an SQL interface on today's ToDo list - constructing the application's queries and the data transfers 'in' and 'out', without the 'weight' of SQLAlchemy. Perhaps? -- Regards =dn -- https://mail.python.org/mailman/listinfo/python-list
Re: WedWonder: Scripts and Modules
On Thu, Sep 12, 2019 at 6:34 AM DL Neil via Python-list wrote: > > In this day-and-age do you have a script in live/production-use, which > is also a module? What is the justification/use case? > Yes, absolutely. It's the easiest way to share code between two scripts. Here's an example that I created recently: https://github.com/Rosuav/shed/blob/master/BL1_find_items.py https://github.com/Rosuav/shed/blob/master/BL2_find_items.py These programs do similar jobs on very different formats of file, so there's a small amount of common code and a large amount that isn't common. One of the common sections is the FunctionArg class, which ties in with argparse; it's implemented in BL1_find_items, and then imported into BL2_find_items. Of course I could break this out into its own dedicated file... but why bother? It's not significant enough to warrant its own module, and I don't see any value in an importable file of "all the stuff that I might feel like importing"; it's just these two files that will need this. Basically, the script/module distinction is a convenient way to simplify a common situation that doesn't need the overhead of anything else. If the code starts getting used in lots more places, it'll eventually get promoted to actual module. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
WedWonder: Scripts and Modules
In this day-and-age do you have a script in live/production-use, which is also a module? What is the justification/use case? (discounting distutils and similar installation tools, or unit testing methodology) There are over 500 questions on StackOverflow which refer to Python's if __name__ == __main__: construct. Even more if you include the idea of a main() multiple entry-point. This construct enables code to distinguish between being "run" as a "script", and being imported as a "module". (which is not in question) It intrigues me how often (per SO, quoted above) this causes anything from a pause to complete consternation amongst Python neophytes. In my own case, I accepted it as a "Pythonic" idiom, adopted its use in my scripts, and moved on. Until I adopted unittest/pytest and took-on the closer definitions of TDD, I used to use the script/module switch as a means of testing modules (see also 'good, old days', below). These days, pytest requires a different approach and splits 'test code' from (production) "delivered code". Oh yeah! However, I can't recall ever gaining similar benefit from using the 'switch' within code designed to be a "script"! Ages ago some beginner asked me the 'script/module switch' question, and I gave the standard/formulaic/Pythonic answer. However, reversing the question in my mind led me to ask (myself): how many scripts do I have (in "production use") which are ever used (also) as a module by some other script? I think the answer is/was: "none"! Accordingly, (spoiler alert: this statement may be heresy) I stopped using the "if". Back in the ?good, old days of mainframes we wrote "monolithic" programs. Then we developed "modular programming" and started to split code into functional units, ie functions, subroutines, paragraphs, procedures. Thus "code re-use" was born. (see what a new idea it is! Subroutine libraries on mag-tape, anyone?) We distinguished a subroutine or "called code" (importable module in Python) from the calling-code by calling the latter the "main-line" (nothing to do with/say no to drugs). We even developed little "stub" programs; which would exercise or test specific subroutines to ensure that they worked (wow, how new is (much of) Test-Driven Development?) So (putting nostalgia aside), these days my Python *scripts* are pure "main-line" code. Faced with the perennial "main/name" question again yesterday, led me to review the above policy/laziness, and thus this "Wednesday Wondering": - modules aside, how often do we write multiple-entry code these days, as opposed to having multiple scripts/main-lines which call re-usable functional modules, as-and-when? I still don't have a single file containing a combination script/module amongst my applications. Do you? Why/who/how/when? -- Regards, =dn -- https://mail.python.org/mailman/listinfo/python-list
