Re: python 2.7.12 on Linux behaving differently than on Windows
BartC wrote: Which means that input of A B* *C will all end up running together so that you have no idea what is what or which file corresponds to which expansion. So if you intend your program to be used on unix, you need to design it so that it doesn't depend on such distinctions. Now the problem is to determine whether processed input of U V W X Y Z are intended bona fide inputs, or whether they just happened to result from some random associations with the files in the current directory. You're entitled to assume that a Unix user using your program from the shell is reasonably intelligent and at least moderately familiar with how Unix shells behave. Such a user will *not* be using unescaped wildcards unless they *intend* them to be expanded. So you are not going to get "random associations with the files in the current directory". That's another imaginary bogeyman. But this is what happens when you have to work to the lowest common denominator. If you're unwilling to do things differently on different platforms, then yes, you need to restrict yourself to their intersection. That's an inescapable fact of logic. (I'm in the middle of porting my console editor to Linux. But one problem is that on one Linux, half the key combinations (eg. Shift+Ctrl+B) are not recognised. If you're reading characters from a tty device in raw mode (which I assume is what you mean by "console editor") I'm not aware of *any* Unix system that will let you distinguish between Ctrl+B and Shift+Ctrl+B that way. That's because the tty driver delivers ASCII characters, and there are no separate ASCII codes for shifted control characters. Except that was only two Linuxes; perhaps on others, the keyboard will likely be crippled in some other way. No, they'll all be the same -- if it has an ASCII code, you'll be able to get it from a tty device, otherwise you won't. How people manage to do anything on such an OS I've no idea. Programs that need to be able to distinguish all of the modifiers are normally implemented as GUI applications, which get keyboard input a different way. -- Greg -- https://mail.python.org/mailman/listinfo/python-list
Re: python 2.7.12 on Linux behaving differently than on Windows
On 12/07/2016 11:26 PM, Gregory Ewing wrote: > BartC wrote: >> println dirlist(cmdparams[2])# Windows >> println tail(cmdparams) # Linux > > I'm not familiar with your language, so I'll reply > in Python. If you write it like this: > > for arg in sys.argv[1:]: >for name in glob(arg): > print(name) > > then the user will be able to invoke it the same way > on both systems. With the caveat that certain filenames may cause issues on Linux. For example, a file like "Chicago - Will You Still Love Me?.ogg". Or a filename with a "*" in the name. Probably it would still work since each of these filenames when taken as a glob actually match themselves when expanded. But they could match other files too that the user did not want to pass in. Personally I'd include logic to only run args through glob if I was on Windows. These are the perils of cross-platform compatibility. -- https://mail.python.org/mailman/listinfo/python-list
Re: python 2.7.12 on Linux behaving differently than on Windows
BartC wrote: println dirlist(cmdparams[2])# Windows println tail(cmdparams) # Linux I'm not familiar with your language, so I'll reply in Python. If you write it like this: for arg in sys.argv[1:]: for name in glob(arg): print(name) then the user will be able to invoke it the same way on both systems. -- Greg -- https://mail.python.org/mailman/listinfo/python-list
Re: python 2.7.12 on Linux behaving differently than on Windows
BartC wrote: Only the tiny minority that can be meaningfully invoked on an arbitrary number of files at the same time. Um... you mean the "tiny minority" that includes just about *every* unix utility that operates on files? -- Greg -- https://mail.python.org/mailman/listinfo/python-list
Re: python 2.7.12 on Linux behaving differently than on Windows
BartC wrote: And globbing doesn't take care of all of it: a Linux program still has to iterate over a loop of filenames. The same as on Windows, except the latter will need to call a function to deliver the next filename. Actually, most of them will require *two* loops, one to iterate over a sequence of filespecs, and another for the files matching each filespec. Whereas the unix program only requires one loop. -- Greg -- https://mail.python.org/mailman/listinfo/python-list
Re: MacOSX SpeechRecognition installation problems
On 12/07/2016 11:09 PM, 3dB wrote: > trying to install SpeechRecognition for Python results in error: > > running install_lib > creating /Library/Python/2.7/site-packages/speech_recognition > error: could not create > '/Library/Python/2.7/site-packages/speech_recognition': Permission denied > > > Any advice on how to fix? Are you using sudo? > > Follow information may be helpful: > > MacOSX 10.12.1 > Python 2.7.10 (default, Jul 30 2016, 18:31:42) > [GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.34)] on darwin > > /usr/local/bin/brew > /usr/bin/python > > pip 9.0.1 from /Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg (python > 2.7) > -- https://mail.python.org/mailman/listinfo/python-list
Re: python 2.7.12 on Linux behaving differently than on Windows
Paul Moore wrote: On Tuesday, 6 December 2016 21:44:18 UTC, Gregory Ewing wrote: What you need to understand is that, to a Unix user, * and ? are *just as well known* as |, < and >. And to Windows users. Just because BartC is proposing a silly distinction between what "Unix users" and "Windows users" expect, doesn't mean there is such a distinction. Yes, Windows users will know about * too, but their understanding of it will be different. The real point is that Unix users will know that it has a special meaning to the shell and work accordingly. They won't be tripped up by the kind of problems Bart imagines they will have with arguments containing *. -- Greg -- https://mail.python.org/mailman/listinfo/python-list
MacOSX SpeechRecognition installation problems
trying to install SpeechRecognition for Python results in error: running install_lib creating /Library/Python/2.7/site-packages/speech_recognition error: could not create '/Library/Python/2.7/site-packages/speech_recognition': Permission denied Any advice on how to fix? Follow information may be helpful: MacOSX 10.12.1 Python 2.7.10 (default, Jul 30 2016, 18:31:42) [GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.34)] on darwin /usr/local/bin/brew /usr/bin/python pip 9.0.1 from /Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg (python 2.7) -- https://mail.python.org/mailman/listinfo/python-list
Re: python 2.7.12 on Linux behaving differently than on Windows
Steve D'Aprano wrote: I don't know any Unix programs that provide file spec processing. 'find' does, but only for its -name argument, and only because it does something different with it from what the shell would do. And you do need to quote it if it contains glob characters. -- Greg -- https://mail.python.org/mailman/listinfo/python-list
Re: sets anomaly
On Thursday 08 December 2016 02:17, Rustom Mody wrote:
> Trying to write some code using sets (well frozen sets)
> And was hit by this anomaly
>
> This is the behavior of lists I analogously expect in sets:
>
[]
> []
[[]]
> [[]]
>
> ie the empty list and the list of the empty list are different things
That's a property of the *list display*, [], not the list() constructor. The
list constructor takes an iterable of items, so if you pass it an empty
iterable, you get an empty list:
py> list([])
[]
Since there's no frozenset display, there's no analogy with [[]], and you
similarly get a single empty frozen set:
py> frozenset([])
frozenset()
Notice the repr()? Like list(), tuple(), set() and dict(), calling frozenset()
with no arguments returns an empty frozenset:
py> frozenset()
frozenset()
> And then some figuring out how to get an empty set into a set
> This is the best I get:
f([f([])])
> frozenset({frozenset()})
py> Ø = frozenset()
py> frozenset([Ø])
frozenset({frozenset()})
--
Steven
"Ever since I learned about confirmation bias, I've been seeing
it everywhere." - Jon Ronson
--
https://mail.python.org/mailman/listinfo/python-list
Re: Get min and max dates
On Thursday 08 December 2016 03:15, DFS wrote:
> dts= ['10-Mar-1998',
> '20-Aug-1997',
> '06-Sep-2009',
> '23-Jan-2010',
> '12-Feb-2010',
> '05-Nov-2010',
> '03-Sep-2009',
> '07-Nov-2014',
> '08-Mar-2013']
>
> Of course, the naive:
> min(dates) = '03-Sep-2009'
> max(dates) = '23-Jan-2010'
> is wrong.
>
> Not wanting to use any date parsing libraries, I came up with:
That's where you have gone wrong. By not using a date parsing library which
works and has been tested, you have to write your own dodgy and possibly buggy
parsing routine yourself.
Why reinvent the wheel?
>
> m=[('Dec','12'),('Nov','11'),('Oct','10'),('Sep','09'),
> ('Aug','08'),('Jul','07'),('Jun','06'),('May','05'),
> ('Apr','04'),('Mar','03'),('Feb','02'),('Jan','01')]
>
> #create new list with properly sortable date (MMDD)
> dts2 = []
> for d in dts:
> dts2.append((d[-4:]+dict(m)[d[3:6]]+d[:2],d))
Why do you convert m into a dict each and every time through the loop? If you
have a million items, you convert m to a dict a million times, and then throw
away the dict a million times. And then you'll complain that Python is slow...
m = {'Jan': 1, 'Feb': 2, 'Mar': 3, 'Apr': 4, 'May': 5, 'Jun': 6,
'Jul': 7, 'Aug': 8, 'Sep': 9, 'Oct': 10, 'Nov': 11, 'Dec': 12}
There you go, fixed that.
As for the rest, I haven't bothered to check your code. But here's a much
simpler, more Pythonic, way:
def date_to_seconds(string):
return time.mktime(time.strptime(string, '%d-%b-%Y'))
min(dts, key=date_to_seconds) # returns '20-Aug-1997'
max(dts, key=date_to_seconds) # returns '07-Nov-2014'
Works for at least Python 3.3 or better.
The best part of this is that you can now support any time format you like,
just by changing the format string '%d-%b-%Y'.
--
Steven
"Ever since I learned about confirmation bias, I've been seeing
it everywhere." - Jon Ronson
--
https://mail.python.org/mailman/listinfo/python-list
Re: python 2.7.12 on Linux behaving differently than on Windows
On Thursday 08 December 2016 12:15, BartC wrote:
> On 08/12/2016 00:09, Steve D'Aprano wrote:
>> On Thu, 8 Dec 2016 02:48 am, BartC wrote:
>
>>> You make it sound like a big deal. Here's a program (in my language not
>>> Python) which prints the list of files matching a file-spec:
>>>
>>> println dirlist(cmdparams[2])
>>
>> Does dirlist provide an escaping mechanism so that you can disable filename
>> globbing?
>
> No. It simply produces a list of files in a path matching a particular
> wildcard pattern using * and ?.
>
> That's all. I know the value of keeping things straightforward instead
> of throwing in everything you can think of. The file-matching is done by
> WinAPI functions. The Linux version was done last week and I stopped
> when I proved the concept (of having a dirlist() that worked - to
> certain specs - on both OSes, 32- and 64-bit, without change).
So you're happy with the fact that there are legitimate file names that your
program simply has no way of dealing with?
Thank you for making my point for me! If you leave the implementation of
metacharacter expansion up to the individual program, then you get the
situation where individual programs will fail to support certain features, or
even basic functionality.
Python's fnmatch lib is a good example. It has, or at least had, no support for
escaping metacharacters. Anyone relying on Python's fnmatch and glob modules
alone for globbing will be unable to handle legitimate file names.
> In another post, you claimed that in your programs, you wouldn't
>> use anything as clumsy and ambiguous as globbing.
>
>> My program wouldn't need anything so crude. The input syntax
>> would be more carefully designed so as to not have the ambiguity.
>
> I meant designing a CLI where *.* could be used in different parameters
> to mean different things.
>
> Reading that post again, you presumably meant either *.* used to match
> all files (with an embedded "." in Linux, with or without in Windows;
> another difference), or *.* used to match a specific file called *.*.
Correct.
It doesn't matter whether metacharacters are expanded by the shell or the
program itself, there needs to be an escaping mechanism for those cases where
you want the metacharacters to be treated literally.
>> So presumably your dirlist() command can distinguish between the file called
>> literally "*.*" and the file spec "*.*" that should be expanded,
>
> No. I won't support that (not unless it's supported by Posix's
> fnmatch()). Because it's the thin end of the wedge. I can show a few
> lines of code and then you will say, Ah, but what about this...
>
> And I first used this function in early 90s I think, I don't recall it
> ever not working.
If you can't use it to specify a file called literally "*.*", then its not
working.
>> And of course your program is also capable of variable and arithmetic
>> expansion, right?
>
> Um, dirlist() is used within a language, of course it can do all that.
> If you mean within the file-pattern string submitted to dirlist, then I
> don't even know what that would mean.
I showed you an example in another post:
[steve@ando ~]$ export base="thefile"
[steve@ando ~]$ ls -l "$base$((1000 + 234))"
-rw-rw-r-- 1 steve steve 0 Dec 8 10:51 thefile1234
Of course it is silly to write 1000 + 234 as a literal constant like that, but
either or both of those could just as easily be variables.
The point here is not that *you* should build a calculator into your version of
ls. The complete opposite: the point is that application authors don't have to
build in more and more non-core functionality into their own applications,
because the shell handles all this peripheral functionality for them.
The user then can choose whether to use the shell's extra functionality or not,
and your application doesn't need to care one way or the other.
>>> (Here are implementations of that dirlist() for both OSes:
>>> http://pastebin.com/knciRULE
>
> [Sorry about that name; I didn't choose it!]
>
>> I have no idea how good the Windows globbing support is, or whether it can
>> be escaped.
>
> Why such a preoccupation with 'globbing'? It's something I may use from
> time to time with DIR or COPY or something, and that's it.
Right -- and that's why you're not the target audience for Unix shells. Unix
sys admins use globs and other shell expansion features extensively, probably
hundreds of times a day. Even me, hardly a sys admin at all, use it dozens of
times a day.
For example, a quick and simple way of backing up a file with a date stamp:
steve@runes:~$ cp alloc.txt{,-`date +%Y%m%d`}
steve@runes:~$ ls alloc.txt*
alloc.txt alloc.txt-20161208
> I just didn't know someone could also use it with any user program and
> that program's command line would then fill with loads of unexpected
> files that can mess things up.
You're treating this as some unknown third party dumping junk into the command
line. Its not. Its me, the use
PonyORM: generators as a query syntax
I was just made aware of a very interesting ORM project that has been around since about 2013, while listening to a recent episode of the Talk Python To Me podcast. The idea of using generators to build queries is really cool. I'm sure PonyORM has its limitations and drawbacks, as all ORM models do. But I like what see so far, particularly how simple it is from a programmer's point of view. Thought others may not have heard of it either, but would find it interesting: https://ponyorm.com/ Anyone have any real-world experience with it vs some other ORM like SQAlchemy? I have a small database project that I may use PonyORM with so we'll see how it goes. I have not yet determined which versions of Python it works with. Hopefully it supports Python 3. I have no idea what kind of black magic is used under the hood to build the SQL queries. -- https://mail.python.org/mailman/listinfo/python-list
Re: python 2.7.12 on Linux behaving differently than on Windows
On Thursday 08 December 2016 11:34, BartC wrote: > On 07/12/2016 23:35, Steve D'Aprano wrote: >> On Thu, 8 Dec 2016 02:19 am, BartC wrote: >> >>> On 07/12/2016 14:34, Steve D'Aprano wrote: >> [...] I don't know why you are so hung up over the number of characters here, or this bogeyman of "one million files" in a directory. >>> >>> Because /you/ brought it up as a reason why 'globbing' would help when >>> there are command limits? >> >> I did? >> >> I don't remember saying any such thing. Citation required. > > In the part you snipped: > > On 07/12/2016 05:15, Steven D'Aprano wrote: > > On Wednesday 07 December 2016 12:55, BartC wrote: > > > >> But even Linux's 128KB will fill [fail?] if someone wanted a command > line that > >> listed 20,000 files individually. But it would be spectacularly bad use > >> of a command line interface which was designed for humans. > > > > That's EXACTLY the point of having the shell do globbing. Right. I meant that rather than have to list 20,000 files by hand, which would be a "spectacularly bad use of a command line interface", I can use globbing. I think in this particular question we're just responding to different things. I didn't mean that globbing was a solution to the limits of command line length. I meant it was a solution to the poor command line interface (i.e. expecting the user to write out all 20,000 files by hand). In fact, I think I was the first person to mention the command line limits, and linked to a page that discussed them in detail. (The "Too Many Arguments" limit that people occasionally run into isn't a shell limitation, but the Unix exec command limit.) -- Steven "Ever since I learned about confirmation bias, I've been seeing it everywhere." - Jon Ronson -- https://mail.python.org/mailman/listinfo/python-list
Re: python 2.7.12 on Linux behaving differently than on Windows
On 08/12/2016 00:09, Steve D'Aprano wrote: On Thu, 8 Dec 2016 02:48 am, BartC wrote: You make it sound like a big deal. Here's a program (in my language not Python) which prints the list of files matching a file-spec: println dirlist(cmdparams[2]) Does dirlist provide an escaping mechanism so that you can disable filename globbing? No. It simply produces a list of files in a path matching a particular wildcard pattern using * and ?. That's all. I know the value of keeping things straightforward instead of throwing in everything you can think of. The file-matching is done by WinAPI functions. The Linux version was done last week and I stopped when I proved the concept (of having a dirlist() that worked - to certain specs - on both OSes, 32- and 64-bit, without change). In another post, you claimed that in your programs, you wouldn't use anything as clumsy and ambiguous as globbing. My program wouldn't need anything so crude. The input syntax would be more carefully designed so as to not have the ambiguity. I meant designing a CLI where *.* could be used in different parameters to mean different things. Reading that post again, you presumably meant either *.* used to match all files (with an embedded "." in Linux, with or without in Windows; another difference), or *.* used to match a specific file called *.*. So presumably your dirlist() command can distinguish between the file called literally "*.*" and the file spec "*.*" that should be expanded, No. I won't support that (not unless it's supported by Posix's fnmatch()). Because it's the thin end of the wedge. I can show a few lines of code and then you will say, Ah, but what about this... And I first used this function in early 90s I think, I don't recall it ever not working. And of course your program is also capable of variable and arithmetic expansion, right? Um, dirlist() is used within a language, of course it can do all that. If you mean within the file-pattern string submitted to dirlist, then I don't even know what that would mean. (Here are implementations of that dirlist() for both OSes: http://pastebin.com/knciRULE [Sorry about that name; I didn't choose it!] I have no idea how good the Windows globbing support is, or whether it can be escaped. Why such a preoccupation with 'globbing'? It's something I may use from time to time with DIR or COPY or something, and that's it. I just didn't know someone could also use it with any user program and that program's command line would then fill with loads of unexpected files that can mess things up. (Presumably its unambiguous because * ? [ ] are not legal in Windows file names?) For the Linux implementation, I don't see how extractpath and extractfile are defined, or fnmatch, man fnmatch Looks like you have a lot of wheels that need re-inventing before you come even close to parity with the features of the Linux shell. These are wheels I don't *want* to re-invent! I'm not writing a shell. I suspect we might be talking at cross-purposes though. -- Bartc -- https://mail.python.org/mailman/listinfo/python-list
Re: python 2.7.12 on Linux behaving differently than on Windows
On Thu, Dec 8, 2016 at 10:35 AM, Steve D'Aprano wrote: > In the Unix world, I'm at the mercy of *one* program, the shell. In your > ideal world, I'm at the mercy of EVERY program that implements globbing: > each and every one has to independently offer a way to disable their own > internal globbing. How is that an improvement? Moreover, you're at the mercy of the one program *you chose* out of a set of similar options (shells). There are some times when bash's style of tab completion feels a bit clunky, and I'd really like the "show a few options and let you pick with the arrow keys" style of... is it zsh? And I have the freedom to choose that if I want to, without reference to the program being invoked. Try doing THAT with the Windows model. Can you make it so that, instead of expanding the whole glob, you pop up something that lets you multiselect from within that pattern? You type "cp *.c dest/" and it opens up a thing listing all .c files and lets you choose which to copy? I don't know if there are any shells like that, but *one could be written*, and it would instantly apply not just to the copy command, but to every other command you might want to use. Want to make symlinks to a whole bunch of things? The ln command accepts multiple targets and a destination directory. In Windows, you'd have to add this feature to every program individually, or (more likely) produce a single overarching file management program that has all these features. Oh, but then it doesn't work if you want to invoke VLC on all those files, because the programmer didn't think of that. Whoops. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: python 2.7.12 on Linux behaving differently than on Windows
On 07/12/2016 23:35, Steve D'Aprano wrote: On Thu, 8 Dec 2016 02:19 am, BartC wrote: On 07/12/2016 14:34, Steve D'Aprano wrote: [...] I don't know why you are so hung up over the number of characters here, or this bogeyman of "one million files" in a directory. Because /you/ brought it up as a reason why 'globbing' would help when there are command limits? I did? I don't remember saying any such thing. Citation required. In the part you snipped: On 07/12/2016 05:15, Steven D'Aprano wrote: > On Wednesday 07 December 2016 12:55, BartC wrote: > >> But even Linux's 128KB will fill [fail?] if someone wanted a command line that >> listed 20,000 files individually. But it would be spectacularly bad use >> of a command line interface which was designed for humans. > > That's EXACTLY the point of having the shell do globbing. -- Bartc -- https://mail.python.org/mailman/listinfo/python-list
Re: python 2.7.12 on Linux behaving differently than on Windows
On Thu, 8 Dec 2016 07:01 am, Skip Montanaro wrote: > Some things are better just left alone. File globbing is probably one > of those things. There's no right answer, and the people in the two > camps will never come to a compromise. Be fair Skip, we've already repeatedly acknowledged that the Unix model of shell expansion doesn't suit all use-cases for a shell. As have the designers of the shells, as they typically provide ways of turning off expansions. Its only Bart who insists that his way is always the best way, and that 40 years of Unix sys admins who have designed the shell to work the way they want are wrong. -- Steve “Cheer up,” they said, “things could be worse.” So I cheered up, and sure enough, things got worse. -- https://mail.python.org/mailman/listinfo/python-list
Re: python 2.7.12 on Linux behaving differently than on Windows
On Thu, 8 Dec 2016 02:48 am, BartC wrote: > On 07/12/2016 13:58, Dennis Lee Bieber wrote: >> On Wed, 7 Dec 2016 11:54:35 +, BartC declaimed the >> following: >> >>> With automatic expansion, then EVERY program can become dangerous. Can >>> no one else see this? >>> >> >> With your scheme, EVERY PROGRAM HAS TO IMPLEMENT ITS OWN GLOBBING. > > Only the tiny minority that can be meaningfully invoked on an arbitrary > number of files at the same time. Its not just filename globbing. There's tilde expansion, brace expansion, arithmetic expansion, parameter expansion, variable expansion, and I dare say I've forgotten some others. [steve@ando ~]$ echo ~ /home/steve [steve@ando ~]$ echo $((1+7*5)) 36 [steve@ando ~]$ echo $http_proxy http://ando:3128 The echo command doesn't have to understand anything about tildes, contain its own arithmetic expression evaluator, or even be able to parse variable names. The shell does all that for it. Filename globbing was just a simple example that would be familiar to everyone discussing the topic. But the full range of expansions performed is much, much richer than that. [...] > You make it sound like a big deal. Here's a program (in my language not > Python) which prints the list of files matching a file-spec: > > println dirlist(cmdparams[2]) Does dirlist provide an escaping mechanism so that you can disable filename globbing? In another post, you claimed that in your programs, you wouldn't use anything as clumsy and ambiguous as globbing. My program wouldn't need anything so crude. The input syntax would be more carefully designed so as to not have the ambiguity. So presumably your dirlist() command can distinguish between the file called literally "*.*" and the file spec "*.*" that should be expanded, only of course you don't actually use globbing syntax, as that's too crude, you use your own ever-so-clever syntax which is completely ambiguity free. Right? And of course your program is also capable of variable and arithmetic expansion, right? [steve@ando ~]$ export base="thefile" [steve@ando ~]$ ls -l "$base$((1000 + 234))" -rw-rw-r-- 1 steve steve 0 Dec 8 10:51 thefile1234 > (Here are implementations of that dirlist() for both OSes: > http://pastebin.com/knciRULE I have no idea how good the Windows globbing support is, or whether it can be escaped. (Presumably its unambiguous because * ? [ ] are not legal in Windows file names?) For the Linux implementation, I don't see how extractpath and extractfile are defined, or fnmatch, so who knows how many bugs are in your globbing implementation. I see no obvious way of disabling globs across the entire command line, equivalent to the bash "setopt -o noglob". I see no brace expansion. I see no arithmetic expansion. I see no variable expansion. I don't even see partial support for ~ expansion, let alone full support. Looks like you have a lot of wheels that need re-inventing before you come even close to parity with the features of the Linux shell. -- Steve “Cheer up,” they said, “things could be worse.” So I cheered up, and sure enough, things got worse. -- https://mail.python.org/mailman/listinfo/python-list
Re: python 2.7.12 on Linux behaving differently than on Windows
On Thu, 8 Dec 2016 02:19 am, BartC wrote:
> On 07/12/2016 14:34, Steve D'Aprano wrote:
[...]
>> I don't know why you are so hung up over the number of characters here,
>> or this bogeyman of "one million files" in a directory.
>
> Because /you/ brought it up as a reason why 'globbing' would help when
> there are command limits?
I did?
I don't remember saying any such thing. Citation required.
[...]
>> But... how does your program distinguish between the file spec *.* which
>> should be expanded, and the literal file *.* which shouldn't be expanded?
>
> Do you mean a *.* parameter that refers to all the files in the
> directory (whether the intention is to build a list of those files or
> not) and a *.* parameter that means something entirely different?
We're talking Linux here, where "*.*" is a legal (if stupid) file name.
Existing Linux programs (mostly?) don't do their own globbing, so if I want
to list such a file, I just need to protect the metacharacters from the
shell:
[steve@ando ~]$ ls -l "*.*"
-rw-rw-r-- 1 steve steve 7 Dec 8 09:30 *.*
and there it is. Now, you *insist* that the one and only correct way to do
this is to disable shell globbing, and have each and every program do its
own globbing. Okay, I can do the first part, and because ls doesn't in
reality do its own globbing, it works fine:
[steve@ando ~]$ set -o noglob
[steve@ando ~]$ ls -l *.*
-rw-rw-r-- 1 steve steve 7 Dec 8 09:30 *.*
But now imagine that you are the author of ls, and that following your
strategy of having programs do their own globbing, ls expands that as a
glob. The shell is no longer involved here, but globbing still needs to be
defeated, only this time I'm at the mercy of each individual program, not
just a single shell.
So my question is, in your programs that you write, where you implement your
own globbing, do you support escaping metacharacters? If you were the
author of Linux ls, could I write
ls -l \*.\*
or possibly
ls -l "*.*"
or even:
ls -l --noglob *.*
to get a listing of the file named "*.*"?
In the Unix world, I'm at the mercy of *one* program, the shell. In your
ideal world, I'm at the mercy of EVERY program that implements globbing:
each and every one has to independently offer a way to disable their own
internal globbing. How is that an improvement?
Hence my rhetorical question. When you write a program that supports
globbing, do you always remember to provide an escape mechanism?
The string quoting, escaping and expansion rules for bash are pretty darn
complicated. You have parameter expansion, aliases, two distinct types of
quoting, backslashes, shell variables, environment variables, and more,
which are applied in a certain order. Once you get past the simple cases,
bash programming is hairy and frankly scarily complex.
But at least the rules are consistent since they're applied by the shell. I
don't have to learn a separate set of rules for each and every program I
use:
"ls supports globbing, but not brace expansion..."
"rm supports globbing and brace expansion, but not parameter expansion..."
"mkdir supports tilde expansion, but incompletely..."
"touch supports only a subset of arithmetic expansion, but with bugs..."
> OK, but you go first: how does a Linux program tell the difference
> between:
>
>A B C
>
> which signifies three files, and:
>
>A B C
>
> which means two files, and an option C? And without using escapes!
First of all, as you well know, the convention is for options to start with
either a single or double hyphen, or sometimes a plus, and usually to come
first. So I would write:
program -C A B
But let's suppose that this particular program doesn't follow that standard
convention, and instead has the argument signature:
argument1 argument2 option
Fine. Where's the difficulty? The program would read the command line,
collect the first argument as `argument1`, the second argument as
`argument2`, and the third argument as `option`.
If I was particularly awful at designing user interfaces, I could even have
the signature where the third parameter could be either:
argument1 argument2 option_or_argument3
# Python syntax
if option_or_argument3 in ("off", "on"):
option = option_or_argument3
argument3 = ""
else:
option = "off"
argument3 = option_or_argument3
I don't see what point you think you are making. How the program interprets
its command arguments is independent of the shell.
>> You need an escape mechanism too. You've done that, of course. Right?
>
> My program wouldn't need anything so crude. The input syntax would be
> more carefully designed so as to not have the ambiguity.
Oh of course it would. How silly of me to even ask.
*rolls eyes*
I'll take that as a "No".
> And actually I generally avoid command line interfaces (ones used by
> shells) because they are such a pain. (Why, for example, aren't the
> contents of the command line - even after preprocessing by the shell -
> just
Re: python 2.7.12 on Linux behaving differently than on Windows
BartC writes: > On 07/12/2016 15:25, Ben Bacarisse wrote: >> BartC writes: >> >>> [...] But remember: >>> >>> cp *.c >>> >>> There might be some irate users out there if it can't detect a simple >>> user error like that. >> >> There might be. They are ill-served by current Unix shells. But, >> oddly, the flow has gone the other way. People have wanted Unix shells >> and utilities ported to Windows, not cmd.exe and friends ported to >> Linux. You clearly see a gap in the provision here. Wouldn't it be >> better for the world if you wrote cmd.exe for Linux rather than >> complaining about what other shells do? > > It's not my job to write OSes. A shell is not an OS. The kind you want is not a big program and if you are not alone (as I think you almost certainly are) in wanting it, you will get help to make it happen. Writing a simple shell used to be an introductory exercise in system programming classes. But maybe it's not really much of a problem for you are your main concern is to have jolly good whinge about it? > But you might have seen my objections; even putting aside bias in > favour of one OS or another, I think many of them are valid. That fact > Linux is getting more popular does not mean it is always better.) It's not an OS issue -- Windows has bash and Linux does not have a compatible cmd.exe only because no one seems to have considered it enough of a problem to be worth solving. But, yes, I've seen all these objections before. The problem is not their validity by the absence of a solution with which they can be compared. MS's cmd.exe is not the solution (for me). I've never had a problem from accidentally typing cp *.c but when I had to use Windows I had a problem almost every day getting anything done. -- Ben. -- https://mail.python.org/mailman/listinfo/python-list
Re: python 2.7.12 on Linux behaving differently than on Windows
On Thu, Dec 8, 2016 at 7:17 AM, eryk sun wrote:
> On Wed, Dec 7, 2016 at 12:39 PM, Chris Angelico wrote:
>> Note that two of the Beauty Stone tracks include quotes as well as
>> question marks. How do you identify those? Let's say you want to play
>> one of these in VLC, and then maybe you decide that the track in
>> Pirates of Penzance/MusicOnly is slightly mis-cropped, so you rebuild
>> it from the one in the parent directory. How does that work on
>> Windows? If you say "it doesn't", then (a) you have taken away choice
>> on a fundamental level, (b) you have your head in the sand, and (c)
>> you still haven't solved the problem of percent signs, carets, and so
>> on, which are perfectly legal in file names, but have meaning to the
>> shell.
>
> The five wildcard characters ("<>*?) aren't allowed in the names of
> files and directories -- at least not by any Windows filesystem that
> I've used. This makes it easy for a filesystem to support globbing in
> its implementation of NtQueryDirectoryFile.
Yep - see my last sentence in the quoted paragraph.
ChrisA
--
https://mail.python.org/mailman/listinfo/python-list
Re: python 2.7.12 on Linux behaving differently than on Windows
Op 07-12-16 om 19:02 schreef BartC: > On 07/12/2016 16:53, Michael Torrie wrote: >> On 12/07/2016 08:48 AM, BartC wrote: >>> I would prefer that the program "t" can be invoked exactly the same way >>> under both systems. I don't want different instructions for Linux, or >>> for the user (of my language) to have to write two lots of code, as that >>> is my job... >> >> Ahh now we come to the crux of your argument. You want all potential >> platforms to conform to your own idea of what is normal. And indeed >> this is the actual issue that started the original thread. But that's >> not the way it works for any platform. I develop a lot of code on Linux >> that I'd like to get running on Windows. I prefer the Linux way of >> doing things but I'm not going to make any headway if I just try to >> brow-beat Windows and Windows' users over the things that aren't >> implement the same way. > > There is a lack of balance. > > If I run a Linux program on Windows, then the application gets to see > parameters such as *.* and can expand them if it wants. > > If I want to run a Windows program on Linux, and that program needs to see > *.* unexpanded, then it can't undo that expansion. > The cat is already out of the bag. You should stop trying to frame this as a need of the program. If it was just about the program needing to see *.* as an argument. That is possible. Just quote the argument. But this isn't about what a program would need, this is about how you would like things to work. You have some points I agree with, but I don't think those points weight heavy enough to get people changing things on the unix side. > It seems the only choice, if someone wants a cross-platform application that > is invoked in the same way, is to dumb it > down and make it assume that any parameters such as A* have been expanded, or > will be expanded by that extra step. cross-platform applications will always need to come to some kind of compromise. > Which means that input of A B* *C will all end up running together so that > you have no idea what is what or which file > corresponds to which expansion. That's assuming these parameters have any > connection to the current set of files at all; if not, then > the input you end up is going to be meaningless. Yes and the point is? If you write a cross-platform application and you don't consider how the environments differ then you might end up with something meaningless. > Now the problem is to determine whether processed input of U V W X Y Z are > intended bona fide inputs, or whether > they just happened to result from some random associations with the files in > the current directory. No it is not. Making this your problem is just an untractable mess and it is not limited to globbing. Suppose I have a file named -rf and want it among other files removed. I don't consider it more closely and just type: rm -rf ... How is your program going to decide between me wanting to remove the file '-rf' and me wanting the options? You don't like how it generally works on unix. Too bad for you when you have to work on a unix box. Me I don't like how it generally works on a windows. Too bad for me when I find myself without a choice but to work on a windows box. Life just isn't easy and fair. -- https://mail.python.org/mailman/listinfo/python-list
Re: python 2.7.12 on Linux behaving differently than on Windows
On Wednesday December 7 2016 14:57, in comp.lang.python, "BartC" wrote: > On 07/12/2016 18:02, BartC wrote: > >> If I want to run a Windows program on Linux, and that program needs to >> see *.* unexpanded, then it can't undo that expansion. The cat is >> already out of the bag. > > I mean, the genie is out of the bottle, whatever. You know what I mean, > the operation is not reversible. So, don't pull the cork. Look, Bart, in a Unix shell, the user has the option to enable or disable globbing (that action that converts the argument * into a list of files). A Unix shell can give a program exactly the same list of unaltered arguments that a Windows shell can. But, point of fact is that the feature to disable globbing is not often needed. Most Unix programs that accept filenames are happy to accept a list of filenames. There is not much call for a program to perform it's own globbing, like is required in Windows. In fact, I can only think of three Unix "commandline" programs that need shell globbing disabled: find - which performs it's own filename matching grep - which uses regular expressions to search the contents of files, and sed - which uses regular expressions to edit the contents of files. (I'm sure that there are a few more examples, though). If your argument is with Unix shell globbing, I say that you can disable it. If your argument is with Python's use of the default shell options to parse commandlines in the Popen() method, then you need to talk to the authors of Python to see if they can give you a way to alter this behaviour. Otherwise, you are just being cranky and argumentative. Stop it. Suck it up and move on. The world doesn't revolve around BartC, no matter how loud he yells. -- Lew Pitcher "In Skills, We Trust" PGP public key available upon request -- https://mail.python.org/mailman/listinfo/python-list
Re: python 2.7.12 on Linux behaving differently than on Windows
On Wed, Dec 7, 2016 at 12:39 PM, Chris Angelico wrote:
> Note that two of the Beauty Stone tracks include quotes as well as
> question marks. How do you identify those? Let's say you want to play
> one of these in VLC, and then maybe you decide that the track in
> Pirates of Penzance/MusicOnly is slightly mis-cropped, so you rebuild
> it from the one in the parent directory. How does that work on
> Windows? If you say "it doesn't", then (a) you have taken away choice
> on a fundamental level, (b) you have your head in the sand, and (c)
> you still haven't solved the problem of percent signs, carets, and so
> on, which are perfectly legal in file names, but have meaning to the
> shell.
The five wildcard characters ("<>*?) aren't allowed in the names of
files and directories -- at least not by any Windows filesystem that
I've used. This makes it easy for a filesystem to support globbing in
its implementation of NtQueryDirectoryFile. Filenames also can't
contain control characters, slash, backslash, pipe, and colon (the
latter delimits a fully-qualified NTFS name, e.g.
filename:streamname:streamtype). NTFS stream names are less limited.
They only disallow NUL, slash, backslash, and colon.
The filesystem runtime library provides the macro
FsRtlIsAnsiCharacterLegal [1], among other related macros, which
allows filesystem drivers to be consistent with FAT or NTFS. To my
knowledge this is voluntary, but going against the grain is only
asking for headaches.
[1]: https://msdn.microsoft.com/en-us/library/ff546731
This macro depends on the array FsRtlLegalAnsiCharacterArray, which
indicates whether each ASCII character is valid for a fixed set of
filesystems. The flag values are as follows:
0x01 - FAT
0x02 - OS/2 HPFS
0x04 - NTFS/Stream
0x08 - Wildcard
0x10 - Stream
Here's the array dumped from the kernel debugger. For convenience I've
added the printable ASCII characters above each line.
lkd> db poi(nt!FsRtlLegalAnsiCharacterArray)
f801`fc0e8550 00 10 10 10 10 10 10 10-10 10 10 10 10 10 10 10
f801`fc0e8560 10 10 10 10 10 10 10 10-10 10 10 10 10 10 10 10
! " # $ % & ' ( ) * + , - . /
f801`fc0e8570 17 07 18 17 17 17 17 17-17 17 18 16 16 17 07 00
0 1 2 3 4 5 6 7 8 9 : ; < = > ?
f801`fc0e8580 17 17 17 17 17 17 17 17-17 17 04 16 18 16 18 18
@ A B C D E F G H I J K L M N O
f801`fc0e8590 17 17 17 17 17 17 17 17-17 17 17 17 17 17 17 17
P Q R S T U V W X Y Z [ \ ] ^ _
f801`fc0e85a0 17 17 17 17 17 17 17 17-17 17 17 16 00 16 17 17
` a b c d e f g h i j k l m n o
f801`fc0e85b0 17 17 17 17 17 17 17 17-17 17 17 17 17 17 17 17
p q r s t u v w x y z { | } ~
f801`fc0e85c0 17 17 17 17 17 17 17 17-17 17 17 17 10 17 17 17
NTFS stream names are the least restricted, allowing everything except
NUL, slash, and backslash. For example:
>>> open('test:\x01|*?<>"', 'w').close()
>>> win32file.FindStreams('test')
[(0, '::$DATA'), (0, ':\x01|*?<>":$DATA')]
The first stream listed above is the anonymous data stream
"test::$DATA", which is the same as simply opening "test".
Technically by the above table ":" is allowed in NTFS names, but its
use is reserved as the delimiter of the fully-qualified name. For
example, the fully-qualified name of a directory is
"dirname:$I30:$INDEX_ALLOCATION". The stream name is "$I30" and the
stream type is "$INDEX_ALLOCATION". A directory can also have multiple
named $DATA streams, because NTFS is weird like that.
--
https://mail.python.org/mailman/listinfo/python-list
Re: python 2.7.12 on Linux behaving differently than on Windows
On 07/12/2016 18:02, BartC wrote: If I want to run a Windows program on Linux, and that program needs to see *.* unexpanded, then it can't undo that expansion. The cat is already out of the bag. I mean, the genie is out of the bottle, whatever. You know what I mean, the operation is not reversible. -- bartc -- https://mail.python.org/mailman/listinfo/python-list
Re: python 2.7.12 on Linux behaving differently than on Windows
On Wed, Dec 7, 2016 at 12:17 PM, Anton Mamaenko wrote: > Wow... this thread gets to finally become a holy war. It's not like this particular holy war is new, either. I'm sure it was beaten to death in the days before Windows was a thing. As I indicated in an earlier note, VMS and Unix did command line parsing (well, globbing at least) differently. There were holy wars about that on Usenet before many here were born, before many more were out of grade school. Add to that the holy wars between Apollo and Sun Microsystems (network topology (Token Ring v. Ethernet), as I recall, probably remote procedure calls and networked file systems as well). I remember being at one meeting where the Apollo and Sun gangs were going at it hammer-and-tong, in person, not with the anonymity of the Internet to obscure their views of each other. I can't now remember if that was about RPC or networked file systems. Those were the good old days. :-) Some things are better just left alone. File globbing is probably one of those things. There's no right answer, and the people in the two camps will never come to a compromise. Skip -- https://mail.python.org/mailman/listinfo/python-list
Re: calling a program from Python batch file
This might be totally irrelevant, but, if (a) the data to be read by the program during a given run is known when the program is run/launched, (b) that data is purely textual and (c) that data can be read by the program from the stdin channel, then my idea is (1) before the launch, put that data into a file named, say, inputfile.dat, and then (2) launch the program with a command like call program.bat outputfile.dat Finally, if the program can read its data from a given file and possibly needs to output its data, again, to a named file, and those files' names can be read from the parameters to the call, then call program.bat inputfile.dat outputfile.dat can be used. On Wed, Dec 7, 2016 at 6:23 PM, Michael Torrie wrote: > On 12/07/2016 10:59 AM, John Gordon wrote: > > In Karim > Farokhnia writes: > > > >> Hi there, > > > >> I am writing a batch file in Python. The batch file, in part, calls a > >> program named "oq-console.bat" to run. Then once the program comes up > >> (it looks like windows CMD), I need the batch file to type some commands > >> to make it run (just like I would do it by myself). > > > > If you need to start up a program, provide interactive input to it, and > > perhaps examine its interactive output, then you want the "pexpect" > module: > > > > Pexpect is a pure Python module for spawning child applications; > > controlling them; and responding to expected patterns in their > output. > > Pexpect allows your script to spawn a child application and control > it > > as if a human were typing commands. > > > > https://pexpect.readthedocs.io/en/stable/ > > Does Pexpect work on Windows? > > In the OP's case it looks like the standard in pipe is sufficient. > > -- > https://mail.python.org/mailman/listinfo/python-list > -- https://mail.python.org/mailman/listinfo/python-list
Re: python 2.7.12 on Linux behaving differently than on Windows
On 07/12/2016 15:25, Ben Bacarisse wrote: BartC writes: [...] But remember: cp *.c There might be some irate users out there if it can't detect a simple user error like that. There might be. They are ill-served by current Unix shells. But, oddly, the flow has gone the other way. People have wanted Unix shells and utilities ported to Windows, not cmd.exe and friends ported to Linux. You clearly see a gap in the provision here. Wouldn't it be better for the world if you wrote cmd.exe for Linux rather than complaining about what other shells do? It's not my job to write OSes. When I started doing this stuff, either there was no OS, or there was just a minimal one providing file services and allowing you to launch applications. Now it seems to have trouble doing even that! (As for people wanting Linux, I don't know why that is. Maybe Linux is used more in colleges and people want to continue working the same way, maybe Linux is perceived as free and Windows a product of evil capitalist M$. Your guess is as good as mine. But you might have seen my objections; even putting aside bias in favour of one OS or another, I think many of them are valid. That fact Linux is getting more popular does not mean it is always better.) OK, I understand. Having a program launcher indiscriminately transform input A into A' is /much/ better than having it done by the program, even if the program doesn't want it and it is not meaningful. Yes, you've explained that you understand the value of such transformations, it's just that the ones you like happen to be exactly those in cmd.exe. You are very lucky that cmd.exe happened to get exactly the right interpretation of the input or you would be raining against /every/ shell on both systems. I can do that as well if you like! But it just seems odd in that I've complained in the past about Python's method for reading interactive lines of input. Apparently, having line input presented in one lump (a single string) which is then explicitly chopped up by the program into tokens, is far superior to any other scheme, including any of mine. But here, having input from the command line presented as a ready-separated set of arguments, and with (possibly meaningless) expansion into multiple files to boot, is far superior to getting the input in one lump (a single string). So for one kind of line input, Black is better than White; for the other, White is better than Black! -- Bartc -- https://mail.python.org/mailman/listinfo/python-list
Re: sets anomaly
On 2016-12-07 15:33, Ned Batchelder wrote:
On Wednesday, December 7, 2016 at 10:18:32 AM UTC-5, Rustom Mody wrote:
Trying to write some code using sets (well frozen sets)
And was hit by this anomaly
This is the behavior of lists I analogously expect in sets:
>>> []
[]
>>> [[]]
[[]]
>>>
ie the empty list and the list of the empty list are different things
However
(with
f= frozenset
)
>>> f()
frozenset()
>>> f([])
frozenset()
>>> f(f([]))
frozenset()
>>>
The difference is more about the difference between the behavior of a
callable constructor, and a list literal. Lists will behave the same
as frozenset if you use list():
>>> list()
[]
>>> list(list())
[]
>>> list(list(list()))
[]
This is because the constructors can take a sequence, and use those
elements as the new contents. List literals don't work that way.
Spent a good ½ hour finding out this strangeness
And then some figuring out how to get an empty set into a set
This is the best I get:
>>> f([f([])])
frozenset({frozenset()})
That is the way I would have done it also. Or:
>>> s = set()
>>> s.add(frozenset())
>>> frozenset(s)
frozenset([frozenset([])])
Notice the repr output of the result shows how to make it! :)
You must be using Python 2 because on Python 3 I get:
>>> frozenset([frozenset()])
frozenset({frozenset()})
--
https://mail.python.org/mailman/listinfo/python-list
Re: calling a program from Python batch file
On 12/07/2016 10:59 AM, John Gordon wrote: > In Karim Farokhnia > writes: > >> Hi there, > >> I am writing a batch file in Python. The batch file, in part, calls a >> program named "oq-console.bat" to run. Then once the program comes up >> (it looks like windows CMD), I need the batch file to type some commands >> to make it run (just like I would do it by myself). > > If you need to start up a program, provide interactive input to it, and > perhaps examine its interactive output, then you want the "pexpect" module: > > Pexpect is a pure Python module for spawning child applications; > controlling them; and responding to expected patterns in their output. > Pexpect allows your script to spawn a child application and control it > as if a human were typing commands. > > https://pexpect.readthedocs.io/en/stable/ Does Pexpect work on Windows? In the OP's case it looks like the standard in pipe is sufficient. -- https://mail.python.org/mailman/listinfo/python-list
Re: python 2.7.12 on Linux behaving differently than on Windows
Wow... this thread gets to finally become a holy war. One method is dealing with this is take the environment as granted (Windows/Unix/Linux/MacOS/Dos sucks while my Windows/Unix/Linux/MacOS/Dos is the best) and figure if the hassle of porting (time, tech, AND moral) worth it. He just "do not try. Do or not do." So stop complaining. Just live with it. Regards, Anton > On 7 Dec 2016, at 21:02, BartC wrote: > >> On 07/12/2016 16:53, Michael Torrie wrote: >>> On 12/07/2016 08:48 AM, BartC wrote: >>> I would prefer that the program "t" can be invoked exactly the same way >>> under both systems. I don't want different instructions for Linux, or >>> for the user (of my language) to have to write two lots of code, as that >>> is my job... >> >> Ahh now we come to the crux of your argument. You want all potential >> platforms to conform to your own idea of what is normal. And indeed >> this is the actual issue that started the original thread. But that's >> not the way it works for any platform. I develop a lot of code on Linux >> that I'd like to get running on Windows. I prefer the Linux way of >> doing things but I'm not going to make any headway if I just try to >> brow-beat Windows and Windows' users over the things that aren't >> implement the same way. > > There is a lack of balance. > > If I run a Linux program on Windows, then the application gets to see > parameters such as *.* and can expand them if it wants. > > If I want to run a Windows program on Linux, and that program needs to see > *.* unexpanded, then it can't undo that expansion. The cat is already out of > the bag. > > It seems the only choice, if someone wants a cross-platform application that > is invoked in the same way, is to dumb it down and make it assume that any > parameters such as A* have been expanded, or will be expanded by that extra > step. > > Which means that input of A B* *C will all end up running together so that > you have no idea what is what or which file corresponds to which expansion. > That's assuming these parameters have any connection to the current set of > files at all; if not, then the input you end up is going to be meaningless. > > Now the problem is to determine whether processed input of U V W X Y Z are > intended bona fide inputs, or whether they just happened to result from some > random associations with the files in the current directory. > > But this is what happens when you have to work to the lowest common > denominator. The only proper solution is ban any unescaped ? or * from > inputs, and to add a warning that the program could behave unexpectedly if > they are used inadvertently. > > (I'm in the middle of porting my console editor to Linux. But one problem is > that on one Linux, half the key combinations (eg. Shift+Ctrl+B) are not > recognised. On other Linux, nearly all of those are missing too, plus most of > the rest! Which means I have to adapt the editor, again, to the lowest common > denominator, the minimum set of keys that are guaranteed to work on both. > > Except that was only two Linuxes; perhaps on others, the keyboard will likely > be crippled in some other way. That is, the same universal USB keyboard that > you can buy anywhere for $5 or $10, which has keys marked Shift, Ctrl and > Alt, but whose combinations cannot be recognised consistently. > > How people manage to do anything on such an OS I've no idea. Or maybe they > spend all day just typing elaborate file-matching patterns to pipe between > applications.) > > -- > Bartc > -- > https://mail.python.org/mailman/listinfo/python-list -- https://mail.python.org/mailman/listinfo/python-list
Re: python 2.7.12 on Linux behaving differently than on Windows
On 07/12/2016 16:53, Michael Torrie wrote: On 12/07/2016 08:48 AM, BartC wrote: I would prefer that the program "t" can be invoked exactly the same way under both systems. I don't want different instructions for Linux, or for the user (of my language) to have to write two lots of code, as that is my job... Ahh now we come to the crux of your argument. You want all potential platforms to conform to your own idea of what is normal. And indeed this is the actual issue that started the original thread. But that's not the way it works for any platform. I develop a lot of code on Linux that I'd like to get running on Windows. I prefer the Linux way of doing things but I'm not going to make any headway if I just try to brow-beat Windows and Windows' users over the things that aren't implement the same way. There is a lack of balance. If I run a Linux program on Windows, then the application gets to see parameters such as *.* and can expand them if it wants. If I want to run a Windows program on Linux, and that program needs to see *.* unexpanded, then it can't undo that expansion. The cat is already out of the bag. It seems the only choice, if someone wants a cross-platform application that is invoked in the same way, is to dumb it down and make it assume that any parameters such as A* have been expanded, or will be expanded by that extra step. Which means that input of A B* *C will all end up running together so that you have no idea what is what or which file corresponds to which expansion. That's assuming these parameters have any connection to the current set of files at all; if not, then the input you end up is going to be meaningless. Now the problem is to determine whether processed input of U V W X Y Z are intended bona fide inputs, or whether they just happened to result from some random associations with the files in the current directory. But this is what happens when you have to work to the lowest common denominator. The only proper solution is ban any unescaped ? or * from inputs, and to add a warning that the program could behave unexpectedly if they are used inadvertently. (I'm in the middle of porting my console editor to Linux. But one problem is that on one Linux, half the key combinations (eg. Shift+Ctrl+B) are not recognised. On other Linux, nearly all of those are missing too, plus most of the rest! Which means I have to adapt the editor, again, to the lowest common denominator, the minimum set of keys that are guaranteed to work on both. Except that was only two Linuxes; perhaps on others, the keyboard will likely be crippled in some other way. That is, the same universal USB keyboard that you can buy anywhere for $5 or $10, which has keys marked Shift, Ctrl and Alt, but whose combinations cannot be recognised consistently. How people manage to do anything on such an OS I've no idea. Or maybe they spend all day just typing elaborate file-matching patterns to pipe between applications.) -- Bartc -- https://mail.python.org/mailman/listinfo/python-list
Re: calling a program from Python batch file
In Karim Farokhnia writes: > Hi there, > I am writing a batch file in Python. The batch file, in part, calls a > program named "oq-console.bat" to run. Then once the program comes up > (it looks like windows CMD), I need the batch file to type some commands > to make it run (just like I would do it by myself). If you need to start up a program, provide interactive input to it, and perhaps examine its interactive output, then you want the "pexpect" module: Pexpect is a pure Python module for spawning child applications; controlling them; and responding to expected patterns in their output. Pexpect allows your script to spawn a child application and control it as if a human were typing commands. https://pexpect.readthedocs.io/en/stable/ -- John Gordon A is for Amy, who fell down the stairs gor...@panix.com B is for Basil, assaulted by bears -- Edward Gorey, "The Gashlycrumb Tinies" -- https://mail.python.org/mailman/listinfo/python-list
Re: calling a program from Python batch file
On 12/07/2016 10:43 AM, Karim Farokhnia wrote: >> Is the program that oq-console.bat runs interactive? After it >> launches into its own console window it asks for more input? > The program doesn't ask for more input parameters as all parameters > are in input files and the program will automatically read the files, > but I need to type in some required commands and press enter. think what you mean is that it *is* interactive, at least requiring some input before it begins to work. As the other person who replied said, it sounds like this program will accept input on standard in. From python you will probably have to use subprocess.popen and use the .communicate() method. You will be able to pass in a string that will get fed to the subprocess on it's standard in pipe (the equivalent of typing and pressing enter). Check the online docs, particularly for the .communicate() method. -- https://mail.python.org/mailman/listinfo/python-list
Re: calling a program from Python batch file
On Wednesday, December 7, 2016 at 12:18:51 PM UTC-5, Michael Torrie wrote: > On 12/07/2016 09:38 AM, Karim Farokhnia wrote: > > I am writing a batch file in Python. The batch file, in part, calls a > > program named "oq-console.bat" to run. Then once the program comes up > > (it looks like windows CMD), I need the batch file to type some > > commands to make it run (just like I would do it by myself). > > > > I need the batch file does the whole things automatically, I mean > > calls the program, types in commands and press enter! > > Is the program that oq-console.bat runs interactive? After it launches > into its own console window it asks for more input? The program doesn't ask for more input parameters as all parameters are in input files and the program will automatically read the files, but I need to type in some required commands and press enter. > > > So far, I have used the below Python codes in my batch file to do > > that; > > > > > > import subprocess > > > > subprocess.call(["C:\Program Files (x86)\OpenQuake > > Engine\oq-console.bat", "oq --run", "job.ini"]) > > > > > > Note that the first string is the program, and the last two are the > > commands which needs to be typed in. > > > > It pulls up the program well, but doesn't type in the commands to > > make it to run. > > You'll have to explain what you mean by "[it] doesn't type ni the > commands to make it run." By [it] I meant the program. -- https://mail.python.org/mailman/listinfo/python-list
Re: calling a program from Python batch file
On Dec 7, 2016 11:40 AM, "Karim Farokhnia" wrote: > > Hi there, > > I am writing a batch file in Python. The batch file, in part, calls a program named "oq-console.bat" to run. Then once the program comes up (it looks like windows CMD), I need the batch file to type some commands to make it run (just like I would do it by myself). > > I need the batch file does the whole things automatically, I mean calls the program, types in commands and press enter! Try using the stdin argument of subprocess. > > So far, I have used the below Python codes in my batch file to do that; > > > import subprocess > > subprocess.call(["C:\Program Files (x86)\OpenQuake Engine\oq-console.bat", "oq --run", "job.ini"]) > > > Note that the first string is the program, and the last two are the commands which needs to be typed in. > > It pulls up the program well, but doesn't type in the commands to make it to run. > > > Any ideas on how I can get the program to run the commands as well? > > Thank you! > -- > https://mail.python.org/mailman/listinfo/python-list -- https://mail.python.org/mailman/listinfo/python-list
Re: "Best" websocket implementation for Python 2.x?
On Wed, Dec 7, 2016 at 8:50 AM, Skip Montanaro wrote: > PyPI came back. A bit more sleuthing suggests that the > websocket-client package on PyPI is Ohtani's package, and is more > up-to-date than the copyright notices would suggest. The package was > updated a few days ago on GitHub. > > Taking the path of least resistance (no changes necessary to the > code), I suspect I will just stick with that unless there are > overriding inputs from the community suggesting something else is way > better... > > S > -- > https://mail.python.org/mailman/listinfo/python-list > I can't vouch for it being better, but if you're using asgi, you may want to take a look at Daphne: https://github.com/django/daphne. I believe it uses autobahn to power the websocket side of things. Daphne was developed for the django-channels project. I have used channels before and I can say that it works pretty well. -- https://mail.python.org/mailman/listinfo/python-list
Re: python 2.7.12 on Linux behaving differently than on Windows
On 12/07/2016 07:57 AM, BartC wrote: > I don't follow you. "?" is problematical on both systems. I think > Windows disallows it completely: I get 'Protocol error' if I copy such a > file from Linux to Windows. Presumably there is an escaping system, but > I don't know what it is. As Chris says, no it's not problematical. Only / and null are not allowed in filenames on Unix (Posix) systems including Linux. Many filename characters do have special meaning to shells, so they must be escaped when specifying them on a shell prompt (but they don't have to be escaped if another program is calling the program with popen). > But we're talking about Linux. Obviously if a program is developed on > Linux, can work on multiple files and DEPENDS on on being presented with > that list of files as though it had been invoked as: > >program file1 file2 file3 file4 file5 file6 file7 fileN > > then it will have problems if that feature is taken away. I can see how > it can still be there after all these years! 40 years is a long time, plenty of time to change what doesn't work. This idea of specifying multiple files individually on a command-line not only works, it's there for good reason. I've alluded to it several times. This idea is part of how we can string together small scripts to do batch processing and filtering (of data or files or whatever) using the shell. You could say it's part of the interprocess communication protocol that shell scripting relies upon. Globbing by each individual app would break this protocol and make it harder to string these bits and bobs together. This protocol involves piping of standard in and standard out, or to pipe the output from one command onto the command-line of another app (multiple channels). This is the power of the unix shell and this is why I and many others use it for our daily work every day. And this is why these concepts that trip you up have had such staying power. We simply haven't found a better way of doing it. Also the unix shell protocol and ideal allows you to completely replace the shell with something else, and the scripts processes one has developed still function! C Shell, Bourne shell, Bash, zsh, fish. Just pick the one you want (each with their own quirks and expansion rules and quoting and escaping rules). I've seen attempts as using something based on Python to form a shell, but ultimately they never feel quite right to me. Mainly because there's an impedance mismatch between python's objects and files on a filesystem. Does such an impedance mismatch come with PowerShell too? I dunno. -- https://mail.python.org/mailman/listinfo/python-list
Re: calling a program from Python batch file
On 12/07/2016 09:38 AM, Karim Farokhnia wrote: > I am writing a batch file in Python. The batch file, in part, calls a > program named "oq-console.bat" to run. Then once the program comes up > (it looks like windows CMD), I need the batch file to type some > commands to make it run (just like I would do it by myself). > > I need the batch file does the whole things automatically, I mean > calls the program, types in commands and press enter! Is the program that oq-console.bat runs interactive? After it launches into its own console window it asks for more input? > So far, I have used the below Python codes in my batch file to do > that; > > > import subprocess > > subprocess.call(["C:\Program Files (x86)\OpenQuake > Engine\oq-console.bat", "oq --run", "job.ini"]) > > > Note that the first string is the program, and the last two are the > commands which needs to be typed in. > > It pulls up the program well, but doesn't type in the commands to > make it to run. You'll have to explain what you mean by "[it] doesn't type ni the commands to make it run." -- https://mail.python.org/mailman/listinfo/python-list
Re: Django broken pipe error
On Wed, Dec 7, 2016 at 1:08 AM, wrote:
> Thank you Justin,
>
> I'm on the dev server and should present results in this way.
>
> Yes, I use manage.py runserver --insecure to start the server (from
> PyCharm).
>
> My views.py call:
>
> @detail_route(methods=['post'], permission_classes=[permissions.AllowAny])
> def export_report(request):
> body_unicode = request.body.decode('utf-8')
> body_str = body_unicode.encode('ascii','ignore')
> attr_list = body_str.split('&')
> attr_dict = {}
> if (len(attr_list) > 0):
> for attr in attr_list:
>...
>key = key_value_pair[0]
>attr_dict[key] = key_value_pair[1]
> trend = trends.calculate_trend(
> attr_dict['search_phrase']
> , attr_dict['time_from']
> , attr_dict['time_to']
> , attr_dict['time_scale']
> )
> attr_dict['trend'] = trend
> res = str(json.dumps(attr_dict))
> return HttpResponse(res, content_type="text/plain; charset=utf-8")
>
> and trend calculation in trends.py with database calls:
>
> def calculate_trend(query_phrase, time_from, time_to, time_scale):
> # check in database if trend already exists
> try:
> db_trend = Trend.objects.get(pk=query_phrase)
> if db_trend.from_time.strftime("%Y-%m-%d") == time_from \
> and db_trend.to_time.strftime("%Y-%m-%d") == time_to \
> and db_trend.granularity == time_scale:
> logger.info("trend already exists.")
> existing_trend_dict = ast.literal_eval(db_trend.content)
> return existing_trend_dict
> except Trend.DoesNotExist:
> logger.info("It is a new trend search.")
> trend_dict = {}
> start_time = pd.Timestamp(value[0])
> end_time = pd.Timestamp(value[-1])
> freq = ... get frequency using pandas lib
> trend_dict[key] = freq
> json_trend_content = trend_dict_to_sorted_json_str(trend_dict)
> trend = Trend(
> phrase=query_phrase,
> content=json_trend_content,
> from_time=time_from,
> to_time=time_to,
> granularity=time_scale,
> )
> if trend is not None:
> try:
> db_trend = Trend.objects.get(pk=query_phrase)
> db_trend.delete()
> logger.info("delete old trend: %s. " % trend)
> except Trend.DoesNotExist:
> logger.info("create trend: %s. " % trend)
> trend.save()
> return trend_dict
>
> Thank you in advance!
>
> Roman
> --
> https://mail.python.org/mailman/listinfo/python-list
>
It looks like you can probably get rid of the try/except block at the end
of the calculate_trend
method as any existing Trend object will have already been caught in the
first try/except block.
>From what I'm reading here:
http://stackoverflow.com/questions/11866792/how-to-prevent-errno-32-broken-pipe
,
this issue can be caused by the client closing the connection before
sendall() finishes writing. Can you estimate
how long it takes for a request to this endpoint takes to resolve? If it's
a long time(maybe due to the pandas call?),
your browser/client may be timing out. It could be because it takes a while
for the Db to find an existing Trend object
as well.
I can't give you any advice on how to fix it exactly, but I can tell you
what the problem is: The client is closing the
connection before socket.sendall() has finished writing to the socket. My
guess is that the calculate_trend() method
takes a long time to complete and the client is timing out.
--
https://mail.python.org/mailman/listinfo/python-list
Re: python 2.7.12 on Linux behaving differently than on Windows
On 12/07/2016 08:48 AM, BartC wrote: > I would prefer that the program "t" can be invoked exactly the same way > under both systems. I don't want different instructions for Linux, or > for the user (of my language) to have to write two lots of code, as that > is my job... Ahh now we come to the crux of your argument. You want all potential platforms to conform to your own idea of what is normal. And indeed this is the actual issue that started the original thread. But that's not the way it works for any platform. I develop a lot of code on Linux that I'd like to get running on Windows. I prefer the Linux way of doing things but I'm not going to make any headway if I just try to brow-beat Windows and Windows' users over the things that aren't implement the same way. And unfortunately neither are you. If I write a script that's going to take some filenames on the command-line, I'm going to have to implement a custom path to expand globs on Windows (if that's the behavior I need). You may prefer "t" to be invoked example the same way on Linux as Windows, but unless your program feels natural on the Linux command-line, it hardly matters to Linux users whether your program is invoked in the exact same way on Linux and Windows. If Linux is indeed going to be a first-class target for an application, you will want to do things the Linux or unix way. This may mean disabling globbing internally and just allowing arbitrary filenames on the command-line. Whatever. But this strange idea of yours that every OS should be like Windows is unrealistic and even highly offensive to some of our sensibilities. It doesn't matter what is better or more logical in your opinion. Python may provide some nice abstractions, but at the end of the day, there are always OS-specific things you'll have to and will want to deal with. -- https://mail.python.org/mailman/listinfo/python-list
Re: "Best" websocket implementation for Python 2.x?
PyPI came back. A bit more sleuthing suggests that the websocket-client package on PyPI is Ohtani's package, and is more up-to-date than the copyright notices would suggest. The package was updated a few days ago on GitHub. Taking the path of least resistance (no changes necessary to the code), I suspect I will just stick with that unless there are overriding inputs from the community suggesting something else is way better... S -- https://mail.python.org/mailman/listinfo/python-list
calling a program from Python batch file
Hi there, I am writing a batch file in Python. The batch file, in part, calls a program named "oq-console.bat" to run. Then once the program comes up (it looks like windows CMD), I need the batch file to type some commands to make it run (just like I would do it by myself). I need the batch file does the whole things automatically, I mean calls the program, types in commands and press enter! So far, I have used the below Python codes in my batch file to do that; import subprocess subprocess.call(["C:\Program Files (x86)\OpenQuake Engine\oq-console.bat", "oq --run", "job.ini"]) Note that the first string is the program, and the last two are the commands which needs to be typed in. It pulls up the program well, but doesn't type in the commands to make it to run. Any ideas on how I can get the program to run the commands as well? Thank you! -- https://mail.python.org/mailman/listinfo/python-list
Re: python 2.7.12 on Linux behaving differently than on Windows
On 12/07/2016 07:58 AM, Marko Rauhamaa wrote: > Steve D'Aprano : > >> I don't know any Unix programs that provide file spec processing. >> (That's not to say that there are absolutely none, only that I don't >> know of any.) > > I can think of "find", "git" and "scp". > > Git supports the recursive "**" wildcard, as well. And rsync. -- https://mail.python.org/mailman/listinfo/python-list
"Best" websocket implementation for Python 2.x?
I've recently inherited a small bit of code which uses websockets. I think the package it uses is Horoki Ohtani's package dating from 2010. I have v. 0.34.0. Looking on PyPI, I see another websocket package (not Ohtani's) with a version of 0.2.1 (also dating from 2010, I believe). These two packages would appear to have stalled. Looking around, I also found ws4py websockets - Python 3.3 or later autobahn.websocket There are probably others I haven't yet stumbled upon. (I'd continue poking around PyPI, but it seems the server is having problems.) Any suggestions as to which version is "best?" I leave it to you to decide what "best" means to you. Having never used websockets before, I'm not sure what criteria to use. Thx, Skip -- https://mail.python.org/mailman/listinfo/python-list
Re: python 2.7.12 on Linux behaving differently than on Windows
On Thu, Dec 8, 2016 at 2:19 AM, BartC wrote: > OK, but you go first: how does a Linux program tell the difference between: > > A B C > > which signifies three files, and: > > A B C > > which means two files, and an option C? And without using escapes! The option would be called "-C", which means you need only distinguish between filenames *that begin with a hyphen* and options. Given that this is fundamentally ambiguous, the solution has to be at the application level, and by convention, you can say "-- A B -C" to mean that these are all file names, or you can put an explicit path on it "A B ./-C". Does a leading "./" count as an escape? ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: python 2.7.12 on Linux behaving differently than on Windows
On Thu, Dec 8, 2016 at 1:57 AM, BartC wrote: > On 07/12/2016 12:39, Chris Angelico wrote: >> >> On Wed, Dec 7, 2016 at 10:54 PM, BartC wrote: >>> >>> But if a program existed that took N filename parameters with the purpose >>> of >>> deleting each of them, then it can't tell if they were typed in >>> individually >>> (so indicating a stronger intent), or whether a finger slipped on a >>> single >>> filename and a "*" was added, in which case it could choose to ask for >>> confirmation *if* it saw that this is a wildcard). >>> >>> 'Globbing' (if that means auto expansion) is taking away choice. >> >> >> And app globbing takes away choice, too. How would you, with the >> Windows 'del' command, remove a single file that has a question mark >> in the name? Or are you depending entirely on the fact that Windows >> doesn't let you do that? Here's some of my music collection: > > > I don't follow you. "?" is problematical on both systems. I think Windows > disallows it completely: I get 'Protocol error' if I copy such a file from > Linux to Windows. Presumably there is an escaping system, but I don't know > what it is. Nope. Not at all problematical on my Linux box. I can invoke that file in any way I like, and it just works. Tab completion works. Globbing works. Invoking it through Popen (or equivalent) works. It is a perfectly acceptable file name. The only time I have trouble with those files is when I copy them onto an external drive so my brother can play them... on his Windows computer. It's not the file name that has the problem. It's Windows. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: python 2.7.12 on Linux behaving differently than on Windows
On 07/12/2016 13:58, Dennis Lee Bieber wrote: On Wed, 7 Dec 2016 11:54:35 +, BartC declaimed the following: With automatic expansion, then EVERY program can become dangerous. Can no one else see this? With your scheme, EVERY PROGRAM HAS TO IMPLEMENT ITS OWN GLOBBING. Only the tiny minority that can be meaningfully invoked on an arbitrary number of files at the same time. And globbing doesn't take care of all of it: a Linux program still has to iterate over a loop of filenames. The same as on Windows, except the latter will need to call a function to deliver the next filename. The result is BLOAT and INCONSISTANT BEHAVIOR. UNIX shell behavior is consist ant -- you learn the rules for ONE shell, and they apply to ALL programs started with that shell. You'll respond that a library should be provided for programs to do that... But a C library would have to be wrapped to be used in Python, Ada, You make it sound like a big deal. Here's a program (in my language not Python) which prints the list of files matching a file-spec: println dirlist(cmdparams[2]) It's invoked like this in Windows (pcc is interpreter, t is the program): pcc t *.c and like this in Linux: ./pcc t "*.c" In both cases is displays a list of .c files. The Linux needs the quotes because otherwise it is expanded to a list of N files, of which the first is taken to be the filespec, so it will print only the first. I would prefer that the program "t" can be invoked exactly the same way under both systems. I don't want different instructions for Linux, or for the user (of my language) to have to write two lots of code, as that is my job: println dirlist(cmdparams[2])# Windows println tail(cmdparams) # Linux (Here are implementations of that dirlist() for both OSes: http://pastebin.com/knciRULE Code is my dynamic language, not C. But you can see there is not much to it. No 3rd party libraries apart from what comes with the OS, or that is part of Posix.) what-have-you. Thereby one now has a possibly different API in each language for doing something that should be the same action. Or even multiple APIs in one language (just look at Python's library for command line argument parsing: Just look at *any* of Python's thousands of libraries. It's always seem strange that there are libraries to do any conceivable thing, but people here baulk at the basics (simple ways of reading stuff on a line, reading a single key, and now reading /the input to a program/, which is pretty important). -- Bartc -- https://mail.python.org/mailman/listinfo/python-list
Re: Detect Linux Runlevel
Grant Edwards : > There are a few things in Unix that are fundamentally broken and > really just can't be used for the things they are intended for (serial > ports come to mind). > > However, that doesn't seem to prevent them from having been used > sucessfully that way way for 40 years. ;) Sigh. Those issues give me grief literally every day in the office. Marko -- https://mail.python.org/mailman/listinfo/python-list
Re: sets anomaly
On Wednesday, December 7, 2016 at 10:18:32 AM UTC-5, Rustom Mody wrote:
> Trying to write some code using sets (well frozen sets)
> And was hit by this anomaly
>
> This is the behavior of lists I analogously expect in sets:
>
> >>> []
> []
> >>> [[]]
> [[]]
> >>>
>
> ie the empty list and the list of the empty list are different things
>
> However
> (with
> f= frozenset
> )
>
> >>> f()
> frozenset()
> >>> f([])
> frozenset()
> >>> f(f([]))
> frozenset()
> >>>
The difference is more about the difference between the behavior of a
callable constructor, and a list literal. Lists will behave the same
as frozenset if you use list():
>>> list()
[]
>>> list(list())
[]
>>> list(list(list()))
[]
This is because the constructors can take a sequence, and use those
elements as the new contents. List literals don't work that way.
> Spent a good ½ hour finding out this strangeness
> And then some figuring out how to get an empty set into a set
> This is the best I get:
> >>> f([f([])])
> frozenset({frozenset()})
That is the way I would have done it also. Or:
>>> s = set()
>>> s.add(frozenset())
>>> frozenset(s)
frozenset([frozenset([])])
Notice the repr output of the result shows how to make it! :)
--Ned.
--
https://mail.python.org/mailman/listinfo/python-list
Re: python 2.7.12 on Linux behaving differently than on Windows
BartC writes: > [...] But remember: > > cp *.c > > There might be some irate users out there if it can't detect a simple > user error like that. There might be. They are ill-served by current Unix shells. But, oddly, the flow has gone the other way. People have wanted Unix shells and utilities ported to Windows, not cmd.exe and friends ported to Linux. You clearly see a gap in the provision here. Wouldn't it be better for the world if you wrote cmd.exe for Linux rather than complaining about what other shells do? > OK, I understand. Having a program launcher indiscriminately transform > input A into A' is /much/ better than having it done by the program, > even if the program doesn't want it and it is not meaningful. Yes, you've explained that you understand the value of such transformations, it's just that the ones you like happen to be exactly those in cmd.exe. You are very lucky that cmd.exe happened to get exactly the right interpretation of the input or you would be raining against /every/ shell on both systems. -- Ben. -- https://mail.python.org/mailman/listinfo/python-list
Re: python 2.7.12 on Linux behaving differently than on Windows
On 07/12/2016 14:34, Steve D'Aprano wrote:
On Wed, 7 Dec 2016 11:23 pm, BartC wrote:
On 07/12/2016 05:15, Steven D'Aprano wrote:
On Wednesday 07 December 2016 12:55, BartC wrote:
But even Linux's 128KB will fill if someone wanted a command line that
listed 20,000 files individually. But it would be spectacularly bad use
of a command line interface which was designed for humans.
That's EXACTLY the point of having the shell do globbing.
Sorry, it's got NOTHING to do with it!
Here's a command that works on all one million files in the current path:
program *.*
11 characters, so Linux still has 131061 left (assuming 'program ' is
part of it).
I don't know why you are so hung up over the number of characters here, or
this bogeyman of "one million files" in a directory.
Because /you/ brought it up as a reason why 'globbing' would help when
there are command limits?
Who does such a thing?
Most sys admins will go their entire career without ever needing to process
a million files in a single directory, or even split over a dozen
directories.
They probably never need to use a 128KB command line either.
But... how does your program distinguish between the file spec *.* which
should be expanded, and the literal file *.* which shouldn't be expanded?
Do you mean a *.* parameter that refers to all the files in the
directory (whether the intention is to build a list of those files or
not) and a *.* parameter that means something entirely different?
OK, but you go first: how does a Linux program tell the difference between:
A B C
which signifies three files, and:
A B C
which means two files, and an option C? And without using escapes!
You need an escape mechanism too. You've done that, of course. Right?
My program wouldn't need anything so crude. The input syntax would be
more carefully designed so as to not have the ambiguity.
And actually I generally avoid command line interfaces (ones used by
shells) because they are such a pain. (Why, for example, aren't the
contents of the command line - even after preprocessing by the shell -
just read using exactly the same mechanism as subsequent lines of input?)
list fooba?.jpg {here,there}/*.jpg another/place/*.{png,jpg} \
[a-z]/IMG* > files
command @files
*Requiring* this would be completely unacceptable. Forcing the user to write
two lines (one to create a variable, then to use it) every time they needed
a glob expansion would go down among sys admins like a lead balloon.
Suppose there is a pattern that can't be expressed as a one-liner?
And the shell is supposed to be a full-fledged programming language,
people keep saying. What can be more natural than to make use of
variables? As you go on to demonstrate!
--
Bartc
--
https://mail.python.org/mailman/listinfo/python-list
sets anomaly
Trying to write some code using sets (well frozen sets)
And was hit by this anomaly
This is the behavior of lists I analogously expect in sets:
>>> []
[]
>>> [[]]
[[]]
>>>
ie the empty list and the list of the empty list are different things
However
(with
f= frozenset
)
>>> f()
frozenset()
>>> f([])
frozenset()
>>> f(f([]))
frozenset()
>>>
Spent a good ½ hour finding out this strangeness
And then some figuring out how to get an empty set into a set
This is the best I get:
>>> f([f([])])
frozenset({frozenset()})
Context:
Trying to write the simple combinatorics code which does the enumeration
corresponding to the identity:
ⁿCr + ⁿCr-1 = ⁿ⁺¹Cr
The Haskell code is very simple:
[`c` means function c used in infix]
[] `c` (r+1) = []
xs `c` 0 = [[]]
(x::xs) `c` r = (xs `c` r) ++ [x::ys | ys <- xs `c` (r-1)]
And its usage:
? [1,2,3,4] `c` 2
[[3, 4], [2, 4], [2, 3], [1, 4], [1, 3], [1, 2]] : [[Int]]
I wanted to do it in python to discuss the question of sets and lists
The best I get is this.
Can it be improved??
---
f = frozenset
def c(zs,r):
""" returns list of lists """
if not zs and r > 0 : return []
if r == 0 : return [[]]
x,xs = zs[0], zs[1:]
return c(xs,r) + [[x]+ys for ys in c(xs,r-1)]
def css(zs,r):
""" returns set of sets """
if not zs and r > 0 : return f([])
if r == 0 : return f([f([])])
x,xs = zs[0], zs[1:]
return css(xs,r) | f([(f([x]) | ys) for ys in css(xs,r-1)])
def cls(zs,r):
""" returns list of sets """
if not zs and r > 0 : return []
if r == 0 : return [f([])]
x,xs = zs[0], zs[1:]
return cls(xs,r) + [f([x]) | ys for ys in cls(xs,r-1)]
And usage:
>>> c("ABC",2)
[['B', 'C'], ['A', 'C'], ['A', 'B']]
>>> css("ABC",2)
frozenset({frozenset({'C', 'A'}), frozenset({'C', 'B'}), frozenset({'B', 'A'})})
>>> cls("ABC",2)
[frozenset({'C', 'B'}), frozenset({'C', 'A'}), frozenset({'B', 'A'})]
--
https://mail.python.org/mailman/listinfo/python-list
Re: python 2.7.12 on Linux behaving differently than on Windows
On Wednesday, 7 December 2016 11:55:01 UTC, BartC wrote: > With automatic expansion, then EVERY program can become dangerous. Can > no one else see this? Please start an elevated command prompt on your PC, and type the command rmdir /s /q C:\WINDOWS There are no special characters here, so no possibility of "automatic expansion", so this is bound to be completely safe[1]. Can you not see that danger is unrelated to the existence of globbing? Danger is a result of people not knowing what the commands they type do. Paul [1] To anyone following along who isn't familiar with the command line, DON'T TRY THIS - IT WILL WRECK YOUR PC! -- https://mail.python.org/mailman/listinfo/python-list
Re: python 2.7.12 on Linux behaving differently than on Windows
On Tuesday, 6 December 2016 21:44:18 UTC, Gregory Ewing wrote: > BartC wrote: > > But those would be silly. > > > > Some special syntax is known about: | < and > for example. % less so > > What you need to understand is that, to a Unix user, > * and ? are *just as well known* as |, < and >. And to Windows users. Just because BartC is proposing a silly distinction between what "Unix users" and "Windows users" expect, doesn't mean there is such a distinction. Users with limited experience of command line environments may well be unclear about special characters like * ? > < |. It's possibly even true that Windows users as a group are more likely to be unfamiliar with the command line than Unix users (although people using iPads and Android phones are technically "Unix users", so that point isn't as true as you'd expect). But claiming that special syntax is an issue because people who are unfamiliar with it, are unfamiliar with it, is silly at best, and deliberate trolling at worst. Paul -- https://mail.python.org/mailman/listinfo/python-list
Re: python 2.7.12 on Linux behaving differently than on Windows
Steve D'Aprano : > I don't know any Unix programs that provide file spec processing. > (That's not to say that there are absolutely none, only that I don't > know of any.) I can think of "find", "git" and "scp". Git supports the recursive "**" wildcard, as well. Marko -- https://mail.python.org/mailman/listinfo/python-list
Re: python 2.7.12 on Linux behaving differently than on Windows
On 07/12/2016 12:39, Chris Angelico wrote: On Wed, Dec 7, 2016 at 10:54 PM, BartC wrote: But if a program existed that took N filename parameters with the purpose of deleting each of them, then it can't tell if they were typed in individually (so indicating a stronger intent), or whether a finger slipped on a single filename and a "*" was added, in which case it could choose to ask for confirmation *if* it saw that this is a wildcard). 'Globbing' (if that means auto expansion) is taking away choice. And app globbing takes away choice, too. How would you, with the Windows 'del' command, remove a single file that has a question mark in the name? Or are you depending entirely on the fact that Windows doesn't let you do that? Here's some of my music collection: I don't follow you. "?" is problematical on both systems. I think Windows disallows it completely: I get 'Protocol error' if I copy such a file from Linux to Windows. Presumably there is an escaping system, but I don't know what it is. But we're talking about Linux. Obviously if a program is developed on Linux, can work on multiple files and DEPENDS on on being presented with that list of files as though it had been invoked as: program file1 file2 file3 file4 file5 file6 file7 fileN then it will have problems if that feature is taken away. I can see how it can still be there after all these years! -- Bartc -- https://mail.python.org/mailman/listinfo/python-list
Re: python 2.7.12 on Linux behaving differently than on Windows
On Tuesday, 6 December 2016 16:34:10 UTC, Michael Torrie wrote: > On 12/06/2016 06:52 AM, BartC wrote: > > Then you don't get utterly ridiculous and dangerous behaviour such as > > the cp example Paul Moore came up with (that trumps most of mine actually): > > It's potentially dangerous agreed. Note, by the way, that BartC is misquoting me - I never said that behaviour was "ridiculous and dangerous". I pointed out that it was potentially dangerous and needs to be learned and understood. And that it's the result of a trade-off made by Unix, where Windows makes a different trade-off. My point was in fact that BartC's position, that expecting completely different environments (Unix shells vs Windows shells vs Popen) to behave identically, is silly. *All* environments make usability trade-offs, and do so in different ways. Users need to learn these, or accept that any mistakes they make are due to their own lack of understanding, and not because, as BartC repeatedly (and rather offensively) asserts, the environments they are not familiar with are "stupid". Paul -- https://mail.python.org/mailman/listinfo/python-list
Re: python 2.7.12 on Linux behaving differently than on Windows
On Wed, 7 Dec 2016 11:23 pm, BartC wrote:
> On 07/12/2016 05:15, Steven D'Aprano wrote:
>> On Wednesday 07 December 2016 12:55, BartC wrote:
>>
>>> But even Linux's 128KB will fill if someone wanted a command line that
>>> listed 20,000 files individually. But it would be spectacularly bad use
>>> of a command line interface which was designed for humans.
>>
>> That's EXACTLY the point of having the shell do globbing.
>
> Sorry, it's got NOTHING to do with it!
>
> Here's a command that works on all one million files in the current path:
>
> program *.*
No, sorry, you're wrong. That specific program is buggy, and its
implementation of globbing is unreliable. I think you meant this one:
prog *.*
except, no, that one uses regular expressions, because the author is an
opinionated PITA who dislikes globs. So you would have to write:
prog .*?\..*
but that's okay, you remembered that, didn't you? I hope so, otherwise it
will BSOD if you call it with an invalid regex.
Perhaps you meant to say
appl *.*
but only version 4.7 or higher, it didn't get support for globbing before
then.
(If you're going to make up imaginary programs that behave the way you
expect, I'm allowed to do the same.)
> 11 characters, so Linux still has 131061 left (assuming 'program ' is
> part of it).
I don't know why you are so hung up over the number of characters here, or
this bogeyman of "one million files" in a directory. Who does such a thing?
Most sys admins will go their entire career without ever needing to process
a million files in a single directory, or even split over a dozen
directories.
But for those who do need to do something like that, there are ways to deal
with it. Nobody (except in your imagination) says that the Unix way of
doing things is optimal for all possible scenarios. As I keep saying, and
you keep ignoring, the Unix way is the product of forty years of directed
evolution by people who have chosen this way because it works best for 90,
95% of what they do. Of course there are exceptions.
I understand that *your* needs are different. I've acknowledged that
repeatedly. And I've repeatedly told you how, for the sake of just a few
extra keystrokes, you can get the results you want on Linux:
- use backslashes to escape individual metacharacters;
- quote the arguments;
(but beware that "x" and 'x' are different in bash, and I'm afraid I'm not
expert enough to remember what the differences are. I just always use "x"
and hope :-)
- disable globbing, using "set -o noglob" in bash.
We get it that your needs are not those of the typical Unix sys admin. Unix
command line tools are designed for Unix sys admins, not you. But they're
not idiots, despite what you apparently think, and provide a way to bail
out of the normal pre-processing when needed.
[...]
> In Windows, argc is 2 (name of program, and the "*.*" parameter). If the
> program wants to process all one million files, it will have to scan
> them one at a time, but it does not need to build a list first.
If you want to pass a file spec to a Unix program, all you need do is quote
the file spec:
ls "*.*"
But it won't do you any good. I don't know any Unix programs that provide
file spec processing. (That's not to say that there are absolutely none,
only that I don't know of any.) That's simply not considered a necessary or
useful feature. But if you want to provide that feature in your own
programs, go right ahead.
But... how does your program distinguish between the file spec *.* which
should be expanded, and the literal file *.* which shouldn't be expanded?
You need an escape mechanism too. You've done that, of course. Right?
[...]
>> command fooba?.jpg {here,there}/*.jpg another/place/*.{png,jpg}
>> [a-z]/IMG*
>
>
>list fooba?.jpg {here,there}/*.jpg another/place/*.{png,jpg} \
> [a-z]/IMG* > files
>
>command @files
*Requiring* this would be completely unacceptable. Forcing the user to write
two lines (one to create a variable, then to use it) every time they needed
a glob expansion would go down among sys admins like a lead balloon.
But for those times when you really need to repeat a complex set of
arguments, bash has you covered:
[steve@ando ~]$ export spec="a* z*"
[steve@ando ~]$ echo "$spec"
a* z*
[steve@ando ~]$ ls $spec
aaa-bbb.odt addressesararatarpaio.txt aß zip
--
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.
--
https://mail.python.org/mailman/listinfo/python-list
Re: Detect Linux Runlevel
On 2016-12-07, Marko Rauhamaa wrote: > Steven D'Aprano : > >> Whereas if it were mandatory locking, enforced by the OS, it >> wouldn't be a problem? > > The point is, the utmp scheme seems to be fundamentally broken. You > can't use a regular file for this kind of communication. There are a few things in Unix that are fundamentally broken and really just can't be used for the things they are intended for (serial ports come to mind). However, that doesn't seem to prevent them from having been used sucessfully that way way for 40 years. ;) -- Grant Edwards grant.b.edwardsYow! I KAISER ROLL?! at What good is a Kaiser Roll gmail.comwithout a little COLE SLAW on the SIDE? -- https://mail.python.org/mailman/listinfo/python-list
Re: python 2.7.12 on Linux behaving differently than on Windows
On Wed, Dec 7, 2016 at 9:43 AM, Chris Angelico wrote: > Exactly. Back in the 1990s, we had the beginnings of Windows NT, which > was designed for servers. It had (if I recall correctly) no concept of > file/directory permissions, little or no notion of process privilege, > and definitely no way to restrict one process to X amount of memory > and Y amount of processing. Do you have a name or a link for this system that pre-dates NT? At the time I thought Microsoft only had MS-DOS and OS/2 (and maybe still Xenix?). I know they hired Dave Cutler (the designer of VMS) and several other DEC programmers in late 1988 to develop NT OS/2. It took them 5 years, and along the way the design switched from NT OS/2 to Windows NT (DOS-based Windows had surged in popularity with the release of Windows 3.x, and the business relationship with IBM fell apart). They relegated the OS/2 and POSIX subsystems to the console only, without GUI support (similar to the new Linux subsystem). > Since then, Windows has progressively > added features that Unix has had for decades, because those features > are critical. Those who don't understand Unix are forced to reinvent > it, piece by piece, badly, and then finally rip it out and start > over... NT was designed to run a Unix subsystem and meet U.S. DoD security requirements, so it's naturally going to share a lot of capabilities with Unix. But the similarities are superficial; the design decisions are fundamentally different in many ways. (The differences are more obvious in native NT and kernel development.) No one could say the NT design team was copying Unix. OTOH, how much they pilfered from their previous work on VMS is debatable. IIRC, Microsoft ultimately reached a settlement out of court with DEC on this issue, which was the main reason NT was ported to the DEC Alpha processor in the 90s. -- https://mail.python.org/mailman/listinfo/python-list
Re: python 2.7.12 on Linux behaving differently than on Windows
On Thu, 8 Dec 2016 12:07 am, eryk sun wrote: >> latest step is Microsoft's partnership with Canonical (the company behind >> Ubuntu Linux) to bring bash to Windows. > > The partnership with Canonical is to provide an entire Linux > distribution, sans the kernel. The Windows Subsystem for Linux (WSL) > works by translating Linux system calls to the NT kernel. It directly > executes unmodified ELF binaries. I think it was a mistake to call it > "bash on Windows". It seems like people think it's just a Windows port > of bash. /slaps forehead Of course it is! Yes, you're right -- its more than just bash. I actually knew that, but forgot, so thanks for the reminder. The reason I misremembered it as just bash is when I learned about this, after the initial chorus of people saying "hang on, its a day early for April's Fool", the majority of comments I saw were from dual Windows/Linux system administrators saying "At last I can use bash on Windows!". -- Steve “Cheer up,” they said, “things could be worse.” So I cheered up, and sure enough, things got worse. -- https://mail.python.org/mailman/listinfo/python-list
Re: python 2.7.12 on Linux behaving differently than on Windows
Chris Angelico : > $ find Music/ -name *\\?* I do recommend quoting your wildcard characters. Yet further nasty security pitfalls in bash programming: 5. Wildcards that don't match any file are left unexpanded. 6. Commands behave unexpectedly when given null lists: $ ls a b c a b c $ ls a b a b $ ls a a $ ls a b c d Marko -- https://mail.python.org/mailman/listinfo/python-list
Re: python 2.7.12 on Linux behaving differently than on Windows
On Wed, Dec 7, 2016 at 8:30 AM, Steven D'Aprano wrote: > > There are Windows programs which understand globs, like dir. Do you honestly > expect us to believe that it is okay to use "dir a*" on Windows because dir > itself expands the glob, but not okay to use "dir a*" on Linux because the > shell has expanded the glob before dir sees it? "dir" is a built-in shell command. But yes, there are programs that understand globbing. A C/C++ program can link with wsetargv.obj to get this feature. That said, I hadn't actually ever used this feature prior to reading this thread. I linked it into a custom of python.exe, and I'm not happy with the implementation because it doesn't allow escaping wildcards (confirmed in the source distributed with VS 2015). > latest step is Microsoft's partnership with Canonical (the company behind > Ubuntu Linux) to bring bash to Windows. The partnership with Canonical is to provide an entire Linux distribution, sans the kernel. The Windows Subsystem for Linux (WSL) works by translating Linux system calls to the NT kernel. It directly executes unmodified ELF binaries. I think it was a mistake to call it "bash on Windows". It seems like people think it's just a Windows port of bash. -- https://mail.python.org/mailman/listinfo/python-list
Re: python 2.7.12 on Linux behaving differently than on Windows
On Wed, Dec 7, 2016 at 10:54 PM, BartC wrote: > But if a program existed that took N filename parameters with the purpose of > deleting each of them, then it can't tell if they were typed in individually > (so indicating a stronger intent), or whether a finger slipped on a single > filename and a "*" was added, in which case it could choose to ask for > confirmation *if* it saw that this is a wildcard). > > 'Globbing' (if that means auto expansion) is taking away choice. And app globbing takes away choice, too. How would you, with the Windows 'del' command, remove a single file that has a question mark in the name? Or are you depending entirely on the fact that Windows doesn't let you do that? Here's some of my music collection: $ find Music/ -name *\\?* Music/BeautyStone2/25: Act III. Change of scene?.ogg Music/BeautyStone2/08: Act II, Scene 1. No. 13 continued. Philip:"How say you?".ogg Music/Seekers/CD2 - 1964-1965/17. What Have They Done To The Rain?.mp3 Music/Seekers/CD4 - Hits. B-sides & 90s/14. When Will The Good Apples Fall?.mp3 Music/BeautyStone1/06: Act I, Scene 1. No. 4. Quartet. Simon:"Who stands within?".ogg Music/G&S/Pirates of Penzance/111 What ought we to do, Gentle sisters, say?.mp3 Music/G&S/Pirates of Penzance/MusicOnly/111 What ought we to do, Gentle sisters, say?.ogg Note that two of the Beauty Stone tracks include quotes as well as question marks. How do you identify those? Let's say you want to play one of these in VLC, and then maybe you decide that the track in Pirates of Penzance/MusicOnly is slightly mis-cropped, so you rebuild it from the one in the parent directory. How does that work on Windows? If you say "it doesn't", then (a) you have taken away choice on a fundamental level, (b) you have your head in the sand, and (c) you still haven't solved the problem of percent signs, carets, and so on, which are perfectly legal in file names, but have meaning to the shell. > They can be dangerous with certain commands and programs that accept > wildcards (which aren't many actually; I don't think any of my programs do). And that right there is a mark against the Windows model. Your program has to consciously accept wildcards. Mine (on Unix) just has to accept file names, and *everything will work*. > If the default is to have it on, then that's not satisfactory. One line in your shell initialization script will change that default. >> If *your* design was paramount, it would be up to each and every >> application to >> choose whether or not to support globbing and environment variable >> expansion, >> and hard luck if the application I needed to use didn't understand the >> globs I >> wanted. > > > How many programs take parameters that specify multiple files and work on > them together? As I said, I think hardly any of mine do. Once again, you assume that because YOU PERSONALLY write no programs that need this feature, the feature ITSELF is a bad thing. Can you not see the arrogance of that? > It took me a while to recognise the significance of the "\". In Windows > that's a path separator. You call yourself a Python programmer and are surprised to find that backslash is the escape character? It's a pretty common convention. > Most open source software seems to be written in C and seems to require > executing a 20,000 or 30,000-line bash 'configure' script, itself dependent > on a dozen Unix utilities, before it's possible to build it. Even if the > resulting product is cross-platform. Not everything is like that. Here, find the configure scripts in these projects of mine: https://github.com/Rosuav/Gypsum https://github.com/Rosuav/StilleBot https://github.com/Rosuav/FrozenOST https://github.com/Rosuav/LetMeKnow https://github.com/Rosuav/zawinski https://github.com/Rosuav/full-stack-app https://github.com/Rosuav/miniature-succotash https://github.com/Rosuav/Yosemite https://github.com/Rosuav/Hymns https://github.com/Rosuav/pike-alsa I think the longest config file is the Hymns collection, with a twenty-line makefile. Don't judge all open source on what you see in Mozilla Thunderbird. :) ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: python 2.7.12 on Linux behaving differently than on Windows
On 07/12/2016 05:15, Steven D'Aprano wrote:
On Wednesday 07 December 2016 12:55, BartC wrote:
But even Linux's 128KB will fill if someone wanted a command line that
listed 20,000 files individually. But it would be spectacularly bad use
of a command line interface which was designed for humans.
That's EXACTLY the point of having the shell do globbing.
Sorry, it's got NOTHING to do with it!
Here's a command that works on all one million files in the current path:
program *.*
11 characters, so Linux still has 131061 left (assuming 'program ' is
part of it).
But Windows will also have 8181 characters left (or 8180, whatever). And
the shell hasn't needed to do any 'globbing'.
Now we come to the next bit, where the program in question has an entry
point that looks like 'main(argc, argv)', where some function splits up
the command line.
In Windows, argc is 2 (name of program, and the "*.*" parameter). If the
program wants to process all one million files, it will have to scan
them one at a time, but it does not need to build a list first.
In Linx, argc will be 101, and the program will still need scan them
one at a time (although it won't need to call an OS function to do the
iteration).
In neither case was the command line limit an issue.
But I can see where you're coming from: if someone was silly enough to
want to specify all one million filenames on the command line, then they
will eventually come up to the limit. On both systems.
That is not going to happen however. And is nothing to do with how or
where *.* expansion is done.
command fooba?.jpg {here,there}/*.jpg another/place/*.{png,jpg} [a-z]/IMG*
list fooba?.jpg {here,there}/*.jpg another/place/*.{png,jpg} \
[a-z]/IMG* > files
command @files
Only 'list' needs to understand all this crap (and more besides).
Applications only need to understand '@'. Or maybe @files /can/ be
expanded by the shell, more acceptable here because it's explicit.
--
Bartc
--
https://mail.python.org/mailman/listinfo/python-list
Re: python 2.7.12 on Linux behaving differently than on Windows
On 07/12/2016 08:30, Steven D'Aprano wrote: On Wednesday 07 December 2016 03:41, BartC wrote: On 06/12/2016 15:44, Michael Torrie wrote: On 12/06/2016 04:43 AM, BartC wrote: Yes shell expansion has it's gotchas. But those can all be learned, Yes, learn to avoid wildcards in command parameters at all costs. But we both know that is not satisfactory. Now you're being ridiculous. Can't you at least debate this sensibly? There are Windows programs which understand globs, like dir. Do you honestly expect us to believe that it is okay to use "dir a*" on Windows because dir itself expands the glob, but not okay to use "dir a*" on Linux because the shell has expanded the glob before dir sees it? You don't get it. The problem is the program not seeing the original unexpanded parameters that it wants, with extra information that is lost when indiscriminately expanded. Tell me Bart, what's the difference between these? # Linux rm * and # Windows del * It asks 'Are your sure?'. But that is beside the point. And commands such as 'del' /are/ crude. But if a program existed that took N filename parameters with the purpose of deleting each of them, then it can't tell if they were typed in individually (so indicating a stronger intent), or whether a finger slipped on a single filename and a "*" was added, in which case it could choose to ask for confirmation *if* it saw that this is a wildcard). 'Globbing' (if that means auto expansion) is taking away choice. I see that the Windows version of copy avoids this *by accident*: C:\Documents and Settings\Steve>copy a* a1 The file cannot be copied onto itself. 0 file(s) copied. No, even if it did copy the file, it would be harmless. 'copy' chose to make the second parameter option, and to make it default to "." Instead of accidentally copying a1 over a2, it accidentally tries to copy a1 over a1, and fails. At least, that's what it looks like to me. Any Windows expert like to chime in? a1 would be copied to a1, a2 to a2. That's how it has to work otherwise all files would be renamed when copied elsewhere! The bottom line is, it doesn't matter where the wildcards are expanded. Whether the shell does it, or the application itself, they're dangerous by their very nature. They can be dangerous with certain commands and programs that accept wildcards (which aren't many actually; I don't think any of my programs do). With automatic expansion, then EVERY program can become dangerous. Can no one else see this? I get that its not the way that you like, but fortunately for you, they provided not one, but TWO easy ways to get the results you want: - escaping the metacharacters you want to be left untouched; - disabling globbing (for either a single command, a session, or permanently). If the default is to have it on, then that's not satisfactory. If *your* design was paramount, it would be up to each and every application to choose whether or not to support globbing and environment variable expansion, and hard luck if the application I needed to use didn't understand the globs I wanted. How many programs take parameters that specify multiple files and work on them together? As I said, I think hardly any of mine do. The fact that you also lose format information (is it three parameters, or one parameter transformed into three) is an extra bonus. Here's a two-line Python program to prove what nonsense that is. steve@runes:~$ cat argv.py import sys print( sys.argv ) And in use: steve@runes:~$ python argv.py a\* z\* ['argv.py', 'a*', 'z*'] But in real use it will be: python argv.py a* z* It took me a while to recognise the significance of the "\". In Windows that's a path separator. Yes, there are any number of ways that *you* or *I* can get AROUND the problem. But my background in the past has been with users (and talking through bunches of console commands on the phone). But the problem is still there. And it looks like Windows is ever-so-slowly converging on the same, or at least similar, sets of conventions: by copying Unix globbing metacharacters in the (I'm sure I used these in the 1970s. Or was DEC copying Unix?) first place, then by adding them to the C library to make it easy for applications to do their own globbing, then by moving to PowerShell, and the latest step is Microsoft's partnership with Canonical (the company behind Ubuntu Linux) to bring bash to Windows. Most open source software seems to be written in C and seems to require executing a 20,000 or 30,000-line bash 'configure' script, itself dependent on a dozen Unix utilities, before it's possible to build it. Even if the resulting product is cross-platform. So that's not surprising, if everyone is obliged to install Msys or Cygwin anyway. (Apart from me, as I refuse to do anything like that. If a program is in C, then I should need only a C compiler. Any software I make available takes exactl
Re: Detect Linux Runlevel
Steven D'Aprano : > Whereas if it were mandatory locking, enforced by the OS, it wouldn't > be a problem? The point is, the utmp scheme seems to be fundamentally broken. You can't use a regular file for this kind of communication. Marko -- https://mail.python.org/mailman/listinfo/python-list
Re: python 2.7.12 on Linux behaving differently than on Windows
On Wed, Dec 7, 2016 at 7:30 PM, Steven D'Aprano wrote: > If you accidentally say "del *" on Windows, say, when you meant "z*" for > example, and Windows deletes all the files in the current directory, do you > say > to yourself...? > >"Thank the deities that * was expanded by the del command rather >than the shell! Imagine how awful it would have been if the shell >did the expansion!" There is one special case: if you use "*.*" on Windows, the del command will prompt. However, there are plenty of other examples where this special case doesn't apply. > I see that the Windows version of copy avoids this *by accident*: > > C:\Documents and Settings\Steve>copy a* > a1 > The file cannot be copied onto itself. > 0 file(s) copied. > > > > Instead of accidentally copying a1 over a2, it accidentally tries to copy a1 > over a1, and fails. At least, that's what it looks like to me. Any Windows > expert like to chime in? Not a Windows expert, but I happen to know this one; and it's another special case. If you provide only one argument (and a glob does count as a single argument), copy assumes you mean "to the current directory" - ie an implicit "." as a second argument. So you've landed on two special cases in Windows commands that take advantage of the fact that the app has to expand globs. They give a marginally better result at the cost of consistency. I believe the Zen of Python has something to say about special cases; while there definitely are times when that line is wrong, there are a lot of times when it's right, and forfeiting consistency is usually too high a cost for the benefit gained. >> (2) There have been 40 years of it working this way and there have never >> been any problems. (That is, 40 years of writing programs with >> stultified command line interfaces to make sure that is always the case. >> [As for 'problems' such as the 'cp *.c' one, that's a feature!] > > Don't be absurd. > > Any Unix sys admin can tell you stories of accidentally globbing too much and > screwing something up. And any Windows sys admin can tell you the same > stories, > even though globbing is handled by the individual programs rather than the > shell. And a lot of the over-globbing examples depend on very specific situations. With "cp *.c" and no destination, Unix cp will silently do the wrong thing ONLY if there are exactly two files matching that glob, or if everything matching that glob is a file except the last, which is a directory. (With most globs, that's unlikely.) With Windows "copy *.c", it gives an obscure error message ONLY because you didn't have a path on the glob; if you say "copy source\*.c", it'll silently do the wrong thing. Let's face it. You should be using git anyway. Then you have a protection *no matter what* you mess up - just check out any files you messed up. Tada! > (At least, any Windows sys admin who uses the command line, or writes batch > files. I dare say those who use nothing but GUI tools have their own, > different, horror stories.) Oh, absolutely. How about a Windows domain controller that BSODed multiple times just trying to pull up the config program? http://thedailywtf.com/articles/the-infrastructure Part of that WTF is that the domain controller had no remote access. But what's the cost (in server load) of a GUI-based remote admin facility? Decidedly non-zero. How about good ol' SSH? So light that I've been able to use it to remote in to a computer that had gotten itself into some sort of infinite loop, and find out what was going wrong. And fix it without rebooting, too. If your solution to everything is "bah, just reboot", then you don't have a solution. > But Unix systems are the end product of 40 years of directed evolution by > thousands of sys admins who have designed the shells to work the way they > want. > > And it looks like Windows is ever-so-slowly converging on the same, or at > least > similar, sets of conventions: by copying Unix globbing metacharacters in the > first place, then by adding them to the C library to make it easy for > applications to do their own globbing, then by moving to PowerShell, and the > latest step is Microsoft's partnership with Canonical (the company behind > Ubuntu Linux) to bring bash to Windows. Exactly. Back in the 1990s, we had the beginnings of Windows NT, which was designed for servers. It had (if I recall correctly) no concept of file/directory permissions, little or no notion of process privilege, and definitely no way to restrict one process to X amount of memory and Y amount of processing. Since then, Windows has progressively added features that Unix has had for decades, because those features are critical. Those who don't understand Unix are forced to reinvent it, piece by piece, badly, and then finally rip it out and start over... ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Django broken pipe error
On Tuesday, December 6, 2016 at 3:22:13 PM UTC+1, dr.rom...@gmail.com wrote:
> Hi,
>
> I'm facing strange Django broken pipe error (Python 2.7 on Ubuntu) that
> apparently is a not fixed Django bug. Does anybody now how to fix it? I've
> been searching a lot and didn't find any solution.
>
> This error happens very irregularly by Post request in Django. Sometimes it
> works sometimes not:
>
> [06/Dec/2016 13:33:57] "POST /export_report/ HTTP/1.1" 500 59
> Traceback (most recent call last):
> File "/usr/lib/python2.7/SocketServer.py", line 593, in
> process_request_thread
> self.finish_request(request, client_address)
> File "/usr/lib/python2.7/SocketServer.py", line 334, in finish_request
> self.RequestHandlerClass(request, client_address, self)
> File
> "/home/ait/.virtualenvs/env1/local/lib/python2.7/site-packages/django/core/servers/basehttp.py",
> line 129, in __init__
> super(WSGIRequestHandler, self).__init__(*args, **kwargs)
> File "/usr/lib/python2.7/SocketServer.py", line 651, in __init__
> self.finish()
> File "/usr/lib/python2.7/SocketServer.py", line 710, in finish
> self.wfile.close()
> File "/usr/lib/python2.7/socket.py", line 279, in close
> self.flush()
> File "/usr/lib/python2.7/socket.py", line 303, in flush
> self._sock.sendall(view[write_offset:write_offset+buffer_size])
> error: [Errno 32] Broken pipe
>
> Exception happened during processing of request from ('127.0.0.1', 38224)
>
> It is also described in:
> https://www.reddit.com/r/joinmarket/comments/4atqrm/is_this_exception_normal_exception_happened/
>
> On https://bugs.python.org/issue14574 is stated that this error should
> already be fixed but apparently not.
>
> Best regards,
>
> Roman
Thank you Justin,
I'm on the dev server and should present results in this way.
Yes, I use manage.py runserver --insecure to start the server (from PyCharm).
My views.py call:
@detail_route(methods=['post'], permission_classes=[permissions.AllowAny])
def export_report(request):
body_unicode = request.body.decode('utf-8')
body_str = body_unicode.encode('ascii','ignore')
attr_list = body_str.split('&')
attr_dict = {}
if (len(attr_list) > 0):
for attr in attr_list:
...
key = key_value_pair[0]
attr_dict[key] = key_value_pair[1]
trend = trends.calculate_trend(
attr_dict['search_phrase']
, attr_dict['time_from']
, attr_dict['time_to']
, attr_dict['time_scale']
)
attr_dict['trend'] = trend
res = str(json.dumps(attr_dict))
return HttpResponse(res, content_type="text/plain; charset=utf-8")
and trend calculation in trends.py with database calls:
def calculate_trend(query_phrase, time_from, time_to, time_scale):
# check in database if trend already exists
try:
db_trend = Trend.objects.get(pk=query_phrase)
if db_trend.from_time.strftime("%Y-%m-%d") == time_from \
and db_trend.to_time.strftime("%Y-%m-%d") == time_to \
and db_trend.granularity == time_scale:
logger.info("trend already exists.")
existing_trend_dict = ast.literal_eval(db_trend.content)
return existing_trend_dict
except Trend.DoesNotExist:
logger.info("It is a new trend search.")
trend_dict = {}
start_time = pd.Timestamp(value[0])
end_time = pd.Timestamp(value[-1])
freq = ... get frequency using pandas lib
trend_dict[key] = freq
json_trend_content = trend_dict_to_sorted_json_str(trend_dict)
trend = Trend(
phrase=query_phrase,
content=json_trend_content,
from_time=time_from,
to_time=time_to,
granularity=time_scale,
)
if trend is not None:
try:
db_trend = Trend.objects.get(pk=query_phrase)
db_trend.delete()
logger.info("delete old trend: %s. " % trend)
except Trend.DoesNotExist:
logger.info("create trend: %s. " % trend)
trend.save()
return trend_dict
Thank you in advance!
Roman
--
https://mail.python.org/mailman/listinfo/python-list
Re: python 2.7.12 on Linux behaving differently than on Windows
Steven D'Aprano wrote: > Tell me Bart, what's the difference between these? > > # Linux > rm * > > and > > # Windows > del * Is there an equivalent to # touch -- -r on Windows? Like in $ tree . |-- -r |-- a | `-- b | `-- c | `-- bar.txt `-- foo.txt 3 directories, 3 files $ rm * $ tree . `-- -r 0 directories, 1 file $ -- https://mail.python.org/mailman/listinfo/python-list
Re: python 2.7.12 on Linux behaving differently than on Windows
On Wednesday 07 December 2016 03:41, BartC wrote: > On 06/12/2016 15:44, Michael Torrie wrote: >> On 12/06/2016 04:43 AM, BartC wrote: > >> Yes shell expansion has it's gotchas. But those can all be learned, > > Yes, learn to avoid wildcards in command parameters at all costs. But we > both know that is not satisfactory. Now you're being ridiculous. Can't you at least debate this sensibly? There are Windows programs which understand globs, like dir. Do you honestly expect us to believe that it is okay to use "dir a*" on Windows because dir itself expands the glob, but not okay to use "dir a*" on Linux because the shell has expanded the glob before dir sees it? (Yes, there's a Linux dir. It appears to be almost identical to ls.) > And it's not much help when someone types in: > > program * Indeed. Tell me Bart, what's the difference between these? # Linux rm * and # Windows del * If you accidentally say "del *" on Windows, say, when you meant "z*" for example, and Windows deletes all the files in the current directory, do you say to yourself...? "Thank the deities that * was expanded by the del command rather than the shell! Imagine how awful it would have been if the shell did the expansion!" > and the program has to try and clean up the mess, if it can see it is a > mess. But remember: > >cp *.c > > There might be some irate users out there if it can't detect a simple > user error like that. *shrug* You can blow your foot off with any sufficiently powerful set of tools, if you're not paying attention. I see that the Windows version of copy avoids this *by accident*: C:\Documents and Settings\Steve>dir a* Volume in drive C has no label. Volume Serial Number is 9CAB-3B4A Directory of C:\Documents and Settings\Steve 07/12/2016 06:05 PM 4 a1 07/12/2016 06:05 PM 4 a2 2 File(s) 8 bytes 0 Dir(s) 1,672,613,888 bytes free C:\Documents and Settings\Steve>copy a* a1 The file cannot be copied onto itself. 0 file(s) copied. Instead of accidentally copying a1 over a2, it accidentally tries to copy a1 over a1, and fails. At least, that's what it looks like to me. Any Windows expert like to chime in? The bottom line is, it doesn't matter where the wildcards are expanded. Whether the shell does it, or the application itself, they're dangerous by their very nature. You're asking the computer to operate on a set of files you specify using fuzzy matching. It is easy to match too much, or too little, if you aren't careful. >> whereas it's much harder to learn and remember all the gotchas and bugs >> of many individual applications' unique ways of dealing with globs. I'd >> rather deal with shells. >> > > OK, I understand. Having a program launcher indiscriminately transform > input A into A' is /much/ better than having it done by the program, For the use-cases that guided the design and evolution of Unix, ABSOLUTELY. That doesn't mean that the same will apply to all uses of the computer command line. But it does apply to the great majority of uses by sys admins on Unix systems. They designed the system to work the way they want. I get that its not the way that you like, but fortunately for you, they provided not one, but TWO easy ways to get the results you want: - escaping the metacharacters you want to be left untouched; - disabling globbing (for either a single command, a session, or permanently). If *your* design was paramount, it would be up to each and every application to choose whether or not to support globbing and environment variable expansion, and hard luck if the application I needed to use didn't understand the globs I wanted. > even if the program doesn't want it and it is not meaningful. *shrug* That's the cost you pay. It's not a big cost, since it costs as little as a single character (backslash) to escape something, but it is a cost. > The fact that you also lose format information (is it three parameters, > or one parameter transformed into three) is an extra bonus. Here's a two-line Python program to prove what nonsense that is. steve@runes:~$ cat argv.py import sys print( sys.argv ) And in use: steve@runes:~$ python argv.py a\* z\* ['argv.py', 'a*', 'z*'] > This is clearly much better than any other scheme because: > > (1) It's Unix not Windows; everything in Unix is always better and > always make sense. [snarky, sarcastic answer to match the tone of your post] Everything except the tar command. > (2) There have been 40 years of it working this way and there have never > been any problems. (That is, 40 years of writing programs with > stultified command line interfaces to make sure that is always the case. > [As for 'problems' such as the 'cp *.c' one, that's a feature!] Don't be absurd. Any Unix sys admin can tell you stories of accidentally globbing too much and screwing something up. And any Wi
Re: Detect Linux Runlevel
On Wednesday 07 December 2016 18:08, Marko Rauhamaa wrote: >This is a problem with advisory locking. The fact that anyone can >create an exclusive lock on a file they can only read! Is this >behavior appropriate? > >https://www.redhat.com/archives/linux-security/1996-Novembe >r/msg00026.html> Whereas if it were mandatory locking, enforced by the OS, it wouldn't be a problem? Here's that URL without the indent and word-wrapping: https://www.redhat.com/archives/linux-security/1996-November/msg00026.html -- Steven "Ever since I learned about confirmation bias, I've been seeing it everywhere." - Jon Ronson -- https://mail.python.org/mailman/listinfo/python-list
