Re: [Tutor] subprocess.getstatusoutput : UnicodeDecodeError
>
> getstatusoutput is a "legacy" function. It still exists for code that
> has already been using it, but it is not recommended for new code.
>
> https://docs.python.org/3.5/library/subprocess.html#using-the-subprocess-module
>
> Since you're using Python 3.5, let's try using the brand new `run`
> function and see if it does better:
>
> import subprocess
> result = subprocess.run(["tail", "-3", "/tmp/pmaster.db"],
> stdout=subprocess.PIPE)
> print("return code is", result.returncode)
> print("output is", result.stdout)
>
>
> It should do better than getstatusoutput, since it returns plain bytes
> without assuming they are ASCII. You can then decode them yourself:
>
> # try this and see if it is sensible
> print("output is", result.stdout.decode('latin1'))
>
> # otherwise this
> print("output is", result.stdout.decode('utf-8', errors='replace'))
>
>
>
>> >>> subprocess.getstatusoutput("tail -3 /tmp/pmaster.db",)
>> Traceback (most recent call last):
> [...]
>> File "/usr/lib/python3.5/encodings/ascii.py", line 26, in decode
>> return codecs.ascii_decode(input, self.errors)[0]
>> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe0 in position 189:
>> ordinal not in range(128)
>
> Let's look at the error message. getstatusoutput apparently expects only
> pure ASCII output, because it is choking on a non-ASCII byte, namely
> 0xe0. Obviously 0xe0 (or in decimal, 224) is not an ASCII value, since
> ASCII goes from 0 to 127 only.
>
> If there's one non-ASCII byte in the file, there are probably more.
>
> So what is that mystery 0xe0 byte? It is hard to be sure, because it
> depends on the source. If pmaster.db is a binary file, it could mean
> anything or nothing. If it is a text file, it depends on the encoding
> that the file uses. If it comes from a Mac, it might be:
>
> py> b'\xe0'.decode('macroman')
> '‡'
>
> If it comes from Windows in Western Europe, it might be:
>
> py> b'\xe0'.decode('latin1')
> 'à'
>
> If it comes from Windows in Greece, it might be:
>
> py> b'\xe0'.decode('iso 8859-7')
> 'ΰ'
>
> and so forth. There's no absolutely reliable way to tell. This is the
> sort of nightmare that Unicode was invented to fix, but unfortunately
> there still exist millions of files, data formats and applications which
> insist on using rubbish "extended ASCII" encodings instead.
>
>
>
>> That file's content is kryptonite for python apparently. Other shell
>> operations work.
>>
>> >>> subprocess.getstatusoutput("file /tmp/pmaster.db",)
>> (0, '/tmp/pmaster.db: Non-ISO extended-ASCII text, with very long lines,
>> with LF, NEL line terminators')
>
> The `file` command agrees with me: it is not ASCII.
Thank you Steve! subprocess.run handles it better.
>>> subprocess.getstatusoutput("tail -400 /tmp/pmaster.txt",)
Traceback (most recent call last):
File "", line 1, in
File "/usr/lib/python3.5/subprocess.py", line 805, in getstatusoutput
data = check_output(cmd, shell=True, universal_newlines=True, stderr=STDOUT)
File "/usr/lib/python3.5/subprocess.py", line 626, in check_output
**kwargs).stdout
File "/usr/lib/python3.5/subprocess.py", line 695, in run
stdout, stderr = process.communicate(input, timeout=timeout)
File "/usr/lib/python3.5/subprocess.py", line 1059, in communicate
stdout = self.stdout.read()
File "/usr/lib/python3.5/codecs.py", line 321, in decode
(result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe0 in position
60942: invalid continuation byte
as opposed to:
>>> result = subprocess.run(["tail", "-400", "/tmp/pmaster.txt"],
>>> stdout=subprocess.PIPE)
>>> result.returncode
0
>>> subprocess.getstatusoutput("file /tmp/pmaster.txt",)
(0, '/tmp/pmaster.txt: Non-ISO extended-ASCII text, with very long
lines, with LF, NEL line terminators')
>>>
That was awesome! :)
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] subprocess.getstatusoutput : UnicodeDecodeError
On Thu, Sep 21, 2017 at 03:46:29PM -0700, Evuraan wrote:
> How can I work around this issue where subprocess.getstatusoutput gives
> up, on Python 3.5.2:
getstatusoutput is a "legacy" function. It still exists for code that
has already been using it, but it is not recommended for new code.
https://docs.python.org/3.5/library/subprocess.html#using-the-subprocess-module
Since you're using Python 3.5, let's try using the brand new `run`
function and see if it does better:
import subprocess
result = subprocess.run(["tail", "-3", "/tmp/pmaster.db"],
stdout=subprocess.PIPE)
print("return code is", result.returncode)
print("output is", result.stdout)
It should do better than getstatusoutput, since it returns plain bytes
without assuming they are ASCII. You can then decode them yourself:
# try this and see if it is sensible
print("output is", result.stdout.decode('latin1'))
# otherwise this
print("output is", result.stdout.decode('utf-8', errors='replace'))
> >>> subprocess.getstatusoutput("tail -3 /tmp/pmaster.db",)
> Traceback (most recent call last):
[...]
> File "/usr/lib/python3.5/encodings/ascii.py", line 26, in decode
> return codecs.ascii_decode(input, self.errors)[0]
> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe0 in position 189:
> ordinal not in range(128)
Let's look at the error message. getstatusoutput apparently expects only
pure ASCII output, because it is choking on a non-ASCII byte, namely
0xe0. Obviously 0xe0 (or in decimal, 224) is not an ASCII value, since
ASCII goes from 0 to 127 only.
If there's one non-ASCII byte in the file, there are probably more.
So what is that mystery 0xe0 byte? It is hard to be sure, because it
depends on the source. If pmaster.db is a binary file, it could mean
anything or nothing. If it is a text file, it depends on the encoding
that the file uses. If it comes from a Mac, it might be:
py> b'\xe0'.decode('macroman')
'‡'
If it comes from Windows in Western Europe, it might be:
py> b'\xe0'.decode('latin1')
'à'
If it comes from Windows in Greece, it might be:
py> b'\xe0'.decode('iso 8859-7')
'ΰ'
and so forth. There's no absolutely reliable way to tell. This is the
sort of nightmare that Unicode was invented to fix, but unfortunately
there still exist millions of files, data formats and applications which
insist on using rubbish "extended ASCII" encodings instead.
> That file's content is kryptonite for python apparently. Other shell
> operations work.
>
> >>> subprocess.getstatusoutput("file /tmp/pmaster.db",)
> (0, '/tmp/pmaster.db: Non-ISO extended-ASCII text, with very long lines,
> with LF, NEL line terminators')
The `file` command agrees with me: it is not ASCII.
--
Steve
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] subprocess.getstatusoutput : UnicodeDecodeError
>
> But: do you really want to "tail" what's probably not really a plaintext
> file? Just guessing, but the .db as well as the error msgs are a hint.
although the filename ends with a ".db", it is just a text file. not
tailing a SQLite or a binary file, just happened to name it so.
I work around the same sort elsewhere thusly:
with open("/tmp/pmaster.db",encoding="utf-8",errors="ignore") as fobj:
or even with codecs
with codecs.open("/tmp/pmaster.db",encoding="utf-8",errors="ignore") as fobj:
I think I've to follow suit here, instead of tail, apparently there's
no "errors=ignore" for subprocess.
I was hoping subprocess would be impervious to decoding errors, as the
text is coming from tail.
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] subprocess.getstatusoutput : UnicodeDecodeError
On 09/21/2017 04:46 PM, Evuraan wrote: > Greetings! > > My search-fu failed me, so thought of finally asking this question here. > > > How can I work around this issue where subprocess.getstatusoutput gives > up, on Python 3.5.2: You picked a fun one! First off, subprocess.getstatusoutput is a dubious API, because if you call a command on a UNIX/Linux flavored system, it has the possibility of writing to both the standard output and standard error streams, but this API cannot separate them. I remember fighting this in something years and years ago. Second, this thing has changed even in the Python 3.x series, which is a bad sign (see docs): first it didn't exist, then it was put back, then the first element of the return tuple changed. Third, according to the docs, "none of the guarantees described above regarding security and exception handling consistency are valid for these functions" (referring to "legacy" getstatusoutput and getoutput) Probably - don't use it. You have your main hint in the error messages: "Non-ISO extended-ASCII text" along with failures claiming a problem decoding something that is not in the range of standard ASCII: 'ascii' codec can't decode byte 0xe0 in position 189: ordinal not in range(128)' Maybe try using subprocess.check_output, although it's not a direct replacement. But: do you really want to "tail" what's probably not really a plaintext file? Just guessing, but the .db as well as the error msgs are a hint. You'll need to fill us in more on what you want to accomplish. ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
[Tutor] subprocess.getstatusoutput : UnicodeDecodeError
Greetings!
My search-fu failed me, so thought of finally asking this question here.
How can I work around this issue where subprocess.getstatusoutput gives
up, on Python 3.5.2:
>>> subprocess.getstatusoutput("tail -3 /tmp/pmaster.db",)
Traceback (most recent call last):
File "", line 1, in
File "/usr/lib/python3.5/subprocess.py", line 805, in getstatusoutput
data = check_output(cmd, shell=True, universal_newlines=True,
stderr=STDOUT)
File "/usr/lib/python3.5/subprocess.py", line 626, in check_output
**kwargs).stdout
File "/usr/lib/python3.5/subprocess.py", line 695, in run
stdout, stderr = process.communicate(input, timeout=timeout)
File "/usr/lib/python3.5/subprocess.py", line 1059, in communicate
stdout = self.stdout.read()
File "/usr/lib/python3.5/encodings/ascii.py", line 26, in decode
return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe0 in position 189:
ordinal not in range(128)
>>>
That file's content is kryptonite for python apparently. Other shell
operations work.
>>> subprocess.getstatusoutput("file /tmp/pmaster.db",)
(0, '/tmp/pmaster.db: Non-ISO extended-ASCII text, with very long lines,
with LF, NEL line terminators')
>>> subprocess.getstatusoutput("wc -l /tmp/pmaster.db",)
(0, '155065 /tmp/pmaster.db')
>>> subprocess.getstatusoutput("tail -3 /tmp/pmaster.db",)
Thanks in advance for your time and advice!
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] running or calling a module from the desktop
On 20/09/17 22:06, orpha.pen...@outlook.com wrote: > I would like to import, run or set path to desktop. OK, On Windows go to Windows Exploder and type in the address %HOMEPATH%\Desktop Go there and you should be able to see/figure out the full path to your personal desktop folder. Then add that path to your PYTHONPATH environment variable. Python will then look in your desktop folder for modules you try to import. > So that I can use things there were I usually work > on new items, I know how > the rest of you are different from that. Yes because your desktop will get awfully messy very quickly once you start working on bigger projects with lots of files. Its much better to create a folder for your code. But its your PC, you do what you want! :-) > I don't know if you could use the cloud as a > means to import items? I don;t think so, not yet anyway. > HOW would I explain to python-chevron I assume you mean the Python >>> prompt? If not you'll need to explain what Pyhon chevron is. If you do mean >>> then the above addition to PYTHONPATH should do the trick. > Iron python capable, whatever that means. Its the version of Python that runs on .NET. I've never used it but I think it pincludes a compiler that can generate .NET code that can be used with C#, VB.Net etc. > the first time I heard SQL as an acronym. I thought > they said Server-Query-Language. If they did, they were wrong. SQL has always stood for Structured Query Language. It has nothing to do with servers. -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ http://www.amazon.com/author/alan_gauld Follow my photo-blog on Flickr at: http://www.flickr.com/photos/alangauldphotos ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
[Tutor] running or calling a module from the desktop
Mmkay. I know that you know how to do this and I would appreciate it if you would tell me. Please. I would like to import, run or set path to desktop. So that I can use things there were I usually work on new items, I know how the rest of you are different from that. (I'm going to stop joking around right here, hello, I am a new user.) I'm Jason. I don't know if you could use the cloud as a means to import items? Sounds like more trouble than its worth at first... But seriously, If I have a module on my desktop, HOW would I explain to python-chevron that I would like to use this (resource, file, module, extension from it's current location?) I began typing importlib and then realized I have no-idea how to tell python what a path is.. other than the first page of the on-line tutorial. I'm 3.6Python, on Win10Pro64. Umm, I do not yet know, but Iron python capable, whatever that means. Honestly I think it has to do with VB which, they don't call that that, and forms, or something. Just another quick question, the first time I heard SQL as an acronym. I thought they said Server-Query-Language. Does anyone else remember it having been called that? I'm pretty sure a teacher said that. Was it a slip of the tongue? Possibly a thorough length of time ago in human years. Please do not forget to address my main question. (I can't help it, I haven't had company in a while.) I'm trying to call a user created Llist.py into python IDLE while using py-prompt, not interested in URL's but interested only on system files most notably those on the desktop. ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
