[issue40885] Cannot pipe GzipFile into subprocess

[Michael Herrmann]

Michael Herrmann  added the comment:

I just encountered what seems to be the inverse problem of this issue: #45585

--
nosy: +mherrmann.at

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40885] Cannot pipe GzipFile into subprocess

[Nehal Patel]

Nehal Patel  added the comment:

In my use case, I was actually trying to stream a large gzip file from the 
cloud directly into subprocess without spilling onto disk or RAM i.e. the code 
actually  looked something more like:

r, w = os.pipe()
# ... launch a thread to feed r
with gzip.open(os.fdopen(w, 'rb')) as gz:
res = subprocess.run("myexe", stdin=gz, capture_output=True)
## fyi, expected output is tiny
 
(In my case, I could modify the executable to expect compressed input, so I 
chose that solution.  Another possibility would have been to use 
subprocess.POpen twice, once with  'gzcat' and second with 'myexe')

I agree that given how libgz works, it would be difficult to fix the  problem.  
I would suggest finding a way to alert the user about this issue because it 
will in general be a very confusing situation when this happens.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40885] Cannot pipe GzipFile into subprocess

[SilentGhost]

SilentGhost  added the comment:

> subprocess somehow gets a hold of the underlying file descriptor pointing to 
> the compressed file, and ends up being fed the compressed bytes

That is exactly what happens, and I'd wager this is not going to change. You 
could easily pass the decoded bytes into the process using input parameter.

--
components: +Library (Lib)
nosy: +SilentGhost, giampaolo.rodola

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40885] Cannot pipe GzipFile into subprocess

[Nehal Patel]

New submission from Nehal Patel :

The following code produces incorrect behavior:

with gzip.open("foo.gz") as gz:
res = subprocess.run("cat", stdin=gz, capture_output=True)

the contents of res.stdout are identical to the contents of "foo.gz" 

It seems the subprocess somehow gets a hold of the underlying file descriptor 
pointing to the compressed file, and ends up being fed the compressed bytes.

--
components: IO
messages: 370804
nosy: Nehal Patel
priority: normal
severity: normal
status: open
title: Cannot pipe GzipFile into subprocess
type: behavior
versions: Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com