Re: tarfile : read from a socket?

2016-02-12 Thread Antoon Pardon 
On 02/11/2016 06:27 PM, Lars Gustäbel wrote:
> On Thu, Feb 11, 2016 at 04:41:43PM +, Ulli Horlacher wrote:
>>   sfo = sock.makefile('r')
>>   taro = tarfile.open(fileobj=sfo,mode='r|')
>>   taro.extractall(path=edir)
> What about using an iterator?
>
> def myiter(tar):
> for t in tar:
> print "extracting", t.name
> yield t
>
> sfo = sock.makefile('r')
> taro = tarfile.open(fileobj=sfo,mode='r|')
> taro.extractall(members=myiter(taro),path=edir)
>
> Cheers,

The tarfile is already an iterator. Just do the following: for ti in
taro: print "extracting", ti.name taro.extract(ti)


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: tarfile : read from a socket?

2016-02-12 Thread Lars Gustäbel 
On Fri, Feb 12, 2016 at 09:35:40AM +0100, Antoon Pardon wrote:
> On 02/11/2016 06:27 PM, Lars Gustäbel wrote:
> > What about using an iterator?
> >
> > def myiter(tar):
> > for t in tar:
> > print "extracting", t.name
> > yield t
> >
> > sfo = sock.makefile('r')
> > taro = tarfile.open(fileobj=sfo,mode='r|')
> > taro.extractall(members=myiter(taro),path=edir)
> >
> The tarfile is already an iterator. Just do the following: for ti in
> taro: print "extracting", ti.name taro.extract(ti)

The extractall() method does a little bit more than just extract(), i.e.
setting directory mtimes, see
https://docs.python.org/3.5/library/tarfile.html#tarfile.TarFile.extractall

-- 
Lars Gustäbel
l...@gustaebel.de
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: tarfile : read from a socket?

2016-02-12 Thread Ulli Horlacher 
Lars Gustäbel  wrote:
> On Fri, Feb 12, 2016 at 09:35:40AM +0100, Antoon Pardon wrote:
> > On 02/11/2016 06:27 PM, Lars Gustäbel wrote:
> > > What about using an iterator?
> > >
> > > def myiter(tar):
> > > for t in tar:
> > > print "extracting", t.name
> > > yield t
> > >
> > > sfo = sock.makefile('r')
> > > taro = tarfile.open(fileobj=sfo,mode='r|')
> > > taro.extractall(members=myiter(taro),path=edir)
> > >
> > The tarfile is already an iterator. Just do the following: for ti in
> > taro: print "extracting", ti.name taro.extract(ti)
> 
> The extractall() method does a little bit more than just extract(), i.e.
> setting directory mtimes, see
> https://docs.python.org/3.5/library/tarfile.html#tarfile.TarFile.extractall

This is an important hint!
Thanks!


-- 
Ullrich Horlacher  Server und Virtualisierung
Rechenzentrum IZUS/TIK E-Mail: horlac...@tik.uni-stuttgart.de
Universitaet Stuttgart Tel:++49-711-68565868
Allmandring 30aFax:++49-711-682357
70550 Stuttgart (Germany)  WWW:http://www.tik.uni-stuttgart.de/
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: modifying a standard module? (was: Re: tarfile : read from a socket?)

2016-02-12 Thread Matt Wheeler 
On 11 February 2016 at 17:10, Ulli Horlacher
 wrote:
>
> Ulli Horlacher  wrote:
> As a hack, I modified the standard library module tarfile.py:
>
> root@diaspora:/usr/lib/python2.7# vv -d
> --- ./.versions/tarfile.py~1~   2015-06-22 21:59:27.0 +0200
> +++ tarfile.py  2016-02-11 18:01:50.18952 +0100
> @@ -2045,6 +2045,7 @@
>  directories.append(tarinfo)
>  tarinfo = copy.copy(tarinfo)
>  tarinfo.mode = 0700
> +print('untar "%s"' % tarinfo.name)
>  self.extract(tarinfo, path)
>
>  # Reverse sort directories.
>
>
> This gives me exact the output I want :-)
>
> BUT I want to distribute my program and all others will not see the tar
> extracting information.
>
> Now my question:
>
> How can I substitute the standard module function tarfile.extractall() with
> my own function?

import tarfile
def new_extractall(self, *args, **kwargs):
print("I am a function. Woohoo!")

tarfile.TarFile.extractall = new_extractall

But bear in mind that that will change tarfile.extractall for every
single module that imports it within the same python process. Is that
really what you want?


Is there a reason you can't subclass TarFile as others have suggested?

Perhaps even this is enough:

class NoisyTarFile(TarFile):
"""untested, sorry"""
def extract(self, member, *args, **kwargs):
print('extracting "%s"' % member.name)
super(NoisyTarFile, self).extract(member, *args, **kwargs)

As the very next step after your print in extractall is a call to
extract anyway?


If you must patch the standard library tarfile module then I would
suggest patching it to have an extra, default False, argument to
enable your printing behaviour, so you don't risk messing up anyone
else's use of it.


-- 
Matt Wheeler
http://funkyh.at
-- 
https://mail.python.org/mailman/listinfo/python-list

tarfile : read from a socket?

2016-02-11 Thread Ulli Horlacher 
https://docs.python.org/2/library/tarfile.html says:

 tarfile.open(name=None, mode='r', fileobj=None, bufsize=10240, **kwargs)

Return a TarFile object for the pathname name.


(How) can I read a tar file from a (tcp) socket?
I do not have a pathname but a socket object from socket.create_connection() 



-- 
Ullrich Horlacher  Server und Virtualisierung
Rechenzentrum IZUS/TIK E-Mail: horlac...@tik.uni-stuttgart.de
Universitaet Stuttgart Tel:++49-711-68565868
Allmandring 30aFax:++49-711-682357
70550 Stuttgart (Germany)  WWW:http://www.tik.uni-stuttgart.de/
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: tarfile : read from a socket?

2016-02-11 Thread Ulli Horlacher 
Antoon Pardon  wrote:

> > (How) can I read a tar file from a (tcp) socket?
> > I do not have a pathname but a socket object from socket.create_connection
> 
> # First you construct a file object with makefile.
> 
> fo = socket.makefile()
> 
> # Then you use the fileobj argument with tarfile.open.
> 
> tarfile.open(mode='r', fileobj = fo)


I have:

  sock = socket.create_connection((server,port))
  bs = kB64
  taro = tarfile.open(fileobj=sock.makefile('w',kB64),mode='w')
  
  

Traceback (most recent call last):
  (...)
  File "./fexit.py", line 1838, in sex_send
taro = tarfile.open(fileobj=sock.makefile('w',kB64),mode='w')
  File "/usr/lib/python2.7/tarfile.py", line 1695, in open
return cls.taropen(name, mode, fileobj, **kwargs)
  File "/usr/lib/python2.7/tarfile.py", line 1705, in taropen
return cls(name, mode, fileobj, **kwargs)
  File "/usr/lib/python2.7/tarfile.py", line 1566, in __init__
self.offset = self.fileobj.tell()
AttributeError: '_fileobject' object has no attribute 'tell'

-- 
Ullrich Horlacher  Server und Virtualisierung
Rechenzentrum IZUS/TIK E-Mail: horlac...@tik.uni-stuttgart.de
Universitaet Stuttgart Tel:++49-711-68565868
Allmandring 30aFax:++49-711-682357
70550 Stuttgart (Germany)  WWW:http://www.tik.uni-stuttgart.de/
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: tarfile : read from a socket?

2016-02-11 Thread INADA Naoki 
Ｈave you tried socket.makefile() method?
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: tarfile : read from a socket?

2016-02-11 Thread Antoon Pardon 
On 02/11/2016 09:31 AM, Ulli Horlacher wrote:
> https://docs.python.org/2/library/tarfile.html says:
>
>  tarfile.open(name=None, mode='r', fileobj=None, bufsize=10240, **kwargs)
>
> Return a TarFile object for the pathname name.
>
>
> (How) can I read a tar file from a (tcp) socket?
> I do not have a pathname but a socket object from socket.create_connection

# First you construct a file object with makefile.

fo = socket.makefile()

# Then you use the fileobj argument with tarfile.open.

tarfile.open(mode='r', fileobj = fo)

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: tarfile : read from a socket?

2016-02-11 Thread Chris Angelico 
On Thu, Feb 11, 2016 at 11:53 PM, Ulli Horlacher
 wrote:
> I have:
>
>   sock = socket.create_connection((server,port))
>   bs = kB64
>   taro = tarfile.open(fileobj=sock.makefile('w',kB64),mode='w')
>
>
>
> Traceback (most recent call last):
>   (...)
>   File "./fexit.py", line 1838, in sex_send
> taro = tarfile.open(fileobj=sock.makefile('w',kB64),mode='w')
>   File "/usr/lib/python2.7/tarfile.py", line 1695, in open
> return cls.taropen(name, mode, fileobj, **kwargs)
>   File "/usr/lib/python2.7/tarfile.py", line 1705, in taropen
> return cls(name, mode, fileobj, **kwargs)
>   File "/usr/lib/python2.7/tarfile.py", line 1566, in __init__
> self.offset = self.fileobj.tell()
> AttributeError: '_fileobject' object has no attribute 'tell'

Sounds like tarfile needs a seekable file. How big is this file you're
reading? Can you simply read the whole thing into memory, then use
io.BytesIO? I had a quick glance at help(BytesIO) but didn't find a
simple way to make a buffer that reads from an upstream file when it
needs more content, but it should be possible to build one.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: tarfile : read from a socket?

2016-02-11 Thread MRAB 

On 2016-02-11 12:53, Ulli Horlacher wrote:

Antoon Pardon  wrote:


> (How) can I read a tar file from a (tcp) socket?
> I do not have a pathname but a socket object from socket.create_connection

# First you construct a file object with makefile.

fo = socket.makefile()

# Then you use the fileobj argument with tarfile.open.

tarfile.open(mode='r', fileobj = fo)



I have:

   sock = socket.create_connection((server,port))
   bs = kB64
   taro = tarfile.open(fileobj=sock.makefile('w',kB64),mode='w')



Traceback (most recent call last):
   (...)
   File "./fexit.py", line 1838, in sex_send
 taro = tarfile.open(fileobj=sock.makefile('w',kB64),mode='w')
   File "/usr/lib/python2.7/tarfile.py", line 1695, in open
 return cls.taropen(name, mode, fileobj, **kwargs)
   File "/usr/lib/python2.7/tarfile.py", line 1705, in taropen
 return cls(name, mode, fileobj, **kwargs)
   File "/usr/lib/python2.7/tarfile.py", line 1566, in __init__
 self.offset = self.fileobj.tell()
AttributeError: '_fileobject' object has no attribute 'tell'

I suppose you could write your own class to wrap the socket and provide 
the required methods.


--
https://mail.python.org/mailman/listinfo/python-list

Re: tarfile : read from a socket?

2016-02-11 Thread Ulli Horlacher 
Chris Angelico  wrote:

> Sounds like tarfile needs a seekable file. How big is this file you're
> reading?

No limits. It can be many TBs...

The use case is:

http://fex.rus.uni-stuttgart.de:8080/


-- 
Ullrich Horlacher  Server und Virtualisierung
Rechenzentrum IZUS/TIK E-Mail: horlac...@tik.uni-stuttgart.de
Universitaet Stuttgart Tel:++49-711-68565868
Allmandring 30aFax:++49-711-682357
70550 Stuttgart (Germany)  WWW:http://www.tik.uni-stuttgart.de/
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: tarfile : read from a socket?

2016-02-11 Thread Ulli Horlacher 
Ulli Horlacher  wrote:

> I have:
> 
>   sock = socket.create_connection((server,port))
>   bs = kB64
>   taro = tarfile.open(fileobj=sock.makefile('w',kB64),mode='w')
>   
>   
> 
> Traceback (most recent call last):
>   (...)
>   File "./fexit.py", line 1838, in sex_send
> taro = tarfile.open(fileobj=sock.makefile('w',kB64),mode='w')
>   File "/usr/lib/python2.7/tarfile.py", line 1695, in open
> return cls.taropen(name, mode, fileobj, **kwargs)
>   File "/usr/lib/python2.7/tarfile.py", line 1705, in taropen
> return cls(name, mode, fileobj, **kwargs)
>   File "/usr/lib/python2.7/tarfile.py", line 1566, in __init__
> self.offset = self.fileobj.tell()
> AttributeError: '_fileobject' object has no attribute 'tell'

Reading the doc helps :-)

https://docs.python.org/2/library/tarfile.html

  For special purposes, there is a second format for mode:
  'filemode|[compression]'. tarfile.open() will return a TarFile object
  that processes its data as a stream of blocks.

With 

  taro = tarfile.open(fileobj=sock.makefile('w',kB64),mode='w|')

I get no more error.


-- 
Ullrich Horlacher  Server und Virtualisierung
Rechenzentrum IZUS/TIK E-Mail: horlac...@tik.uni-stuttgart.de
Universitaet Stuttgart Tel:++49-711-68565868
Allmandring 30aFax:++49-711-682357
70550 Stuttgart (Germany)  WWW:http://www.tik.uni-stuttgart.de/
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: tarfile : read from a socket?

2016-02-11 Thread Random832 
On Thu, Feb 11, 2016, at 11:41, Ulli Horlacher wrote:
> When I use:
> 
>   for member in taro.getmembers():
>   print('extracting "%s"' % member.name)
>   taro.extract(member)
> 
> I get the error:
> 
>   File "/usr/lib/python2.7/tarfile.py", line 556, in seek
> raise StreamError("seeking backwards is not allowed")
> 
> Of course, a stream is not seekable.
> 
> Any ideas?

Try this:

while True:
member = taro.next()
if member is None: break
print('extracting "%s"' % member.name)
taro.extract(member)
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: tarfile : read from a socket?

2016-02-11 Thread Ulli Horlacher 
Ulli Horlacher  wrote:

> With 
> 
>   taro = tarfile.open(fileobj=sock.makefile('w',kB64),mode='w|')
> 
> I get no more error.

Of course, this is the writing client.

Now I have a small problem with the reading client.

This code works so far:

  sfo = sock.makefile('r')
  taro = tarfile.open(fileobj=sfo,mode='r|')
  taro.extractall(path=edir)

But it does not writes anything to the terminal to inform the user.

When I use:

  for member in taro.getmembers():
  print('extracting "%s"' % member.name)
  taro.extract(member)

I get the error:

  File "/usr/lib/python2.7/tarfile.py", line 556, in seek
raise StreamError("seeking backwards is not allowed")

Of course, a stream is not seekable.

Any ideas?

-- 
Ullrich Horlacher  Server und Virtualisierung
Rechenzentrum IZUS/TIK E-Mail: horlac...@tik.uni-stuttgart.de
Universitaet Stuttgart Tel:++49-711-68565868
Allmandring 30aFax:++49-711-682357
70550 Stuttgart (Germany)  WWW:http://www.tik.uni-stuttgart.de/
-- 
https://mail.python.org/mailman/listinfo/python-list

modifying a standard module? (was: Re: tarfile : read from a socket?)

2016-02-11 Thread Ulli Horlacher 

Ulli Horlacher  wrote:

> This code works so far:
> 
>   sfo = sock.makefile('r')
>   taro = tarfile.open(fileobj=sfo,mode='r|')
>   taro.extractall(path=edir)
> 
> But it does not writes anything to the terminal to inform the user.
> 
> When I use:
> 
>   for member in taro.getmembers():
>   print('extracting "%s"' % member.name)
>   taro.extract(member)
> 
> I get the error:
> 
>   File "/usr/lib/python2.7/tarfile.py", line 556, in seek
> raise StreamError("seeking backwards is not allowed")
> 
> Of course, a stream is not seekable.
> 
> Any ideas?

As a hack, I modified the standard library module tarfile.py:

root@diaspora:/usr/lib/python2.7# vv -d
--- ./.versions/tarfile.py~1~   2015-06-22 21:59:27.0 +0200
+++ tarfile.py  2016-02-11 18:01:50.18952 +0100
@@ -2045,6 +2045,7 @@
 directories.append(tarinfo)
 tarinfo = copy.copy(tarinfo)
 tarinfo.mode = 0700
+print('untar "%s"' % tarinfo.name)
 self.extract(tarinfo, path)
 
 # Reverse sort directories.


This gives me exact the output I want :-)

BUT I want to distribute my program and all others will not see the tar
extracting information.

Now my question:

How can I substitute the standard module function tarfile.extractall() with
my own function?



-- 
Ullrich Horlacher  Server und Virtualisierung
Rechenzentrum IZUS/TIK E-Mail: horlac...@tik.uni-stuttgart.de
Universitaet Stuttgart Tel:++49-711-68565868
Allmandring 30aFax:++49-711-682357
70550 Stuttgart (Germany)  WWW:http://www.tik.uni-stuttgart.de/
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: tarfile : read from a socket?

2016-02-11 Thread MRAB 

On 2016-02-11 16:41, Ulli Horlacher wrote:

Ulli Horlacher  wrote:


With

  taro = tarfile.open(fileobj=sock.makefile('w',kB64),mode='w|')

I get no more error.


Of course, this is the writing client.

Now I have a small problem with the reading client.

This code works so far:

   sfo = sock.makefile('r')
   taro = tarfile.open(fileobj=sfo,mode='r|')
   taro.extractall(path=edir)

But it does not writes anything to the terminal to inform the user.

When I use:

   for member in taro.getmembers():
   print('extracting "%s"' % member.name)
   taro.extract(member)

I get the error:

   File "/usr/lib/python2.7/tarfile.py", line 556, in seek
 raise StreamError("seeking backwards is not allowed")

Of course, a stream is not seekable.

Any ideas?


Try this:

member = taro.next()
while member is not None:
print('extracting "%s"' % member.name)
taro.extract(member)
member = tar.next()

--
https://mail.python.org/mailman/listinfo/python-list

Re: tarfile : read from a socket?

2016-02-11 Thread Peter Otten 
Ulli Horlacher wrote:

> Ulli Horlacher  wrote:
> 
>> With
>> 
>>   taro = tarfile.open(fileobj=sock.makefile('w',kB64),mode='w|')
>> 
>> I get no more error.
> 
> Of course, this is the writing client.
> 
> Now I have a small problem with the reading client.
> 
> This code works so far:
> 
>   sfo = sock.makefile('r')
>   taro = tarfile.open(fileobj=sfo,mode='r|')
>   taro.extractall(path=edir)
> 
> But it does not writes anything to the terminal to inform the user.
> 
> When I use:
> 
>   for member in taro.getmembers():
>   print('extracting "%s"' % member.name)
>   taro.extract(member)
> 
> I get the error:
> 
>   File "/usr/lib/python2.7/tarfile.py", line 556, in seek
> raise StreamError("seeking backwards is not allowed")
> 
> Of course, a stream is not seekable.
> 
> Any ideas?

A look into the source is often helpful ;)

$ cat extract_from_stream.py
import sys
from tarfile import TarFile

class MyTarFile(TarFile):
def extract(self, member, path="."):
print "extracting", member
return TarFile.extract(self, member, path)

tf = MyTarFile.open(fileobj=sys.stdin, mode="r|")
tf.extractall()
$ touch foo bar
$ tar -cf archive.tar foo bar
$ python extract_from_stream.py < archive.tar 
extracting 
extracting 


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: tarfile : read from a socket?

2016-02-11 Thread Lars Gustäbel 
On Thu, Feb 11, 2016 at 04:41:43PM +, Ulli Horlacher wrote:
>   sfo = sock.makefile('r')
>   taro = tarfile.open(fileobj=sfo,mode='r|')
>   taro.extractall(path=edir)

What about using an iterator?

def myiter(tar):
for t in tar:
print "extracting", t.name
yield t

sfo = sock.makefile('r')
taro = tarfile.open(fileobj=sfo,mode='r|')
taro.extractall(members=myiter(taro),path=edir)

Cheers,

-- 
Lars Gustäbel
l...@gustaebel.de
-- 
https://mail.python.org/mailman/listinfo/python-list