Re: [HACKERS] Possible TODO item: copy to/from pipe

2006-06-04 Thread Greg Stark
Andreas Pflug <[EMAIL PROTECTED]> writes: > Bruce Momjian wrote: > > > > For use case, consider this: > > > > COPY mytable TO '| rsh [EMAIL PROTECTED] > test '; > > > > so you can COPY to another server directly. > > > Why not rsh psql -c "\copy foobar to test" ? Who knows? The sysadmin could

Re: [HACKERS] Possible TODO item: copy to/from pipe

2006-06-04 Thread Andreas Pflug
Bruce Momjian wrote: For use case, consider this: COPY mytable TO '| rsh [EMAIL PROTECTED] > test '; so you can COPY to another server directly. Why not rsh psql -c "\copy foobar to test" ? Regards, Andreas ---(end of broadcast)---

Re: [HACKERS] Possible TODO item: copy to/from pipe

2006-06-03 Thread Bruce Momjian
Andreas Pflug wrote: > Tom Lane wrote: > > After re-reading what I just wrote to Andreas about how compression of > > COPY data would be better done outside the backend than inside, it > > struck me that we are missing a feature that's fairly common in Unix > > programs. Perhaps COPY ought to have

Re: [HACKERS] Possible TODO item: copy to/from pipe

2006-06-03 Thread Bruce Momjian
Martijn van Oosterhout wrote: -- Start of PGP signed section. > On Wed, May 31, 2006 at 01:08:28PM -0700, Steve Atkins wrote: > > On May 31, 2006, at 12:58 PM, Dave Page wrote: > > >On 31/5/06 19:13, "Andreas Pflug" <[EMAIL PROTECTED]> wrote: > > > > > >>I wonder if we'd be able to ship gzip with t

Re: [HACKERS] Possible TODO item: copy to/from pipe

2006-06-01 Thread Mark Woodward
> After re-reading what I just wrote to Andreas about how compression of > COPY data would be better done outside the backend than inside, it > struck me that we are missing a feature that's fairly common in Unix > programs. Perhaps COPY ought to have the ability to pipe its output > to a shell co

Re: [HACKERS] Possible TODO item: copy to/from pipe

2006-06-01 Thread Dawid Kuroczko
On 5/31/06, Tom Lane <[EMAIL PROTECTED]> wrote: After re-reading what I just wrote to Andreas about how compression of COPY data would be better done outside the backend than inside, it struck me that we are missing a feature that's fairly common in Unix programs. Perhaps COPY ought to have the

Re: [HACKERS] Possible TODO item: copy to/from pipe

2006-05-31 Thread Martijn van Oosterhout
On Wed, May 31, 2006 at 11:20:21PM +0200, Magnus Hagander wrote: > Uh. The installer does *not* include readline. Terribly sorry, I misinterpreted the thread about it at the beginning of the year. http://archives.postgresql.org/pgsql-hackers/2006-02/msg00539.php Have a nice day, -- Martijn van

Re: [HACKERS] Possible TODO item: copy to/from pipe

2006-05-31 Thread Joshua D. Drake
Uh. The installer does *not* include readline. We do include PostGIS, but the PostGIS people themselves don't consider us GPLed because of that ;-) That is a tad different. PostgreSQL does not link to Postgis. Readline does :) Sincerely, Joshua D. Drake Everything else is != GPL. //

Re: [HACKERS] Possible TODO item: copy to/from pipe

2006-05-31 Thread Magnus Hagander
> > >>I wonder if we'd be able to ship gzip with the windows > installer, to > > >>insure proper integration. > > > > > >'Fraid not. It's GPL'd. > > > > Well, one implementation of it is. zlib is new-bsd-ish, though, and > > includes minigzip, which should be just fine for use in a pipe on > >

Re: [HACKERS] Possible TODO item: copy to/from pipe

2006-05-31 Thread Andreas Pflug
Dave Page wrote: It's not about a primarily GUI based OS not being able to do everything a traditionally command line based OS can do on the command line, it's about providing a solution that will work on either and remain portable. Whilst I agree with your objection to using pg_lzcompress,

Re: [HACKERS] Possible TODO item: copy to/from pipe

2006-05-31 Thread Martijn van Oosterhout
On Wed, May 31, 2006 at 01:08:28PM -0700, Steve Atkins wrote: > On May 31, 2006, at 12:58 PM, Dave Page wrote: > >On 31/5/06 19:13, "Andreas Pflug" <[EMAIL PROTECTED]> wrote: > > > >>I wonder if we'd be able to ship gzip with the windows installer, to > >>insure proper integration. > > > >'Fraid no

Re: [HACKERS] Possible TODO item: copy to/from pipe

2006-05-31 Thread Dave Page
On 31/5/06 21:10, "Tom Lane" <[EMAIL PROTECTED]> wrote: > Dave Page writes: >> Exactly my point; how many production Windows servers do you have with gzip >> anywhere near them? Andreas' point about pipes is also valid though - it's >> simply not the norm on Windows as I found when we were por

Re: [HACKERS] Possible TODO item: copy to/from pipe

2006-05-31 Thread Andrew Dunstan
Alvaro Herrera wrote: Andrew Dunstan wrote: But why is that hugely better than piping psql output to gzip? psql output has already travelled over the network. As I understand Tom's suggestion, it does not involve compression of over the wire data. He suggested that on the serve

Re: [HACKERS] Possible TODO item: copy to/from pipe

2006-05-31 Thread Tom Lane
Dave Page writes: > Exactly my point; how many production Windows servers do you have with gzip > anywhere near them? Andreas' point about pipes is also valid though - it's > simply not the norm on Windows as I found when we were porting Slony > (more.exe barfs at >8MB being pipe in). I don't see

Re: [HACKERS] Possible TODO item: copy to/from pipe

2006-05-31 Thread Tom Lane
Andreas Pflug <[EMAIL PROTECTED]> writes: > My COPY WITH COMPRESSION is not the same as taking a copy file and > zipping it; it creates a copy file with BinarySignature that has > compressed bytes in the data part, thus it can be handled by any client > app that can stream binary copy files from

Re: [HACKERS] Possible TODO item: copy to/from pipe

2006-05-31 Thread Magnus Hagander
> > I wonder if we'd be able to ship gzip with the windows > installer, to > > insure proper integration. > > 'Fraid not. It's GPL'd. > Well, if we want to go down that route, we could probably hack up something simple around zlib. IIRC, there's even sample code in there for how to write a gzi

Re: [HACKERS] Possible TODO item: copy to/from pipe

2006-05-31 Thread Steve Atkins
On May 31, 2006, at 12:58 PM, Dave Page wrote: On 31/5/06 19:13, "Andreas Pflug" <[EMAIL PROTECTED]> wrote: I wonder if we'd be able to ship gzip with the windows installer, to insure proper integration. 'Fraid not. It's GPL'd. Well, one implementation of it is. zlib is new-bsd-ish, th

Re: [HACKERS] Possible TODO item: copy to/from pipe

2006-05-31 Thread Alvaro Herrera
Andrew Dunstan wrote: > David Fetter wrote: > >As with "in-place upgrades,"[1] the compelling use case is being short > >on disk space. For somebody with a multi-TB (or whatever figure > >sounds big this week) PostgreSQL database, it may be impossible to get > >space for twice or more that. Givi

Re: [HACKERS] Possible TODO item: copy to/from pipe

2006-05-31 Thread Dave Page
On 31/5/06 19:13, "Andreas Pflug" <[EMAIL PROTECTED]> wrote: > I wonder if we'd be able to ship gzip with the windows installer, to > insure proper integration. 'Fraid not. It's GPL'd. Regards, Dave. ---(end of broadcast)--- TIP 9: In versions

Re: [HACKERS] Possible TODO item: copy to/from pipe

2006-05-31 Thread Dave Page
On 31/5/06 18:28, "Magnus Hagander" <[EMAIL PROTECTED]> wrote: > Won't help too much, until gzip's output is piped back too, so a > replacement for COPY .. TO STDOUT COMPRESSED would be >> COPY ... TO '| > /bin/gzip |' STDOUT, to enable clients to receive the

Re: [HACKERS] Possible TODO item: copy to/from pipe

2006-05-31 Thread Andrew Dunstan
David Fetter wrote: On Wed, May 31, 2006 at 02:46:29PM -0400, Andrew Dunstan wrote: I wish somebody would explain to me the compelling use case for this. As with "in-place upgrades,"[1] the compelling use case is being short on disk space. For somebody with a multi-TB (or whatever fi

Re: [HACKERS] Possible TODO item: copy to/from pipe

2006-05-31 Thread David Fetter
On Wed, May 31, 2006 at 02:46:29PM -0400, Andrew Dunstan wrote: > I wish somebody would explain to me the compelling use case for > this. As with "in-place upgrades,"[1] the compelling use case is being short on disk space. For somebody with a multi-TB (or whatever figure sounds big this week) P

Re: [HACKERS] Possible TODO item: copy to/from pipe

2006-05-31 Thread Andrew Dunstan
: [HACKERS] Possible TODO item: copy to/from pipe Andreas Pflug wrote: Won't help too much, until gzip's output is piped back too, so a replacement for COPY .. TO STDOUT COMPRESSED would be COPY ... TO '| /bin/gzip |' STDOUT, to enable clients to receive the reduced stuff.

Re: [HACKERS] Possible TODO item: copy to/from pipe

2006-05-31 Thread Andreas Pflug
Chris Browne wrote: [EMAIL PROTECTED] (Andreas Pflug) writes: Dave Page wrote: -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Andreas Pflug Sent: 31 May 2006 16:41 Cc: Tom Lane; pgsql-hackers@postgresql.org Subject: Re: [HACKERS] Possible TODO item

Re: [HACKERS] Possible TODO item: copy to/from pipe

2006-05-31 Thread Chris Browne
-hackers@postgresql.org >>>Subject: Re: [HACKERS] Possible TODO item: copy to/from pipe >>> >>>Andreas Pflug wrote: >>> >>> >>>> Won't help too much, until gzip's output is piped back too, so a >>>> replacement for

Re: [HACKERS] Possible TODO item: copy to/from pipe

2006-05-31 Thread Magnus Hagander
> >>>Won't help too much, until gzip's output is piped back too, so a > >>>replacement for COPY .. TO STDOUT COMPRESSED would be > COPY ... TO '| > >>>/bin/gzip |' STDOUT, to enable clients to > >> > >>receive the > >> > >>>reduced stuff. > >> > >>Forgot to mention: > >>COPY COMPRESSED was also

Re: [HACKERS] Possible TODO item: copy to/from pipe

2006-05-31 Thread Andreas Pflug
Joshua D. Drake wrote: I dislike putting this into the backend precisely because it's trying to impose a one-size-fits-all compression solution. Someone might wish to use bzip2 instead of gzip, for instance, or tweak the compression level options of gzip. It's trivial for the user to do that i

Re: [HACKERS] Possible TODO item: copy to/from pipe

2006-05-31 Thread Joshua D. Drake
I dislike putting this into the backend precisely because it's trying to impose a one-size-fits-all compression solution. Someone might wish to use bzip2 instead of gzip, for instance, or tweak the compression level options of gzip. It's trivial for the user to do that if the compression progra

Re: [HACKERS] Possible TODO item: copy to/from pipe

2006-05-31 Thread Andreas Pflug
Dave Page wrote: -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Andreas Pflug Sent: 31 May 2006 16:41 Cc: Tom Lane; pgsql-hackers@postgresql.org Subject: Re: [HACKERS] Possible TODO item: copy to/from pipe Andreas Pflug wrote: Won't hel

Re: [HACKERS] Possible TODO item: copy to/from pipe

2006-05-31 Thread Tom Lane
Andreas Pflug <[EMAIL PROTECTED]> writes: > Forgot to mention: > COPY COMPRESSED was also meant to introduce a portable format that's > efficient for both text and binary data. Relying on some external XYZzip > version seems not too portable to me. I dislike putting this into the backend precise

Re: [HACKERS] Possible TODO item: copy to/from pipe

2006-05-31 Thread Dave Page
> -Original Message- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf Of Andreas Pflug > Sent: 31 May 2006 16:41 > Cc: Tom Lane; pgsql-hackers@postgresql.org > Subject: Re: [HACKERS] Possible TODO item: copy to/from pipe > > Andreas Pflug wrote

Re: [HACKERS] Possible TODO item: copy to/from pipe

2006-05-31 Thread Andreas Pflug
Andreas Pflug wrote: Won't help too much, until gzip's output is piped back too, so a replacement for COPY .. TO STDOUT COMPRESSED would be COPY ... TO '| /bin/gzip |' STDOUT, to enable clients to receive the reduced stuff. Forgot to mention: COPY COMPRESSED was also meant to introduce a p

Re: [HACKERS] Possible TODO item: copy to/from pipe

2006-05-31 Thread Andreas Pflug
Tom Lane wrote: After re-reading what I just wrote to Andreas about how compression of COPY data would be better done outside the backend than inside, it struck me that we are missing a feature that's fairly common in Unix programs. Perhaps COPY ought to have the ability to pipe its output to a

Re: [HACKERS] Possible TODO item: copy to/from pipe

2006-05-31 Thread David Fetter
On Wed, May 31, 2006 at 11:03:14AM -0400, Tom Lane wrote: > After re-reading what I just wrote to Andreas about how compression > of COPY data would be better done outside the backend than inside, > it struck me that we are missing a feature that's fairly common in > Unix programs. Perhaps COPY ou

Re: [HACKERS] Possible TODO item: copy to/from pipe

2006-05-31 Thread Andrew Dunstan
Tom Lane wrote: You can accomplish COPY piping now through psql, but it's a bit awkward: psql -c "COPY mytable TO stdout" mydb | gzip ... Thoughts? Is this worth doing, or is the psql -c approach good enough? I think it's good enough. And there is also pg_dump -F c -t b

[HACKERS] Possible TODO item: copy to/from pipe

2006-05-31 Thread Tom Lane
After re-reading what I just wrote to Andreas about how compression of COPY data would be better done outside the backend than inside, it struck me that we are missing a feature that's fairly common in Unix programs. Perhaps COPY ought to have the ability to pipe its output to a shell command, or