On 1/15/13 2:53 PM, Stephen Frost wrote:
You're right, to clarify, for *file_fdw*, which is a backend-only
operation, the popen patch is great (thought I made that clear before).
I would think that if we get writable FDWs, you would want file_fdw to
go through zlib so that it can write directly
* Peter Eisentraut (pete...@gmx.net) wrote:
On 1/15/13 2:53 PM, Stephen Frost wrote:
You're right, to clarify, for *file_fdw*, which is a backend-only
operation, the popen patch is great (thought I made that clear before).
I would think that if we get writable FDWs, you would want file_fdw
On Mon, Jan 14, 2013 at 11:28 AM, Stephen Frost sfr...@snowman.net wrote:
* Tom Lane (t...@sss.pgh.pa.us) wrote:
Stephen Frost sfr...@snowman.net writes:
I do like the idea of a generalized answer which just runs a
user-provided command on the server but that's always going to require
On Tue, Jan 15, 2013 at 3:37 PM, Christopher Browne cbbro...@gmail.com wrote:
That points towards a fix that involves having a set of non-arbitrary commands
that we allow plain users to use.
Hmm. There's an interesting thought...
How about having a pg_filters table in pg_catalog which
On Wed, Jan 16, 2013 at 5:15 PM, Robert Haas robertmh...@gmail.com wrote:
On Tue, Jan 15, 2013 at 3:37 PM, Christopher Browne cbbro...@gmail.com
wrote:
That points towards a fix that involves having a set of non-arbitrary
commands
that we allow plain users to use.
Hmm. There's an
On Tue, Jan 15, 2013 at 4:50 PM, Tom Lane t...@sss.pgh.pa.us wrote:
I find the argument that this supports compression-over-the-wire to be
quite weak, because COPY is only one form of bulk data transfer, and
one that a lot of applications don't ever use. If we think we need to
support
* Bruce Momjian (br...@momjian.us) wrote:
On Wed, Jan 16, 2013 at 06:19:09PM -0500, Robert Haas wrote:
But presumably this would transparently compress at one end and
decompress at the other end, which is again a somewhat different use
case. To get compressed output on the client side, you
On Wed, Jan 16, 2013 at 06:19:09PM -0500, Robert Haas wrote:
On Tue, Jan 15, 2013 at 4:50 PM, Tom Lane t...@sss.pgh.pa.us wrote:
I find the argument that this supports compression-over-the-wire to be
quite weak, because COPY is only one form of bulk data transfer, and
one that a lot of
On Wed, Jan 16, 2013 at 8:19 PM, Robert Haas robertmh...@gmail.com wrote:
On Tue, Jan 15, 2013 at 4:50 PM, Tom Lane t...@sss.pgh.pa.us wrote:
I find the argument that this supports compression-over-the-wire to be
quite weak, because COPY is only one form of bulk data transfer, and
one that a
On Tue, Jan 15, 2013 at 2:33 AM, Stephen Frost sfr...@snowman.net wrote:
* Peter Eisentraut (pete...@gmx.net) wrote:
On 1/14/13 11:28 AM, Stephen Frost wrote:
While there is no option currently for having the server do the
compression before sending the data over the wire.
OpenSSL?
To be
On Mon, Jan 14, 2013 at 11:33 PM, Stephen Frost sfr...@snowman.net wrote:
Now, protocol-level on-the-wire compression
is another option, but there's quite a few drawbacks to that and quite a
bit of work involved. Having support for COPY-based compression could
be an answer for many cases
On 1/13/13 9:16 PM, Stephen Frost wrote:
On top of this I plan to submit a trivial patch to add support for
this to file_fdw, allowing creation of FDW tables which operate
directly on compressed files (including CSVs, which is what I need
this patch for).
I've also begun working on
* Peter Eisentraut (pete...@gmx.net) wrote:
Operating on compressed files transparently in file_fdw is obviously
useful, but why only gzip?
This isn't really an argument, imv. It's only gzip *right this moment*
because that's all that I implemented. I've already offered to add
bzip2 or
* Claudio Freire (klaussfre...@gmail.com) wrote:
Postgres' packetized protocol lends itself quite well for this kind of
thing. It could even be done on a packet-by-packet basis. The only
drawback I see, is that it pretty much rules out piping through
arbitrary commands (a protocol needs to be
On Tue, Jan 15, 2013 at 1:08 PM, Stephen Frost sfr...@snowman.net wrote:
Where it does work well is when you move into a bulk-data mode (ala
COPY) and can compress relatively large amounts of data into a smaller
number of full-size packets to be sent.
Well... exactly. COPY is one case, big
On Tue, Jan 15, 2013 at 01:35:57PM +, Greg Stark wrote:
On Tue, Jan 15, 2013 at 2:33 AM, Stephen Frost sfr...@snowman.net wrote:
* Peter Eisentraut (pete...@gmx.net) wrote:
On 1/14/13 11:28 AM, Stephen Frost wrote:
While there is no option currently for having the server do the
* Claudio Freire (klaussfre...@gmail.com) wrote:
On Tue, Jan 15, 2013 at 1:08 PM, Stephen Frost sfr...@snowman.net wrote:
Where it does work well is when you move into a bulk-data mode (ala
COPY) and can compress relatively large amounts of data into a smaller
number of full-size packets to
On Tue, Jan 15, 2013 at 10:55:04AM -0500, Stephen Frost wrote:
* Peter Eisentraut (pete...@gmx.net) wrote:
Operating on compressed files transparently in file_fdw is obviously
useful, but why only gzip?
This isn't really an argument, imv. It's only gzip *right this moment*
because that's
* Bruce Momjian (br...@momjian.us) wrote:
On Tue, Jan 15, 2013 at 10:55:04AM -0500, Stephen Frost wrote:
* Peter Eisentraut (pete...@gmx.net) wrote:
Operating on compressed files transparently in file_fdw is obviously
useful, but why only gzip?
This isn't really an argument, imv.
On Tue, Jan 15, 2013 at 2:53 PM, Stephen Frost sfr...@snowman.net wrote:
The popen patch doesn't support the '|compression-binary' option through
the FE protocol. Even if it did, it would only be available for
superusers as we can't allow regular users to run arbitrary commands on
the
On Tue, Jan 15, 2013 at 03:37:07PM -0500, Christopher Browne wrote:
On Tue, Jan 15, 2013 at 2:53 PM, Stephen Frost sfr...@snowman.net wrote:
The popen patch doesn't support the '|compression-binary' option through
the FE protocol. Even if it did, it would only be available for
superusers
* Christopher Browne (cbbro...@gmail.com) wrote:
How about having a pg_filters table in pg_catalog which allows capturing
labels and names of known-to-be-safe binary filters:
I was considering that (though I was thinking they'd be
transformations rather than filters; filter implies that you're
* Bruce Momjian (br...@momjian.us) wrote:
Well, COPY is super-user only, so it seems only useful for FDW, no? We
already have lots of user-configuration FDW commands, so I can see
adding this one too.
COPY is most certainly not superuser-only.. COPY w/ popen, if that
popen can call anything,
Stephen Frost sfr...@snowman.net writes:
* Bruce Momjian (br...@momjian.us) wrote:
And this leads to support-my-compression-binary-of-the-day mess. Why
not just allow them to do '|compression-binary'?
The popen patch doesn't support the '|compression-binary' option through
the FE protocol.
* Tom Lane (t...@sss.pgh.pa.us) wrote:
I find the argument that this supports compression-over-the-wire to be
quite weak, because COPY is only one form of bulk data transfer, and
one that a lot of applications don't ever use. If we think we need to
support transmission compression for
On Tue, Jan 15, 2013 at 04:22:48PM -0500, Stephen Frost wrote:
* Bruce Momjian (br...@momjian.us) wrote:
Well, COPY is super-user only, so it seems only useful for FDW, no? We
already have lots of user-configuration FDW commands, so I can see
adding this one too.
COPY is most certainly
* Bruce Momjian (br...@momjian.us) wrote:
COPY with a file name is super-user-only. I am unclear how you would
use STDIN/STDOUT in any meaningful way with binary data produced by
compression. I guess you could with libpq.
The patch that I posted provided this:
psql -h myhost -c COPY mytable
Stephen Frost sfr...@snowman.net writes:
* Tom Lane (t...@sss.pgh.pa.us) wrote:
I find the argument that this supports compression-over-the-wire to be
quite weak, because COPY is only one form of bulk data transfer, and
one that a lot of applications don't ever use. If we think we need to
* Tom Lane (t...@sss.pgh.pa.us) wrote:
Really? Given that libpq provides no useful support for doing anything
with COPY data, much less higher-level packages such as Perl DBI, I'd
venture that the real-world ratio is more like 90/10. If not 99/1.
Perhaps I'm taking a bit too narrow view of
Stephen Frost sfr...@snowman.net writes:
* Tom Lane (t...@sss.pgh.pa.us) wrote:
Really? Given that libpq provides no useful support for doing anything
with COPY data, much less higher-level packages such as Perl DBI, I'd
venture that the real-world ratio is more like 90/10. If not 99/1.
Stephen Frost sfr...@snowman.net writes:
Perhaps I'm taking a bit too narrow view of the world, but my thinking
is OLTP won't want things compressed, as it increases latency of
requests, while OLAP users are operating with enough data that they'll
go through the effort to use COPY.
Also, if
On 01/15/2013 06:22 PM, Tom Lane wrote:
Stephen Frost sfr...@snowman.net writes:
Perhaps I'm taking a bit too narrow view of the world, but my thinking
is OLTP won't want things compressed, as it increases latency of
requests, while OLAP users are operating with enough data that they'll
go
On Tue, Jan 15, 2013 at 7:46 PM, Tom Lane t...@sss.pgh.pa.us wrote:
Compressing every small packet seems like it'd be overkill and might
surprise people by actually reducing performance in the case of lots of
small requests.
Yeah, proper selection and integration of a compression method would
Tom,
* Tom Lane (t...@sss.pgh.pa.us) wrote:
Stephen Frost sfr...@snowman.net writes:
Attached is a patch to add a 'COMPRESSED' option to COPY which will
cause COPY to expect a gzip'd file on input and which will output a
gzip'd file on output. Included is support for backend COPY,
On 14 January 2013 13:43, Stephen Frost sfr...@snowman.net wrote:
Tom,
* Tom Lane (t...@sss.pgh.pa.us) wrote:
Stephen Frost sfr...@snowman.net writes:
Attached is a patch to add a 'COMPRESSED' option to COPY which will
cause COPY to expect a gzip'd file on input and which will output a
* Simon Riggs (si...@2ndquadrant.com) wrote:
There is a new option being added to pre/post process data, so it
seems like the best way to add new features - in general.
That structure appears to have no option for passing compressed data to
or from a client connection. Instead, it actually
On Mon, Jan 14, 2013 at 1:01 PM, Stephen Frost sfr...@snowman.net wrote:
I do like the idea of a generalized answer which just runs a
user-provided command on the server but that's always going to require
superuser privileges.
Unless it's one of a set of superuser-authorized compression tools.
Stephen Frost sfr...@snowman.net writes:
I do like the idea of a generalized answer which just runs a
user-provided command on the server but that's always going to require
superuser privileges.
The design that was being kicked around allowed pipes to be used on the
client side too, ie \copy
* Claudio Freire (klaussfre...@gmail.com) wrote:
On Mon, Jan 14, 2013 at 1:01 PM, Stephen Frost sfr...@snowman.net wrote:
I do like the idea of a generalized answer which just runs a
user-provided command on the server but that's always going to require
superuser privileges.
Unless it's
* Tom Lane (t...@sss.pgh.pa.us) wrote:
Stephen Frost sfr...@snowman.net writes:
I do like the idea of a generalized answer which just runs a
user-provided command on the server but that's always going to require
superuser privileges.
The design that was being kicked around allowed pipes
On 1/14/13 11:28 AM, Stephen Frost wrote:
While there is no option currently for having the server do the
compression before sending the data over the wire.
OpenSSL?
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
* Peter Eisentraut (pete...@gmx.net) wrote:
On 1/14/13 11:28 AM, Stephen Frost wrote:
While there is no option currently for having the server do the
compression before sending the data over the wire.
OpenSSL?
To be honest, I expected that to come up earlier in this discussion.
It'd be
Stephen Frost sfr...@snowman.net writes:
Attached is a patch to add a 'COMPRESSED' option to COPY which will
cause COPY to expect a gzip'd file on input and which will output a
gzip'd file on output. Included is support for backend COPY, psql's
\copy, regression tests for both, and
43 matches
Mail list logo
