Re: [HACKERS] raw output from copy

2016-12-06 16:59 GMT+09:00 Pavel Stehule : > > > 2016-12-06 1:50 GMT+01:00 Kohei KaiGai : >> >> 2016-12-05 22:45 GMT+09:00 Pavel Stehule : >> > >> > There are more goals: >> > >> > 1. user friendly import of text or binary data

Re: [HACKERS] raw output from copy

2016-12-06 1:50 GMT+01:00 Kohei KaiGai : > 2016-12-05 22:45 GMT+09:00 Pavel Stehule : > > > > There are more goals: > > > > 1. user friendly import of text or binary data - import text data (with > > psql) from file is possible - but you have to load

Re: [HACKERS] raw output from copy

2016-12-05 22:45 GMT+09:00 Pavel Stehule : > > There are more goals: > > 1. user friendly import of text or binary data - import text data (with > psql) from file is possible - but you have to load a content to psql > variable. For binary data you should to use workaround

Re: [HACKERS] raw output from copy

Hi 2016-12-05 14:19 GMT+01:00 Kohei KaiGai : > Sorry for my late response. > > I've briefly checked a series of discussion in the past. > I understood the target/purpose of this patch is provision of a fast > interface > to import/export a particular cell of a relation, by

Re: [HACKERS] raw output from copy

Sorry for my late response. I've briefly checked a series of discussion in the past. I understood the target/purpose of this patch is provision of a fast interface to import/export a particular cell of a relation, by skip of text<->binary transformation. Its typical use case are XML and JSON data

Re: [HACKERS] raw output from copy

On Tue, Nov 22, 2016 at 10:48 PM, Haribabu Kommi wrote: > Hi, > > This is a gentle reminder. > > you assigned as reviewer to the current patch in the 11-2016 commitfest. > But you haven't shared your review yet. Please share your review about > the patch. This will

Re: [HACKERS] raw output from copy

Hi, This is a gentle reminder. you assigned as reviewer to the current patch in the 11-2016 commitfest. But you haven't shared your review yet. Please share your review about the patch. This will help us in smoother operation of commitfest. Please Ignore if you already shared your review.

Re: [HACKERS] raw output from copy

On Sat, Jul 16, 2016 at 5:55 PM, Pavel Stehule wrote: > I am sending fresh version of COPY RAW patch. Moved to next CF per this status. +++ b/src/interfaces/libpq/test/copy-raw-regress.pl @@ -0,0 +1,48 @@ +#!/usr/bin/perl -w + +use strict; I don't understand why this is

Re: [HACKERS] raw output from copy

Hi 2016-04-05 10:45 GMT+02:00 Pavel Stehule : > Hi > > here is cleaned/finished previous implementation of RAW_TEXT/RAW_BINARY > formats for COPY statements. > > The RAW with text formats means unescaped data, but with correct encoding > - input/output is realised with

Re: [HACKERS] raw output from copy

2016-04-12 22:48 GMT+02:00 Tom Lane : > Pavel Stehule writes: > > I had a idea about additional options of COPY RAW statements. One can be > > CAST function. These CAST functions can be used to any for any format. > > Uh, what? CAST() is not about

Re: [HACKERS] raw output from copy

Pavel Stehule writes: > I had a idea about additional options of COPY RAW statements. One can be > CAST function. These CAST functions can be used to any for any format. Uh, what? CAST() is not about external representations of values, and overloading it for that

Re: [HACKERS] raw output from copy

2016-04-12 12:22 GMT+02:00 Ants Aasma : > On 8 Apr 2016 9:14 pm, "Pavel Stehule" wrote: > > 2016-04-08 20:54 GMT+02:00 Andrew Dunstan : > >> I should add that I've been thinking about this some more, and that I > now agree that

Re: [HACKERS] raw output from copy

On 8 Apr 2016 9:14 pm, "Pavel Stehule" wrote: > 2016-04-08 20:54 GMT+02:00 Andrew Dunstan : >> I should add that I've been thinking about this some more, and that I now >> agree that something should be done to support this at the SQL level, mainly

Re: [HACKERS] raw output from copy

2016-04-08 20:54 GMT+02:00 Andrew Dunstan : > > > On 04/08/2016 02:13 PM, Robert Haas wrote: > >> On Tue, Apr 5, 2016 at 4:45 AM, Pavel Stehule >> wrote: >> >>> here is cleaned/finished previous implementation of RAW_TEXT/RAW_BINARY >>> formats for

Re: [HACKERS] raw output from copy

On 04/08/2016 02:13 PM, Robert Haas wrote: On Tue, Apr 5, 2016 at 4:45 AM, Pavel Stehule wrote: here is cleaned/finished previous implementation of RAW_TEXT/RAW_BINARY formats for COPY statements. The RAW with text formats means unescaped data, but with correct

Re: [HACKERS] raw output from copy

2016-04-08 20:13 GMT+02:00 Robert Haas : > On Tue, Apr 5, 2016 at 4:45 AM, Pavel Stehule > wrote: > > here is cleaned/finished previous implementation of RAW_TEXT/RAW_BINARY > > formats for COPY statements. > > > > The RAW with text formats means

Re: [HACKERS] raw output from copy

On Tue, Apr 5, 2016 at 4:45 AM, Pavel Stehule wrote: > here is cleaned/finished previous implementation of RAW_TEXT/RAW_BINARY > formats for COPY statements. > > The RAW with text formats means unescaped data, but with correct encoding - > input/output is realised with

Re: [HACKERS] raw output from copy

Hi here is cleaned/finished previous implementation of RAW_TEXT/RAW_BINARY formats for COPY statements. The RAW with text formats means unescaped data, but with correct encoding - input/output is realised with input/output function. RAW binary means content produced/received by sending/received

Re: [HACKERS] raw output from copy

Tom Lane wrote: > >> Code that uses PQexecParams() binary "resultFormat", or the > >> binary format of copy doesn't have that problem, but most > >> client-side drivers don't do that. > > > And maybe they just can't realistically, because getting result > > format in binary is exposed

Re: [HACKERS] raw output from copy

On Fri, Apr 1, 2016 at 8:42 AM, Daniel Verite wrote: > Andrew Dunstan wrote: > > > If someone can make a good case that this is going to be of > > general use I'll happily go along, but I haven't seen one so far. > > About COPY FROM with a raw format, for

Re: [HACKERS] raw output from copy

"Daniel Verite" writes: > One reason of adding the format to COPY is that it's where users > are looking for it. It's the canonical way of importing contents > from files so that's where it makes more sense. I'm not sure I buy that argument, because it could be used to

Re: [HACKERS] raw output from copy

Andrew Dunstan wrote: > Inserting the whole contents of a text file unchanged is insanely easy > in psql. > > \set file `cat /path/to/file` > insert into mytable(contents) values(:'file'); That's assuming psql but the asker of that question never mentioned using psql. The

Re: [HACKERS] raw output from copy

On 04/01/2016 11:42 AM, Daniel Verite wrote: Andrew Dunstan wrote: If someone can make a good case that this is going to be of general use I'll happily go along, but I haven't seen one so far. About COPY FROM with a raw format, for instance just yesterday there was this user

Re: [HACKERS] raw output from copy

Andrew Dunstan wrote: > If someone can make a good case that this is going to be of > general use I'll happily go along, but I haven't seen one so far. About COPY FROM with a raw format, for instance just yesterday there was this user question on stackoverflow:

Re: [HACKERS] raw output from copy

2016-03-31 9:48 GMT+02:00 Craig Ringer : > On 31 March 2016 at 14:40, Pavel Stehule wrote: > > >> this patch doesn't break any old application. Accepting new feature >> depends on binary method detection. PQbinaryTuples based clients should >> to

Re: [HACKERS] raw output from copy

On 31 March 2016 at 14:40, Pavel Stehule wrote: > this patch doesn't break any old application. Accepting new feature > depends on binary method detection. PQbinaryTuples based clients should > to support COPY RAW* without problems, PQfformat() should to report >

Re: [HACKERS] raw output from copy

2016-03-31 8:34 GMT+02:00 Craig Ringer : > On 30 March 2016 at 00:19, Tom Lane wrote: > >> Pavel Stehule writes: >> > I tested COPY RAW on old psql clients - and it is working without any >> > problem - so when the client uses

Re: [HACKERS] raw output from copy

On 30 March 2016 at 00:19, Tom Lane wrote: > Pavel Stehule writes: > > I tested COPY RAW on old psql clients - and it is working without any > > problem - so when the client uses same logic as psql, then it should to > > work. Sure, there can be

Re: [HACKERS] raw output from copy

Hi 2016-03-29 20:59 GMT+02:00 Tom Lane : > Pavel Stehule writes: > > I am writing few lines as summary: > > > 1. invention RAW_TEXT and RAW_BINARY > > 2. for RAW_BINARY: PQbinaryTuples() returns 1 and PQfformat() returns 1 > > 3.a for RAW_TEXT:

Re: [HACKERS] raw output from copy

Andrew Dunstan writes: > The I would suggest we try to invent something for psql which does help > with it. I just don't see this as an SQL problem. There's certainly a lot to be said for that approach. I'm still not convinced that we can make COPY do this without creating

Re: [HACKERS] raw output from copy

On 03/28/2016 11:18 PM, Pavel Stehule wrote: Anyway this is certainly not committable as-is, so I'm setting it back to Waiting on Author. But the fact that both libpq and ecpg would need updates makes me question whether we can safely pretend that

Re: [HACKERS] raw output from copy

2016-03-29 20:59 GMT+02:00 Tom Lane : > Pavel Stehule writes: > > I am writing few lines as summary: > > > 1. invention RAW_TEXT and RAW_BINARY > > 2. for RAW_BINARY: PQbinaryTuples() returns 1 and PQfformat() returns 1 > > 3.a for RAW_TEXT:

Re: [HACKERS] raw output from copy

Pavel Stehule writes: > I am writing few lines as summary: > 1. invention RAW_TEXT and RAW_BINARY > 2. for RAW_BINARY: PQbinaryTuples() returns 1 and PQfformat() returns 1 > 3.a for RAW_TEXT: PQbinaryTuples() returns 0 and PQfformat() returns 0, but > the client should

Re: [HACKERS] raw output from copy

Hi 2016-03-29 18:19 GMT+02:00 Tom Lane : > Pavel Stehule writes: > > I tested COPY RAW on old psql clients - and it is working without any > > problem - so when the client uses same logic as psql, then it should to > > work. Sure, there can be

Re: [HACKERS] raw output from copy

Pavel Stehule writes: > I tested COPY RAW on old psql clients - and it is working without any > problem - so when the client uses same logic as psql, then it should to > work. Sure, there can be differently implemented clients, but the COPY > client side is usually simple

Re: [HACKERS] raw output from copy

Hi 2016-03-29 0:26 GMT+02:00 Tom Lane : > Pavel Stehule writes: > > [ copy-raw-format-20160227-03.patch ] > > I looked at this patch. I'm having a hard time accepting that it has > a use-case large enough to justify it, and here's the reason: it's >

Re: [HACKERS] raw output from copy

Hi >>> Anyway this is certainly not committable as-is, so I'm setting it back >>> to Waiting on Author. But the fact that both libpq and ecpg would need >>> updates makes me question whether we can safely pretend that this isn't >>> a protocol break. >>> >>> >>> >> >> >> In that case I humbly

Re: [HACKERS] raw output from copy

Hi 2016-03-29 0:26 GMT+02:00 Tom Lane : > Pavel Stehule writes: > > [ copy-raw-format-20160227-03.patch ] > > I looked at this patch. I'm having a hard time accepting that it has > a use-case large enough to justify it, and here's the reason: it's >

Re: [HACKERS] raw output from copy

2016-03-29 5:12 GMT+02:00 Andrew Dunstan : > > > On 03/28/2016 06:26 PM, Tom Lane wrote: > >> Pavel Stehule writes: >> >>> [ copy-raw-format-20160227-03.patch ] >>> >> I looked at this patch. I'm having a hard time accepting that it has >> a

Re: [HACKERS] raw output from copy

On 03/28/2016 06:26 PM, Tom Lane wrote: Pavel Stehule writes: [ copy-raw-format-20160227-03.patch ] I looked at this patch. I'm having a hard time accepting that it has a use-case large enough to justify it, and here's the reason: it's a protocol break.

Re: [HACKERS] raw output from copy

Pavel Stehule writes: > [ copy-raw-format-20160227-03.patch ] I looked at this patch. I'm having a hard time accepting that it has a use-case large enough to justify it, and here's the reason: it's a protocol break. Conveniently omitting to update protocol.sgml doesn't

Re: [HACKERS] raw output from copy

On 3/12/16 1:24 AM, Pavel Stehule wrote: Great, thank you very much. I hope so this feature really useful. It allow to simple export/import XML doc in different encodings, JSONs and can be enhanced future via options. The nice feature (but not for this release) can be additional cast info for

Re: [HACKERS] raw output from copy

2016-03-18 16:32 GMT+01:00 David Steele : > On 3/12/16 1:24 AM, Pavel Stehule wrote: > > Great, thank you very much. I hope so this feature really useful. It >> allow to simple export/import XML doc in different encodings, JSONs and >> can be enhanced future via options. The

Re: [HACKERS] raw output from copy

Hi 2016-03-09 18:41 GMT+01:00 Corey Huinker : > >>> The regression tests seem to adequately cover all new functionality, >>> though I wonder if we should add some cases that highlight situations where >>> BINARY mode is insufficient. >>> >>> > One thing I tried to test

Re: [HACKERS] raw output from copy

> > >> The regression tests seem to adequately cover all new functionality, >> though I wonder if we should add some cases that highlight situations where >> BINARY mode is insufficient. >> >> One thing I tried to test RAW was to load an existing json file. My own personal test was to load an

Re: [HACKERS] raw output from copy

2016-03-04 18:06 GMT+01:00 Pavel Stehule : > > > 2016-03-04 15:54 GMT+01:00 Daniel Verite : > >> Corey Huinker wrote: >> >> > So, for me, RAW is the right solution, or at least *a* right solution. >> >> Questions on how to extract from a

Re: [HACKERS] raw output from copy

2016-03-04 15:54 GMT+01:00 Daniel Verite : > Corey Huinker wrote: > > > So, for me, RAW is the right solution, or at least *a* right solution. > > Questions on how to extract from a bytea column come up on a regular > basis, as in [1] [2] [3], or [4] a few days

Re: [HACKERS] raw output from copy

Corey Huinker wrote: > So, for me, RAW is the right solution, or at least *a* right solution. Questions on how to extract from a bytea column come up on a regular basis, as in [1] [2] [3], or [4] a few days ago, and so far the answers are to encode the contents in text and decode them in

Re: [HACKERS] raw output from copy

Hi Pavel 27/02/16 10:26, Pavel Stehule пишет: Hi 2015-08-06 10:37 GMT+02:00 Pavel Stehule >: Hi, Psql based implementation needs new infrastructure (more than few lines) Missing: * binary mode support *

Re: [HACKERS] raw output from copy

2016-03-04 3:13 GMT+01:00 Corey Huinker : > On Sat, Feb 27, 2016 at 2:26 AM, Pavel Stehule > wrote: > >> Hi >> >> 2015-08-06 10:37 GMT+02:00 Pavel Stehule : >> >>> Hi, >>> >>> Psql based implementation needs new

Re: [HACKERS] raw output from copy

On Sat, Feb 27, 2016 at 2:26 AM, Pavel Stehule wrote: > Hi > > 2015-08-06 10:37 GMT+02:00 Pavel Stehule : > >> Hi, >> >> Psql based implementation needs new infrastructure (more than few lines) >> >> Missing: >> >> * binary mode support >> *

Re: [HACKERS] raw output from copy

Hi 2015-08-06 10:37 GMT+02:00 Pavel Stehule : > Hi, > > Psql based implementation needs new infrastructure (more than few lines) > > Missing: > > * binary mode support > * parametrized query support, > > I am not against, but both points I proposed, and both was

Re: [HACKERS] raw output from copy

Hi, Psql based implementation needs new infrastructure (more than few lines) Missing: * binary mode support * parametrized query support, I am not against, but both points I proposed, and both was rejected. So why dont use current infrastructure? Raw copy is trivial patch. Dne 6.8.2015 0:09

Re: [HACKERS] raw output from copy

On 07/27/2015 02:28 PM, Pavel Stehule wrote: 2015-07-27 10:41 GMT+02:00 Heikki Linnakangas hlinn...@iki.fi: What about input? This is a whole new feature, but it would be nice to be able to pass the file contents as a query parameter. Something like: \P /tmp/foo binary INSERT INTO foo VALUES

Re: [HACKERS] raw output from copy

On 08/05/2015 04:59 PM, Heikki Linnakangas wrote: On 07/27/2015 02:28 PM, Pavel Stehule wrote: 2015-07-27 10:41 GMT+02:00 Heikki Linnakangas hlinn...@iki.fi: What about input? This is a whole new feature, but it would be nice to be able to pass the file contents as a query parameter.

Re: [HACKERS] raw output from copy

2015-07-27 10:41 GMT+02:00 Heikki Linnakangas hlinn...@iki.fi: On 07/27/2015 06:55 AM, Craig Ringer wrote: On 7 July 2015 at 14:32, Pavel Stehule pavel.steh...@gmail.com wrote: Hi previous patch was broken, and buggy Here is new version with fixed upload and more tests I routinely see

Re: [HACKERS] raw output from copy

On 07/27/2015 06:55 AM, Craig Ringer wrote: On 7 July 2015 at 14:32, Pavel Stehule pavel.steh...@gmail.com wrote: Hi previous patch was broken, and buggy Here is new version with fixed upload and more tests I routinely see people trying to use COPY ... FORMAT binary to export a single

Re: [HACKERS] raw output from copy

On 7 July 2015 at 14:32, Pavel Stehule pavel.steh...@gmail.com wrote: Hi previous patch was broken, and buggy Here is new version with fixed upload and more tests I routinely see people trying to use COPY ... FORMAT binary to export a single binary field (like an image, for example) and

Re: [HACKERS] raw output from copy

2015-07-23 22:05 GMT+02:00 Dickson S. Guedes lis...@guedesoft.net: 2015-07-07 3:32 GMT-03:00 Pavel Stehule pavel.steh...@gmail.com: Hi previous patch was broken, and buggy Here is new version with fixed upload and more tests The interesting is so I should not to modify interface

Re: [HACKERS] raw output from copy

2015-07-07 3:32 GMT-03:00 Pavel Stehule pavel.steh...@gmail.com: Hi previous patch was broken, and buggy Here is new version with fixed upload and more tests The interesting is so I should not to modify interface or client - so it should to work with any current driver with protocol

Re: [HACKERS] raw output from copy

Hi previous patch was broken, and buggy Here is new version with fixed upload and more tests The interesting is so I should not to modify interface or client - so it should to work with any current driver with protocol support = 3. Regards Pavel 2015-07-06 23:34 GMT+02:00 Pavel Stehule

Re: [HACKERS] raw output from copy

Hi here is a version with both direction support. postgres=# copy foo from '/tmp/1.jpg' (format raw); COPY 1 Time: 93.021 ms postgres=# \dt+ foo List of relations ┌┬──┬───┬───┬┬─┐ │ Schema │ Name │ Type │ Owner │ Size │ Description │

Re: [HACKERS] raw output from copy

2015-07-02 17:02 GMT+02:00 Tom Lane t...@sss.pgh.pa.us: Andrew Dunstan and...@dunslane.net writes: Does the COPY line protocol even support binary data? The protocol, per se, just transmits a byte stream. There is a field in the CopyInResponse/CopyOutResponse messages that indicates

Re: [HACKERS] raw output from copy

Hi I'll do it today evening Pavel 2015-07-02 12:55 GMT+02:00 Simon Riggs si...@2ndquadrant.com: On 1 July 2015 at 07:42, Pavel Golub pa...@microolap.com wrote: I looked through the patch. Sources are OK. However I didn't find any docs and test cases. Would you please provide me with short

Re: [HACKERS] raw output from copy

On 07/02/2015 07:14 AM, Pavel Stehule wrote: Hi I'll do it today evening Pavel, Please don't top-post on the PostgreSQL lists. You've been around here long enough to know that bottom posting is our custom. I posted a patch for this in 2013 at

Re: [HACKERS] raw output from copy

On 07/02/2015 09:43 AM, Simon Riggs wrote: On 2 July 2015 at 14:02, Andrew Dunstan and...@dunslane.net mailto:and...@dunslane.net wrote: Please don't top-post on the PostgreSQL lists. You've been around here long enough to know that bottom posting is our custom. I posted a patch

Re: [HACKERS] raw output from copy

2015-07-02 16:02 GMT+02:00 Andrew Dunstan and...@dunslane.net: On 07/02/2015 09:43 AM, Simon Riggs wrote: On 2 July 2015 at 14:02, Andrew Dunstan and...@dunslane.net mailto: and...@dunslane.net wrote: Please don't top-post on the PostgreSQL lists. You've been around here long

Re: [HACKERS] raw output from copy

On 07/02/2015 09:02 AM, Andrew Dunstan wrote: On 07/02/2015 07:14 AM, Pavel Stehule wrote: Hi I'll do it today evening Pavel, Please don't top-post on the PostgreSQL lists. You've been around here long enough to know that bottom posting is our custom. I posted a patch for this in

Re: [HACKERS] raw output from copy

On 2 July 2015 at 14:02, Andrew Dunstan and...@dunslane.net wrote: Please don't top-post on the PostgreSQL lists. You've been around here long enough to know that bottom posting is our custom. I posted a patch for this in 2013 at

Re: [HACKERS] raw output from copy

2015-07-02 15:43 GMT+02:00 Simon Riggs si...@2ndquadrant.com: On 2 July 2015 at 14:02, Andrew Dunstan and...@dunslane.net wrote: Please don't top-post on the PostgreSQL lists. You've been around here long enough to know that bottom posting is our custom. I posted a patch for this in 2013

Re: [HACKERS] raw output from copy

On 07/02/2015 11:02 AM, Tom Lane wrote: Andrew Dunstan and...@dunslane.net writes: Does the COPY line protocol even support binary data? The protocol, per se, just transmits a byte stream. There is a field in the CopyInResponse/CopyOutResponse messages that indicates whether a text or binary

Re: [HACKERS] raw output from copy

On 07/02/2015 10:07 AM, Pavel Stehule wrote: 2015-07-02 16:02 GMT+02:00 Andrew Dunstan and...@dunslane.net mailto:and...@dunslane.net: On 07/02/2015 09:43 AM, Simon Riggs wrote: On 2 July 2015 at 14:02, Andrew Dunstan and...@dunslane.net mailto:and...@dunslane.net

Re: [HACKERS] raw output from copy

On 2 July 2015 at 15:07, Pavel Stehule pavel.steh...@gmail.com wrote: It can be used from psql without any problems. It can, but your patch does not yet do that, while Andrew's does. We want a solution that works from psql and other clients. Hopefully the same-ish solution. -- Simon Riggs

Re: [HACKERS] raw output from copy

Hello Pavel. I looked through the patch. Sources are OK. However I didn't find any docs and test cases. Would you please provide me with short description on this feature and why it is important. Because I didn't manage to find the old Andrew Dunstan's post either. On Sat, Apr 11, 2015 at 12:26

Re: [HACKERS] raw output from copy

On 4/10/15 5:26 PM, Pavel Stehule wrote: Hi I wrote a prototype of this patch, and it works well postgres=# set client_encoding to 'latin2'; SET Time: 1.488 ms postgres=# \copy (select xmlelement(name xx, d) from d) to ~/d.xml (format 'raw') COPY 1 Time: 1.108 ms postgres=# copy

Re: [HACKERS] raw output from copy

It would be nice, but it is not true. You can get correct non utf8 xml with encoding specification only when binary mode is used. Psql doesn't support binary mode. Regards Pavel Dne 15. 4. 2015 22:06 napsal uživatel Peter Eisentraut pete...@gmx.net: On 4/10/15 5:26 PM, Pavel Stehule wrote:

Re: [HACKERS] raw output from copy

Hi I wrote a prototype of this patch, and it works well postgres=# set client_encoding to 'latin2'; SET Time: 1.488 ms postgres=# \copy (select xmlelement(name xx, d) from d) to ~/d.xml (format 'raw') COPY 1 Time: 1.108 ms postgres=# copy (select xmlelement(name xx, d) from d) to stdout (format

[HACKERS] raw output from copy

Hi This thread was finished without real work. I have a real use case - export XML doc in non utf8 encoding. http://www.postgresql.org/message-id/16174.1319228...@sss.pgh.pa.us I propose to implement new format option RAW like Tom proposed. It requires only one row, one column result - and