Re: [HACKERS] Built-in plugin for logical decoding output

It seems test_decoding.c could be easily changed to support JSON by using the built in PostgreSQL functions (json.c composite_to_json) to convert a Datum into SQL. It's use of OidOutputFunctionCall could be modified to emit arrays and composite types as JSON. This might be enough to enable

Re: [HACKERS] Built-in plugin for logical decoding output

On 26/09/17 17:50, Craig Ringer wrote: On 26 September 2017 at 22:14, Magnus Hagander > wrote: On Tue, Sep 26, 2017 at 2:16 PM, Alvaro Hernandez > wrote:     But what about earlier

Re: [HACKERS] Built-in plugin for logical decoding output

On 26 September 2017 at 22:14, Magnus Hagander wrote: > > > On Tue, Sep 26, 2017 at 2:16 PM, Alvaro Hernandez wrote: > >> >> >> >> But what about earlier versions? Any chance it could be backported >> down to 9.4? If that would be acceptable, I could

Re: [HACKERS] Built-in plugin for logical decoding output

2017-09-26 2:46 GMT-03:00 Alvaro Hernandez : > I think that's awesome. Now... back to my original question: what is the > *list* of output plugins supported by managed PostgreSQL solutions? So far > it looks like wal2json for 9.5-9.6 on RDS, and nothing else (it may just be >

Re: [HACKERS] Built-in plugin for logical decoding output

On Tue, Sep 26, 2017 at 2:16 PM, Alvaro Hernandez wrote: > > > On 26/09/17 12:57, Petr Jelinek wrote: > >> On 26/09/17 09:26, Alvaro Hernandez wrote: >> >>> On 26/09/17 10:03, Craig Ringer wrote: >>> On 26 September 2017 at 14:08, Alvaro Hernandez

Re: [HACKERS] Built-in plugin for logical decoding output

On 26/09/17 10:55, Craig Ringer wrote: On 26 September 2017 at 15:26, Alvaro Hernandez > wrote:     That's better than nothing. But as much as interoperable json may be, people still need to talk the (binary) replication protocol to use

Re: [HACKERS] Built-in plugin for logical decoding output

On 26/09/17 12:57, Petr Jelinek wrote: On 26/09/17 09:26, Alvaro Hernandez wrote: On 26/09/17 10:03, Craig Ringer wrote: On 26 September 2017 at 14:08, Alvaro Hernandez > wrote: - If you stick to in-core plugins, then you need to support at

Re: [HACKERS] Built-in plugin for logical decoding output

On 26/09/17 09:26, Alvaro Hernandez wrote: > On 26/09/17 10:03, Craig Ringer wrote: >> On 26 September 2017 at 14:08, Alvaro Hernandez > > wrote: >> - If you stick to in-core plugins, then you need to support at >> least three different output

Re: [HACKERS] Built-in plugin for logical decoding output

On Tue, Sep 26, 2017 at 7:42 AM, Alvaro Hernandez wrote: > > > On 25/09/17 22:13, Magnus Hagander wrote: > > On Mon, Sep 25, 2017 at 8:20 PM, Alvaro Hernandez wrote: > >> >> >> On 25/09/17 20:18, Andres Freund wrote: >> >>> On 2017-09-24 13:36:56 +0300, Alvaro

Re: [HACKERS] Built-in plugin for logical decoding output

On 26 September 2017 at 15:26, Alvaro Hernandez wrote: > > That's better than nothing. But as much as interoperable json may be, > people still need to talk the (binary) replication protocol to use it. > No, they don't. They can use the SQL interface to logical decoding.

Re: [HACKERS] Built-in plugin for logical decoding output

On 26/09/17 10:03, Craig Ringer wrote: On 26 September 2017 at 14:08, Alvaro Hernandez > wrote:     OK, let me try to do that. I believe data integration is a priority. Definitely agree so far. - If you want to develop your own output

Re: [HACKERS] Built-in plugin for logical decoding output

On 26 September 2017 at 14:08, Alvaro Hernandez wrote: > >> > OK, let me try to do that. I believe data integration is a priority. Definitely agree so far. > - If you want to develop your own output plugin, then your market is > reduced as you have to exclude all the

Re: [HACKERS] Built-in plugin for logical decoding output

On 25/09/17 21:38, Andres Freund wrote: On 2017-09-25 21:31:11 +0300, Alvaro Hernandez wrote: - Distribution and testing are non-trivial: many OS/archs combinations. Yes, it is. Why would we want to increase that burden to this community?     That's a different story, and one I cannot

Re: [HACKERS] Built-in plugin for logical decoding output

On 25/09/17 22:08, Jignesh Shah wrote: On Mon, Sep 25, 2017 at 11:37 AM, Joshua D. Drake > wrote: On 09/25/2017 11:31 AM, Alvaro Hernandez wrote: Whether or not they are included in a managed environment

Re: [HACKERS] Built-in plugin for logical decoding output

On 25/09/17 22:13, Magnus Hagander wrote: On Mon, Sep 25, 2017 at 8:20 PM, Alvaro Hernandez > wrote: On 25/09/17 20:18, Andres Freund wrote: On 2017-09-24 13:36:56 +0300, Alvaro Hernandez wrote: However, if DMS uses it

Re: [HACKERS] Built-in plugin for logical decoding output

On 26 September 2017 at 01:48, Joshua D. Drake wrote: > On 09/25/2017 10:43 AM, Andres Freund wrote: > >> On 2017-09-25 10:38:52 -0700, Joshua D. Drake wrote: >> >>> On 09/25/2017 10:32 AM, Petr Jelinek wrote: >>> On 25/09/17 19:26, Tom Lane wrote: > Alvaro

Re: [HACKERS] Built-in plugin for logical decoding output

On 26 September 2017 at 01:53, Andres Freund wrote: > On 2017-09-25 13:50:29 -0400, Tom Lane wrote: > > Andres Freund writes: > > >> On 25/09/17 19:26, Tom Lane wrote: > > >>> The problem with this type of argument is that it leads directly to > the > >

Re: [HACKERS] Built-in plugin for logical decoding output

On 25/09/17 19:48, Joshua D. Drake wrote: > On 09/25/2017 10:43 AM, Andres Freund wrote: >> On 2017-09-25 10:38:52 -0700, Joshua D. Drake wrote: >>> On 09/25/2017 10:32 AM, Petr Jelinek wrote: On 25/09/17 19:26, Tom Lane wrote: > Alvaro Hernandez writes: >>>

Re: [HACKERS] Built-in plugin for logical decoding output

On Sep 25, 2017 1:39 PM, "Joshua D. Drake" wrote: > On 09/25/2017 10:32 AM, Petr Jelinek wrote: > >> On 25/09/17 19:26, Tom Lane wrote: >> >>> Alvaro Hernandez writes: >>> >> > >> There is already about 3 million output plugins out there so I think we >>

Re: [HACKERS] Built-in plugin for logical decoding output

On Mon, Sep 25, 2017 at 8:20 PM, Alvaro Hernandez wrote: > > > On 25/09/17 20:18, Andres Freund wrote: > >> On 2017-09-24 13:36:56 +0300, Alvaro Hernandez wrote: >> >>> However, if DMS uses it for what I'd call production use, I assume >>> it is >>> actually production

Re: [HACKERS] Built-in plugin for logical decoding output

On Mon, Sep 25, 2017 at 11:37 AM, Joshua D. Drake wrote: > On 09/25/2017 11:31 AM, Alvaro Hernandez wrote: > >> >> > Whether or not they are included in a managed environment is generally >>> based on two things: >>> >>> 1. Safety (why RDS doesn't allow certain C

Re: [HACKERS] Built-in plugin for logical decoding output

On 2017-09-25 21:31:11 +0300, Alvaro Hernandez wrote: > > > > - Distribution and testing are non-trivial: many OS/archs combinations. > > > > > > > > Yes, it is. Why would we want to increase that burden to this community? > > >     That's a different story, and one I cannot argue against. If

Re: [HACKERS] Built-in plugin for logical decoding output

On 09/25/2017 11:31 AM, Alvaro Hernandez wrote: Whether or not they are included in a managed environment is generally based on two things: 1. Safety (why RDS doesn't allow certain C extensions) 2. Community/Popularity (Exactly why RDS has PostGIS)     A. Demand with a

Re: [HACKERS] Built-in plugin for logical decoding output

On 25/09/17 20:31, Joshua D. Drake wrote: On 09/25/2017 10:19 AM, Petr Jelinek wrote: On 25/09/17 18:48, Alvaro Hernandez wrote: In my opinion, logical decoding plugins that don't come with core are close to worthless (don't get me wrong): I respectfully disagree. As do I.    

Re: [HACKERS] Built-in plugin for logical decoding output

On 25/09/17 20:18, Andres Freund wrote: On 2017-09-24 13:36:56 +0300, Alvaro Hernandez wrote:     However, if DMS uses it for what I'd call production use, I assume it is actually production quality. I bet they do enough testing, and don't ship software to potentially millions of customers

Re: [HACKERS] Built-in plugin for logical decoding output

Andres, all, * Andres Freund (and...@anarazel.de) wrote: > On 2017-09-25 19:32:29 +0200, Petr Jelinek wrote: > > On 25/09/17 19:26, Tom Lane wrote: > > > Alvaro Hernandez writes: > > >> In my opinion, logical decoding plugins that don't come with core > > >> are close to

Re: [HACKERS] Built-in plugin for logical decoding output

On 2017-09-25 13:50:29 -0400, Tom Lane wrote: > Andres Freund writes: > >> On 25/09/17 19:26, Tom Lane wrote: > >>> The problem with this type of argument is that it leads directly to the > >>> conclusion that every feature users want must be in core. > > > ... I don't think

Re: [HACKERS] Built-in plugin for logical decoding output

Andres Freund writes: >> On 25/09/17 19:26, Tom Lane wrote: >>> The problem with this type of argument is that it leads directly to the >>> conclusion that every feature users want must be in core. > ... I don't think that should mean that there's no possible output > plugin

Re: [HACKERS] Built-in plugin for logical decoding output

On 09/25/2017 10:43 AM, Andres Freund wrote: On 2017-09-25 10:38:52 -0700, Joshua D. Drake wrote: On 09/25/2017 10:32 AM, Petr Jelinek wrote: On 25/09/17 19:26, Tom Lane wrote: Alvaro Hernandez writes: There is already about 3 million output plugins out there so I think

Re: [HACKERS] Built-in plugin for logical decoding output

On 2017-09-25 19:32:29 +0200, Petr Jelinek wrote: > On 25/09/17 19:26, Tom Lane wrote: > > Alvaro Hernandez writes: > >> In my opinion, logical decoding plugins that don't come with core > >> are close to worthless (don't get me wrong): > > > >> - They very unlikely will be

Re: [HACKERS] Built-in plugin for logical decoding output

On 2017-09-25 10:38:52 -0700, Joshua D. Drake wrote: > On 09/25/2017 10:32 AM, Petr Jelinek wrote: > > On 25/09/17 19:26, Tom Lane wrote: > > > Alvaro Hernandez writes: > > > > > There is already about 3 million output plugins out there so I think we > > did reasonable job

Re: [HACKERS] Built-in plugin for logical decoding output

On 09/25/2017 10:32 AM, Petr Jelinek wrote: On 25/09/17 19:26, Tom Lane wrote: Alvaro Hernandez writes: There is already about 3 million output plugins out there so I think we did reasonable job there. The fact that vast majority of that are various json ones gives

Re: [HACKERS] Built-in plugin for logical decoding output

On 25/09/17 19:26, Tom Lane wrote: > Alvaro Hernandez writes: >> In my opinion, logical decoding plugins that don't come with core >> are close to worthless (don't get me wrong): > >> - They very unlikely will be installed in managed environments (an area >> growing

Re: [HACKERS] Built-in plugin for logical decoding output

On 09/25/2017 10:19 AM, Petr Jelinek wrote: On 25/09/17 18:48, Alvaro Hernandez wrote:     In my opinion, logical decoding plugins that don't come with core are close to worthless (don't get me wrong): I respectfully disagree. As do I. - They very unlikely will be installed in

Re: [HACKERS] Built-in plugin for logical decoding output

Alvaro Hernandez writes: > In my opinion, logical decoding plugins that don't come with core > are close to worthless (don't get me wrong): > - They very unlikely will be installed in managed environments (an area > growing significantly). > - As anything that is not in core,

Re: [HACKERS] Built-in plugin for logical decoding output

On 09/25/2017 10:15 AM, Gregory Brail wrote: Yes. I'm advocating something "built-in" to Postgres. Any or all of those are likely a great starting point. I don't see a benefit to having this "in postgres". The whole reason we have built out a mature and extensible product is so that not

Re: [HACKERS] Built-in plugin for logical decoding output

On 25/09/17 18:48, Alvaro Hernandez wrote: > > > On 25/09/17 19:39, Petr Jelinek wrote: >> >> Well, test_decoding is not meant for production use anyway, no need for >> middleware to support it. The pgoutput is primarily used for internal >> replication purposes, which is why we need something

Re: [HACKERS] Built-in plugin for logical decoding output

Hi, On 2017-09-25 12:56:00 -0400, Andrew Dunstan wrote: > A general purpose lower bandwidth plugin might one supporting Protocol > Buffers. The downside is that unlike json it's not self-contained, you > need the message definitions to interpret the stream, AIUI. I think that makes it a

Re: [HACKERS] Built-in plugin for logical decoding output

On 2017-09-24 13:36:56 +0300, Alvaro Hernandez wrote: >     However, if DMS uses it for what I'd call production use, I assume it is > actually production quality. I bet they do enough testing, and don't ship > software to potentially millions of customers if it doesn't work well. So... > first,

Re: [HACKERS] Built-in plugin for logical decoding output

Yes. I'm advocating something "built-in" to Postgres. Any or all of those are likely a great starting point. As for protobuf, I'm a big advocate -- it is easy to use, fast, extensible, runs on lots of platforms, and produces very compact output. However it introduces a few dependencies to the

Re: [HACKERS] Built-in plugin for logical decoding output

On 25/09/17 19:56, Andrew Dunstan wrote: On 09/25/2017 12:48 PM, Alvaro Hernandez wrote: On 25/09/17 19:39, Petr Jelinek wrote: Well, test_decoding is not meant for production use anyway, no need for middleware to support it. The pgoutput is primarily used for internal replication

Re: [HACKERS] Built-in plugin for logical decoding output

On 09/25/2017 09:59 AM, Gregory Brail wrote: However, I can't find any docs for the output format of pgoutput, which is going to make it less likely for people to be able to consume it. Is anyone working on docs? I know that it's a painful process. I also think that a JSON-format (or

Re: [HACKERS] Built-in plugin for logical decoding output

I'm encouraged that pgoutput exists and I'm sorry that I missed it before. I think it's fine as a binary-only format. If someone can write a client for the Postgres wire protocol as documented in Chapter 52 of the docs, then they should have no trouble consuming the output from pgoutput. However,

Re: [HACKERS] Built-in plugin for logical decoding output

On 09/25/2017 12:48 PM, Alvaro Hernandez wrote: > > > On 25/09/17 19:39, Petr Jelinek wrote: >> >> Well, test_decoding is not meant for production use anyway, no need for >> middleware to support it. The pgoutput is primarily used for internal >> replication purposes, which is why we need

Re: [HACKERS] Built-in plugin for logical decoding output

On 25/09/17 19:39, Petr Jelinek wrote: Well, test_decoding is not meant for production use anyway, no need for middleware to support it. The pgoutput is primarily used for internal replication purposes, which is why we need something with more interoperability in mind in the first place. The

Re: [HACKERS] Built-in plugin for logical decoding output

On 23/09/17 19:01, Alvaro Hernandez wro> On 23/09/17 18:42, Euler Taveira wrote: >> 2017-09-22 19:28 GMT-03:00 Gregory Brail : >>> We have been working on a project that makes extensive use of logical >>> replication for use inside Apigee (which is a very small part of >>>

Re: [HACKERS] Built-in plugin for logical decoding output

On 24 September 2017 at 15:15, Craig Ringer wrote: > On 23 September 2017 at 06:28, Gregory Brail wrote: > >> >> Would the community support the development of another plugin that is >> distributed as part of "contrib" that addresses these issues? > >

Re: [HACKERS] Built-in plugin for logical decoding output

On 24 September 2017 at 07:41, Euler Taveira wrote: > It is difficult to > coordinate a change like that having only one-way communication). > > I really think we need to fix that at some point, such that: * Downstream

Re: [HACKERS] Built-in plugin for logical decoding output

On 23 September 2017 at 06:28, Gregory Brail wrote: > Would the community support the development of another plugin that is > distributed as part of "contrib" that addresses these issues? > Petr Jelinek and I tried just that with pglogical. Our submission was knocked back

Re: [HACKERS] Built-in plugin for logical decoding output

On 24/09/17 02:41, Euler Taveira wrote: 2017-09-23 14:01 GMT-03:00 Alvaro Hernandez : However, AFAIK, AWS's DMS uses it for production purposes (see http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_PostgreSQL.html). It seems a bad idea. AFAICS test_decoding

Re: [HACKERS] Built-in plugin for logical decoding output

2017-09-23 14:01 GMT-03:00 Alvaro Hernandez : > However, AFAIK, AWS's DMS uses it for production purposes (see > http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_PostgreSQL.html). > It seems a bad idea. AFAICS test_decoding was not designed to be a

Re: [HACKERS] Built-in plugin for logical decoding output

On 23/09/17 18:42, Euler Taveira wrote: 2017-09-22 19:28 GMT-03:00 Gregory Brail : We have been working on a project that makes extensive use of logical replication for use inside Apigee (which is a very small part of Google): https://github.com/apigee-labs/transicator

Re: [HACKERS] Built-in plugin for logical decoding output

2017-09-22 19:28 GMT-03:00 Gregory Brail : > We have been working on a project that makes extensive use of logical > replication for use inside Apigee (which is a very small part of Google): > > https://github.com/apigee-labs/transicator > > In order to do this, we had to

Re: [HACKERS] Built-in plugin for logical decoding output

Hi, On 2017-09-22 17:11:47 -0700, Gregory Brail wrote: > Also in lieu of the new snapshot mechanism for logical replication, which > might not work for us This needs context... >, we were using the transaction ID to calculate what > was committed in a client's snapshot and what they need to

Re: [HACKERS] Built-in plugin for logical decoding output

Thanks! I didn't read the version 10 RC docs carefully enough. I found the code that generates the protocol message (looks like its replication/proto/logical.c). Are there docs somewhere on the format, or is it just the code? Also in lieu of the new snapshot mechanism for logical replication,

Re: [HACKERS] Built-in plugin for logical decoding output

On 23/09/17 00:28, Gregory Brail wrote: We have been working on a project that makes extensive use of logical replication for use inside Apigee (which is a very small part of Google): https://github.com/apigee-labs/transicator In order to do this, we had to write our own logical replication