Re: How to redirect output from PostgreSQL pg_recvlogical to a file or a pipe?

2024-01-16 Thread David Ventimiglia
Whoops! Wrong SO link. Here's the correct SO link: https://stackoverflow.com/questions/77808615/how-to-use-logical-decoding-with-pg-recvlogical-to-pass-changes-through-a-non-tr On Tue, Jan 16, 2024 at 1:15 PM David Ventimiglia < davidaventimig...@hasura.io> wrote: > Thanks for the reply, Jim.

Re: How to redirect output from PostgreSQL pg_recvlogical to a file or a pipe?

2024-01-16 Thread David Ventimiglia
Thanks for the reply, Jim. No, I'm afraid that's not the missing piece. I knew enough to use jq to transform the JSON output into SQL statements. What I didn't know enough was about jq. No, the missing piece turned out not to have anything to do with PostgreSQL or pg_recvlogical (I guessed incor

Re: How to redirect output from PostgreSQL pg_recvlogical to a file or a pipe?

2024-01-16 Thread Jim Nasby
On 1/13/24 3:34 PM, David Ventimiglia wrote: The business problem I'm trying to solve is: "How do I capture logical decoding events with the wal2json output encoder, filter them with jq, and pipe them to psql, using pg_recvlogical?" I think the missing piece here is that you can't simply pipe

Re: How to redirect output from PostgreSQL pg_recvlogical to a file or a pipe?

2024-01-13 Thread David Ventimiglia
I'm asking a question about technology. It has an answer. Whatever that answer is, it's independent of anyone's purpose. On Sat, Jan 13, 2024, 4:53 PM Ron Johnson wrote: > *No,* that's a technology problem. What is the purpose of storing them > back in the database using psql? > > On Sat, Jan 1

Re: How to redirect output from PostgreSQL pg_recvlogical to a file or a pipe?

2024-01-13 Thread David Ventimiglia
It satisfies business constraints. On Sat, Jan 13, 2024, 5:01 PM Karsten Hilbert wrote: > Am Sat, Jan 13, 2024 at 05:53:14PM -0500 schrieb Ron Johnson: > > > *No,* that's a technology problem. What is the purpose of storing them > > back in the database using psql? > > Or even the end goal to b

Re: How to redirect output from PostgreSQL pg_recvlogical to a file or a pipe?

2024-01-13 Thread Karsten Hilbert
Am Sat, Jan 13, 2024 at 05:53:14PM -0500 schrieb Ron Johnson: > *No,* that's a technology problem. What is the purpose of storing them > back in the database using psql? Or even the end goal to be achieved by that ? Karsten -- GPG 40BE 5B0E C98E 1713 AFA6 5BC0 3BEA AC80 7D4F C89B

Re: How to redirect output from PostgreSQL pg_recvlogical to a file or a pipe?

2024-01-13 Thread Ron Johnson
*No,* that's a technology problem. What is the purpose of storing them back in the database using psql? On Sat, Jan 13, 2024 at 4:34 PM David Ventimiglia < davidaventimig...@hasura.io> wrote: > The business problem I'm trying to solve is: > > "How do I capture logical decoding events with the wa

Re: How to redirect output from PostgreSQL pg_recvlogical to a file or a pipe?

2024-01-13 Thread David Ventimiglia
The business problem I'm trying to solve is: "How do I capture logical decoding events with the wal2json output encoder, filter them with jq, and pipe them to psql, using pg_recvlogical?" On Sat, Jan 13, 2024, 1:04 PM Ron Johnson wrote: > I think this might be an A-B problem. Tell us the "busi

Re: How to redirect output from PostgreSQL pg_recvlogical to a file or a pipe?

2024-01-13 Thread Ron Johnson
I think this might be an A-B problem. Tell us the "business problem" you are trying to solve, not the problem you're having with your solution to the "business problem". (If you've already mentioned it, please restate it.) On Sat, Jan 13, 2024 at 11:49 AM David Ventimiglia < davidaventimig...@ha

Re: How to redirect output from PostgreSQL pg_recvlogical to a file or a pipe?

2024-01-13 Thread Adrian Klaver
On 1/13/24 08:48, David Ventimiglia wrote: Thanks.  I'm aware of all of those other alternatives, but the thing is, I'm not trying to answer this broader question: /"What are some options for capturing change events in PostgreSQL?"/ / / Rather, I'm trying to answer a narrower question: /"How d

Re: How to redirect output from PostgreSQL pg_recvlogical to a file or a pipe?

2024-01-13 Thread David Ventimiglia
Thanks. I'm aware of all of those other alternatives, but the thing is, I'm not trying to answer this broader question: *"What are some options for capturing change events in PostgreSQL?"* Rather, I'm trying to answer a narrower question: *"How does one capture output from pg_recvlogical and pi

Re: How to redirect output from PostgreSQL pg_recvlogical to a file or a pipe?

2024-01-13 Thread Adrian Klaver
On 1/12/24 21:23, David Ventimiglia wrote: Let me just lay my cards on the table.  What I'm really trying to do is capture change events with logical decoding and then send them back into the database into a database table.  To do that, I believe I need to process the event records into SQL ins

Re: How to redirect output from PostgreSQL pg_recvlogical to a file or a pipe?

2024-01-12 Thread David Ventimiglia
Let me just lay my cards on the table. What I'm really trying to do is capture change events with logical decoding and then send them back into the database into a database table. To do that, I believe I need to process the event records into SQL insert statements somehow. xargs is one option.

Re: How to redirect output from PostgreSQL pg_recvlogical to a file or a pipe?

2024-01-12 Thread Adrian Klaver
On 1/12/24 14:03, David Ventimiglia wrote: Reply to list also. Ccing list Hi Adrian, I left out the creation of the slot both from the SO question and from this mailing list question, because I believe it's a red herring.  I run into the same problem with the default output plugin as I do wit

Re: How to redirect output from PostgreSQL pg_recvlogical to a file or a pipe?

2024-01-12 Thread Francisco Olarte
David: On Fri, 12 Jan 2024 at 20:35, David Ventimiglia wrote: > pg_recvlogical -d postgres --slot test --start -f - >> sample.jsonl > > Lest there be any confusion, I already created the slot in an earlier step. > Moreover, I can verify that if I omit the output redirection >> sample then > it

Re: How to redirect output from PostgreSQL pg_recvlogical to a file or a pipe?

2024-01-12 Thread Adrian Klaver
On 1/12/24 11:34, David Ventimiglia wrote: Hello! How do I redirect logical decoding output from the PostgreSQL CLI tool |pg_recvlogical| either to a file or to another command via a pipe? I ask because when I try the obvious, no output is recorded or sent: |pg_recvlogical -d postgres --slot t

How to redirect output from PostgreSQL pg_recvlogical to a file or a pipe?

2024-01-12 Thread David Ventimiglia
Hello! How do I redirect logical decoding output from the PostgreSQL CLI tool pg_recvlogical either to a file or to another command via a pipe? I ask because when I try the obvious, no output is recorded or sent: pg_recvlogical -d postgres --slot test --start -f - >> sample.jsonl Lest there be