Re: Non-text mode for pg_dumpall

2025-07-31 Thread Andrew Dunstan
On 2025-07-31 Th 2:22 PM, Nathan Bossart wrote: On Wed, Jul 30, 2025 at 02:51:59PM -0400, Andrew Dunstan wrote: OK, now that's reverted... Can we close the open item for this one now? Or is there something else remaining? Thanks for the reminder. I have marked the item as fixed. cheers

Re: Non-text mode for pg_dumpall

2025-07-31 Thread Nathan Bossart
On Wed, Jul 30, 2025 at 02:51:59PM -0400, Andrew Dunstan wrote: > OK, now that's reverted... Can we close the open item for this one now? Or is there something else remaining? -- nathan

Re: Non-text mode for pg_dumpall

2025-07-31 Thread Christoph Berg
Re: Andrew Dunstan > That file was deleted by the revert. Maybe you have a cache problem? Oh right. This was caused by our snapshot builds using the latest tarball (if available) and putting a patch on top that. I've now bumped the upstream version to 18~beta3, this should avoid the problem. Sorr

Re: Non-text mode for pg_dumpall

2025-07-31 Thread Andrew Dunstan
On 2025-07-31 Th 5:44 AM, Christoph Berg wrote: The 18 branch is broken for apt.pg.o: 00:54:18 # Failed test 'dump_globals_only: pg_dumpall runs' 00:54:18 # at t/006_pg_dumpall.pl line 329. 00:54:18 # Tests were run but no plan was declared and done_testing() was not seen. 00:54:18 # Look

Re: Non-text mode for pg_dumpall

2025-07-31 Thread Christoph Berg
The 18 branch is broken for apt.pg.o: 00:54:18 # Failed test 'dump_globals_only: pg_dumpall runs' 00:54:18 # at t/006_pg_dumpall.pl line 329. 00:54:18 # Tests were run but no plan was declared and done_testing() was not seen. 00:54:18 # Looks like your test exited with 2 just after 1. 00:54:1

Re: Non-text mode for pg_dumpall

2025-07-30 Thread Andrew Dunstan
On 2025-07-29 Tu 4:09 PM, Andrew Dunstan wrote: On 2025-07-28 Mo 8:04 AM, Andrew Dunstan wrote: On 2025-07-27 Su 7:56 PM, Noah Misch wrote: On Fri, Jul 25, 2025 at 04:59:29PM -0400, Tom Lane wrote: Andrew Dunstan writes: Before we throw the baby out with the bathwater, how about this sug

Re: Non-text mode for pg_dumpall

2025-07-29 Thread Andrew Dunstan
On 2025-07-29 Tu 4:34 PM, Noah Misch wrote: On Tue, Jul 29, 2025 at 04:09:13PM -0400, Andrew Dunstan wrote: here's a reversion patch for master. This reverts parts or all of the following commits: I briefly looked through this. The biggest non-reverted part is, I think, c1da728 "Move co

Re: Non-text mode for pg_dumpall

2025-07-29 Thread Noah Misch
On Tue, Jul 29, 2025 at 04:09:13PM -0400, Andrew Dunstan wrote: > here's a reversion patch for master. > This reverts parts or all of the following commits: I briefly looked through this. The biggest non-reverted part is, I think, c1da728 "Move common pg_dump code related to connections to a

Re: Non-text mode for pg_dumpall

2025-07-29 Thread Andrew Dunstan
On 2025-07-28 Mo 8:04 AM, Andrew Dunstan wrote: On 2025-07-27 Su 7:56 PM, Noah Misch wrote: On Fri, Jul 25, 2025 at 04:59:29PM -0400, Tom Lane wrote: Andrew Dunstan writes: Before we throw the baby out with the bathwater, how about this suggestion? pg_dumpall would continue to produce globa

Re: Non-text mode for pg_dumpall

2025-07-28 Thread Andrew Dunstan
On 2025-07-27 Su 7:56 PM, Noah Misch wrote: On Fri, Jul 25, 2025 at 04:59:29PM -0400, Tom Lane wrote: Andrew Dunstan writes: Before we throw the baby out with the bathwater, how about this suggestion? pg_dumpall would continue to produce globals.dat, but it wouldn't be processed by pg_restor

Re: Non-text mode for pg_dumpall

2025-07-27 Thread Noah Misch
On Fri, Jul 25, 2025 at 04:59:29PM -0400, Tom Lane wrote: > Andrew Dunstan writes: > > Before we throw the baby out with the bathwater, how about this > > suggestion? pg_dumpall would continue to produce globals.dat, but it > > wouldn't be processed by pg_restore, which would only restore the >

Re: Non-text mode for pg_dumpall

2025-07-25 Thread Tom Lane
Andrew Dunstan writes: > Before we throw the baby out with the bathwater, how about this > suggestion? pg_dumpall would continue to produce globals.dat, but it > wouldn't be processed by pg_restore, which would only restore the > individual databases. Or else we just don't produce globals.dat a

Re: Non-text mode for pg_dumpall

2025-07-25 Thread Andrew Dunstan
On 2025-07-25 Fr 12:21 PM, Noah Misch wrote: On Thu, Jul 24, 2025 at 04:33:15PM -0400, Andrew Dunstan wrote: On 2025-07-21 Mo 8:53 PM, Noah Misch wrote: I suspect this is going to end with a structured dump like we use on the pg_dump (per-database) side. It's not an accident that v17 pg_resto

Re: Non-text mode for pg_dumpall

2025-07-25 Thread Noah Misch
On Thu, Jul 24, 2025 at 04:33:15PM -0400, Andrew Dunstan wrote: > On 2025-07-21 Mo 8:53 PM, Noah Misch wrote: > > I suspect this is going to end with a structured dump like we use on the > > pg_dump (per-database) side. It's not an accident that v17 pg_restore > > doesn't > > lex text files to do

Re: Non-text mode for pg_dumpall

2025-07-24 Thread Andrew Dunstan
On 2025-07-21 Mo 8:53 PM, Noah Misch wrote: I suspect this is going to end with a structured dump like we use on the pg_dump (per-database) side. It's not an accident that v17 pg_restore doesn't lex text files to do its job. pg_dumpall deals with a more-limited set of statements than pg_dump

Re: Non-text mode for pg_dumpall

2025-07-24 Thread Robert Haas
On Wed, Jul 9, 2025 at 2:51 PM Mahendra Singh Thalor wrote: > > This drops all databases: > > > > pg_dumpall --clean -Fd -f /tmp/dump > > pg_restore -d template1 --globals-only /tmp/dump > > > > That didn't match my expectations given this help text: > > > > $ pg_restore --help|grep global > > -

Re: Non-text mode for pg_dumpall

2025-07-24 Thread Noah Misch
On Thu, Jul 17, 2025 at 03:46:41PM +0530, Mahendra Singh Thalor wrote: > On Wed, 16 Jul 2025 at 05:50, Noah Misch wrote: > > On Thu, Jul 10, 2025 at 12:21:03AM +0530, Mahendra Singh Thalor wrote: > > > On Wed, 9 Jul 2025 at 02:58, Noah Misch wrote: > > > > On Fri, Apr 04, 2025 at 04:11:05PM -0400

Re: Non-text mode for pg_dumpall

2025-07-21 Thread Andrew Dunstan
On 2025-07-21 Mo 8:53 PM, Noah Misch wrote: On Mon, Jul 21, 2025 at 04:41:03PM -0400, Andrew Dunstan wrote: On 2025-07-17 Th 6:18 AM, Mahendra Singh Thalor wrote --- a/src/bin/pg_dump/pg_restore.c +++ b/src/bin/pg_dump/pg_restore.c +/* + * read_one_statement + * + * This will start reading fr

Re: Non-text mode for pg_dumpall

2025-07-21 Thread Noah Misch
On Mon, Jul 21, 2025 at 04:41:03PM -0400, Andrew Dunstan wrote: > On 2025-07-17 Th 6:18 AM, Mahendra Singh Thalor wrote > > > > > > > --- a/src/bin/pg_dump/pg_restore.c > > > > > > > +++ b/src/bin/pg_dump/pg_restore.c > > > > > > > +/* > > > > > > > + * read_one_statement > > > > > > > + * > > > >

Re: Non-text mode for pg_dumpall

2025-07-21 Thread Andrew Dunstan
On 2025-07-17 Th 6:18 AM, Mahendra Singh Thalor wrote --- a/src/bin/pg_dump/pg_restore.c +++ b/src/bin/pg_dump/pg_restore.c +/* + * read_one_statement + * + * This will start reading from passed file pointer using fgetc and read till + * semicolon(sql statement terminator for global.dat file) +

Re: Non-text mode for pg_dumpall

2025-07-17 Thread Andrew Dunstan
On 2025-07-17 Th 7:11 AM, Álvaro Herrera wrote: On 2025-Jul-17, Mahendra Singh Thalor wrote: To pg_restore, we are giving a dump of pg_dumpall which has a global.dat file and we have drop commands in the global.dat file so when we are using 'globals-only', we are dropping databases as we have

Re: Non-text mode for pg_dumpall

2025-07-17 Thread Mahendra Singh Thalor
Thanks Álvaro for the feedback. On Thu, 17 Jul 2025 at 16:41, Álvaro Herrera wrote: > > On 2025-Jul-17, Mahendra Singh Thalor wrote: > > > To pg_restore, we are giving a dump of pg_dumpall which has a > > global.dat file and we have drop commands in the global.dat file so > > when we are using 'g

Re: Non-text mode for pg_dumpall

2025-07-17 Thread Álvaro Herrera
On 2025-Jul-17, Mahendra Singh Thalor wrote: > To pg_restore, we are giving a dump of pg_dumpall which has a > global.dat file and we have drop commands in the global.dat file so > when we are using 'globals-only', we are dropping databases as we have > DROP commands. > As of now, we don't have an

Re: Non-text mode for pg_dumpall

2025-07-17 Thread Mahendra Singh Thalor
Attaching the correct patch. Sorry, I attached the wrong patch in my last email. On Thu, 17 Jul 2025 at 15:46, Mahendra Singh Thalor wrote: > > Thanks Noah for the feedback. > > On Wed, 16 Jul 2025 at 05:50, Noah Misch wrote: > > > > On Thu, Jul 10, 2025 at 12:21:03AM +0530, Mahendra Singh Tha

Re: Non-text mode for pg_dumpall

2025-07-17 Thread Mahendra Singh Thalor
Thanks Noah for the feedback. On Wed, 16 Jul 2025 at 05:50, Noah Misch wrote: > > On Thu, Jul 10, 2025 at 12:21:03AM +0530, Mahendra Singh Thalor wrote: > > On Wed, 9 Jul 2025 at 02:58, Noah Misch wrote: > > > On Fri, Apr 04, 2025 at 04:11:05PM -0400, Andrew Dunstan wrote: > > > > Thanks. I have

Re: Non-text mode for pg_dumpall

2025-07-15 Thread Noah Misch
On Thu, Jul 10, 2025 at 12:21:03AM +0530, Mahendra Singh Thalor wrote: > On Wed, 9 Jul 2025 at 02:58, Noah Misch wrote: > > On Fri, Apr 04, 2025 at 04:11:05PM -0400, Andrew Dunstan wrote: > > > Thanks. I have pushed these now with a few further small tweaks. > > > > This drops all databases: > > >

Re: Non-text mode for pg_dumpall

2025-07-09 Thread Mahendra Singh Thalor
Thanks Noah for the comments. On Wed, 9 Jul 2025 at 02:58, Noah Misch wrote: > > On Fri, Apr 04, 2025 at 04:11:05PM -0400, Andrew Dunstan wrote: > > Thanks. I have pushed these now with a few further small tweaks. > > This drops all databases: > > pg_dumpall --clean -Fd -f /tmp/dump > pg_restore

Re: Non-text mode for pg_dumpall

2025-07-08 Thread Noah Misch
On Fri, Apr 04, 2025 at 04:11:05PM -0400, Andrew Dunstan wrote: > Thanks. I have pushed these now with a few further small tweaks. This drops all databases: pg_dumpall --clean -Fd -f /tmp/dump pg_restore -d template1 --globals-only /tmp/dump That didn't match my expectations given this help text

Re: Non-text mode for pg_dumpall

2025-04-15 Thread Andrew Dunstan
On 2025-04-15 Tu 2:30 PM, Mahendra Singh Thalor wrote: Hi Andrew, I did some refactoring to find out dump file extensions(.dmp/.tar etc) in pg_restore. With the attached patch, we will not try to find out file extension with each database, rather we will find out before the loop. Here, I am at

Re: Non-text mode for pg_dumpall

2025-04-15 Thread Mahendra Singh Thalor
On Sat, 5 Apr 2025 at 01:41, Andrew Dunstan wrote: > > > On 2025-04-04 Fr 5:12 AM, Mahendra Singh Thalor wrote: > > On Fri, 4 Apr 2025 at 13:52, Mahendra Singh Thalor wrote: > > On Fri, 4 Apr 2025 at 01:17, Andrew Dunstan wrote: > > On 2025-04-01 Tu 1:59 AM, Mahendra Singh Thalor wrote: > > On M

Re: Non-text mode for pg_dumpall

2025-04-12 Thread Ranier Vilela
Hi Andrew. I just saw the fix commit. My fault. I'm sorry. best regards, Ranier Vilela

Re: Non-text mode for pg_dumpall

2025-04-10 Thread Ranier Vilela
Em qui., 10 de abr. de 2025 20:09, Andrew Dunstan escreveu: > > On 2025-04-10 Th 5:45 PM, Ranier Vilela wrote: > > > Em qui., 10 de abr. de 2025 às 15:58, Andrew Dunstan > escreveu: > >> >> On 2025-04-10 Th 2:38 PM, Ranier Vilela wrote: >> >> >>> >>> Thanks. I have pushed these now with a few fu

Re: Non-text mode for pg_dumpall

2025-04-10 Thread Andrew Dunstan
On 2025-04-10 Th 5:45 PM, Ranier Vilela wrote: Em qui., 10 de abr. de 2025 às 15:58, Andrew Dunstan escreveu: On 2025-04-10 Th 2:38 PM, Ranier Vilela wrote: Thanks. I have pushed these now with a few further small tweaks. Sorry if it is not the right place. Coverit

Re: Non-text mode for pg_dumpall

2025-04-10 Thread Ranier Vilela
Em qui., 10 de abr. de 2025 às 15:58, Andrew Dunstan escreveu: > > On 2025-04-10 Th 2:38 PM, Ranier Vilela wrote: > > >> >> Thanks. I have pushed these now with a few further small tweaks. >> > Sorry if it is not the right place. > Coverity has another resource leak alert. > > trivial patch attac

Re: Non-text mode for pg_dumpall

2025-04-10 Thread Andrew Dunstan
On 2025-04-10 Th 2:38 PM, Ranier Vilela wrote: Thanks. I have pushed these now with a few further small tweaks. Sorry if it is not the right place. Coverity has another resource leak alert. trivial patch attached. Thanks for checking. Pushed. cheers andrew -- Andrew Dunstan ED

Re: Non-text mode for pg_dumpall

2025-04-10 Thread Ranier Vilela
Hi. Em sex., 4 de abr. de 2025 às 17:11, Andrew Dunstan escreveu: > > On 2025-04-04 Fr 5:12 AM, Mahendra Singh Thalor wrote: > > On Fri, 4 Apr 2025 at 13:52, Mahendra Singh Thalor > wrote: > > On Fri, 4 Apr 2025 at 01:17, Andrew Dunstan > wrote: > > On 2025-04-01 Tu 1:59 AM, Mahendra Singh

Re: Non-text mode for pg_dumpall

2025-04-10 Thread Mahendra Singh Thalor
On Sat, 5 Apr 2025 at 01:41, Andrew Dunstan wrote: > > > On 2025-04-04 Fr 5:12 AM, Mahendra Singh Thalor wrote: > > On Fri, 4 Apr 2025 at 13:52, Mahendra Singh Thalor wrote: > > On Fri, 4 Apr 2025 at 01:17, Andrew Dunstan wrote: > > On 2025-04-01 Tu 1:59 AM, Mahendra Singh Thalor wrote: > > On M

Re: Non-text mode for pg_dumpall

2025-04-05 Thread Mahendra Singh Thalor
On Fri, 4 Apr 2025 at 01:17, Andrew Dunstan wrote: > > > On 2025-04-01 Tu 1:59 AM, Mahendra Singh Thalor wrote: > > On Mon, 31 Mar 2025 at 23:43, Álvaro Herrera > > wrote: > >> Hi > >> > >> FWIW I don't think the on_exit_nicely business is in final shape just > >> yet. We're doing something sup

Re: Non-text mode for pg_dumpall

2025-04-05 Thread Mahendra Singh Thalor
On Mon, 31 Mar 2025 at 19:27, Andrew Dunstan wrote: > > > On 2025-03-31 Mo 5:34 AM, Mahendra Singh Thalor wrote: > > > > > > > > There are a couple of rough edges, though. > > > > > > First, I see this: > > > > > > > > > andrew@ub22arm:inst $ bin/pg_restore -C -d postgres > > > --exclude-database=

Re: Non-text mode for pg_dumpall

2025-04-05 Thread Mahendra Singh Thalor
On Fri, 4 Apr 2025 at 13:52, Mahendra Singh Thalor wrote: > > On Fri, 4 Apr 2025 at 01:17, Andrew Dunstan wrote: > > > > > > On 2025-04-01 Tu 1:59 AM, Mahendra Singh Thalor wrote: > > > On Mon, 31 Mar 2025 at 23:43, Álvaro Herrera > > > wrote: > > >> Hi > > >> > > >> FWIW I don't think the on_e

Re: Non-text mode for pg_dumpall

2025-04-05 Thread Mahendra Singh Thalor
On Sun, 30 Mar 2025 at 22:20, Andrew Dunstan wrote: > > > On 2025-03-29 Sa 1:17 AM, Mahendra Singh Thalor wrote: > > On Sat, 29 Mar 2025 at 03:50, Andrew Dunstan wrote: > >> > >> On 2025-03-27 Th 5:15 PM, Andrew Dunstan wrote: > >>> On 2025-03-19 We 2:41 AM, Mahendra Singh Thalor wrote: > On

Re: Non-text mode for pg_dumpall

2025-04-05 Thread Andrew Dunstan
On 2025-03-31 Mo 12:16 PM, Mahendra Singh Thalor wrote: On Mon, 31 Mar 2025 at 19:27, Andrew Dunstan wrote: On 2025-03-31 Mo 5:34 AM, Mahendra Singh Thalor wrote: There are a couple of rough edges, though. First, I see this: andrew@ub22arm:inst $ bin/pg_restore -C -d postgres --exclude-d

Re: Non-text mode for pg_dumpall

2025-04-04 Thread Andrew Dunstan
On 2025-04-04 Fr 5:12 AM, Mahendra Singh Thalor wrote: On Fri, 4 Apr 2025 at 13:52, Mahendra Singh Thalor wrote: On Fri, 4 Apr 2025 at 01:17, Andrew Dunstan wrote: On 2025-04-01 Tu 1:59 AM, Mahendra Singh Thalor wrote: On Mon, 31 Mar 2025 at 23:43, Álvaro Herrera wrote: Hi FWIW I don't thi

Re: Non-text mode for pg_dumpall

2025-04-03 Thread Andrew Dunstan
On 2025-04-01 Tu 1:59 AM, Mahendra Singh Thalor wrote: On Mon, 31 Mar 2025 at 23:43, Álvaro Herrera wrote: Hi FWIW I don't think the on_exit_nicely business is in final shape just yet. We're doing something super strange and novel about keeping track of an array index, so that we can modify

Re: Non-text mode for pg_dumpall

2025-03-31 Thread Mahendra Singh Thalor
On Mon, 31 Mar 2025 at 23:43, Álvaro Herrera wrote: > > Hi > > FWIW I don't think the on_exit_nicely business is in final shape just > yet. We're doing something super strange and novel about keeping track > of an array index, so that we can modify it later. Or something like > that, I think? T

Re: Non-text mode for pg_dumpall

2025-03-31 Thread Andrew Dunstan
On 2025-03-31 Mo 5:34 AM, Mahendra Singh Thalor wrote: > > There are a couple of rough edges, though. > > First, I see this: > > > andrew@ub22arm:inst $ bin/pg_restore -C -d postgres > --exclude-database=regression_dummy_seclabel > --exclude-database=regression_test_extensions > --exclude-data

Re: Non-text mode for pg_dumpall

2025-03-31 Thread Álvaro Herrera
Hi FWIW I don't think the on_exit_nicely business is in final shape just yet. We're doing something super strange and novel about keeping track of an array index, so that we can modify it later. Or something like that, I think? That doesn't sound all that nice to me. Elsewhere it was suggested

Re: Non-text mode for pg_dumpall

2025-03-31 Thread Andrew Dunstan
On 2025-03-31 Mo 1:16 PM, Andrew Dunstan wrote: Thanks. Here are patches that contain (my version of) all the cleanups. With this I get a clean restore run in my test case with no error messages. This time with patches cheers andrew -- Andrew Dunstan EDB: https://www.enterprised

Re: Non-text mode for pg_dumpall

2025-03-30 Thread Andrew Dunstan
On 2025-03-30 Su 12:50 PM, Andrew Dunstan wrote: On 2025-03-29 Sa 1:17 AM, Mahendra Singh Thalor wrote: On Sat, 29 Mar 2025 at 03:50, Andrew Dunstan wrote: On 2025-03-27 Th 5:15 PM, Andrew Dunstan wrote: On 2025-03-19 We 2:41 AM, Mahendra Singh Thalor wrote: On Wed, 12 Mar 2025 at 21:18,

Re: Non-text mode for pg_dumpall

2025-03-30 Thread Andrew Dunstan
On 2025-03-29 Sa 1:17 AM, Mahendra Singh Thalor wrote: On Sat, 29 Mar 2025 at 03:50, Andrew Dunstan wrote: On 2025-03-27 Th 5:15 PM, Andrew Dunstan wrote: On 2025-03-19 We 2:41 AM, Mahendra Singh Thalor wrote: On Wed, 12 Mar 2025 at 21:18, Andrew Dunstan wrote: On 2025-03-12 We 3:03 AM, j

Re: Non-text mode for pg_dumpall

2025-03-28 Thread Mahendra Singh Thalor
On Sat, 29 Mar 2025 at 03:50, Andrew Dunstan wrote: > > > On 2025-03-27 Th 5:15 PM, Andrew Dunstan wrote: > > > > On 2025-03-19 We 2:41 AM, Mahendra Singh Thalor wrote: > >> On Wed, 12 Mar 2025 at 21:18, Andrew Dunstan > >> wrote: > >>> > >>> On 2025-03-12 We 3:03 AM, jian he wrote: > On Wed

Re: Non-text mode for pg_dumpall

2025-03-27 Thread Andrew Dunstan
On 2025-03-19 We 2:41 AM, Mahendra Singh Thalor wrote: On Wed, 12 Mar 2025 at 21:18, Andrew Dunstan wrote: On 2025-03-12 We 3:03 AM, jian he wrote: On Wed, Mar 12, 2025 at 1:06 AM Álvaro Herrera wrote: Hello, On 2025-Mar-11, Mahendra Singh Thalor wrote: In map.dat file, I tried to fix t

Re: Non-text mode for pg_dumpall

2025-03-18 Thread Mahendra Singh Thalor
On Wed, 12 Mar 2025 at 21:18, Andrew Dunstan wrote: > > > On 2025-03-12 We 3:03 AM, jian he wrote: > > On Wed, Mar 12, 2025 at 1:06 AM Álvaro Herrera > > wrote: > >> Hello, > >> > >> On 2025-Mar-11, Mahendra Singh Thalor wrote: > >> > >>> In map.dat file, I tried to fix this issue by adding numb

Re: Non-text mode for pg_dumpall

2025-03-12 Thread Andrew Dunstan
On 2025-03-12 We 3:03 AM, jian he wrote: On Wed, Mar 12, 2025 at 1:06 AM Álvaro Herrera wrote: Hello, On 2025-Mar-11, Mahendra Singh Thalor wrote: In map.dat file, I tried to fix this issue by adding number of characters in dbname but as per code comments, as of now, we are not supporting

Re: Non-text mode for pg_dumpall

2025-03-12 Thread Laurenz Albe
On Tue, 2025-03-11 at 19:14 -0400, Isaac Morland wrote: > On Tue, 11 Mar 2025 at 18:37, Andrew Dunstan wrote: > > > Well, JSON is supposed to be UTF8. What should we do about database > > names that are not UTF8? > > How can you have a database name that isn't encodeable in UTF-8? At this point

Re: Non-text mode for pg_dumpall

2025-03-12 Thread jian he
On Wed, Mar 12, 2025 at 1:06 AM Álvaro Herrera wrote: > > Hello, > > On 2025-Mar-11, Mahendra Singh Thalor wrote: > > > In map.dat file, I tried to fix this issue by adding number of characters > > in dbname but as per code comments, as of now, we are not supporting \n\r > > in dbnames so i remove

Re: Non-text mode for pg_dumpall

2025-03-11 Thread Isaac Morland
On Tue, 11 Mar 2025 at 18:37, Andrew Dunstan wrote: Well, JSON is supposed to be UTF8. What should we do about database > names that are not UTF8? > How can you have a database name that isn't encodeable in UTF-8? At this point I'm pretty sure Unicode has subsumed essentially every character eve

Re: Non-text mode for pg_dumpall

2025-03-11 Thread Andrew Dunstan
On 2025-03-11 Tu 5:03 PM, Álvaro Herrera wrote: On 2025-Mar-11, Andrew Dunstan wrote: I'm not sure everyone thinks that was a good idea for backup manifests (in fact I know some don't), and it seems somewhat like overkill for a simple map of oids to database names. If such a simple system ca

Re: Non-text mode for pg_dumpall

2025-03-11 Thread Álvaro Herrera
On 2025-Mar-11, Andrew Dunstan wrote: > I'm not sure everyone thinks that was a good idea for backup manifests (in > fact I know some don't), and it seems somewhat like overkill for a simple > map of oids to database names. If such a simple system can be made to work for all possible valid databa

Re: Non-text mode for pg_dumpall

2025-03-11 Thread Andrew Dunstan
On 2025-03-11 Tu 1:52 PM, Dagfinn Ilmari Mannsåker wrote: Álvaro Herrera writes: Hello, On 2025-Mar-11, Mahendra Singh Thalor wrote: In map.dat file, I tried to fix this issue by adding number of characters in dbname but as per code comments, as of now, we are not supporting \n\r in dbnam

Re: Non-text mode for pg_dumpall

2025-03-11 Thread Dagfinn Ilmari Mannsåker
Álvaro Herrera writes: > Hello, > > On 2025-Mar-11, Mahendra Singh Thalor wrote: > >> In map.dat file, I tried to fix this issue by adding number of characters >> in dbname but as per code comments, as of now, we are not supporting \n\r >> in dbnames so i removed handling. >> I will do some more

Re: Non-text mode for pg_dumpall

2025-03-11 Thread Álvaro Herrera
Hello, On 2025-Mar-11, Mahendra Singh Thalor wrote: > In map.dat file, I tried to fix this issue by adding number of characters > in dbname but as per code comments, as of now, we are not supporting \n\r > in dbnames so i removed handling. > I will do some more study to fix this issue. Yeah, I t

Re: Non-text mode for pg_dumpall

2025-03-11 Thread Mahendra Singh Thalor
On Tue, 11 Mar 2025 at 20:12, Álvaro Herrera wrote: > > On 2025-Mar-11, Mahendra Singh Thalor wrote: > > > On Wed, 5 Mar 2025 at 20:42, Álvaro Herrera wrote: > > > > Okay, we should probably fix that, but I think the new map.dat file your > > > patch adds is going to make the problem worse, becau

Re: Non-text mode for pg_dumpall

2025-03-11 Thread Mahendra Singh Thalor
Thanks Alvaro and Jian for the review and feedback. On Wed, 5 Mar 2025 at 20:42, Álvaro Herrera wrote: > > Disclaimer: I didn't review these patches fully. > > On 2025-Mar-05, Mahendra Singh Thalor wrote: > > > On Wed, 5 Mar 2025 at 01:02, Álvaro Herrera wrote: > > > > > A database name containi

Re: Non-text mode for pg_dumpall

2025-03-11 Thread Álvaro Herrera
On 2025-Mar-11, Mahendra Singh Thalor wrote: > On Wed, 5 Mar 2025 at 20:42, Álvaro Herrera wrote: > > Okay, we should probably fix that, but I think the new map.dat file your > > patch adds is going to make the problem worse, because it doesn't look > > like you handled that case in any particul

Re: Non-text mode for pg_dumpall

2025-03-10 Thread jian he
On Thu, Mar 6, 2025 at 12:49 AM Mahendra Singh Thalor wrote: > > Thanks Alvaro for feedback and review. > > On Wed, 5 Mar 2025 at 20:42, Álvaro Herrera wrote: > > > > Disclaimer: I didn't review these patches fully. > > > > On 2025-Mar-05, Mahendra Singh Thalor wrote: > > > > > On Wed, 5 Mar 2025

Re: Non-text mode for pg_dumpall

2025-03-05 Thread Mahendra Singh Thalor
Thanks Alvaro for feedback and review. On Wed, 5 Mar 2025 at 20:42, Álvaro Herrera wrote: > > Disclaimer: I didn't review these patches fully. > > On 2025-Mar-05, Mahendra Singh Thalor wrote: > > > On Wed, 5 Mar 2025 at 01:02, Álvaro Herrera wrote: > > > > > A database name containing a newline

Re: Non-text mode for pg_dumpall

2025-03-05 Thread Álvaro Herrera
Disclaimer: I didn't review these patches fully. On 2025-Mar-05, Mahendra Singh Thalor wrote: > On Wed, 5 Mar 2025 at 01:02, Álvaro Herrera wrote: > > > A database name containing a newline breaks things for this patch: > > > > CREATE DATABASE "foo > > bar"; > I also reported this issue on 29-0

Re: Non-text mode for pg_dumpall

2025-03-04 Thread Mahendra Singh Thalor
On Wed, 5 Mar 2025 at 01:02, Álvaro Herrera wrote: > > A database name containing a newline breaks things for this patch: > > CREATE DATABASE "foo > bar"; > > > $ pg_dumpall -Fc --file test > shell command argument contains a newline or carriage return: " dbname='foo > bar'" > > -- > Álvaro Herrer

Re: Non-text mode for pg_dumpall

2025-03-04 Thread Álvaro Herrera
A database name containing a newline breaks things for this patch: CREATE DATABASE "foo bar"; $ pg_dumpall -Fc --file test shell command argument contains a newline or carriage return: " dbname='foo bar'" -- Álvaro Herrera 48°01'N 7°57'E — https://www.EnterpriseDB.com/

Re: Non-text mode for pg_dumpall

2025-02-24 Thread jian he
hi. some documentation issue: doc/src/sgml/ref/pg_dumpall.sgml d directory Output a directory-format archive suitable for input into pg_restore. Under dboid subdirectory, this will create a directory with one file for ea

Re: Non-text mode for pg_dumpall

2025-02-22 Thread jian he
hi. v20-0001 in src/bin/pg_dump/pg_dumpall.c, we have: static const char *connstr = ""; case 'd': connstr = pg_strdup(optarg); break; i am not sure you can declare it as "const" for connstr. since connstr value can be changed. ``#include "pg_backup.h"`

Re: Non-text mode for pg_dumpall

2025-02-20 Thread Mahendra Singh Thalor
On Thu, 20 Feb 2025 at 14:48, jian he wrote: > > hi. > about 0001 > > /* > * connectDatabase > * > * Make a database connection with the given parameters. An > * interactive password prompt is automatically issued if required. > * > * If fail_on_error is false, we return NULL without printi

Re: Non-text mode for pg_dumpall

2025-02-20 Thread jian he
hi. about 0001 /* * connectDatabase * * Make a database connection with the given parameters. An * interactive password prompt is automatically issued if required. * * If fail_on_error is false, we return NULL without printing any message * on failure, but preserve any prompted password fo

Re: Non-text mode for pg_dumpall

2025-02-19 Thread Mahendra Singh Thalor
Thanks Álvaro for feedback. On Thu, 20 Feb 2025 at 02:39, Álvaro Herrera wrote: > > Hello, > > I think the business with an evergrowing on_exit list needs a different > solution than a gigantic array of entries. Maybe it would make sense to > restructure that code so that there's a single on_exi

Re: Non-text mode for pg_dumpall

2025-02-19 Thread Álvaro Herrera
Hello, I think the business with an evergrowing on_exit list needs a different solution than a gigantic array of entries. Maybe it would make sense to restructure that code so that there's a single on_exit item, but there exists a list of per-database entries to clean up which are all done in one

Re: Non-text mode for pg_dumpall

2025-02-19 Thread Mahendra Singh Thalor
On Tue, 18 Feb 2025 at 10:00, Srinath Reddy wrote: > > Hi, > i think during restore we should not force user to use -C during cases like > ./pg_restore pdd -g -f - > ./pg_restore pdd -a -f - > ./pg_restore pdd -s -f - > because its not good to use -C to create database every time when we are > us

Re: Non-text mode for pg_dumpall

2025-02-19 Thread jian he
hi. Currently, pg_retore says --exit-on-error Exit if an error is encountered while sending SQL commands to the database. The default is to continue and to display a count of errors at the end of the restoration. Do we need to apply this to restore executing global commands (create role, create ta

Re: Non-text mode for pg_dumpall

2025-02-19 Thread Mahendra Singh Thalor
On Tue, 18 Feb 2025 at 10:00, Srinath Reddy wrote: > > Hi, > i think during restore we should not force user to use -C during cases like > ./pg_restore pdd -g -f - > ./pg_restore pdd -a -f - > ./pg_restore pdd -s -f - > because its not good to use -C to create database every time when we are > us

Re: Non-text mode for pg_dumpall

2025-02-18 Thread jian he
On Tue, Feb 18, 2025 at 2:10 PM jian he wrote: > > hi. hi. more cosmetic minor issues. +static int +get_dbname_oid_list_from_mfile(const char *dumpdirpath, SimpleDatabaseOidList *dbname_oid_list) ... + /* + * XXX : before adding dbname into list, we can verify that this db + * needs to skipped f

Re: Non-text mode for pg_dumpall

2025-02-17 Thread jian he
hi. pg_restore restore a PostgreSQL database from an archive file created by pg_dump or restore multiple PostgreSQL database from an archive directory created by pg_dumpall i think it's way too verbose. we can change it to: restore PostgreSQL database from an archiv

Re: Non-text mode for pg_dumpall

2025-02-17 Thread Srinath Reddy
Hi, i think during restore we should not force user to use -C during cases like ./pg_restore pdd -g -f - ./pg_restore pdd -a -f - ./pg_restore pdd -s -f - because its not good to use -C to create database every time when we are using these options individually. latest patch throws following error f

Re: Non-text mode for pg_dumpall

2025-02-13 Thread Mahendra Singh Thalor
Thanks Jian. On Wed, 12 Feb 2025 at 12:45, jian he wrote: > > On Wed, Feb 12, 2025 at 1:17 AM Mahendra Singh Thalor > wrote: > > > > > > > > There are some tests per https://commitfest.postgresql.org/52/5495, I > > > will check it later. > > hi. > the cfbot failure is related to function _tocEnt

Re: Non-text mode for pg_dumpall

2025-02-11 Thread jian he
On Wed, Feb 12, 2025 at 1:17 AM Mahendra Singh Thalor wrote: > > > > > There are some tests per https://commitfest.postgresql.org/52/5495, I > > will check it later. hi. the cfbot failure is related to function _tocEntryRequired if (strcmp(te->desc, "DATABASE") == 0 || strcmp(te->desc, "DATA

Re: Non-text mode for pg_dumpall

2025-02-11 Thread Mahendra Singh Thalor
On Tue, 11 Feb 2025 at 20:40, jian he wrote: > > hi. > review based on v16. > > because of > https://postgr.es/m/cafc+b6pwqisl+3rvlxn9vhc8aonp4ov9c6u+bvd6kmwmdbd...@mail.gmail.com > > in copy_global_file_to_out_file, now it is: > if (strcmp(outfile, "-") == 0) > OPF = stdout; > I am co

Re: Non-text mode for pg_dumpall

2025-02-11 Thread jian he
hi. review based on v16. because of https://postgr.es/m/cafc+b6pwqisl+3rvlxn9vhc8aonp4ov9c6u+bvd6kmwmdbd...@mail.gmail.com in copy_global_file_to_out_file, now it is: if (strcmp(outfile, "-") == 0) OPF = stdout; I am confused, why "-" means stdout. ``touch ./- `` command works fine. i

Re: Non-text mode for pg_dumpall

2025-02-10 Thread Mahendra Singh Thalor
Thanks Jian. On Tue, 4 Feb 2025 at 07:35, jian he wrote: > > hi. > > just a quick response for v15. > > the pg_restore man page says option --list as "List the table of > contents of the archive". > but > $BIN10/pg_restore --format=directory --list --file=1.sql dir10 > also output the contents of

Re: Non-text mode for pg_dumpall

2025-02-03 Thread jian he
hi. This attached patch solves problems mentioned in [1]. so pg_restore --file restoring multiple databases will have the ```\connect dbname``` command in it. the output plain text file can be used in psql. pg_restore --file output will be: -- -- Database "template1" dump -- -- Dumped from data

Re: Non-text mode for pg_dumpall

2025-02-03 Thread jian he
hi. just a quick response for v15. the pg_restore man page says option --list as "List the table of contents of the archive". but $BIN10/pg_restore --format=directory --list --file=1.sql dir10 also output the contents of "global.dat", we should not output it. in restoreAllDatabases, we can do th

Re: Non-text mode for pg_dumpall

2025-02-03 Thread Mahendra Singh Thalor
On Mon, 3 Feb 2025 at 14:23, Srinath Reddy wrote: > > Hi, > I found a bug ,while using "./pg_restore pdd -f -" actually it has to copy > everything(global sql commands + remaining dump ) into stdout as per the "-f, > --file=FILENAME output file name (- for stdout)" but it is copying > glob

Re: Non-text mode for pg_dumpall

2025-02-03 Thread jian he
On Mon, Feb 3, 2025 at 5:14 PM jian he wrote: > > there is some corner like num_db_restore == 0, num_db_restore >= 100 > in that scarenio, the execute_global_sql_commands already executed, > which is not ideal, since you have pg_fatal and some sql commands > already executed. > maybe we can be if

Re: Non-text mode for pg_dumpall

2025-02-03 Thread jian he
hi. git clean -fdx && $BIN10/pg_dumpall --format=directory --file=dir10 $BIN10/pg_restore --format=directory --file=1.sql --verbose dir10 > dir_format 2>&1 there is no "\connect dbname" command. pipe 1.sql to psql will execute all the database dump into a single database, which is not good. we ne

Re: Non-text mode for pg_dumpall

2025-02-03 Thread Srinath Reddy
here's the whole version of delta patch diff --git a/src/bin/pg_dump/pg_restore.c b/src/bin/pg_dump/pg_restore.c index 42c4fe3ce2..90e6b71a50 100644 --- a/src/bin/pg_dump/pg_restore.c +++ b/src/bin/pg_dump/pg_restore.c @@ -84,7 +84,7 @@ static int restoreAllDatabases(PGconn *conn, const char *dump

Re: Non-text mode for pg_dumpall

2025-02-03 Thread Srinath Reddy
Hi, I found a bug ,while using "./pg_restore pdd -f -" actually it has to copy everything(global sql commands + remaining dump ) into stdout as per the "-f, --file=FILENAME output file name (- for stdout)" but it is copying global sql commands to a file literally naming it as "-" and remaining

Re: Non-text mode for pg_dumpall

2025-02-02 Thread Mahendra Singh Thalor
Thanks Jian for review, testing and delta patches. On Wed, 29 Jan 2025 at 15:09, jian he wrote: > > hi. > > we need to escape the semicolon within the single quotes or double quotes. > I think my patch in [1] is correct. > > we can have "ERROR: role "z" already exists > but > error message like

Re: Non-text mode for pg_dumpall

2025-02-01 Thread Srinath Reddy
Hi, i think we have to change the pg_dumpall "--help" message similar to pg_dump's specifying that now pg_dumpall dumps cluster into to other non-text formats. Need similar "--help" message change in pg_restore to specify that now pg_restore supports restoring whole cluster from archive created fro

Re: Non-text mode for pg_dumpall

2025-01-31 Thread jian he
hi. more small issues. + count_db++; /* Increment db couter. */ + dboidprecell = dboid_cell; + } + typo, "couter" should be "counter". + +/* + * get_dbname_oid_list_from_mfile + * + * Open map.dat file and read line by line and then prepare a list of database + * names and correspoding db_oid. +

Re: Non-text mode for pg_dumpall

2025-01-30 Thread jian he
hi. -extern void RestoreArchive(Archive *AHX); +extern void RestoreArchive(Archive *AHX, bool append_data); Can we spare some words to explain the purpose of append_data. in get_dbname_oid_list_from_mfile pg_log_info("map.dat file is not present in dump of pg_dumpall, so nothing to resto

Re: Non-text mode for pg_dumpall

2025-01-29 Thread jian he
hi. we need to escape the semicolon within the single quotes or double quotes. I think my patch in [1] is correct. we can have "ERROR: role "z" already exists but error message like pg_restore: error: could not execute query: "ERROR: unterminated quoted string at or near "'; should not be accep

Re: Non-text mode for pg_dumpall

2025-01-28 Thread Mahendra Singh Thalor
On Fri, 24 Jan 2025 at 20:50, jian he wrote: > > On Thu, Jan 23, 2025 at 6:35 PM Mahendra Singh Thalor > wrote: > > hi. > After some tests and thinking about your reply, I admit that using > expand_dbname_patterns > in pg_restore will not work. > We need to do pattern matching against the map.dat

  1   2   >