On Tue, Nov 11, 2025 at 9:00 PM Hannu Krosing wrote:
>
> Attached is a patch that adds the ability to dump table data in multiple
> chunks.
>
> Looking for feedback at this point:
> 1) what have I missed
> 2) should I implement something to avoid single-page chunks
>
> The flag --huge-table-chu
Added to https://commitfest.postgresql.org/patch/6219/
On Thu, Nov 13, 2025 at 9:26 PM Hannu Krosing wrote:
>
> The reason for small chunk sizes is that they are determined by main
> heap table, and that was just over 1GB
>
> largetoastdb=> SELECT format('%I.%I', t.schemaname, t.relname) as table
The reason for small chunk sizes is that they are determined by main
heap table, and that was just over 1GB
largetoastdb=> SELECT format('%I.%I', t.schemaname, t.relname) as table_name,
pg_table_size(t.relid) AS table_size,
sum(pg_relation_size(i.indexrelid)) AS total_index_size,
Ran another test with a 53GB database where most of the data is in TOAST
CREATE TABLE just_toasted(
id serial primary key,
toasted1 char(2200) STORAGE EXTERNAL,
toasted2 char(2200) STORAGE EXTERNAL,
toasted3 char(2200) STORAGE EXTERNAL,
toasted4 char(2200) STORAGE EXTERNAL
);
and the to
Going up to 16 workers did not improve performance , but this is
expected, as the disk behind the database can only do 4TB/hour of
reads, which is now the bottleneck. (408/352/*3600 = 4172 GB/h)
$ time ./pg_dump --format=directory -h 10.58.80.2 -U postgres
--huge-table-chunk-pages=131072 -j 16 -f
I just ran a test by generating a 408GB table and then dumping it both ways
$ time pg_dump --format=directory -h 10.58.80.2 -U postgres -f
/tmp/plain.dump largedb
real39m54.968s
user37m21.557s
sys 2m32.422s
$ time ./pg_dump --format=directory -h 10.58.80.2 -U postgres
--huge-table-ch
Hi Hannu,
On Tue, Nov 11, 2025 at 9:00 PM Hannu Krosing wrote:
>
> Attached is a patch that adds the ability to dump table data in multiple
> chunks.
>
> Looking for feedback at this point:
> 1) what have I missed
> 2) should I implement something to avoid single-page chunks
>
> The flag --hug
Attached is a patch that adds the ability to dump table data in multiple chunks.
Looking for feedback at this point:
1) what have I missed
2) should I implement something to avoid single-page chunks
The flag --huge-table-chunk-pages which tells the directory format
dump to dump tables where the
