Hi Daniel,
I've prepared the updated LZ4 patch for the truncated pg_dump/pg_restore
issue.
The patch adds detection of an incomplete LZ4 frame and a regression test
for a truncated LZ4 directory archive.
The src/bin/pg_dump tests pass successfully:
Files=7, Tests=13257
Result: PASS
git diff --check is also clean.
I've attached the patch as:
v1-0001-fix-truncated-lz4-pg-dump-nocfbot.patch
Regards,
Osama Abdul Qader
On Fri, Aug 14, 2026 at 9:58 AM Chao Li <[email protected]> wrote:
>
>
> > On Aug 14, 2026, at 06:59, Chao Li <[email protected]> wrote:
> >
> >
> >
> >> On Aug 14, 2026, at 04:57, Daniel Gustafsson <[email protected]> wrote:
> >>
> >>> On 13 Aug 2026, at 07:11, Chao Li <[email protected]> wrote:
> >>
> >>> PFA v7: addressed Jipan’s comment.
> >>
> >> Sorry for being slow on this, things are quite busy but I hope to have
> a review
> >> soon. While poking at this I realized that our compression code in
> pg_dump
> >> likely has the same issue. I hacked up a quick PoC diff (attached) but
> it's
> >> untested (can one actually test the data-in-zstd-internal-buffers case
> at all?)
> >> and mainly a sketch. If you want to pick it up and rework into this
> patchset
> >> to tackle it treewide then that would be fantastic.
> >>
> >> --
> >> Daniel Gustafsson
> >>
> >> <pg_dump.diff.txt>
> >
> > I can work on this today.
>
> I just checked pg_dump/pg_restore. The problem exists only with zstd and
> lz4, gzip doesn't have the problem.
>
> Daniel’s PoC covers the custom-archive-format path, but not the
> directory-format path.
>
> For the directory-format path, we can reproduce the problem by simply
> truncating one byte from a compressed data file. For the custom-archive
> path, reproducing the problem is less straightforward because the
> compressed data is stored inside length-prefixed archive blocks. Simply
> truncating the file can make archive parsing fail before the decompressor
> sees the truncated frame. I created a repro script, see the attached shell
> script.
>
> Before the fix, the output contains:
> ```
> custom zstd: exit status 0
> directory zstd: exit status 0
> custom lz4: exit status 0
> directory lz4: exit status 0
> pg_restore: error: could not uncompress data: (null)
> custom gzip: exit status 1
> pg_restore: error: could not close data file
> "/tmp/pgdump-trunc.1rxbf0/gzip-dir-bad/3931.dat": Undefined error: 0
> directory gzip: exit status 1
> ```
>
> This shows that gzip reports failure, while zstd and lz4 silently accept
> the truncated dump files.
>
> After the fix, zstd and lz4 report failures as well:
> ```
> pg_restore: error: could not decompress data: compressed stream is
> incomplete
> custom zstd: exit status 1
> pg_restore: error: could not decompress data: compressed stream is
> incomplete
> directory zstd: exit status 1
> pg_restore: error: could not decompress data: compressed stream is
> incomplete
> custom lz4: exit status 1
> pg_restore: error: could not read from input file: Input/output error
> directory lz4: exit status 1
> pg_restore: error: could not uncompress data: (null)
> custom gzip: exit status 1
> pg_restore: error: could not close data file
> "/tmp/pgdump-trunc.XjUiJ4/gzip-dir-bad/3931.dat": Undefined error: 0
> directory gzip: exit status 1
> ```
>
> While testing, I also found a small issue in LZ4Stream_read_internal().
> Its error branches call pg_log_error() and then return -1, but callers
> immediately call pg_fatal() when the return value <0. This results in
> duplicate error messages. So, I removed those pg_log_error() calls.
>
> See 0002 for the fix. I added tests only for zstd and lz4, since gzip is
> not changed.
>
> Best regards,
> --
> Chao Li (Evan)
> HighGo Software Co., Ltd.
> https://www.highgo.com/
>
>
>
>
>
diff --git a/src/bin/pg_dump/compress_lz4.c b/src/bin/pg_dump/compress_lz4.c
index 500d5e16a6d..1dc3a7f05dd 100644
--- a/src/bin/pg_dump/compress_lz4.c
+++ b/src/bin/pg_dump/compress_lz4.c
@@ -58,6 +58,7 @@ typedef struct LZ4State
* decompression operations.
*/
bool compressing;
+ bool frame_finished;
/*
* I/O buffer area.
@@ -471,7 +472,14 @@ LZ4Stream_read_internal(LZ4State *state, void *ptr, int ptrsize, bool eol_flag)
return -1;
}
if (rsize == 0)
+ {
+ if (!state->frame_finished)
+ {
+ state->errcode = EIO;
+ return -1;
+ }
break; /* must be EOF */
+ }
state->bufdata = rsize;
state->bufnext = 0;
}
@@ -499,6 +507,8 @@ LZ4Stream_read_internal(LZ4State *state, void *ptr, int ptrsize, bool eol_flag)
state->bufnext += inlen;
state->outbufdata = outlen;
state->outbufnext = 0;
+ if (status == 0)
+ state->frame_finished = true;
}
}
diff --git a/src/bin/pg_dump/t/006_pg_dump_compress.pl b/src/bin/pg_dump/t/006_pg_dump_compress.pl
index d4ce6b18077..98c2870e92c 100644
--- a/src/bin/pg_dump/t/006_pg_dump_compress.pl
+++ b/src/bin/pg_dump/t/006_pg_dump_compress.pl
@@ -631,6 +631,33 @@ foreach my $run (sort keys %pgdump_runs)
}
}
+#########################################
+# Test that pg_restore rejects a truncated LZ4 stream.
+
+if ($supports_lz4)
+{
+ my $source_dir = "$tempdir/compression_lz4_dir";
+ my $truncated_dir = "$tempdir/compression_lz4_dir_truncated";
+
+ system('cp', '-a', $source_dir, $truncated_dir) == 0
+ or die "could not copy LZ4 directory archive";
+
+ my $toc_file = "$truncated_dir/toc.dat.lz4";
+ my $size = -s $toc_file;
+
+ truncate($toc_file, $size - 10)
+ or die "could not truncate $toc_file: $!";
+
+ $node->command_fails(
+ [
+ 'pg_restore',
+ '--file' => "$tempdir/compression_lz4_dir_truncated.sql",
+ '--statistics',
+ $truncated_dir,
+ ],
+ 'pg_restore rejects truncated LZ4 archive');
+}
+
#########################################
# Stop the database instance, which will be removed at the end of the tests.
