Bug#1065751: pristine-tar: diff for NMU version 1.50+nmu2
On 2024-03-31 19:42:24 [+], tony mancill wrote: > Given what has unfolded over the past few days regarding xz-utils and > CVE-2024-3094 [0], should we revisit the patches applied here and for > #1063252? Are they still needed? Not with the fallback to pre 5.4.x series but *I* don't think this will remain forever. The requirement for the change was different default value for the -T parameter. Recording the -T parameter by default and relying on defaults is good. > Thank you, > tony Sebastian
Bug#1065751: pristine-tar: diff for NMU version 1.50+nmu2
On Wed, Mar 13, 2024 at 01:50:47PM -0400, Jeremy Bícha wrote: > The 3 test cases pass for me now with the uploaded 1.50+nmu2. Maybe my > earlier test build used the old version of xz-utils. Thank you for > your upload! Given what has unfolded over the past few days regarding xz-utils and CVE-2024-3094 [0], should we revisit the patches applied here and for #1063252? Are they still needed? I'm not making any assertions about the validity of the changes, only suggesting that we should review changes made to accommodate the xz-utils versions related to the CVE. (It is still not clear why some gbp workflows using pristine-tar started failing [1] around the same time that changes were being introduced to xz-utils and pristine-tar. Perhaps it is a coicidence, but it seems potentially related.) Thank you, tony [0] https://security-tracker.debian.org/tracker/CVE-2024-3094 [1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1065445 signature.asc Description: PGP signature
Bug#1065751: pristine-tar: diff for NMU version 1.50+nmu2
On Tue, Mar 12, 2024 at 5:16 PM Jeremy Bícha wrote: > > On Tue, Mar 12, 2024 at 4:13 PM Sebastian Andrzej Siewior > wrote: > > > > On 2024-03-12 09:26:32 [-0400], Jeremy Bícha wrote: > > > > Could someone check this, please? > > > > > > Did you try running autopkgtests on this version? The autopkgtests fail > > > for me. > > > > autopkgtests were the first thing that pointed me here and they passed. > > If you say they fail for you then I may have used the wrong xz version… > > I tried running the autopkgtests locally using a version of 1.50+nmu2 > that I built using your patch and the autopkgtests failed. > > None of the 3 test cases passed for me with that version or with 1.50+nmu1. The 3 test cases pass for me now with the uploaded 1.50+nmu2. Maybe my earlier test build used the old version of xz-utils. Thank you for your upload! Jeremy Bícha
Bug#1065751: pristine-tar: diff for NMU version 1.50+nmu2
On Tue, Mar 12, 2024 at 09:13:09PM +0100, Sebastian Andrzej Siewior wrote: > tried with 1.50+nmu2 which is in deferred for the next 11h. So tomorrow > it should be all good. Please make sure to commit the patch to the pristine-tar git repository as well. signature.asc Description: PGP signature
Bug#1065751: pristine-tar: diff for NMU version 1.50+nmu2
On Tue, Mar 12, 2024 at 4:13 PM Sebastian Andrzej Siewior wrote: > > On 2024-03-12 09:26:32 [-0400], Jeremy Bícha wrote: > > > Could someone check this, please? > > > > Did you try running autopkgtests on this version? The autopkgtests fail for > > me. > > autopkgtests were the first thing that pointed me here and they passed. > If you say they fail for you then I may have used the wrong xz version… I tried running the autopkgtests locally using a version of 1.50+nmu2 that I built using your patch and the autopkgtests failed. None of the 3 test cases passed for me with that version or with 1.50+nmu1. Thank you, Jeremy Bícha
Bug#1065751: pristine-tar: diff for NMU version 1.50+nmu2
On 2024-03-12 09:26:32 [-0400], Jeremy Bícha wrote: > > Could someone check this, please? > > Did you try running autopkgtests on this version? The autopkgtests fail for > me. autopkgtests were the first thing that pointed me here and they passed. If you say they fail for you then I may have used the wrong xz version… > I assume that the largest use of pristine-tar in Debian is with > git-buildpackage. The 1.50+nmu1 upload **caused** pristine-tar to > break in many cases for me. If I revert back to 1.50, I no longer get > mismatched tarballs errors. Here are some test cases to demonstrate: > > Test Case 1 > == > gbp clone --add-upstream-vcs https://salsa.debian.org/jbicha/pangomm2.48 > > cd pangomm2.48 > > gbp import-orig --uscan > > gbp buildpackage > > What happens > -- > The exact hashes will probably vary but I get an error like this: > > gbp:error: Pristine-tar couldn't verify > "pangomm2.48_2.50.2.orig.tar.xz": pristine-tar: > /home/jeremy/build-area/pangomm2.48_2.50.2.orig.tar.xz does not match > stored hash (expected > e99b6a9c89e9c284bf44f5ae8125c06515d6ab8f8577d75d2887726dacb5a372, got > 826ad52f53ac8e15c9ceba4dc6e616efddae5e089f36bf4e60081c177d80d4b6) | (sid)bigeasy@debbuildd:~/pristine$ gbp clone --add-upstream-vcs https://salsa.debian.org/jbicha/pangomm2.48 | gbp:info: Cloning from 'https://salsa.debian.org/jbicha/pangomm2.48' | gbp:info: Adding upstream vcs at https://gitlab.gnome.org/GNOME/pangomm.git as additional remote | (sid)bigeasy@debbuildd:~/pristine$ cd pangomm2.48 | (sid)bigeasy@debbuildd:~/pristine/pangomm2.48$ gbp import-orig --uscan | gbp:info: Launching uscan... | Newest version of pangomm2.48 on remote site is 2.50.2, local version is 2.50.1 | => Newer package available from: | => https://download.gnome.org/sources/pangomm/2.50/pangomm-2.50.2.tar.xz | Successfully repacked ../pangomm-2.50.2.tar.xz as ../pangomm2.48_2.50.2.orig.tar.xz, deleting 416 files from it. | gbp:info: Using uscan downloaded tarball ../pangomm2.48_2.50.2.orig.tar.xz | What is the upstream version? [2.50.2] | gbp:info: Importing '../pangomm2.48_2.50.2.orig.tar.xz' to branch 'upstream/latest'... | gbp:info: Source package is pangomm2.48 | gbp:info: Upstream version is 2.50.2 | gbp:info: Replacing upstream source on 'debian/latest' | gbp:info: Running Postimport hook | dch warning: neither DEBEMAIL nor EMAIL environment variable is set | dch warning: building email address from username and FQDN | dch: Did you see those 2 warnings? Press RETURN to continue... | | git commit -m 'New upstream release' | [debian/latest f5c4f6d14a78] New upstream release | 1 file changed, 5 insertions(+), 2 deletions(-) | gbp:info: Successfully imported version 2.50.2 of ../pangomm2.48_2.50.2.orig.tar.xz passed. > Other info > - > pangomm2.48 uses Files-Excluded in debian/copyright so uscan will > rebuild a tarball and its hash will vary depending on the time it was > created. (Perhaps the hash should be reproducible but that's not > relevant to this bug.) > > Test Case 2 > = > gbp clone https://salsa.debian.org/gnome-team/gtk4 > cd gtk4 > gbp buildpackage > > What happens > > gbp:error: Pristine-tar couldn't verify "gtk4_4.12.5+ds.orig.tar.xz": > pristine-tar: > /home/jeremy/devel/pkg-gnome/temp/build-area/gtk4_4.12.5+ds.orig.tar.xz > does not match stored hash (expected > 3338a691d774ae031af65299e9a1c6207f543f13b256539717a1970f752358cb, got > 70ac33e0f37dc1b657d6560f1b8a40b3f4b67e956936633ced495d8b880d3fb0) | (sid)bigeasy@debbuildd:~/pristine$ gbp clone https://salsa.debian.org/gnome-team/gtk4 | gbp:info: Cloning from 'https://salsa.debian.org/gnome-team/gtk4' | (sid)bigeasy@debbuildd:~/pristine$ cd gtk4 | (sid)bigeasy@debbuildd:~/pristine/gtk4$ gbp buildpackage | gbp:info: Creating /home/bigeasy/pristine/gtk4_4.12.5+ds.orig.tar.xz | gbp:info: Performing the build | dpkg-buildpackage -us -uc -ui -i -I | dpkg-buildpackage: info: source package gtk4 | dpkg-buildpackage: info: source version 4.12.5+ds-4 passed. > Other info > > This pristine-tar tarball was committed January 19 so it did not use > either the new xz-utils or pristine-tar. > > Test Case 3 > = > gbp clone https://salsa.debian.org/gnome-team/pango > cd pango > gbp buildpackage > > What happens > --- > gbp:error: Pristine-tar couldn't verify > "pango1.0_1.52.1+ds.orig.tar.xz": pristine-tar: > /home/jeremy/devel/pkg-gnome/temp/build-area/pango1.0_1.52.1+ds.orig.tar.xz > does not match stored hash (expected > 12d67d8182cbb2ae427406df9bab5ce2ff5619102bf2a0fc6331d80a9914b139, got > a641d29d2d7df7843e44762a0733987dc8220d238b697b382dd96fafe5fc890a) | (sid)bigeasy@debbuildd:~/pristine$ gbp clone https://salsa.debian.org/gnome-team/pango | gbp:info: Cloning from 'https://salsa.debian.org/gnome-team/pango' |
Bug#1065751: pristine-tar: diff for NMU version 1.50+nmu2
On Sun, Mar 10, 2024 at 4:46 PM Sebastian Andrzej Siewior wrote: > I've prepared an NMU for pristine-tar (versioned as 1.50+nmu2) and > uploaded it to DELAYED/2. Please feel free to tell me if I > should delay it longer. > > Could someone check this, please? Did you try running autopkgtests on this version? The autopkgtests fail for me. I assume that the largest use of pristine-tar in Debian is with git-buildpackage. The 1.50+nmu1 upload **caused** pristine-tar to break in many cases for me. If I revert back to 1.50, I no longer get mismatched tarballs errors. Here are some test cases to demonstrate: Test Case 1 == gbp clone --add-upstream-vcs https://salsa.debian.org/jbicha/pangomm2.48 cd pangomm2.48 gbp import-orig --uscan gbp buildpackage What happens -- The exact hashes will probably vary but I get an error like this: gbp:error: Pristine-tar couldn't verify "pangomm2.48_2.50.2.orig.tar.xz": pristine-tar: /home/jeremy/build-area/pangomm2.48_2.50.2.orig.tar.xz does not match stored hash (expected e99b6a9c89e9c284bf44f5ae8125c06515d6ab8f8577d75d2887726dacb5a372, got 826ad52f53ac8e15c9ceba4dc6e616efddae5e089f36bf4e60081c177d80d4b6) Other info - pangomm2.48 uses Files-Excluded in debian/copyright so uscan will rebuild a tarball and its hash will vary depending on the time it was created. (Perhaps the hash should be reproducible but that's not relevant to this bug.) Test Case 2 = gbp clone https://salsa.debian.org/gnome-team/gtk4 cd gtk4 gbp buildpackage What happens gbp:error: Pristine-tar couldn't verify "gtk4_4.12.5+ds.orig.tar.xz": pristine-tar: /home/jeremy/devel/pkg-gnome/temp/build-area/gtk4_4.12.5+ds.orig.tar.xz does not match stored hash (expected 3338a691d774ae031af65299e9a1c6207f543f13b256539717a1970f752358cb, got 70ac33e0f37dc1b657d6560f1b8a40b3f4b67e956936633ced495d8b880d3fb0) Other info This pristine-tar tarball was committed January 19 so it did not use either the new xz-utils or pristine-tar. Test Case 3 = gbp clone https://salsa.debian.org/gnome-team/pango cd pango gbp buildpackage What happens --- gbp:error: Pristine-tar couldn't verify "pango1.0_1.52.1+ds.orig.tar.xz": pristine-tar: /home/jeremy/devel/pkg-gnome/temp/build-area/pango1.0_1.52.1+ds.orig.tar.xz does not match stored hash (expected 12d67d8182cbb2ae427406df9bab5ce2ff5619102bf2a0fc6331d80a9914b139, got a641d29d2d7df7843e44762a0733987dc8220d238b697b382dd96fafe5fc890a) Other info - This tarball was committed a few days ago with the new xz-utils and pristine-tar 1.50+nmu1. pango also uses Files-Excluded Conclusion Test cases 1, 2, and 3 pass with pristine-tar 1.50 but fail with pristine-tar 1.50+nmu1 Thank you, Jeremy Bícha
Bug#1065751: pristine-tar: diff for NMU version 1.50+nmu2
On 2024-03-11 21:23:03 [+], Amin Bandali wrote: > Hi, Hi, > On Mon, Mar 11, 2024 at 05:55:31PM +0100, Sebastian Andrzej Siewior wrote: > > On 2024-03-11 00:05:54 [+], Amin Bandali wrote: > > > Hi Sebastian, all, > > Hi, > > > > > Will this fix be enough for addressing all cases, though? > > > > I think so. Do you have a test case for me to check? > > Not about pristine-xz specifically; I meant more in the context of > other devel tools like gbp et al. ah okay. pristine-tar was the only tool that had CI failures during the upload of new xz-utils to exp. I wouldn't know other tools that require to recreate the same binary file. > > Who is handling the compression in the first place here? > > In the case of "gbp import-orig --uscan", gbp invokes uscan, part of > the devscripts package which has several perl modules including > Devscripts::Compression which is a sort of a wrapper around dpkg's > Dpkg::Compression, which will ultimately run the 'xz' executable. > > In some other cases like "gbp import-orig --filter" mentioned by > Andrey, gbp does the compression itself. Which is why I suggested > that 'Opts' in gbp.pkg.compressor may need to be updated to add '-T1' > for calls to 'xz'. okay. I wouldn't recomment doing -T1. This forces xz doing a single block and using a signle thread. The default (without passing the -T argument) will allow xz to use multiple threads and compress into multiple blocks which in turn can be decompressed using multiple threads. Forcing -T1 will force single threaded compression and decompression. pristine-tar can handle both cases. > > The idea is to pass -T1 to xz if nothing was recorded in pristine-tar's > > delta information. If the -T argument then everything keeps working > > as-is. If you use gbp to repack the tar archive then I would recommend > > to no pass -T1 and to use multi-threaded compression. pristine-tar > > will recongnise this and record this information. > > Sorry I don't think I fully understood this bit. Could you please > explain again, perhaps a bit more verbosely? If you do "pristine-tar gendelta" then pristine tar creates a .delta file which is tar.gz file containing a few files including the actual delta from `xdelta' and a file called `wrapper'. The `wrapper' file is also a tar.gz file including files regarding the invocation of the compressing tool which includes the arguments required to produce the exact output of the resulting .xz (from the tar input). Prior 1.50+nmu1 pristine-tar didn't record here the -T argument unless multi-threaded compression was used and pristine-tar used -Tcpus and recorded this. Since 1.50+nmu1 I made pristine-tar to always record the -T argument in the wrapper file, either -Tcpus in the multi threaded case as it did or by using -T1 in the single threaded one block case. That means the reproduce case has always the fitting -T argument. If you get an older archive which lacks the -T argument, pristine-tar will assume -T1 which was the old default. > To clarify, the use-cases described earlier involving gbp and > devscripts aren't necessarily related to pristine-xz, used for > regenerating pristine xz files; rather, about the generation or > repacking of xz files *before* they are handed to pristine-xz for > processing and storage in the repo. I was trying to imply that > similarly to how you sent patches for pristine-tar to adapt it for > changes in xz-utils, that similar patches are probably also needed > for gbp and devscripts. Does that make sense? So gbp and descripts should be able to deal with xz as-is since they don't have any expectation in the resulting binary file. They are happy once the input compressed/ decompressed. pristine-tar is the only tool, to my best knowledge, that requires binary identical output. Therefore I would keep gbp and devscripts as-is and prefer the multi-threaded compression & decompression. dpkg uses multi-threaded compression since a while and decompression since Bookworm. > Thanks, > -amin Sebastian
Bug#1065751: pristine-tar: diff for NMU version 1.50+nmu2
Hi, On Mon, Mar 11, 2024 at 05:55:31PM +0100, Sebastian Andrzej Siewior wrote: > On 2024-03-11 00:05:54 [+], Amin Bandali wrote: > > Hi Sebastian, all, > Hi, > > > Will this fix be enough for addressing all cases, though? > > I think so. Do you have a test case for me to check? Not about pristine-xz specifically; I meant more in the context of other devel tools like gbp et al. > > I'm thinking specifically of cases where tarball repacking > > is involved, for example when using git-buildpackage's > > "gbp import-orig --uscan" where uscan is used to download and > > repack the upstream tarball, because the package at hand has > > a Files-Excluded field in its debian/copyright header stanza. > > As far as I can tell, Devscripts::Compression would need to be > > updated to specify -T1 for xz compressions. > > > > I believe there are also some cases where git-buildpackage > > itself does repacking, so we'd probably want to update its > > gbp.pkg.compressor's Opts to pass in -T1 for xz. > > Who is handling the compression in the first place here? In the case of "gbp import-orig --uscan", gbp invokes uscan, part of the devscripts package which has several perl modules including Devscripts::Compression which is a sort of a wrapper around dpkg's Dpkg::Compression, which will ultimately run the 'xz' executable. In some other cases like "gbp import-orig --filter" mentioned by Andrey, gbp does the compression itself. Which is why I suggested that 'Opts' in gbp.pkg.compressor may need to be updated to add '-T1' for calls to 'xz'. > The idea is to pass -T1 to xz if nothing was recorded in pristine-tar's > delta information. If the -T argument then everything keeps working > as-is. If you use gbp to repack the tar archive then I would recommend > to no pass -T1 and to use multi-threaded compression. pristine-tar > will recongnise this and record this information. Sorry I don't think I fully understood this bit. Could you please explain again, perhaps a bit more verbosely? To clarify, the use-cases described earlier involving gbp and devscripts aren't necessarily related to pristine-xz, used for regenerating pristine xz files; rather, about the generation or repacking of xz files *before* they are handed to pristine-xz for processing and storage in the repo. I was trying to imply that similarly to how you sent patches for pristine-tar to adapt it for changes in xz-utils, that similar patches are probably also needed for gbp and devscripts. Does that make sense? > Sebastian > Thanks, -amin
Bug#1065751: pristine-tar: diff for NMU version 1.50+nmu2
On 2024-03-11 00:05:54 [+], Amin Bandali wrote: > Hi Sebastian, all, Hi, > Will this fix be enough for addressing all cases, though? I think so. Do you have a test case for me to check? > I'm thinking specifically of cases where tarball repacking > is involved, for example when using git-buildpackage's > "gbp import-orig --uscan" where uscan is used to download and > repack the upstream tarball, because the package at hand has > a Files-Excluded field in its debian/copyright header stanza. > As far as I can tell, Devscripts::Compression would need to be > updated to specify -T1 for xz compressions. > > I believe there are also some cases where git-buildpackage > itself does repacking, so we'd probably want to update its > gbp.pkg.compressor's Opts to pass in -T1 for xz. Who is handling the compression in the first place here? The idea is to pass -T1 to xz if nothing was recorded in pristine-tar's delta information. If the -T argument then everything keeps working as-is. If you use gbp to repack the tar archive then I would recommend to no pass -T1 and to use multi-threaded compression. pristine-tar will recongnise this and record this information. > Thanks, > -a Sebastian
Bug#1065751: pristine-tar: diff for NMU version 1.50+nmu2
On Mon, Mar 11, 2024 at 12:05:54AM +, Amin Bandali wrote: > I believe there are also some cases where git-buildpackage > itself does repacking E.g. `gbp import-orig --filter` -- WBR, wRAR signature.asc Description: PGP signature
Bug#1065751: pristine-tar: diff for NMU version 1.50+nmu2
Hi Sebastian, all, Will this fix be enough for addressing all cases, though? I'm thinking specifically of cases where tarball repacking is involved, for example when using git-buildpackage's "gbp import-orig --uscan" where uscan is used to download and repack the upstream tarball, because the package at hand has a Files-Excluded field in its debian/copyright header stanza. As far as I can tell, Devscripts::Compression would need to be updated to specify -T1 for xz compressions. I believe there are also some cases where git-buildpackage itself does repacking, so we'd probably want to update its gbp.pkg.compressor's Opts to pass in -T1 for xz. Thanks, -a
Bug#1065751: pristine-tar: diff for NMU version 1.50+nmu2
Control: tags 1065751 + patch
Control: tags 1065751 + pending
Dear maintainer,
I've prepared an NMU for pristine-tar (versioned as 1.50+nmu2) and
uploaded it to DELAYED/2. Please feel free to tell me if I
should delay it longer.
Could someone check this, please?
Regards.
Sebastian
diff -Nru pristine-tar-1.50+nmu1/debian/changelog pristine-tar-1.50+nmu2/debian/changelog
--- pristine-tar-1.50+nmu1/debian/changelog 2024-02-25 12:18:32.0 +0100
+++ pristine-tar-1.50+nmu2/debian/changelog 2024-03-10 21:38:16.0 +0100
@@ -1,3 +1,11 @@
+pristine-tar (1.50+nmu2) UNRELEASED; urgency=medium
+
+ * Non-maintainer upload.
+ * Preoperly account -T parameter for xz. Thanks to Jia Tan for the hint.
+(Closes: #1065751).
+
+ -- Sebastian Andrzej Siewior Sun, 10 Mar 2024 21:38:16 +0100
+
pristine-tar (1.50+nmu1) unstable; urgency=medium
* Non-maintainer upload.
diff -Nru pristine-tar-1.50+nmu1/pristine-xz pristine-tar-1.50+nmu2/pristine-xz
--- pristine-tar-1.50+nmu1/pristine-xz 2024-02-25 12:18:06.0 +0100
+++ pristine-tar-1.50+nmu2/pristine-xz 2024-03-10 21:38:12.0 +0100
@@ -416,11 +416,11 @@
next if $param eq '--check=crc64';
next if $param eq '--check=sha256';
next if $param =~ /^(--block-list=[0-9,]+)$/;
- next if $param =~ /^-T[0-9]+$/;
+
if ($param =~ /^-T[0-9]+$/) {
- $threads_set = 1;
- next;
- }
+ $threads_set = 1;
+ next;
+ }
} elsif ($delta->{program} eq 'pixz') {
next if $param eq '-t';
}
@@ -429,8 +429,10 @@
@params = split(' ', $delta->{params});
- if (!$threads_set) {
-push @params, '-T1';
+ if ($delta->{program} eq 'xz') {
+ if (!$threads_set) {
+push @params, '-T1';
+ }
}
doit($program, @params, $file);
