Bug#972543: apt-file: Apt-update downloads full Contents file, not pdiffs
calumlikesapple...@gmail.com: > On Sun, 2020-10-25 at 11:04 +0100, Niels Thykier wrote: >> Hi again, >> >> Thanks for the data. Unfortunately, it did reveal a smoking gun, so >> I >> think we need /tmp/update.log generated from this command. >> >> apt -o Debug::pkgAcquire::Worker=1 \ >> -o Debug::pkgAcquire::Diffs=1 update \ >> 2>&1 | tee /tmp/update.log > > Attached. > Apologies, I should have been more precise. Please use that command next time you need to run an apt update and when you get a run where you should have had PDiffs but did not get any, then submit the log file for that run. The log file provided is for a "no-changes" run. Accordingly, I cannot tell if it skipped PDiffs or not because it concluded there was no need for an update at all. >> Please attach the file rather than copy-paste in the output from it. > > Sure! I just wasn't sure if attaching was standard, or if you'd gotten > my original attachment. > My rule of thumb is more about length and whether the mail programs line wrap long lines. :) For mail programs that don't, "short" inline copy-paste is often nicer, but for long files or zealous line wrapping attachments are often more readable. ~Niels
Bug#972543: apt-file: Apt-update downloads full Contents file, not pdiffs
On Sun, 2020-10-25 at 11:24 -0400, calumlikesapple...@gmail.com wrote: > On Sun, 2020-10-25 at 11:04 +0100, Niels Thykier wrote: > > Hi again, > > > > Thanks for the data. Unfortunately, it did reveal a smoking gun, > > so > > I > > think we need /tmp/update.log generated from this command. > > > > apt -o Debug::pkgAcquire::Worker=1 \ > > -o Debug::pkgAcquire::Diffs=1 update \ > > 2>&1 | tee /tmp/update.log > > Attached. > > > Please attach the file rather than copy-paste in the output from > > it. > > Sure! I just wasn't sure if attaching was standard, or if you'd > gotten > my original attachment. I just realized that I forgot to mention something important: the issue doesn't happen every time I run apt update, and even sometimes when it needs to update other files, it doesn't wind up updating the contents. I'll log my manual updates for the next week (gnome-software sometimes runs it in the background), in case this log is useless. signature.asc Description: This is a digitally signed message part
Bug#972543: apt-file: Apt-update downloads full Contents file, not pdiffs
On Sun, 2020-10-25 at 11:04 +0100, Niels Thykier wrote: > Hi again, > > Thanks for the data. Unfortunately, it did reveal a smoking gun, so > I > think we need /tmp/update.log generated from this command. > > apt -o Debug::pkgAcquire::Worker=1 \ > -o Debug::pkgAcquire::Diffs=1 update \ > 2>&1 | tee /tmp/update.log Attached. > Please attach the file rather than copy-paste in the output from it. Sure! I just wasn't sure if attaching was standard, or if you'd gotten my original attachment. WARNING: apt does not have a stable CLI interface. Use with caution in scripts. Starting method '/usr/lib/apt/methods/http' <- http:100%20Capabilities%0aSend-Config:%20true%0aPipeline:%20true%0aVersion:%201.2 Configured access method http Version:1.2 SingleInstance:0 Pipeline:1 SendConfig:1 LocalOnly: 0 NeedsCleanup: 0 Removable: 0 AuxRequests: 0 Starting method '/usr/lib/apt/methods/https' <- https:100%20Capabilities%0aSend-Config:%20true%0aPipeline:%20true%0aVersion:%201.2 Configured access method https Version:1.2 SingleInstance:0 Pipeline:1 SendConfig:1 LocalOnly: 0 NeedsCleanup: 0 Removable: 0 AuxRequests: 0 Starting method '/usr/lib/apt/methods/http' <- http:100%20Capabilities%0aSend-Config:%20true%0aPipeline:%20true%0aVersion:%201.2 Configured access method http Version:1.2 SingleInstance:0 Pipeline:1 SendConfig:1 LocalOnly: 0 NeedsCleanup: 0 Removable: 0 AuxRequests: 0 ->
Bug#972543: apt-file: Apt-update downloads full Contents file, not pdiffs
Hi again, Thanks for the data. Unfortunately, it did reveal a smoking gun, so I think we need /tmp/update.log generated from this command. apt -o Debug::pkgAcquire::Worker=1 \ -o Debug::pkgAcquire::Diffs=1 update \ 2>&1 | tee /tmp/update.log Please attach the file rather than copy-paste in the output from it. calumlikesapple...@gmail.com: >>> [...] >> [...] > >> $ apt-config dump Acquire::IndexTargets > > I'll paste the full output below > Thanks. Everything looks like it was in order. >> file:/etc/apt/apt-file.conf (if it exists) > > This file doesn't exist, though I attached what I think is it's > equivalent in my initial message. That is fine. I hoped as much as most people do not need it. :) >> This will show the effective fetching configuration for apt and apt- >> file. >> Relatedly, just to confirm: The bug is on the system you are >> reporting >> from (i.e. running testing/sid) and you run "apt update" at least >> once a week. Are these assumptions of mine correct? > Yes, that's correct. > > [...] > Thanks for confirming.
Bug#972543: apt-file: Apt-update downloads full Contents file, not pdiffs
> > Basically, on every call to apt-update that isn't completely > > trivial (ie, at least > > 1 package changed), my computer downloads the full 60 Mb of > > Contents files. This > > is irritating, for while I am often on a fast internet, I > > frequently find myself on > > a very slow connection. > > > > I know my config file enables pdiffs (and I attached it so you can > > know, too). > > The Readme talks a lot about pdiffs, and why I should enable them, > > but except for > > one sentence, it doesn't mention anything about apt update. (The > > sentence in > > question *could* be interpreted to mean that apt update will never > > download > > Pdiffs, but that is unlikely, given its context). > > > > I don't know what's the cause. If you need me to get a log file, I > > can, but > > Apt doesnt include these files in its final output: only while it > > is > > drawing progress bars. > > > > Thank you! > > Calum > > > > [...] > > > > > Hi Calum, > I am sorry to hear it does not work as expected. > All of the Contents downloading is outsourced to apt, so most likely > it is a configuration (or a bug in apt itself). Can you provide the > output/contents of? > $ apt-config dump Acquire::IndexTargets I'll paste the full output below > file:/etc/apt/apt-file.conf (if it exists) This file doesn't exist, though I attached what I think is it's equivalent in my initial message. > This will show the effective fetching configuration for apt and apt- > file. > Relatedly, just to confirm: The bug is on the system you are > reporting > from (i.e. running testing/sid) and you run "apt update" at least > once a week. Are these assumptions of mine correct? Yes, that's correct. apt-config dump Aquire::IndexTargets output: Acquire::IndexTargets ""; Acquire::IndexTargets::deb ""; Acquire::IndexTargets::deb::Packages ""; Acquire::IndexTargets::deb::Packages::MetaKey "$(COMPONENT)/binary- $(ARCHITECTURE)/Packages"; Acquire::IndexTargets::deb::Packages::flatMetaKey "Packages"; Acquire::IndexTargets::deb::Packages::ShortDescription "Packages"; Acquire::IndexTargets::deb::Packages::Description "$(RELEASE)/$(COMPONENT) $(ARCHITECTURE) Packages"; Acquire::IndexTargets::deb::Packages::flatDescription "$(RELEASE) Packages"; Acquire::IndexTargets::deb::Packages::Optional "0"; Acquire::IndexTargets::deb::Translations ""; Acquire::IndexTargets::deb::Translations::MetaKey "$(COMPONENT)/i18n/Translation-$(LANGUAGE)"; Acquire::IndexTargets::deb::Translations::flatMetaKey "$(LANGUAGE)"; Acquire::IndexTargets::deb::Translations::ShortDescription "Translation-$(LANGUAGE)"; Acquire::IndexTargets::deb::Translations::Description "$(RELEASE)/$(COMPONENT) Translation-$(LANGUAGE)"; Acquire::IndexTargets::deb::Translations::flatDescription "$(RELEASE) Translation-$(LANGUAGE)"; Acquire::IndexTargets::deb::DEP-11 ""; Acquire::IndexTargets::deb::DEP-11::MetaKey "$(COMPONENT)/dep11/Components-$(NATIVE_ARCHITECTURE).yml"; Acquire::IndexTargets::deb::DEP-11::ShortDescription "Components- $(NATIVE_ARCHITECTURE)"; Acquire::IndexTargets::deb::DEP-11::Description "$(RELEASE)/$(COMPONENT) $(NATIVE_ARCHITECTURE) DEP-11 Metadata"; Acquire::IndexTargets::deb::DEP-11::KeepCompressed "true"; Acquire::IndexTargets::deb::DEP-11::KeepCompressedAs "gz"; Acquire::IndexTargets::deb::DEP-11-icons-small ""; Acquire::IndexTargets::deb::DEP-11-icons-small::MetaKey "$(COMPONENT)/dep11/icons-48x48.tar"; Acquire::IndexTargets::deb::DEP-11-icons-small::ShortDescription "icons-48x48"; Acquire::IndexTargets::deb::DEP-11-icons-small::Description "$(RELEASE)/$(COMPONENT) DEP-11 48x48 Icons"; Acquire::IndexTargets::deb::DEP-11-icons-small::KeepCompressed "true"; Acquire::IndexTargets::deb::DEP-11-icons-small::KeepCompressedAs "gz"; Acquire::IndexTargets::deb::DEP-11-icons-small::DefaultEnabled "true"; Acquire::IndexTargets::deb::DEP-11-icons ""; Acquire::IndexTargets::deb::DEP-11-icons::MetaKey "$(COMPONENT)/dep11/icons-64x64.tar"; Acquire::IndexTargets::deb::DEP-11-icons::ShortDescription "icons- 64x64"; Acquire::IndexTargets::deb::DEP-11-icons::Description "$(RELEASE)/$(COMPONENT) DEP-11 64x64 Icons"; Acquire::IndexTargets::deb::DEP-11-icons::KeepCompressed "true"; Acquire::IndexTargets::deb::DEP-11-icons::KeepCompressedAs "gz"; Acquire::IndexTargets::deb::DEP-11-icons::DefaultEnabled "true"; Acquire::IndexTargets::deb::DEP-11-icons-hidpi ""; Acquire::IndexTargets::deb::DEP-11-icons-hidpi::MetaKey "$(COMPONENT) /dep11/icons-64...@2.tar"; Acquire::IndexTargets::deb::DEP-11-icons-hidpi::ShortDescription " icons-64x64@2"; Acquire::IndexTargets::deb::DEP-11-icons-hidpi::Description "$(RELEASE)/$(COMPONENT) DEP-11 64x64@2 Icons"; Acquire::IndexTargets::deb::DEP-11-icons-hidpi::KeepCompressed "true"; Acquire::IndexTargets::deb::DEP-11-icons-hidpi::KeepCompressedAs "gz"; Acquire::IndexTargets::deb::DEP-11-icons-hidpi::DefaultEnabled "false"; Acquire::IndexTargets::deb::DEP-11-icons-large ""; Acquire::IndexTargets::deb::DEP-11-icons-large::MetaKey
Bug#972543: apt-file: Apt-update downloads full Contents file, not pdiffs
Control: tags -1 moreinfo unreproducible Calum McConnell: > Package: apt-file > Version: 3.2.2 > Severity: normal > X-Debbugs-Cc: calumlikesapple...@gmail.com > > Basically, on every call to apt-update that isn't completely trivial (ie, at > least > 1 package changed), my computer downloads the full 60 Mb of Contents files. > This > is irritating, for while I am often on a fast internet, I frequently find > myself on > a very slow connection. > > I know my config file enables pdiffs (and I attached it so you can know, too). > The Readme talks a lot about pdiffs, and why I should enable them, but except > for > one sentence, it doesn't mention anything about apt update. (The sentence in > question *could* be interpreted to mean that apt update will never download > Pdiffs, but that is unlikely, given its context). > > I don't know what's the cause. If you need me to get a log file, I can, but > Apt doesnt include these files in its final output: only while it is > drawing progress bars. > > Thank you! > Calum > > [...] Hi Calum, I am sorry to hear it does not work as expected. All of the Contents downloading is outsourced to apt, so most likely it is a configuration (or a bug in apt itself). Can you provide the output/contents of? $ apt-config dump Acquire::IndexTargets file:/etc/apt/apt-file.conf (if it exists) This will show the effective fetching configuration for apt and apt-file. Relatedly, just to confirm: The bug is on the system you are reporting from (i.e. running testing/sid) and you run "apt update" at least once a week. Are these assumptions of mine correct? Thanks, ~Niels
Bug#972543: apt-file: Apt-update downloads full Contents file, not pdiffs
Package: apt-file
Version: 3.2.2
Severity: normal
X-Debbugs-Cc: calumlikesapple...@gmail.com
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA512
Basically, on every call to apt-update that isn't completely trivial (ie, at
least
1 package changed), my computer downloads the full 60 Mb of Contents files.
This
is irritating, for while I am often on a fast internet, I frequently find
myself on
a very slow connection.
I know my config file enables pdiffs (and I attached it so you can know, too).
The Readme talks a lot about pdiffs, and why I should enable them, but except
for
one sentence, it doesn't mention anything about apt update. (The sentence in
question *could* be interpreted to mean that apt update will never download
Pdiffs, but that is unlikely, given its context).
I don't know what's the cause. If you need me to get a log file, I can, but
Apt doesnt include these files in its final output: only while it is
drawing progress bars.
Thank you!
Calum
- -- System Information:
Debian Release: bullseye/sid
APT prefers testing
APT policy: (500, 'testing')
Architecture: amd64 (x86_64)
Foreign Architectures: i386
Kernel: Linux 5.8.0-3-amd64 (SMP w/4 CPU threads)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), LANGUAGE not set
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled
Versions of packages apt-file depends on:
ii apt 2.1.10
ii libapt-pkg-perl 0.1.36+b3
ii liblist-moreutils-perl 0.416-1+b5
ii libregexp-assemble-perl 0.36-1
ii perl 5.30.3-4
apt-file recommends no packages.
apt-file suggests no packages.
- -- no debconf information
-BEGIN PGP SIGNATURE-
iQJRBAEBCgA7FiEE/vC/PEGxsMPJ5u5w7/Xh1+DNmzIFAl+ObBsdHGNhbHVtbGlr
ZXNhcHBsZXBpZUBnbWFpbC5jb20ACgkQ7/Xh1+DNmzKbuQ//SXtp2pvjIgjQa2As
eWwtieJpvjlxG+qsKKJSfeHxLuCdGKoTs2a7/e4/BVbaQhPvNHdsDOPA6tY+QcdU
+iPU48L9fkA8B4YxeQqB6AzUyZM2bNdiaYNMJSyLu+o1WBDbk/an/QclkD23wQGL
E+Yyo9vIzVtmIA8S/NTWT6u9gW0mov4zH4ORHavacR3z/c4Yx17yNhHc5OlFbqmk
4nAuAedgirkj4OTbjwh+xofFkX7iP5P9X1rkz59UUJmKBOti9ZQs4GZYpHd8YLOG
FKJlOjumHbTB4kz16TfLAYvP/w0BIYkqPMTfZMjZxpFfyYllYx31K6j/w5qXaFTi
GYyBUnqB6+zk6doNzZwV0cBTwaO7tW0snptIHwLtM0iU1LyYkTh5M62IcLeqrozL
eNX9WZOk/OSqiwlJqg2mqB7R2ZRYtJaYf4qGmdQzQ2xMft16Vm+1q9WB0RY+RIJ7
RexE8fo+UrewdKkyuzMw6aMjpFm6iSakagx7Szc02RN9LnL9r63AsIbl0nDIxCQh
a933Oto7D2WbF+tl3OO+I0m2+xGt941VWUD+3yp/RBQcjnSieYIRSfwINlXiNVc+
6/+u3W0NhigH/qS88XH0gc3jQY0DixcGROvrtkVe4+1fwQEwGMrjtjG9aH3E2bif
HfPVdM8L2mhoKuUV7lg4rFLXPEk=
=+avA
-END PGP SIGNATURE-
## This file is provided by apt-file(1) to download Contents
## files, which is used by apt-file for searching.
Acquire::IndexTargets {
deb::Contents-deb {
MetaKey "$(COMPONENT)/Contents-$(ARCHITECTURE)";
ShortDescription "Contents-$(ARCHITECTURE)";
Description "$(RELEASE)/$(COMPONENT) $(ARCHITECTURE) Contents (deb)";
flatMetaKey "Contents-$(ARCHITECTURE)";
flatDescription "$(RELEASE) Contents (deb)";
PDiffs "true";
KeepCompressed "true";
};
# Download Contents for source files if there is a deb-src
# line
deb-src::Contents-dsc {
MetaKey "$(COMPONENT)/Contents-source";
ShortDescription "Contents-source";
Description "$(RELEASE)/$(COMPONENT) source Contents (dsc)";
flatMetaKey "Contents-source";
flatDescription "$(RELEASE) Contents (dsc)";
PDiffs "true";
KeepCompressed "true";
DefaultEnabled "false";
};
# Configuration for downloading Contents files for
# debian-installer packages (udebs).
deb::Contents-udeb {
MetaKey "$(COMPONENT)/Contents-udeb-$(ARCHITECTURE)";
ShortDescription "Contents-udeb-$(ARCHITECTURE)";
Description "$(RELEASE)/$(COMPONENT) $(ARCHITECTURE) Contents (udeb)";
flatMetaKey "Contents-udeb-$(ARCHITECTURE)";
flatDescription "$(RELEASE) Contents (udeb)";
KeepCompressed "true";
PDiffs "true";
DefaultEnabled "false";
};
### FALLBACKS
deb::Contents-deb-legacy {
MetaKey "Contents-$(ARCHITECTURE)";
ShortDescription "Contents-$(ARCHITECTURE)";
Description "$(RELEASE) $(ARCHITECTURE) Contents (deb)";
PDiffs "true";
KeepCompressed "true";
Fallback-Of "Contents-deb";
Identifier "Contents-deb";
};
};
Dir::Etc::apt-file-main "apt-file.conf";
# Default for -I/--index-names (comma-separated)
apt-file::Index-Names "deb";
# Set to true, if you are working with Contents files generated by
# older versions of dak or reprepro (<< 5.2.0-1~) that includes a
# descriptive header.
apt-file::Parser::Check-For-Description-Header "false";
