Re: dnf --refresh reverts to older metadata (was: Re: dnf update vs Software Udpates)
On Mon, 24 Aug 2015 16:48:22 +0200, Heinz Diehl wrote: https://fedorahosted.org/fedora-infrastructure/ticket/4866 Maybe it's time to take a look at how other distributions do it. Arch's pacman has worked for me without any trouble a long time. And there is Opensuse Co.. For such a step, evaluating where we are today should come first, IMO. Then you can compare with how others do it, whether they distribute their users to the same number of mirrors world-wide, or whether they prefer a few mirrors run by passioned admins. The master repository is updated. Announcements are mailed out. Users read an announcement, but they don't see the updates yet. WTF? Why not? Forum posts around the world suggest yum clean all and dnf clean all as _the fix_. But it isn't a fix, if the nearby mirrors don't carry the changed repo contents yet. dnf --refresh doesn't force retrieval of latest repodata either. (And that it even reverted to old data is an unrelated bug in the implementation.) Any automatically triggered check of the metadata would have achieved the same, if done at the right time. Mirrors not being up-to-date, nothing new to fetch. Updates become available as soon as they arrive on the nearby mirrors. Is that sent as a clear message to all Fedora users? I don't think so. On average, how long does it take for updates to arrive on the mirrors? Is anything known here? How long does it take for updates to arrive in another country? Are there any high-priority mirrors, who likely carry the latest changes quickly after release? Would it be possible to assign users to nearby repos in a more reliable way? Questions, questions, questions. -- users mailing list users@lists.fedoraproject.org To unsubscribe or change subscription options: https://admin.fedoraproject.org/mailman/listinfo/users Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines Have a question? Ask away: http://ask.fedoraproject.org
Re: dnf --refresh reverts to older metadata (was: Re: dnf update vs Software Udpates)
On 24.08.2015, Michael Schwendt wrote: The feedback in the ticket I've opened is not encouraging so far. https://fedorahosted.org/fedora-infrastructure/ticket/4866 Maybe it's time to take a look at how other distributions do it. Arch's pacman has worked for me without any trouble a long time. And there is Opensuse Co.. -- users mailing list users@lists.fedoraproject.org To unsubscribe or change subscription options: https://admin.fedoraproject.org/mailman/listinfo/users Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines Have a question? Ask away: http://ask.fedoraproject.org
Re: dnf --refresh reverts to older metadata (was: Re: dnf update vs Software Udpates)
On Thu, 20 Aug 2015 07:25:48 -0400 (EDT), Honza Šilhan wrote: The more I think about it DNF does it right. You should report it to Fedora infrastructure. DNF shouldn't inspect all mirrors - you would waste too much resources then. We need a better mechanism. Just 1 reference repomd metadata with the most up-to-date metadata checksums (on one server only). Than we could detect the mirror DNF chose is outdated. Stuff for everyone interested. The feedback in the ticket I've opened is not encouraging so far. https://fedorahosted.org/fedora-infrastructure/ticket/4866 -- users mailing list users@lists.fedoraproject.org To unsubscribe or change subscription options: https://admin.fedoraproject.org/mailman/listinfo/users Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines Have a question? Ask away: http://ask.fedoraproject.org
Re: dnf --refresh reverts to older metadata (was: Re: dnf update vs Software Udpates)
On Tue, 18 Aug 2015 05:54:07 -0400 (EDT), Honza Šilhan wrote: File a bug, if you care, please. And if I don't file a bug, I don't care? That would be an odd way to put it. =:-/ Would you rather prefer silence and people returning to another distribution? I mean, it is not clear yet whether mirror manager is the culprit, is it? Also, it's very unlikely that the dnf developers use only bugzilla for all project management, such as todo lists and milestone target details. Now that you've learned there is some fundamental problem, it should be put onto some internal todo list to schedule some proof-reading of the implementation and sit together with the mirror manager guys. Very important would be to verify all the stuff around the metalink XML parsing and evaluating, especially the age/timestamp/checksum/alternates. Adding more helpful debug output would be an idea, too. Certainly it should be possible to print metalink data age, which would be much more relevant than the time of the last metadata expiration check. + output of these command sequence (to see which mirror is picked) # export LIBREPO_DEBUG=1 dnf update --assumeno # export LIBREPO_DEBUG=1 dnf update --assumeno --refresh # dnf clean metadata # export LIBREPO_DEBUG=1 dnf update --assumeno This does not print which mirror is picked. It does not print anything helpful at all. I would expect debug information to include some details about the metalink server response, the metadata checksum and the metadata age, and stuff like that. | Librepo version: 1.7.16 with CURL_GLOBAL_ACK_EINTR support (libcurl/7.44.0 NSS/3.19.3 Basic ECC zlib/1.2.8 libidn/1.32 libssh2/1.6.0 nghttp2/1.2.0) | lr_download: Target: file:///etc/dnf/dnf.conf (-) | select_next_target: Selecting mirror for: file:///etc/dnf/dnf.conf | prepare_next_transfer: URL: file:///etc/dnf/dnf.conf | add_librepo_xattr: Cannot set xattr user.Librepo.DownloadInProgress (fd: 5): Operation not supported | lr_download: Downloading started | check_transfer_statuses: Transfer finished: file:///etc/dnf/dnf.conf (Effective url: file:///etc/dnf/dnf.conf) | Fedora - Rawhide - Developmental packages for t 1.4 MB/s | 43 MB 00:29 | Adobe Systems Incorporated 23 kB/s | 1.8 kB 00:00 | Last metadata expiration check performed 0:00:00 ago on Tue Aug 18 12:25:42 2015. | Dependencies resolved. Michael, have you run all these commands at one moment or after 1 day breaks? That can be seen in dnf's output: https://lists.fedoraproject.org/pipermail/users/2015-August/464144.html Excerpt: # dnf update Last metadata expiration check performed 1:19:01 ago on Fri Aug 14 11:29:15 2015. [...] Install 3 Packages Upgrade 76 Packages Remove3 Packages [...] # dnf update --refresh Adobe Systems Incorporated 21 kB/s | 1.8 kB 00:00 Fedora - Rawhide - Developmental packages for t 1.5 MB/s | 43 MB 00:29 Last metadata expiration check performed 0:00:13 ago on Fri Aug 14 12:48:43 2015. [...] Install 3 Packages Upgrade 76 Packages Remove3 Packages [...] # dnf update --refresh Fedora - Rawhide - Developmental packages for t 1.4 MB/s | 43 MB 00:30 Adobe Systems Incorporated 21 kB/s | 1.8 kB 00:00 Last metadata expiration check performed 0:00:00 ago on Fri Aug 14 12:50:08 2015. [...] Upgrade 50 Packages [...] As one can see, a bit more than a minute between those two last runs. Then, a day later, only dnf clean metadata lead to fetching new metadata: https://lists.fedoraproject.org/pipermail/users/2015-August/464173.html Some hours later when I replied to another mail in the thread, I could reproduce the --refresh reverts to old metadata problem with two runs one minute after eachother: https://lists.fedoraproject.org/pipermail/users/2015-August/464184.html # dnf update --refresh Adobe Systems Incorporated 21 kB/s | 1.8 kB 00:00 Fedora - Rawhide - Developmental packages for t 1.5 MB/s | 43 MB 00:29 Last metadata expiration check performed 0:00:11 ago on Sat Aug 15 19:21:15 2015. [...] Install4 Packages Upgrade 168 Packages Remove 3 Packages [...] # dnf update --refresh Adobe Systems Incorporated 13 kB/s | 1.8 kB 00:00 Fedora - Rawhide - Developmental packages for t 1.4 MB/s | 43 MB 00:29 Last metadata expiration check performed 0:00:39 ago on Sat Aug 15 19:22:31 2015. [...] Install4 Packages Upgrade 111 Packages Remove 3 Packages [...] -- users mailing list users@lists.fedoraproject.org To unsubscribe or change subscription options: https://admin.fedoraproject.org/mailman/listinfo/users Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines Have a question? Ask away: http://ask.fedoraproject.org
Re: dnf --refresh reverts to older metadata (was: Re: dnf update vs Software Udpates)
And here's proof of what can happen with just --refresh: 1. dnf update 2. dnf update --refresh 3. dnf update --refresh The last run reverts to older metadata with only 50 updates available compared with earlier. Mirror manager assigning to an out-of-date mirror? A day later, no matter how often I run dnf update --refresh, it never gets access to the newer metadata from yesterday again. Not the 76 packages as shown earlier in this thread, only the older 50. So, indeed, there's something seriously wrong here, and I assume it can only be fixed if the developers of mirror manager and dnf come together and look into it. After giving the dnf clean expire-cache option another try, too, and then the hammer dnf clean metadata, there has been a fresh download of even newer metadata compared with yesterday: [...] Install4 Packages Upgrade 111 Packages Remove 3 Packages Total download size: 211 M [...] # rpm -q dnf dnf-1.1.0-2.fc24.noarch -- users mailing list users@lists.fedoraproject.org To unsubscribe or change subscription options: https://admin.fedoraproject.org/mailman/listinfo/users Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines Have a question? Ask away: http://ask.fedoraproject.org
Re: dnf --refresh reverts to older metadata (was: Re: dnf update vs Software Udpates)
On Sat, 15 Aug 2015 13:40:04 +0100, Patrick O'Callaghan wrote: Worth a BZ report surely? Not from me this time. It is my understanding that there have been multiple reports before. A few hours have passed, and meanwhile there are even newer metadata. However, a subsequent run of dnf update --refresh reverted to older metadata. See bottom. I have no special insight into this, but often at the root of this type of problem there is a failure to clearly specify what each of the components is supposed to do, resulting in a mismatch between reality and expectation on either side. Or using ambiguous terminology. I've seen that occasionally. dnf update --refresh currently redownloads the 43 MB metadata *always*. The manual page says: --refresh set metadata as expired before running the command Apparently, it doesn't try to verify/confirm whether the cache matches what the server offers. It seems to be less than clean expire-cache and more like clean metadata. It would need somebody with intimate knowledge of what mirror manager supports. To tell whether the package tool can decide which metadata are the latest. Perhaps timezones are not considered? Or why else did it decide to download something that's older? Did mirror manage tell that the cache metadata are not current? Here's a fresh reproducer. First it fetched the newer metadata, then it refreshed to something older: # dnf update --refresh Adobe Systems Incorporated 21 kB/s | 1.8 kB 00:00 Fedora - Rawhide - Developmental packages for t 1.5 MB/s | 43 MB 00:29 Last metadata expiration check performed 0:00:11 ago on Sat Aug 15 19:21:15 2015. Dependencies resolved. PackageArch VersionRepository Size Installing: kernel x86_64 4.2.0-0.rc6.git1.1.fc24rawhide 72 k kernel-corex86_64 4.2.0-0.rc6.git1.1.fc24rawhide 20 M kernel-modules x86_64 4.2.0-0.rc6.git1.1.fc24rawhide 18 M ncurses-compat-libsx86_64 6.0-1.20150810.fc24rawhide 305 k Upgrading: NetworkManager x86_64 1:1.0.6-0.1.20150813git7e2caa2.fc24 rawhide 2.0 M NetworkManager-adslx86_64 1:1.0.6-0.1.20150813git7e2caa2.fc24 rawhide 110 k NetworkManager-bluetooth x86_64 1:1.0.6-0.1.20150813git7e2caa2.fc24 rawhide 133 k NetworkManager-config-connectivity-fedora x86_64 1:1.0.6-0.1.20150813git7e2caa2.fc24 rawhide 100 k NetworkManager-glibx86_64 1:1.0.6-0.1.20150813git7e2caa2.fc24 rawhide 366 k NetworkManager-libnm x86_64 1:1.0.6-0.1.20150813git7e2caa2.fc24 rawhide 476 k NetworkManager-teamx86_64 1:1.0.6-0.1.20150813git7e2caa2.fc24 rawhide 112 k NetworkManager-wifix86_64 1:1.0.6-0.1.20150813git7e2caa2.fc24 rawhide 140 k NetworkManager-wwanx86_64 1:1.0.6-0.1.20150813git7e2caa2.fc24 rawhide 133 k PackageKit x86_64 1.0.7-3.fc24 rawhide 575 k PackageKit-cached-metadata x86_64 1.0.7-3.fc24 rawhide 73 M PackageKit-command-not-found x86_64 1.0.7-3.fc24 rawhide 27 k PackageKit-glibx86_64 1.0.7-3.fc24 rawhide 131 k PackageKit-gstreamer-pluginx86_64 1.0.7-3.fc24 rawhide 19 k PackageKit-gtk3-module x86_64 1.0.7-3.fc24 rawhide 19 k aclx86_64 2.2.52-10.fc24 rawhide 76 k attr x86_64 2.4.47-13.fc24 rawhide 64 k automake noarch 1.15-5.fc24rawhide 694 k boost x86_64 1.58.0-6.fc24 rawhide 43 k boost-atomic x86_64 1.58.0-6.fc24 rawhide 45 k boost-chrono x86_64 1.58.0-6.fc24 rawhide 52 k boost-containerx86_64 1.58.0-6.fc24 rawhide 66 k boost-context x86_64 1.58.0-6.fc24 rawhide 58 k boost-coroutinex86_64 1.58.0-6.fc24
Re: dnf --refresh reverts to older metadata (was: Re: dnf update vs Software Udpates)
On 15.08.2015, Michael Schwendt wrote: A day later, no matter how often I run dnf update --refresh, it never gets access to the newer metadata from yesterday again. Not the 76 packages as shown earlier in this thread, only the older 50. Jupp! It's exactly what I'm encountering since moving to F22, as shown several times in this list. So, indeed, there's something seriously wrong here, and I assume it can only be fixed if the developers of mirror manager and dnf come together and look into it. Indeed. -- users mailing list users@lists.fedoraproject.org To unsubscribe or change subscription options: https://admin.fedoraproject.org/mailman/listinfo/users Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines Have a question? Ask away: http://ask.fedoraproject.org
Re: dnf --refresh reverts to older metadata (was: Re: dnf update vs Software Udpates)
On Sat, 2015-08-15 at 13:21 +0200, Michael Schwendt wrote: So, indeed, there's something seriously wrong here, and I assume it can only be fixed if the developers of mirror manager and dnf come together and look into it. Worth a BZ report surely? I have no special insight into this, but often at the root of this type of problem there is a failure to clearly specify what each of the components is supposed to do, resulting in a mismatch between reality and expectation on either side. poc -- users mailing list users@lists.fedoraproject.org To unsubscribe or change subscription options: https://admin.fedoraproject.org/mailman/listinfo/users Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines Have a question? Ask away: http://ask.fedoraproject.org
dnf --refresh reverts to older metadata (was: Re: dnf update vs Software Udpates)
And here's proof of what can happen with just --refresh: 1. dnf update 2. dnf update --refresh 3. dnf update --refresh The last run reverts to older metadata with only 50 updates available compared with earlier. Mirror manager assigning to an out-of-date mirror? # dnf update Last metadata expiration check performed 1:19:01 ago on Fri Aug 14 11:29:15 2015. Dependencies resolved. Package Arch Version Repository Size Installing: kernelx86_64 4.2.0-0.rc6.git0.2.fc24 rawhide72 k kernel-core x86_64 4.2.0-0.rc6.git0.2.fc24 rawhide20 M kernel-modulesx86_64 4.2.0-0.rc6.git0.2.fc24 rawhide18 M Upgrading: automake noarch 1.15-5.fc24 rawhide 694 k boost x86_64 1.58.0-6.fc24 rawhide43 k boost-atomic x86_64 1.58.0-6.fc24 rawhide45 k boost-chrono x86_64 1.58.0-6.fc24 rawhide52 k boost-container x86_64 1.58.0-6.fc24 rawhide66 k boost-context x86_64 1.58.0-6.fc24 rawhide58 k boost-coroutine x86_64 1.58.0-6.fc24 rawhide59 k boost-date-time x86_64 1.58.0-6.fc24 rawhide58 k boost-devel x86_64 1.58.0-6.fc24 rawhide 8.1 M boost-filesystem x86_64 1.58.0-6.fc24 rawhide73 k boost-graph x86_64 1.58.0-6.fc24 rawhide 137 k boost-iostreams x86_64 1.58.0-6.fc24 rawhide67 k boost-locale x86_64 1.58.0-6.fc24 rawhide 278 k boost-log x86_64 1.58.0-6.fc24 rawhide 478 k boost-mathx86_64 1.58.0-6.fc24 rawhide 312 k boost-program-options x86_64 1.58.0-6.fc24 rawhide 165 k boost-python x86_64 1.58.0-6.fc24 rawhide 133 k boost-random x86_64 1.58.0-6.fc24 rawhide51 k boost-regex x86_64 1.58.0-6.fc24 rawhide 296 k boost-serialization x86_64 1.58.0-6.fc24 rawhide 156 k boost-signals x86_64 1.58.0-6.fc24 rawhide70 k boost-system x86_64 1.58.0-6.fc24 rawhide48 k boost-testx86_64 1.58.0-6.fc24 rawhide 220 k boost-thread x86_64 1.58.0-6.fc24 rawhide88 k boost-timer x86_64 1.58.0-6.fc24 rawhide50 k boost-wavex86_64 1.58.0-6.fc24 rawhide 234 k crontabs noarch 1.11-11.20150630git.fc24rawhide23 k curl x86_64 7.44.0-1.fc24 rawhide 284 k dbus x86_64 1:1.9.20-1.fc24 rawhide 240 k dbus-develx86_64 1:1.9.20-1.fc24 rawhide57 k dbus-libs x86_64 1:1.9.20-1.fc24 rawhide 171 k dbus-x11 x86_64 1:1.9.20-1.fc24 rawhide51 k dhcp-client x86_64 12:4.3.3-0.2b1.fc24 rawhide 297 k dhcp-common noarch 12:4.3.3-0.2b1.fc24 rawhide 192 k dhcp-libs x86_64 12:4.3.3-0.2b1.fc24 rawhide 136 k dracutx86_64 043-60.git20150811.fc24 rawhide 322 k dracut-config-rescue x86_64 043-60.git20150811.fc24 rawhide44 k dracut-networkx86_64 043-60.git20150811.fc24 rawhide82 k firefox x86_64 40.0-4.fc24 rawhide71 M fontconfigx86_64 2.11.94-3.fc24 rawhide 241 k fontconfig-devel x86_64 2.11.94-3.fc24 rawhide 136 k gnupg2x86_64 2.1.6-1.fc24rawhide 1.8 M hplip x86_64 3.15.7-4.fc24 rawhide12 M hplip-common x86_64 3.15.7-4.fc24 rawhide96 k hplip-compat-libs x86_64 3.15.7-4.fc24 rawhide76 k hplip-libsx86_64 3.15.7-4.fc24 rawhide 185 k kernel-headersx86_64 4.2.0-0.rc6.git0.2.fc24 rawhide 1.0 M libblkid x86_64 2.27-0.2.fc24 rawhide 178 k libcacard x86_64 2:2.4.0-1.fc24 rawhide74 k libcurl x86_64 7.44.0-1.fc24 rawhide 256 k libcurl-devel
