Re: No more deltarpms by default
On Sat, Nov 8, 2014 at 2:25 PM, Reindl Harald h.rei...@thelounge.net wrote: Am 08.11.2014 um 20:21 schrieb Nico Kadel-Garcia: [root@rawhide ~]# rm -rf /var/cache/dnf/* I do think you also flushed all the historical data on what packages were installed from what repositories no - that would be rm -rf /var/lib/yum/* rm -rf /var/lib/dnf/* and yes i delete them regulary on machines which made 10 or more dist-upgrades in the history, typially one week after a dist-upgrade Good point, thanks. -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: No more deltarpms by default
On Fri, Nov 7, 2014 at 3:31 PM, Reindl Harald h.rei...@thelounge.net wrote: the meta-data are fat look below The metadata are bulky, fairly monolithic databases. They cannot easily support incremental updates of the metadata because they are compressed, singular files storing data for all available packages. The bandwidth and storage costs are thus singificant. It's nowhere near as bad as I'd concluded from my back-of-the-envelope calculations because the base OS metadata is stable, the churn is among the updates, and the expiration is set longer than I'd realized on Fedora. (I'd been looking at the defaults and making unwarranted conclusions about update frequency from RHEL experience.) [root@rawhide ~]# rm -rf /var/cache/dnf/* [root@rawhide ~]# df DateisystemTyp Größe Benutzt Verf. Verw% Eingehängt auf /dev/sdb1 ext4 12G598M 11G6% / /dev/sda1 ext4 487M 35M 448M8% /boot [root@rawhide ~]# dnf --disablerepo=koji info kernel 2 /dev/null /dev/null [root@rawhide ~]# df DateisystemTyp Größe Benutzt Verf. Verw% Eingehängt auf /dev/sdb1 ext4 12G713M 11G7% / /dev/sda1 ext4 487M 35M 448M8% /boot [root@rawhide ~]# rm -rf /var/cache/dnf/* I do think you also flushed all the historical data on what packages were installed from what repositories, and local cookies, when you did that. Next time, just use 'dnf clean metadata*' and 'dnf list, to avoid playing with anything else. The du command will also gove you a lot more detail about what is taking up all the space. Overall. Yest, it's bulky. 'deltarpms' does not help with this underlying churn required for even reporting new available packages. I still think that, for most folks, it's a larger bandwidth burden than the occassional update, but that's no longer the case if you update packages frequently. -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: No more deltarpms by default
Am 08.11.2014 um 20:21 schrieb Nico Kadel-Garcia: [root@rawhide ~]# rm -rf /var/cache/dnf/* I do think you also flushed all the historical data on what packages were installed from what repositories no - that would be rm -rf /var/lib/yum/* rm -rf /var/lib/dnf/* and yes i delete them regulary on machines which made 10 or more dist-upgrades in the history, typially one week after a dist-upgrade signature.asc Description: OpenPGP digital signature -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: No more deltarpms by default
Hi On Fri, Nov 7, 2014 at 1:19 AM, Dennis Gilmore wrote: /etc/yum.repos.d/fedora-updates.repo on f21 has metadata_expire=6h I was looking at dnf since the discussion is about dnf and the metadata right now for f21 updates is an empty repo with no packages in it f20 updates repo http://dl.fedoraproject.org/pub/fedora/linux/updates/20/x86_64/repodata/ right now has primary.sqllite of 12M and filelists.sqlite of 19M not sure how you got 32K Not sure. Probably misread something Rahul -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: No more deltarpms by default
Am 07.11.2014 um 17:53 schrieb Rahul Sundaram: On Fri, Nov 7, 2014 at 1:19 AM, Dennis Gilmore wrote: /etc/yum.repos.d/fedora-updates.repo on f21 has metadata_expire=6h I was looking at dnf since the discussion is about dnf where do you see DNF using other than /etc/yum.repos.d/ DNF is using the same directory and repo configs that's part of the fedora release package and the metadata right now for f21 updates is an empty repo with no packages in it f20 updates repo http://dl.fedoraproject.org/pub/fedora/linux/updates/20/x86_64/repodata/ right now has primary.sqllite of 12M and filelists.sqlite of 19M not sure how you got 32K Not sure. Probably misread something the meta-data are fat look below [root@rawhide ~]# rm -rf /var/cache/dnf/* [root@rawhide ~]# df DateisystemTyp Größe Benutzt Verf. Verw% Eingehängt auf /dev/sdb1 ext4 12G598M 11G6% / /dev/sda1 ext4 487M 35M 448M8% /boot [root@rawhide ~]# dnf --disablerepo=koji info kernel 2 /dev/null /dev/null [root@rawhide ~]# df DateisystemTyp Größe Benutzt Verf. Verw% Eingehängt auf /dev/sdb1 ext4 12G713M 11G7% / /dev/sda1 ext4 487M 35M 448M8% /boot [root@rawhide ~]# rm -rf /var/cache/dnf/* [root@rawhide ~]# df DateisystemTyp Größe Benutzt Verf. Verw% Eingehängt auf /dev/sdb1 ext4 12G598M 11G6% / /dev/sda1 ext4 487M 35M 448M8% /boot signature.asc Description: OpenPGP digital signature -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: No more deltarpms by default
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Mon, 6 Oct 2014 08:16:30 -0700 Gerald B. Cox gb...@bzb.us wrote: On Mon, Oct 6, 2014 at 8:00 AM, drago01 drag...@gmail.com wrote: I am not convinced that being fast and download less are mutually exclusive when using deltas. So we should keep deltas *and* make them faster. Exactly. The fact that some users have more bandwidth means exactly what? Most people also have faster processors and disks now. It is more efficient from a networking perspective to minimize unnecessary traffic and use local processing. That was behind the rationale when delta was introduced and made the default. It was valid then, and it is valid now. You don't change the default without following the proper process. even in some countries that are highly developed there is issues. In Australia for instance it is extremely common to have download caps. having deltas even if slower will give a better experience as users will not use up as much of the download cap. What we have is different scenarios we can not possibly know all of the data about. we should err on the side of being conservative in places where the potential cost is highest. while the cost of applying deltarpms can be high I think that the potential cost of bandwidth is higher, as such we should default to deltarpm support and make it easy to disable for those that want to. Dennis -BEGIN PGP SIGNATURE- Version: GnuPG v2 iQIcBAEBAgAGBQJUXGBsAAoJEH7ltONmPFDRcBIP/iYYKF343fgyv6gznyPqdmxm FI3KKTuYEPMNc+Ze+S/A8XM0Oy7NLWHVEn2EuHs7VLMHp31SHulJyO3cfPlAvrWX NBz6R7GOYvkNgL1MGd1Hw/+M0aVTaza7FK451T+mTU13Dy0s3MwjpkTUEBNqs1U5 tagUg77kthrY/R510OS5FK5QZQW4/nZ852wPZLMrUG44e0/aAIIAaDSCTzHLWIt0 AttShV07WbVuk/0cVerBcpaoS6+kU3B7N+mEfy4+3Ra1/t7oKwdhYKhXWioZ6dVw lgWfr1UzkbxfX/5q5Zmcnt8bwbXtK9cssvGAR9srlwMWvMKRqT666n2ZK6nMu2ml Uw/Ue1RdiOsRGEXNuKiFRY5cuDVD18TeflPtAklNn2BWmeAqJx3Wlv4b2T8lTBWC w4YuyKByoe9Ul47F7onSHL0vS1Zzs4lT3rUAsYLKilv7wji7VIphCKlJQ3zzxqtu EEfRQlHGD1stfNAghHeop4/yiAmmUeRFGPo1tUTkhe5+pweVY5cqNaFMZ6eDJdyq sLG1k3nOnXVkqb5w03OkCcyg7a54zBju3XEj0gnmQpOtPNJ4PZAF5/XkWjb3e6Bu psV9z9IOgxYroXP6kPH3aOTcdiUA8XRTYfwln0BCsyQP3XBY5iegPvsslWjkJOza xCNCQtwrI3VyYmGfyTb4 =AloG -END PGP SIGNATURE- -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: No more deltarpms by default
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Sun, 19 Oct 2014 22:56:40 -0400 Rahul Sundaram methe...@gmail.com wrote: Hi On Sun, Oct 19, 2014 at 9:52 PM, Nico Kadel-Garcia wrote: It's a rough figure, from http://mirrors.kernel.org/fedora/releases/20/Everything/x86_64/os/repodata/ That doesn't seem to be the right way to look at the numbers. In a local system, there are two different repositories enabled by default - Fedora and Updates. Fedora repodata is about 40MB but only expires every 7 days . The updates repo is about 32K and relies on the default metadata expiry period which is 2 days. /etc/yum.repos.d/fedora-updates.repo on f21 has metadata_expire=6h and the metadata right now for f21 updates is an empty repo with no packages in it f20 updates repo http://dl.fedoraproject.org/pub/fedora/linux/updates/20/x86_64/repodata/ right now has primary.sqllite of 12M and filelists.sqlite of 19M not sure how you got 32K Dennis references: /usr/lib/systemd/system/dnf-makecache.service /etc/yum.repos.d/fedora.repo /etc/yum.repos.d/fedora-updates.repo man dnf and dnf.conf Rahul -BEGIN PGP SIGNATURE- Version: GnuPG v2 iQIcBAEBAgAGBQJUXGSHAAoJEH7ltONmPFDRdegP/2Gg2BOic7MHxsEduLn1gOMe wwlDrzfoIWDsMI8NrQOhstNaVzWDuXvkRADNczWBva5LWDK57+GVuoWeVKf78qwM 6TbBatIKQQ5Jaa1lOF/zXZxeymclHkaPwg1UOGlRODmmyTJgUV82i5FjLSiaclIC y/AMfytmy+Vk+EgdEjYtyD2L1FKWTrJyeifhanWvxShsSTDOOazoz2nE5boWYEk6 estIfw2BB+AwtxeEWrc3/YYwryxCQFyZrMZ7supLF92ZHybXzDNDT3EpH2aBwqaO txm4FVhhE4q4HQjUgY1QJQGPlHxxrGK9zLjEUx0ABHb84zz1UoMo7vG06g6/p/sj L14fCqNc7tY9HTyjMzT3jG4vG9cluS1AZ8MsjnG8hkA1ouABx6YOqIiJFAPY6i6a BLsPevhZ5YKFZfZ7BxmVu/e3bpHnTNgkL+/HghZz77KIUSKS2Mp2+14jvszYpBMG kfRLThVoVYEcrWiDfULKyyFHTvFsDxEyvQSkctQz/Y/fcnqNUlcVAforE6lrc+l3 Wq4BE2gmvSgsBnqB5X9A/YBwHYi8qxxWHoqY78CxSdZjCay+nViXqWICL6MUghJD h8CWsHkdJrxD075E3FGJFZMmwfMCBHfUVgM56VxwwI2urBXCi968YTAQIczLXsUC 5EvMzsfQX2AfGYxIf3N7 =4K9h -END PGP SIGNATURE- -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: No more deltarpms by default
On 10/17/2014 05:52 AM, poma wrote: On 06.10.2014 16:46, Jaroslav Reznik wrote: - Original Message - On Mon, 2014-10-06 at 10:54 +0100, Ian Malone wrote: On 6 October 2014 09:41, Rahul Sundaram methe...@gmail.com wrote: Hi One of the long standing features that were enabled by default in yum is support for delta rpms. dnf developers have disabled this and I think this change deserves a broader discussion https://bugzilla.redhat.com/show_bug.cgi?id=1148208 I have an internet flatrate at 150 mbs, and downloading the full rpms is ALOT faster than the the work that the delta rpms requires. Wow. Good to see normal users are taken into account. The main argument from a distro point of view is reducing load on servers, but I don't know many people on 150Mbs either. Heck, I've just tested my work janet connection and that's 100Mbs in our office. At home 8Mbps is a good day. (I'm assuming mbs is a typo for Mbps and not milli bit seconds, where the very slow transfer speed declines exponentially as the connection progresses.) The deltarpms were meant to serve two purposes 1) (lesser) Address the needs of users in developing countries (where Fedora is fairly popular) and bandwidth concerns are very considerable. Many of these users have connections that are either metered or extremely slow, so deltarpms provides a way to get the data to them more economically. This of course can be handled with a non-default option, so we can talk about making that more discoverable if we disable them by default. This is a good point but even in developing countries internet access is getting better and better. A few years ago installation DVD was the only way how to access Fedora repo. It's not requested anymore. But yeah, I do not live there. I strongly disagree with this point. The internet access in developing countries may be getting better, but that is primarily city focused, and even in cities, the cost of internet access is still high. In a large developing country like India (where I live), a huge percentage of the population lives away from the cities. Unlike people living in the cities, the people in the small towns are more likely to be open to using Fedora, for reasons of both cost and principles. I work with the Fedora FreeMedia program, and I send out around 50 Fedora DVDs a month, out of which a good number is to individuals and organizations in rural areas. Regarding the use of deltarpms, if it were not for the availability of deltarpms, a large number of Fedora systems in developing countries will remain not updated, or very irregularly updated, which will result in a large number of Fedora users being exposed to security vulnerabilities. The users will not be able to help it, because the cost for internet access here is often tied to bandwidth usage, and even a weekly update of Fedora using the full rpms will be too expensive. Add to that the unreliability of the internet access, leading to download errors for bigger packages, and most users will choose not to update. So from what I know, based on my personal experience, and my experience in the Fedora FreeMedia program, the Fedora install and Live DVDs are still very relevant, and deltarpms are extremely relevant for developing countries. Please do not sound the death knell for these highly essential tools, based just on the experience in developed countries. p.s. Thanks to poma for bringing this mail thread to my attention. I was not on the Fedora devel list so far. I have now subscribed to the Fedora devel list, so as not to miss such important discussions. :-) -- Regards, Rejy M Cyriac (rmc) -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: No more deltarpms by default
On Mon, Oct 6, 2014 at 10:41 AM, Rahul Sundaram methe...@gmail.com wrote: Hi One of the long standing features that were enabled by default in yum is support for delta rpms. dnf developers have disabled this and I think this change deserves a broader discussion https://bugzilla.redhat.com/show_bug.cgi?id=1148208 I don't know if this really will happen, but when I read this I had to think about this discussion. http://uk.reuters.com/article/2014/10/22/uk-hungary-internet-tax-idUKKCN0IB0RI20141022 So deltarpm might be a way of actually saving money ;-) -johannes Rahul -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: No more deltarpms by default
Am 20.10.2014 um 04:02 schrieb Nico Kadel-Garcia:
On Sun, Oct 19, 2014 at 5:26 AM, Reindl Harald h.rei...@thelounge.net wrote:
Am 19.10.2014 um 06:37 schrieb Nico Kadel-Garcia:
On Sat, Oct 18, 2014 at 8:28 PM, Reindl Harald h.rei...@thelounge.net
wrote:
#!/usr/bin/bash
basearch=`uname -i`
releasever=`rpm -q --qf %{version}\n fedora-release`
for g in `ls -1b /var/cache/yum`
do
if [ -d /var/cache/yum/$g/packages ]
then
echo /var/cache/yum/$g/packages/ /repo/cache/fc$releasever/
sudo mv --verbose /var/cache/yum/$g/packages/*.rpm
/repo/cache/fc$releasever/ 2 /dev/null
fi
done
/buildserver/repo-create.sh
And that only works on one machine, and is sensitive to exclusions
written into yum.conf or /etc/yum.repos.d/*, etc., etc., etc. You also
forgot to use 'gpgcheck' to verify the contents of the already
downloaded RPM, since 'repsync' does not track successful versus
partial downloads without it.
Anyway. Now multiply your 'reposync' based tool by 200 thin
provisioned virtual machines, and you're chewing up quite a lot of
disk space doing this
you don't get it
* that single machine has all used packages installed
* that single machine builds a repo below /repo/cache/fc$releasever/
* that repo don't mirror blindly anything, just used RPM's
* that RPMs are the result *after* apply deltarpm out of yum cache
I get it
no
It starts with the the fact that your script has not
verified *any* of your RPM's, and reposync does not ordinarily verify
file contents unless you activate GPG verification. So any partial
transfer or interrupted transfer will effectively corrupted your local
repo, especially if anything else is dependent on that package. Do
take a look at the github scripts I pointed out, they do a more
thorough job.
what are you talking about reposync all the time?
frankly i have posted the whole source above
/buildserver/repo-create.sh can be read as createrepo chown
chmod rsync-to-failover maschine on the cluster
*/var/cache/yum/* is the source *after* ran updates on that machine
ordinary with yum - guess what - after that suceeded the RPM's GPG
verified and *that* is what i try to explain the whole time - yum don't
need to know about deltarpm and so other scripts using the yum-cache
don't too
And running 'reposync still downloads the upstream repodata. You've
really saved very little bandwidth as long as you keep doing this, and
you *need* to keep downloading the bulky repodata to ensure getting
the latest releases of packages. Yum repodata itself has no 'deltarps'
for its own information, it's compressed binary database blobs.
Last, doing this on an individual machine does not scale You're
chewing up disk space on each machine, and bandwitdh, with all the
required 'repodata' downloads
WTF - no single other machine has *any* repo in the WAN configured
signature.asc
Description: OpenPGP digital signature
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: No more deltarpms by default
Hi, 3) People who have a lot of hosts and high bandwidth, high speed local deployment requirements can, and do, set up an internal Fedora mirror with much lower bandwidth costs. Maybe mirrormanager can give a hint in case it directs yum/dnf to a onsite mirror? cheers, Gerd -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: No more deltarpms by default
On Fr, 2014-10-17 at 11:49 +, Andre Robatino wrote: Roberto Ragusa mail at robertoragusa.it writes: Are compressed rpms completely impossible to diff efficiently by rsync? In a word, yes. They're already compressed, so it's unlikely there would be any matching blocks between old and new rpms for rsync to take advantage of. Years ago I saw rsync being patched for this (done by suse IIRC). Basic trick is to teach rsync to unpack/repack rpms, thereby effectively rsyning the files within the rpm, which works *alot* better than using the compressed rpm payload. Obvious downside is that it is very cpu consuming, which likely is a showstopper at least for public mirrors (and I guess it never went upstream for that reason). cheers, Gerd -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: No more deltarpms by default
Am 20.10.2014 um 12:31 schrieb Gerd Hoffmann: On Fr, 2014-10-17 at 11:49 +, Andre Robatino wrote: Roberto Ragusa mail at robertoragusa.it writes: Are compressed rpms completely impossible to diff efficiently by rsync? In a word, yes. They're already compressed, so it's unlikely there would be any matching blocks between old and new rpms for rsync to take advantage of. Years ago I saw rsync being patched for this (done by suse IIRC). Basic trick is to teach rsync to unpack/repack rpms, thereby effectively rsyning the files within the rpm, which works *alot* better than using the compressed rpm payload and has exactly the same result as deltarpm now where the most performance drop is repack the RPM based on the delta - you you are doing exactly the same more complex and introduce additional software in the mix signature.asc Description: OpenPGP digital signature -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: No more deltarpms by default
On Sun, Oct 19, 2014 at 11:07 PM, Kevin Fenzi ke...@scrye.com wrote: On Sun, 19 Oct 2014 21:52:00 -0400 Nico Kadel-Garcia nka...@gmail.com wrote: It's a rough figure, from http://mirrors.kernel.org/fedora/releases/20/Everything/x86_64/os/repodata/ (a useful local mirror). Since the number of packages there changes day to day, but are likely to increase monotonically over time, it's a reasonable working figure for back of the envelope calcualations. To clarify, that repo never changes after release. ;) kevin Thanks for the correction on this. I thought it was a meged repo. -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: No more deltarpms by default
On Mon, Oct 20, 2014 at 4:22 AM, Reindl Harald h.rei...@thelounge.net wrote: Am 20.10.2014 um 04:02 schrieb Nico Kadel-Garcia: On Sun, Oct 19, 2014 at 5:26 AM, Reindl Harald h.rei...@thelounge.net wrote: Am 19.10.2014 um 06:37 schrieb Nico Kadel-Garcia: On Sat, Oct 18, 2014 at 8:28 PM, Reindl Harald h.rei...@thelounge.net wrote: It starts with the the fact that your script has not verified *any* of your RPM's, and reposync does not ordinarily verify file contents unless you activate GPG verification. So any partial transfer or interrupted transfer will effectively corrupted your local repo, especially if anything else is dependent on that package. Do take a look at the github scripts I pointed out, they do a more thorough job. what are you talking about reposync all the time? You've a point, I was misreading your script. frankly i have posted the whole source above /buildserver/repo-create.sh can be read as createrepo chown chmod rsync-to-failover maschine on the cluster Mind you, putting this stuff in places like /repo/cache and /buildserver is a direct violation of the file system hierarchy, but that's your own local policy and potential problem. */var/cache/yum/* is the source *after* ran updates on that machine ordinary with yum - guess what - after that suceeded the RPM's GPG verified and *that* is what i try to explain the whole time - yum don't need to know about deltarpm and so other scripts using the yum-cache don't too I do think you'd be much better off with using rsync and a local mirror. You can point it to architectures and OS releases you aren't currently running, and point it to include all potential updates. Even your your setup, you can use 'yum --doanloadonly' and 'yum install *' to ensure the presence of more packages in /var/cache/yum Last, doing this on an individual machine does not scale You're chewing up disk space on each machine, and bandwitdh, with all the required 'repodata' downloads WTF - no single other machine has *any* repo in the WAN configured *That* part was missing. You've configured your local hosts to all point to an individual mirror? And that local repo has *all* packages you might possibly install on the other machines? -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: No more deltarpms by default
On Mon, Oct 20, 2014 at 6:40 AM, Reindl Harald h.rei...@thelounge.net wrote: Am 20.10.2014 um 12:31 schrieb Gerd Hoffmann: On Fr, 2014-10-17 at 11:49 +, Andre Robatino wrote: Roberto Ragusa mail at robertoragusa.it writes: Are compressed rpms completely impossible to diff efficiently by rsync? In a word, yes. They're already compressed, so it's unlikely there would be any matching blocks between old and new rpms for rsync to take advantage of. Years ago I saw rsync being patched for this (done by suse IIRC). Basic trick is to teach rsync to unpack/repack rpms, thereby effectively rsyning the files within the rpm, which works *alot* better than using the compressed rpm payload and has exactly the same result as deltarpm now where the most performance drop is repack the RPM based on the delta - you you are doing exactly the same more complex and introduce additional software in the mix It also creates a profound computational burden on the client and server, and it also doesn't work well for tools like libreoffice, whose component filea are also pre-compressed into war files. Hey, if we keep compressing it, we'll get it down to size zero and won't use any bandwidth. I'm just not finding justification for deltarpms in scaleable environments, even though my estimates of the cost of 'repodata' synchronization were way overblown. -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: No more deltarpms by default
Am 20.10.2014 um 13:57 schrieb Nico Kadel-Garcia: On Mon, Oct 20, 2014 at 4:22 AM, Reindl Harald h.rei...@thelounge.net wrote: Am 20.10.2014 um 04:02 schrieb Nico Kadel-Garcia: On Sun, Oct 19, 2014 at 5:26 AM, Reindl Harald h.rei...@thelounge.net wrote: Am 19.10.2014 um 06:37 schrieb Nico Kadel-Garcia: On Sat, Oct 18, 2014 at 8:28 PM, Reindl Harald h.rei...@thelounge.net wrote: It starts with the the fact that your script has not verified *any* of your RPM's, and reposync does not ordinarily verify file contents unless you activate GPG verification. So any partial transfer or interrupted transfer will effectively corrupted your local repo, especially if anything else is dependent on that package. Do take a look at the github scripts I pointed out, they do a more thorough job. what are you talking about reposync all the time? You've a point, I was misreading your script. now we come together :-) not that it is a large or complex script frankly i have posted the whole source above /buildserver/repo-create.sh can be read as createrepo chown chmod rsync-to-failover maschine on the cluster Mind you, putting this stuff in places like /repo/cache and /buildserver is a direct violation of the file system hierarchy, but that's your own local policy and potential problem. that are just symlinks and having that folders not in the FHS means never ever some package will start to use them - so violate the FHS is by intention also as mountpoints for datadisks /Volumes/dune */var/cache/yum/* is the source *after* ran updates on that machine ordinary with yum - guess what - after that suceeded the RPM's GPG verified and *that* is what i try to explain the whole time - yum don't need to know about deltarpm and so other scripts using the yum-cache don't too I do think you'd be much better off with using rsync and a local mirror. You can point it to architectures and OS releases you aren't currently running, and point it to include all potential updates. Even your your setup, you can use 'yum --doanloadonly' and 'yum install *' to ensure the presence of more packages in /var/cache/yum no because anything wich touchs the infarstructure is tested or in a lot of cases built on that machine (sample setups and so on) and there is no need for anything more starting 2008 on F9 until now upgraded with yum to F20 on around 30 machines Last, doing this on an individual machine does not scale You're chewing up disk space on each machine, and bandwitdh, with all the required 'repodata' downloads WTF - no single other machine has *any* repo in the WAN configured *That* part was missing. You've configured your local hosts to all point to an individual mirror? And that local repo has *all* packages you might possibly install on the other machines? no it was not, please go back to the first message you responded to the idea doing what i described is to not have any machine touch repos in the internt and so after test a update which don't work on the buildserver results in delete it instead push to the local repo so no other machine can see it - frankly if you build up a maitained infrastructure you don't want that one server has a chance to pull a more recent mirror and update stuff you have never seen in the QA the first message you responded to had the following paragraph you even quoted that few lines below are enough to use createrepo and build up a local cache without mirror the whole upstream, you just need to have one machine with any pakcge you use installed on it - works perfect over 6 years including dist-upgrades *and* benefits from deltarpm in the first step as well follow-up quotes from you contained * that single machine has all used packages installed * that single machine builds a repo below /repo/cache/fc$releasever/ * that repo don't mirror blindly anything, just used RPM's * that RPMs are the result *after* apply deltarpm out of yum cache signature.asc Description: OpenPGP digital signature -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: No more deltarpms by default
On Sun, Oct 19, 2014 at 1:04 AM, Rahul Sundaram methe...@gmail.com wrote:
Hi
On Sun, Oct 19, 2014 at 12:37 AM, Nico Kadel-Garcia wrote:
. And the bandwidth saving is still badly
overshadowed by the necessary repodata bandwith from almost every run
of reposync in the model you've just described. 50 Megabytes, *every
time* it runs unless the metadata for the repo has not expired. .
Since the default is 90 minutes,
It is not for dnf. man dnf.conf
metadata_expire ... The default corresponds to 48 hours
Rahul
Odd, that doesn't match the DNF documentation at
http://rpm-software-management.github.io/dnf/conf_ref.html which lists
a three hour expiration. Wonder if it's been tweaked in the SRP{M?
Even then, though still means up to 60 MB * 15/month, or 900 MB/month
even with no updates ever actually applied, if you leave the update
alert tools in place as found in a default OS installation. This still
overshadows the benefits of deltarpm profoundly unless you're
constantly updating builky tools like Libreoffice. And deltarpm
doesn't help much with those, anyway, due to the mismatch of the zip
compressed binary contents.
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: No more deltarpms by default
Hi On Sun, Oct 19, 2014 at 3:40 AM, Nico Kadel-Garcia wrote: Even then, though still means up to 60 MB * 15/month, Earlier you mentioned 50 MB. Now you claim 60 MB. Can you show where these figures are from? Rahul -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: No more deltarpms by default
Am 19.10.2014 um 06:37 schrieb Nico Kadel-Garcia:
On Sat, Oct 18, 2014 at 8:28 PM, Reindl Harald h.rei...@thelounge.net wrote:
#!/usr/bin/bash
basearch=`uname -i`
releasever=`rpm -q --qf %{version}\n fedora-release`
for g in `ls -1b /var/cache/yum`
do
if [ -d /var/cache/yum/$g/packages ]
then
echo /var/cache/yum/$g/packages/ /repo/cache/fc$releasever/
sudo mv --verbose /var/cache/yum/$g/packages/*.rpm
/repo/cache/fc$releasever/ 2 /dev/null
fi
done
/buildserver/repo-create.sh
And that only works on one machine, and is sensitive to exclusions
written into yum.conf or /etc/yum.repos.d/*, etc., etc., etc. You also
forgot to use 'gpgcheck' to verify the contents of the already
downloaded RPM, since 'repsync' does not track successful versus
partial downloads without it.
Anyway. Now multiply your 'reposync' based tool by 200 thin
provisioned virtual machines, and you're chewing up quite a lot of
disk space doing this
you don't get it
* that single machine has all used packages installed
* that single machine builds a repo below /repo/cache/fc$releasever/
* that repo don't mirror blindly anything, just used RPM's
* that RPMs are the result *after* apply deltarpm out of yum cache
signature.asc
Description: OpenPGP digital signature
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: No more deltarpms by default
On Sun, Oct 19, 2014 at 3:51 AM, Rahul Sundaram methe...@gmail.com wrote: Hi On Sun, Oct 19, 2014 at 3:40 AM, Nico Kadel-Garcia wrote: Even then, though still means up to 60 MB * 15/month, Earlier you mentioned 50 MB. Now you claim 60 MB. Can you show where these figures are from? Rahul It's a rough figure, from http://mirrors.kernel.org/fedora/releases/20/Everything/x86_64/os/repodata/ (a useful local mirror). Since the number of packages there changes day to day, but are likely to increase monotonically over time, it's a reasonable working figure for back of the envelope calcualations. -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: No more deltarpms by default
On Sun, Oct 19, 2014 at 5:26 AM, Reindl Harald h.rei...@thelounge.net wrote:
Am 19.10.2014 um 06:37 schrieb Nico Kadel-Garcia:
On Sat, Oct 18, 2014 at 8:28 PM, Reindl Harald h.rei...@thelounge.net
wrote:
#!/usr/bin/bash
basearch=`uname -i`
releasever=`rpm -q --qf %{version}\n fedora-release`
for g in `ls -1b /var/cache/yum`
do
if [ -d /var/cache/yum/$g/packages ]
then
echo /var/cache/yum/$g/packages/ /repo/cache/fc$releasever/
sudo mv --verbose /var/cache/yum/$g/packages/*.rpm
/repo/cache/fc$releasever/ 2 /dev/null
fi
done
/buildserver/repo-create.sh
And that only works on one machine, and is sensitive to exclusions
written into yum.conf or /etc/yum.repos.d/*, etc., etc., etc. You also
forgot to use 'gpgcheck' to verify the contents of the already
downloaded RPM, since 'repsync' does not track successful versus
partial downloads without it.
Anyway. Now multiply your 'reposync' based tool by 200 thin
provisioned virtual machines, and you're chewing up quite a lot of
disk space doing this
you don't get it
* that single machine has all used packages installed
* that single machine builds a repo below /repo/cache/fc$releasever/
* that repo don't mirror blindly anything, just used RPM's
* that RPMs are the result *after* apply deltarpm out of yum cache
I get it. It starts with the the fact that your script has not
verified *any* of your RPM's, and reposync does not ordinarily verify
file contents unless you activate GPG verification. So any partial
transfer or interrupted transfer will effectively corrupted your local
repo, especially if anything else is dependent on that package. Do
take a look at the github scripts I pointed out, they do a more
thorough job.
And running 'reposync still downloads the upstream repodata. You've
really saved very little bandwidth as long as you keep doing this, and
you *need* to keep downloading the bulky repodata to ensure getting
the latest releases of packages. Yum repodata itself has no 'deltarps'
for its own information, it's compressed binary database blobs.
Last, doing this on an individual machine does not scale You're
chewing up disk space on each machine, and bandwitdh, with all the
required 'repodata' downloads.
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: No more deltarpms by default
Hi On Sun, Oct 19, 2014 at 9:52 PM, Nico Kadel-Garcia wrote: It's a rough figure, from http://mirrors.kernel.org/fedora/releases/20/Everything/x86_64/os/repodata/ That doesn't seem to be the right way to look at the numbers. In a local system, there are two different repositories enabled by default - Fedora and Updates. Fedora repodata is about 40MB but only expires every 7 days . The updates repo is about 32K and relies on the default metadata expiry period which is 2 days. references: /usr/lib/systemd/system/dnf-makecache.service /etc/yum.repos.d/fedora.repo /etc/yum.repos.d/fedora-updates.repo man dnf and dnf.conf Rahul -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: No more deltarpms by default
On Sun, 19 Oct 2014 21:52:00 -0400 Nico Kadel-Garcia nka...@gmail.com wrote: It's a rough figure, from http://mirrors.kernel.org/fedora/releases/20/Everything/x86_64/os/repodata/ (a useful local mirror). Since the number of packages there changes day to day, but are likely to increase monotonically over time, it's a reasonable working figure for back of the envelope calcualations. To clarify, that repo never changes after release. ;) kevin signature.asc Description: PGP signature -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: No more deltarpms by default
On Thu, Oct 16, 2014 at 11:40 PM, Gerald B. Cox gb...@bzb.us wrote: My comment was not meant to be argumentative, but rather tongue-in-cheek. However, I do believe when changing a default, it isn't about what is convenient for me. It's about what is best for the entire community and what are the real world ramifications. I'm not buying the let's change the default because high bandwidth is pervasive argument. I'll go out on a limb here and suggest: 1. Most people who can afford to pay the monthly recurring cost for a high speed bandwidth connection have multi-core machines. 2. People who are running Fedora on multiple machines possess the skill set to easily change the default and turn Presto off if they wish. You left out some but related issues. 3) People who have a lot of hosts and high bandwidth, high speed local deployment requirements can, and do, set up an internal Fedora mirror with much lower bandwidth costs. This reduces the tangible benefits of deltarpms significantly. This is combined with my direct observation that working from plain rpms is much faster and more reliable. Re-assembly from deltarpms is computationally expensive and thus expensive for large numbers of yum clients. It sounds like a great idea at first glance, but it profoundly and unpredicatably increases the disk space and bandwidth needed for mirrors themselves at a relatively small benefit in download bandwidth. The great example of difficult to delta RPM's sems to be libreoffice. Between the compressed and distinct binaries for the war files, and the deltarpm process itself, they save very little bandwidth for many of the builkies projects. 4) The much, much larger source of wasted bandwidth and resources is the yum repodata. That's 50 Meg, every time the repodata expires or is updated, even if there are no actual RPM updates. And for systems that need frequent updates and refreshes to ensure the latest packages, it's consumed *every single time* you flush the metadata. What about the repositories and mirrors? Do they all have unlimited, cheap bandwidth? No one does. But in my direct observation, deltarpms are a theoretically clever attempt to solve the wrong problem, an attempt that requires massive *extra* diskspace and resources for the mirror sites that doesn't address the more consistent and demonstrable problem. There are environments where bandwidth limits are so great that even the bare RPM downloads are vulnerable to interruption, and the marginal improvement of the deltarpm would be noticeable. But I've not seen such an environment in roughly 10 years, and it was usually to some bad network hardware or local misconfiguration. -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: No more deltarpms by default
On Sat, Oct 18, 2014 at 07:00:19PM -0400, Nico Kadel-Garcia wrote: 3) People who have a lot of hosts and high bandwidth, high speed local deployment requirements can, and do, set up an internal Fedora mirror with much lower bandwidth costs. This reduces the tangible benefits of deltarpms significantly. This is combined with my direct Folks that have that sort of environment also typically use kickstart to set systems up, and can trivially disable deltarpms in the process. - Solomon -- Solomon Peachy pizza at shaftnet dot org Delray Beach, FL ^^ (email/xmpp) ^^ Quidquid latine dictum sit, altum viditur. pgpOuoyFvYVvz.pgp Description: PGP signature -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: No more deltarpms by default
On Oct 18, 2014 5:00 PM, Nico Kadel-Garcia nka...@gmail.com wrote: On Thu, Oct 16, 2014 at 11:40 PM, Gerald B. Cox gb...@bzb.us wrote: My comment was not meant to be argumentative, but rather tongue-in-cheek. However, I do believe when changing a default, it isn't about what is convenient for me. It's about what is best for the entire community and what are the real world ramifications. I'm not buying the let's change the default because high bandwidth is pervasive argument. I'll go out on a limb here and suggest: 1. Most people who can afford to pay the monthly recurring cost for a high speed bandwidth connection have multi-core machines. 2. People who are running Fedora on multiple machines possess the skill set to easily change the default and turn Presto off if they wish. You left out some but related issues. 3) People who have a lot of hosts and high bandwidth, high speed local deployment requirements can, and do, set up an internal Fedora mirror with much lower bandwidth costs. This reduces the tangible benefits of deltarpms significantly. This is combined with my direct observation that working from plain rpms is much faster and more reliable. Re-assembly from deltarpms is computationally expensive and thus expensive for large numbers of yum clients. It sounds like a great idea at first glance, but it profoundly and unpredicatably increases the disk space and bandwidth needed for mirrors themselves at a relatively small benefit in download bandwidth. The great example of difficult to delta RPM's sems to be libreoffice. Between the compressed and distinct binaries for the war files, and the deltarpm process itself, they save very little bandwidth for many of the builkies projects. 4) The much, much larger source of wasted bandwidth and resources is the yum repodata. That's 50 Meg, every time the repodata expires or is updated, even if there are no actual RPM updates. And for systems that need frequent updates and refreshes to ensure the latest packages, it's consumed *every single time* you flush the metadata. What about the repositories and mirrors? Do they all have unlimited, cheap bandwidth? No one does. But in my direct observation, deltarpms are a theoretically clever attempt to solve the wrong problem, an attempt that requires massive *extra* diskspace and resources for the mirror sites that doesn't address the more consistent and demonstrable problem. There are environments where bandwidth limits are so great that even the bare RPM downloads are vulnerable to interruption, and the marginal improvement of the deltarpm would be noticeable. But I've not seen such an environment in roughly 10 years, and it was usually to some bad network hardware or local misconfiguration. -- Network operators will experience a real, tangible cost increase from turning off deltarpms. End users with metered bandwidth - off the cuff, I'd guess hundreds of thousands of current users and billions of prospective users - will experience a real financial impact from the 'loss' of this feature. Now, I realize everyone can turn this on, and everyone can turn it off. Changing the default option doesn't take away the feature. Those with metered connections will probably, eventually, discover they can turn deltarpms on. After the first bill arrives, or the second. These are likely to be people with limited financial resources, using computers that you couldn't pay me to work on these days. We're keeping the entire 32bit architecture alive for them; in contrast to that, the infrastructure burden of deltarpms is slight. Even so, we continue to welcome this demographic to use and participate in Fedora. Faster update transactions are good, but anyone that's concerned about the *financial impact* of consuming extra CPU cycles ( as in Nico's case #3) is very likely to be employing a competent administrator (like Nico) that should know how and why to flip the switch. Nobody wants to take this option away, and I doubt it would even be possible. --Pete -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: No more deltarpms by default
Am 19.10.2014 um 02:19 schrieb Solomon Peachy:
On Sat, Oct 18, 2014 at 07:00:19PM -0400, Nico Kadel-Garcia wrote:
3) People who have a lot of hosts and high bandwidth, high speed
local deployment requirements can, and do, set up an internal Fedora
mirror with much lower bandwidth costs. This reduces the tangible
benefits of deltarpms significantly. This is combined with my direct
Folks that have that sort of environment also typically use kickstart to
set systems up, and can trivially disable deltarpms in the process
that people don't fall in that category anyways
they just re-use the result of deltarpm in /var/cache/yum to build up
their local repos and even in that case they benefit from one time saved
downloads - keep in mind the result in /var/cache/yum/ from which you
can build up your local repos is the full RPM
and that is why the current implementation of deltarpm is perfect
designed and any improvement needs to happen on a different layer
that few lines below are enough to use createrepo and build up a local
cache without mirror the whole upstream, you just need to have one
machine with any pakcge you use installed on it - works perfect over 6
years including dist-upgrades *and* benefits from deltarpm in the first step
#!/usr/bin/bash
basearch=`uname -i`
releasever=`rpm -q --qf %{version}\n fedora-release`
for g in `ls -1b /var/cache/yum`
do
if [ -d /var/cache/yum/$g/packages ]
then
echo /var/cache/yum/$g/packages/ /repo/cache/fc$releasever/
sudo mv --verbose /var/cache/yum/$g/packages/*.rpm
/repo/cache/fc$releasever/ 2 /dev/null
fi
done
/buildserver/repo-create.sh
_
everybody can disable it with whatever action
so it has to be enabled by default to save network ressources
99% of people don't realize or know anything about nor do the update 24
hours a day packages and so the rebuild time is not important
signature.asc
Description: OpenPGP digital signature
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: No more deltarpms by default
On Sat, Oct 18, 2014 at 8:28 PM, Reindl Harald h.rei...@thelounge.net wrote:
Am 19.10.2014 um 02:19 schrieb Solomon Peachy:
On Sat, Oct 18, 2014 at 07:00:19PM -0400, Nico Kadel-Garcia wrote:
3) People who have a lot of hosts and high bandwidth, high speed
local deployment requirements can, and do, set up an internal Fedora
mirror with much lower bandwidth costs. This reduces the tangible
benefits of deltarpms significantly. This is combined with my direct
Folks that have that sort of environment also typically use kickstart to
set systems up, and can trivially disable deltarpms in the process
that people don't fall in that category anyways
they just re-use the result of deltarpm in /var/cache/yum to build up their
local repos and even in that case they benefit from one time saved downloads
- keep in mind the result in /var/cache/yum/ from which you can build up
your local repos is the full RPM
and that is why the current implementation of deltarpm is perfect designed
and any improvement needs to happen on a different layer
that few lines below are enough to use createrepo and build up a local cache
without mirror the whole upstream, you just need to have one machine with
any pakcge you use installed on it - works perfect over 6 years including
dist-upgrades *and* benefits from deltarpm in the first step
#!/usr/bin/bash
basearch=`uname -i`
releasever=`rpm -q --qf %{version}\n fedora-release`
for g in `ls -1b /var/cache/yum`
do
if [ -d /var/cache/yum/$g/packages ]
then
echo /var/cache/yum/$g/packages/ /repo/cache/fc$releasever/
sudo mv --verbose /var/cache/yum/$g/packages/*.rpm
/repo/cache/fc$releasever/ 2 /dev/null
fi
done
/buildserver/repo-create.sh
And that only works on one machine, and is sensitive to exclusions
written into yum.conf or /etc/yum.repos.d/*, etc., etc., etc. You also
forgot to use 'gpgcheck' to verify the contents of the already
downloaded RPM, since 'repsync' does not track successful versus
partial downloads without it.
My old script for dong reposync mirrors is at
https://github.com/nkadel/nkadel-rsync-scripts/blob/master/reposync-rhel.sh,
written for RHEL SRPM mirroring for CentOS work, and it's quite
suitable for synchronizing 3rd party repositories without 'rsync'
access. My old tools for rsyncing other repositories are in the same
github repo, do grab them if you feel a need.
Anyway. Now multiply your 'reposync' based tool by 200 thin
provisioned virtual machines, and you're chewing up quite a lot of
disk space doing this. And the bandwidth saving is still badly
overshadowed by the necessary repodata bandwith from almost every run
of reposync in the model you've just described. 50 Megabytes, *every
time* it runs unless the metadata for the repo has not expired. .
Since the default is 90 minutes, if you're running any of the keep me
all the time updated or alerted tools, that's 24 hours / 90 minutes *
50 Meg, or up to 800 Meg/day. Even if it's only run nightly, it's 50
Meg/day with *no* updates applied, That overwhelms the benefits of
deltarpms pretty quickly.
Sorry to rain on parades about the benefits of deltarpms, but it's
like wearing waterproof boots when when you're standing in the shower.
It's just an extra burden when the repodata downloads are already
eating significantly more bandwidth than any but the largest largest
RPM downloads.
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: No more deltarpms by default
Hi On Sun, Oct 19, 2014 at 12:37 AM, Nico Kadel-Garcia wrote: . And the bandwidth saving is still badly overshadowed by the necessary repodata bandwith from almost every run of reposync in the model you've just described. 50 Megabytes, *every time* it runs unless the metadata for the repo has not expired. . Since the default is 90 minutes, It is not for dnf. man dnf.conf metadata_expire ... The default corresponds to 48 hours Rahul -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: No more deltarpms by default
On 10/06/2014 07:57 PM, Reindl Harald wrote: oh no - don't tie all together for reasons which did not destory the world over years - it is a damned good design that the part dealing with rpm packages don't need to know anything aboutt delta rpms because the normal packages are created before that step And, going further on the same road... why don't we just find a way to locally produce the original rpm by downloading through rsync? Are compressed rpms completely impossible to diff efficiently by rsync? (losing a bit of compression efficiency could be acceptable if rpms became rsyncable) -- Roberto Ragusamail at robertoragusa.it -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: No more deltarpms by default
On 10/06/2014 07:25 PM, Reindl Harald wrote: the last discussions suggested that the result needs to be *identical* to the full RPM downloaded for not break signatures Bizarre design; the signature should protect the content (uncompressed), not the transport method (compressed). -- Roberto Ragusamail at robertoragusa.it -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: No more deltarpms by default
Am 17.10.2014 um 10:49 schrieb Roberto Ragusa: On 10/06/2014 07:57 PM, Reindl Harald wrote: oh no - don't tie all together for reasons which did not destory the world over years - it is a damned good design that the part dealing with rpm packages don't need to know anything aboutt delta rpms because the normal packages are created before that step And, going further on the same road... why don't we just find a way to locally produce the original rpm by downloading through rsync? Are compressed rpms completely impossible to diff efficiently by rsync? (losing a bit of compression efficiency could be acceptable if rpms became rsyncable) to rsync something you need both files no we (users) don't want the installed rpms stored forever as package to support this nor do we want anything else than RPM for package management/integrity, otherwise we (the users) could have chosen Debian so please don't suggest technical background changes if it ain't broken don't fix it the switch exists and is useful on a 56kbit modem you don't want to download the full RPMs and on a 150Mbit line the download will always be faster than rebuild the RPM signature.asc Description: OpenPGP digital signature -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: No more deltarpms by default
Am 17.10.2014 um 10:55 schrieb Roberto Ragusa: On 10/06/2014 07:25 PM, Reindl Harald wrote: the last discussions suggested that the result needs to be *identical* to the full RPM downloaded for not break signatures Bizarre design; the signature should protect the content (uncompressed), not the transport method (compressed) no - it is a smart design besides the topic the whole RPM package is signed and so you can verify the integrity without unpack it first - the history showed security flaws more than once just uncompress untrusted archives https://www.google.at/search?q=attack+uncompress and it is a smart design in general: RPM don't need to know anything about deltarpms the way it is implemented just because RPM has not to deal with that and only faces the rebuilt package signature.asc Description: OpenPGP digital signature -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: No more deltarpms by default
Roberto Ragusa mail at robertoragusa.it writes: Are compressed rpms completely impossible to diff efficiently by rsync? In a word, yes. They're already compressed, so it's unlikely there would be any matching blocks between old and new rpms for rsync to take advantage of. (You can verify this by trying a local rsync between two different versions of a large RPM.) -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: No more deltarpms by default
On 10/17/2014 05:09, Reindl Harald wrote: on a 56kbit modem you don't want to download the full RPMs and on a 150Mbit line the download will always be faster than rebuild the RPM Perhaps this has already been suggested, but why not ask a question during the installation of the OS? For example, during the installation process it may be possible to simply ask the user whether they want to set themselves up for full downloads or not. Based on the answer to the question, you can choose delta RPMS or not. It is also possible to put something in the output of running an update that says to some effect, If you want to limit bandwidth usage, insert instructions on how to adjust the settings here. Those are just two examples of how it may be possible to have the best of both worlds. Anyone who has played video games over the past couple of decades knows that questions about one's Internet connection speed are the norm during installation/configuration so perhaps it makes sense to implement something similar in this instance. Also, for what it's worth, for the past 25+ years I've been part of a development team that writes code for products like a physician practice management system and software that automates wholesale distributors. Anytime we introduce a new configuration option we try to offer as much choice as possible without taking functionality away from those that expect it to remain the way it is. Choice is key in this process and every time we've embraced it our customers have been happiest. Tom -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: No more deltarpms by default
On Fri, Oct 17, 2014 at 2:53 PM, Tom Rivers t...@impact-crater.com wrote: On 10/17/2014 05:09, Reindl Harald wrote: on a 56kbit modem you don't want to download the full RPMs and on a 150Mbit line the download will always be faster than rebuild the RPM Perhaps this has already been suggested, but why not ask a question during the installation of the OS? Because it makes no sense and pushes it to the user. The os (i.e we) should handle that. In that case we should do both 1) have lower bandwith requirements (i.e use deltas) *and* 2) have fast installation of updates. Those two goals are not mutually exclusive. Its just the current implementation that is lacking. So instead of messing with questions during the installation we should just fix that. Also, for what it's worth, for the past 25+ years I've been part of a development team that writes code for products like a physician practice management system and software that automates wholesale distributors. Anytime we introduce a new configuration option we try to offer as much choice as possible without taking functionality away from those that expect it to remain the way it is. Choice is key in this process and every time we've embraced it our customers have been happiest. You customers would be *way* happier if stuff would just work without having them to mess with options until the correct configuration is found. Asking the user is the last resort when everything else fails. -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: No more deltarpms by default
On 10/17/2014 10:05, drago01 wrote: Because it makes no sense and pushes it to the user. The os (i.e we) should handle that. In that case we should do both 1) have lower bandwith requirements (i.e use deltas) *and* 2) have fast installation of updates. Those two goals are not mutually exclusive. Its just the current implementation that is lacking. So instead of messing with questions during the installation we should just fix that. If the proper configuration can be determined automagically, then by all means just do it. My point is that users aren't too stupid to understand bandwidth/processor considerations. The configuration of how much bandwidth/processor time will be involved in one configuration versus another is not rocket science. These are Linux users after all. I completely reject the notion that there isn't a simple way to phrase the configuration of how the system is update. After all, setting up a secure wireless connection is much more difficult and these same users do it every day. You customers would be *way* happier if stuff would just work without having them to mess with options until the correct configuration is found. Asking the user is the last resort when everything else fails. We configure the system for them whenever possible, so that's precisely what we do. When it comes to a subjective decision about the way the system operates after we've implemented additional functionality, we always preserve the way the system works and offer an option to change it if they so desire. By the way, considering we have a vast majority of our clients still with us and have kept them happy from the WANG 2200 days, through Xenix, SCO Unix, and on to Windows and Linux all these years from when we first opened our doors in 1985, I'd say we're doing a good job. Tom -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: No more deltarpms by default
Hi On Fri, Oct 17, 2014 at 10:24 AM, Tom Rivers wrote: If the proper configuration can be determined automagically, then by all means just do it. My point is that users aren't too stupid to understand bandwidth/processor considerations. The configuration of how much bandwidth/processor time will be involved in one configuration versus another is not rocket science. Wile users might be able to handle such questions (I would avoid calling them stupid even otherwise), it is contrary to the goals of the installer. The Fedora installer is explicitly designed to do the absolute minimum necessary to get the installation up and running. Everything else should be handled post installation. The reason is that additional complexity increases the chances of bugs in that codebase and installer issues cannot be fixed post release easily and can delay the releases. Pushing it to end users is also often the easy way out and almost always the wrong solution. Rahul -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: No more deltarpms by default
On 10/17/2014 04:24 PM, Tom Rivers wrote: On 10/17/2014 10:05, drago01 wrote: Because it makes no sense and pushes it to the user. The os (i.e we) should handle that. In that case we should do both 1) have lower bandwith requirements (i.e use deltas) *and* 2) have fast installation of updates. Those two goals are not mutually exclusive. Its just the current implementation that is lacking. So instead of messing with questions during the installation we should just fix that. If the proper configuration can be determined automagically, Well a package installer can't have the knowledge which would be required to determine such a decision. E.g. though a user could be connected via a fast connection he may have a limited download contingent or could be charged at a high rate per download volume. I.e. I don't see a possibility but to leave the final decision to the user. Ralf -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: No more deltarpms by default
On Fri, Oct 17, 2014 at 5:02 PM, Ralf Corsepius rc040...@freenet.de wrote: On 10/17/2014 04:24 PM, Tom Rivers wrote: On 10/17/2014 10:05, drago01 wrote: Because it makes no sense and pushes it to the user. The os (i.e we) should handle that. In that case we should do both 1) have lower bandwith requirements (i.e use deltas) *and* 2) have fast installation of updates. Those two goals are not mutually exclusive. Its just the current implementation that is lacking. So instead of messing with questions during the installation we should just fix that. If the proper configuration can be determined automagically, Well a package installer can't have the knowledge which would be required to determine such a decision. E.g. though a user could be connected via a fast connection he may have a limited download contingent or could be charged at a high rate per download volume. I.e. I don't see a possibility but to leave the final decision to the user. Did you even read my mail? The point is there shouldn't be a choice to be made in the first place. Which makes how to choose what moot. -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: No more deltarpms by default
On 06.10.2014 16:46, Jaroslav Reznik wrote: - Original Message - On Mon, 2014-10-06 at 10:54 +0100, Ian Malone wrote: On 6 October 2014 09:41, Rahul Sundaram methe...@gmail.com wrote: Hi One of the long standing features that were enabled by default in yum is support for delta rpms. dnf developers have disabled this and I think this change deserves a broader discussion https://bugzilla.redhat.com/show_bug.cgi?id=1148208 I have an internet flatrate at 150 mbs, and downloading the full rpms is ALOT faster than the the work that the delta rpms requires. Wow. Good to see normal users are taken into account. The main argument from a distro point of view is reducing load on servers, but I don't know many people on 150Mbs either. Heck, I've just tested my work janet connection and that's 100Mbs in our office. At home 8Mbps is a good day. (I'm assuming mbs is a typo for Mbps and not milli bit seconds, where the very slow transfer speed declines exponentially as the connection progresses.) The deltarpms were meant to serve two purposes 1) (lesser) Address the needs of users in developing countries (where Fedora is fairly popular) and bandwidth concerns are very considerable. Many of these users have connections that are either metered or extremely slow, so deltarpms provides a way to get the data to them more economically. This of course can be handled with a non-default option, so we can talk about making that more discoverable if we disable them by default. This is a good point but even in developing countries internet access is getting better and better. A few years ago installation DVD was the only way how to access Fedora repo. It's not requested anymore. But yeah, I do not live there. https://lists.fedoraproject.org/pipermail/test/2013-December/119412.html Rejy, speak! poma -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: No more deltarpms by default
On 10/17/2014 10:43, Rahul Sundaram wrote: Wile users might be able to handle such questions (I would avoid calling them stupid even otherwise) I didn't call them stupid - in fact I suggested just the opposite. Go back and read what I wrote. , it is contrary to the goals of the installer. The Fedora installer is explicitly designed to do the absolute minimum necessary to get the installation up and running. Everything else should be handled post installation. Pre-installation, post-installation, whatever is best. I simply advocated asking a question because users are smart enough to handle it. Tom -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: No more deltarpms by default
Hi On Fri, Oct 17, 2014 at 11:09 AM, Tom Rivers wrote: I didn't call them stupid - in fact I suggested just the opposite. Go back and read what I wrote. I did. You said My point is that users aren't too stupid to understand bandwidth/processor considerations. I am just saying even if one doesn't understand these considerations, they aren't stupid. Pre-installation, post-installation, whatever is best. I simply advocated asking a question because users are smart enough to handle it. Well, it is a configuration option. If one wants to tweak it, they can do so. Rahul -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: No more deltarpms by default
On 10/17/2014 05:07 PM, drago01 wrote: On Fri, Oct 17, 2014 at 5:02 PM, Ralf Corsepius rc040...@freenet.de wrote: On 10/17/2014 04:24 PM, Tom Rivers wrote: On 10/17/2014 10:05, drago01 wrote: Because it makes no sense and pushes it to the user. The os (i.e we) should handle that. In that case we should do both 1) have lower bandwith requirements (i.e use deltas) *and* 2) have fast installation of updates. Those two goals are not mutually exclusive. Its just the current implementation that is lacking. So instead of messing with questions during the installation we should just fix that. If the proper configuration can be determined automagically, Well a package installer can't have the knowledge which would be required to determine such a decision. E.g. though a user could be connected via a fast connection he may have a limited download contingent or could be charged at a high rate per download volume. I.e. I don't see a possibility but to leave the final decision to the user. Did you even read my mail? The point is there shouldn't be a choice to be made in the first place. Which makes how to choose what moot. Did you read my mail? I say, no choice is inapplicable, non-useful, naive non-sense. -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: No more deltarpms by default
Everyone: please remember the code of conduct here, and advance the conversation in a productive way or take it elsewhere. Thank you. -- Matthew Miller mat...@fedoraproject.org Fedora Project Leader -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: No more deltarpms by default
On Mon, Oct 6, 2014 at 8:07 AM, Stephen Gallagher sgall...@redhat.com wrote: [ . . . ] It would be nice to see if we can find ways to improve the performance of the deltarpm reconstruction instead. Much of the time is spent on compression/decompression tasks which *should* be massively parallel; we should be able to speed this up by taking advantage of additional cores (and hyperthreading) on the system, for example. Another option might be not to bother recompressing the RPMs but instead just install an uncompressed RPM (and possibly recompress it out of band if we needed to keep it in the cache). export XZ_OPT=T0 If that were enabled in the environment the XZ compression phase would be significantly faster. -- -Jon Disnard -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: No more deltarpms by default
On Fri, Oct 17, 2014 at 7:24 AM, Tom Rivers t...@impact-crater.com wrote: If the proper configuration can be determined automagically, then by all means just do it. My point is that users aren't too stupid to understand bandwidth/processor considerations. The configuration of how much bandwidth/processor time will be involved in one configuration versus another is not rocket science. These are Linux users after all. I completely reject the notion that there isn't a simple way to phrase the configuration of how the system is update. After all, setting up a secure wireless connection is much more difficult and these same users do it every day. I think you're setting up a false equivalency here. There is a difference between the requirements of say an XBOX or Playstation user and that of updating a Fedora system. If given the choice, most people will say Yes, I have a high speed connection... turn it on, it's more better. They won't give it a second thought. Consider the upstream consequences of that. The bandwidth requirements of repositories and mirrors will increase. They will need to make the decision of whether to upgrade their circuits or just let them become saturated. Ask a CIO what they think about circuit costs. They always want to either reduce or defer upgrades. These things aren't free. We should be looking for ways to improve the performance of Presto...not just throwing it over the wall and assume that the network will handle it. -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: No more deltarpms by default
On 10/17/2014 06:00 PM, Matthew Miller wrote: Everyone: please remember the code of conduct here, and advance the conversation in a productive way or take it elsewhere. Thank you. Well, then please explain, how you expect people to express fundamental disagreement with somebody's rationale, if one thinks a person's/party's rationale is irrational and non-applicable? -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: No more deltarpms by default
On Fri, Oct 17, 2014 at 5:52 PM, Ralf Corsepius rc040...@freenet.de wrote: On 10/17/2014 05:07 PM, drago01 wrote: On Fri, Oct 17, 2014 at 5:02 PM, Ralf Corsepius rc040...@freenet.de wrote: On 10/17/2014 04:24 PM, Tom Rivers wrote: On 10/17/2014 10:05, drago01 wrote: Because it makes no sense and pushes it to the user. The os (i.e we) should handle that. In that case we should do both 1) have lower bandwith requirements (i.e use deltas) *and* 2) have fast installation of updates. Those two goals are not mutually exclusive. Its just the current implementation that is lacking. So instead of messing with questions during the installation we should just fix that. If the proper configuration can be determined automagically, Well a package installer can't have the knowledge which would be required to determine such a decision. E.g. though a user could be connected via a fast connection he may have a limited download contingent or could be charged at a high rate per download volume. I.e. I don't see a possibility but to leave the final decision to the user. Did you even read my mail? The point is there shouldn't be a choice to be made in the first place. Which makes how to choose what moot. Did you read my mail? I say, no choice is inapplicable, non-useful, naive non-sense. You missed the point again. Making a choice only makes sense between mutually exclusive choices. In case you can have both (given a better implementation) the goal should be that. -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: No more deltarpms by default
Am 17.10.2014 um 17:07 schrieb drago01: On Fri, Oct 17, 2014 at 5:02 PM, Ralf Corsepius rc040...@freenet.de wrote: On 10/17/2014 04:24 PM, Tom Rivers wrote: On 10/17/2014 10:05, drago01 wrote: Because it makes no sense and pushes it to the user. The os (i.e we) should handle that. In that case we should do both 1) have lower bandwith requirements (i.e use deltas) *and* 2) have fast installation of updates. Those two goals are not mutually exclusive. Its just the current implementation that is lacking. So instead of messing with questions during the installation we should just fix that. If the proper configuration can be determined automagically, Well a package installer can't have the knowledge which would be required to determine such a decision. E.g. though a user could be connected via a fast connection he may have a limited download contingent or could be charged at a high rate per download volume. I.e. I don't see a possibility but to leave the final decision to the user. Did you even read my mail? The point is there shouldn't be a choice to be made in the first place. Which makes how to choose what moot. so how do you imagine that decision happens later? the user deliberate makes the change? don't get me wrong but than you have no expierience about the ordinary users, they don't do anything, even not react if someone alerts them about a hacked mailaccount abused for spam-sending over days and after security warnings about Heartbleed and please change your passwords another one choses his first name as new secure password *that* is what you can expect from users and if you work in the IT and have a different expierience go out every morning and kiss them! signature.asc Description: OpenPGP digital signature -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: No more deltarpms by default
On 10/17/2014 12:11, Gerald B. Cox wrote: I think you're setting up a false equivalency here. There is a difference between the requirements of say an XBOX or Playstation user and that of updating a Fedora system. If given the choice, most people will say Yes, I have a high speed connection... turn it on, it's more better. They won't give it a second thought. My point was to say Linux users are usually more tech savvy than XBox and Playstation users. If they say they have a high speed connection and they don't and that decision ends up costing them more money in ISP costs, then that's on them. Still, that's really not the crux of what I've been trying to say. We should be looking for ways to improve the performance of Presto...not just throwing it over the wall and assume that the network will handle it. I completely agree. My comments on this topic were meant solely to counter the notion that it must be completely abandoned, low bandwidth/high-cost bandwidth users be damned. I was merely trying to illustrate that there may be a way to make everyone happy. Tom -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: No more deltarpms by default
On 10/17/2014 11:29, Rahul Sundaram wrote: On Fri, Oct 17, 2014 at 11:09 AM, Tom Rivers wrote: I didn't call them stupid - in fact I suggested just the opposite. Go back and read what I wrote. I did. You said My point is that users aren't too stupid to understand bandwidth/processor considerations. I am just saying even if one doesn't understand these considerations, they aren't stupid. If you deconstruct the sentence you quoted above, it boils down to the subject users and the predicate aren't. Therefore I said quite specifically that users ARE NOT too stupid to understand bandwidth/processor considerations. The inverse of that statement is the users ARE smart enough to understand bandwidth/processor considerations. More to the point, I went on to bolster the notion that Linux users are capable of understanding what's at play by saying this: These are Linux users after all. That sentence is meant to infer that using Linux requires a decent amount of skill, something that again goes against the notion that I have somehow called them stupid. Overall, the entire point of me advocating that users should be asked whether they want to endure large amounts of bandwidth/processor usage during system updates is predicated on the notion that they CAN answer the question intelligently. It would make absolutely no sense to posit querying users if they couldn't respond intelligently. Please don't persist in asserting I'm saying people are stupid. It quite simply is not the case. Tom -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: No more deltarpms by default
Hi On Fri, Oct 17, 2014 at 1:03 PM, Tom Rivers wrote: If you deconstruct the sentence you quoted above, it boils down to the subject users and the predicate aren't. Therefore I said quite specifically that users ARE NOT too stupid to understand bandwidth/processor considerations. The inverse of that statement is the users ARE smart enough to understand bandwidth/processor considerations. For the last time, my point is that: smart and stupid is not determined by whether a user understands bandwidth/processor considerations. Rahul -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: No more deltarpms by default
On Fri, Oct 17, 2014 at 6:50 PM, Reindl Harald h.rei...@thelounge.net wrote: Am 17.10.2014 um 17:07 schrieb drago01: On Fri, Oct 17, 2014 at 5:02 PM, Ralf Corsepius rc040...@freenet.de wrote: On 10/17/2014 04:24 PM, Tom Rivers wrote: On 10/17/2014 10:05, drago01 wrote: Because it makes no sense and pushes it to the user. The os (i.e we) should handle that. In that case we should do both 1) have lower bandwith requirements (i.e use deltas) *and* 2) have fast installation of updates. Those two goals are not mutually exclusive. Its just the current implementation that is lacking. So instead of messing with questions during the installation we should just fix that. If the proper configuration can be determined automagically, Well a package installer can't have the knowledge which would be required to determine such a decision. E.g. though a user could be connected via a fast connection he may have a limited download contingent or could be charged at a high rate per download volume. I.e. I don't see a possibility but to leave the final decision to the user. Did you even read my mail? The point is there shouldn't be a choice to be made in the first place. Which makes how to choose what moot. so how do you imagine that decision happens later? the user deliberate makes the change? don't get me wrong but than you have no expierience about the ordinary users, they don't do anything, even not react if someone alerts them about a hacked mailaccount abused for spam-sending over days and after security warnings about Heartbleed and please change your passwords another one choses his first name as new secure password *that* is what you can expect from users and if you work in the IT and have a different expierience go out every morning and kiss them! I have no idea what your mail has anything to do with that but I try to explain it one more time. Currently we have 1) deltarpms - low bandwidth use but takes a while to apply the updates 2) no deltarpms - high bandwidth use but faster updates application. So now people suggest the user (or the distro) has to either choose 1) or 2) ... What I am suggesting is adding a third option: 3) Use less bandwidth *and* apply updates fast (by fixing deltarpm). So if we have 3) the choice between 1) and 2) does not make much sense. -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: No more deltarpms by default
Am 17.10.2014 um 19:45 schrieb drago01: 1) deltarpms - low bandwidth use but takes a while to apply the updates 2) no deltarpms - high bandwidth use but faster updates application. So now people suggest the user (or the distro) has to either choose 1) or 2) ... What I am suggesting is adding a third option: 3) Use less bandwidth *and* apply updates fast (by fixing deltarpm). So if we have 3) the choice between 1) and 2) does not make much sense but we *do not have* option 3 without change fundamentally how things are implemented, so yes - it would be fine - but decisions are already made by what we *have now* by *disable* it as default so just stop arguing in theory when the topic is about what is possible signature.asc Description: OpenPGP digital signature -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: No more deltarpms by default
On 10/17/2014 13:19, Rahul Sundaram wrote: For the last time, my point is that: smart and stupid is not determined by whether a user understands bandwidth/processor considerations. I completely understand that not comprehending all of the complexities of bandwidth/processor considerations doesn't make one stupid. I agree wholeheartedly with you. However, you are going off on a tangent again that really has nothing to do with the point I was trying to make. I would really like to know how you can equate that with my original point: Hey, maybe instead of throwing Presto under the bus we could discuss the option of giving users the choice to determine how much bandwidth/processor resources to throw at system updates instead. At least then we wouldn't be throwing out something that significantly benefits those who DO pay a lot of money and DO have low bandwidth connections to the Internet. It's a simple compromise: keep things configured as efficiently as possible while still allowing those who want to open up the throttle to do just that. Tom -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: No more deltarpms by default
Hi On Fri, Oct 17, 2014 at 2:17 PM, Tom Rivers wrote: I would really like to know how you can equate that with my original point: Hey, maybe instead of throwing Presto under the bus.. We aren't though. Deltarpms is again enabled by default in Dnf a while back if you read the bug report. Everything else has been an tangent, yes. Rahul -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: No more deltarpms by default
On 10/17/2014 14:24, Rahul Sundaram wrote: Deltarpms is again enabled by default in Dnf a while back if you read the bug report. Everything else has been an tangent, yes. I suppose I'm having trouble understanding that when you started this thread with: On 10/6/2014 04:41, Rahul Sundaram wrote: One of the long standing features that were enabled by default in yum is support for delta rpms. dnf developers have disabled this and I think this change deserves a broader discussion https://bugzilla.redhat.com/show_bug.cgi?id=1148208 You wrote dnf developers have disabled this and I think this change deserves a broader discussion. I went back and read your original post on the bug report you referenced and you said this: Since Fedora 12 or so (my blog post on this at http://mether.wordpress.com/2009/10/01/fedora-12-and-yum-presto-plugin/) deltarpm support has been available by default for many users. dnf should enable this configuration by default as well I even went back and read your blog post. I honestly don't understand. You say it needs to be enabled at first and now say that it's already enabled. Color me confused, my friend. Regardless, I don't think anyone, including you and I, wants to expend more bandwidth on tangential non-issues. If you say this is handled, then that's fine by me. I just wanted to be sure that everyone's bandwidth/processor issues were being adequately addressed. Tom -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: No more deltarpms by default
On Fri, Oct 17, 2014 at 2:44 PM, Tom Rivers wrote: I suppose I'm having trouble understanding that when you started this thread with:ble by default for many users. dnf should enable this configuration by default as well Yes, that was true when I posted it. As a result of this discussion, the situation was resolved. So the discussion had already achieved its purpose a while back. Rahul -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: No more deltarpms by default
On Fri, Oct 17, 2014 at 9:58 AM, Tom Rivers t...@impact-crater.com wrote: My point was to say Linux users are usually more tech savvy than XBox and Playstation users. If they say they have a high speed connection and they don't and that decision ends up costing them more money in ISP costs, then that's on them. Still, that's really not the crux of what I've been trying to say. That wasn't quite my point. What I was attempting to convey is that people with high bandwidth connections would be choosing that option because they would believe it would be more better - regardless of whether or not they cared much about speeding up their updates. I would venture to say for the vast majority of users, any time savings really doesn't matter. They just don't care. However, choosing that option would have real implications upstream. All of a sudden more repositories would be called upon to deliver full RPMs... that is alot more bandwidth, which translates to $$$. Regardless of whether or not it is a university or even Redhat... everyone has a budget, and they would be concerned about increased costs. Many people don't understand networks; they think it's just an unlimited resource that is free. Nothing more could be further from the truth. When people see the costs they say WTF and quickly change their tune. I understand that thankfully DNF will keep Presto as the default; but if the issue is revisited people need to realize it isn't just about them - there are implications upstream - and I believe if they were presented the bill, they would quickly say oh, it's not that important to wait an extra few minutes to have my updates applied. The bottom line is Presto needs to be improved. That is the solution. In the meantime if people want to download full RPMs they are free to change the option themselves - and they should be happy it isn't default; because if it were and everyone was downloading full RPMs they would most likely see that repository circuits become saturated and their downloads would come screeching to a halt. -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: No more deltarpms by default
Ian Malone wrote: I have an internet flatrate at 150 mbs, and downloading the full rpms is ALOT faster than the the work that the delta rpms requires. Wow. Good to see normal users are taken into account. I have a normal Austrian broadband connection, and it is still much faster to just download the full RPMs than to use delta RPMs. And parallelization (as others in the thread have suggested) will not help at all on the single-core machine I'm typing this on. Thus, I disabled delta RPMs long ago and agree that they should be off by default. Kevin Kofler -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: No more deltarpms by default
On Thu, Oct 16, 2014 at 8:00 AM, Kevin Kofler kevin.kof...@chello.at wrote: And parallelization (as others in the thread have suggested) will not help at all on the single-core machine I'm typing this on. Single-Core? Really Kevin? Even the One Laptop Per Child machines are dual-core. ;-) -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: No more deltarpms by default
Gerald B. Cox composed on 2014-10-16 12:52 (UTC-0700): Kevin Kofler wrote: And parallelization (as others in the thread have suggested) will not help at all on the single-core machine I'm typing this on. Single-Core? Really Kevin? Even the One Laptop Per Child machines are dual-core. ;-) I have F20, F21 and/or F22 installed on 15 machines, of which one has more than one core, and only 2-3, maybe 4, of which have HT. Some people need to get the most out of their money, and/or take whatever they can get. -- The wise are known for their understanding, and pleasant words are persuasive. Proverbs 16:21 (New Living Translation) Team OS/2 ** Reg. Linux User #211409 ** a11y rocks! Felix Miata *** http://fm.no-ip.com/ -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: No more deltarpms by default
My comment was not meant to be argumentative, but rather tongue-in-cheek. However, I do believe when changing a default, it isn't about what is convenient for me. It's about what is best for the entire community and what are the real world ramifications. I'm not buying the let's change the default because high bandwidth is pervasive argument. I'll go out on a limb here and suggest: 1. Most people who can afford to pay the monthly recurring cost for a high speed bandwidth connection have multi-core machines. 2. People who are running Fedora on multiple machines possess the skill set to easily change the default and turn Presto off if they wish. I can't speak for other countries, but in the USA low cost high speed bandwidth is not pervasive. We're fighting about net neutrality and the FCC is trying to change the definition of a broadband connection. The carriers are fighting it. They are constantly looking for ways to cut bandwidth, shape traffic. Most people would rather use their bandwidth to stream media than to get an update applied a little faster. What about the repositories and mirrors? Do they all have unlimited, cheap bandwidth? Who is the target demographic of Fedora? People with single-core machines and high speed broadband? What about people with slow connections? Is our response to them sucks to be you? Yes, not everyone can afford to buy a new machine - but neither can everyone afford to pay the monthly recurring cost for high speed broadband. The purpose of Presto is still sound. It was a good idea then, it remains a good idea now. http://fedoraproject.org/wiki/Features/Presto Full disclosure: Yes, I have a high speed broadband connection. I'd rather use it for streaming and other downloading than downloading full RPMs. It doesn't bother me if it takes a few more minutes to update my system. I do it when I'm sleeping anyway. On Thu, Oct 16, 2014 at 1:03 PM, Felix Miata mrma...@earthlink.net wrote: Gerald B. Cox composed on 2014-10-16 12:52 (UTC-0700): Kevin Kofler wrote: And parallelization (as others in the thread have suggested) will not help at all on the single-core machine I'm typing this on. Single-Core? Really Kevin? Even the One Laptop Per Child machines are dual-core. ;-) I have F20, F21 and/or F22 installed on 15 machines, of which one has more than one core, and only 2-3, maybe 4, of which have HT. Some people need to get the most out of their money, and/or take whatever they can get. -- The wise are known for their understanding, and pleasant words are persuasive. Proverbs 16:21 (New Living Translation) Team OS/2 ** Reg. Linux User #211409 ** a11y rocks! Felix Miata *** http://fm.no-ip.com/ -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: No more deltarpms by default
On 10/17/2014 05:40 AM, Gerald B. Cox wrote: My comment was not meant to be argumentative, but rather tongue-in-cheek. However, I do believe when changing a default, it isn't about what is convenient for me. It's about what is best for the entire community and what are the real world ramifications. IMO, the default should be about what fits most users needs bests. I'll go out on a limb here and suggest: 1. Most people who can afford to pay the monthly recurring cost for a high speed bandwidth connection have multi-core machines. 2. People who are running Fedora on multiple machines possess the skill set to easily change the default and turn Presto off if they wish. I can't speak for other countries, but in the USA low cost high speed bandwidth is not pervasive. We're fighting about net neutrality and the FCC is trying to change the definition of a broadband connection. The same applies to my home country (Germany) and as I would guess probably most of the Western World. What about the repositories and mirrors? Do they all have unlimited, cheap bandwidth? Probably not all, but I guess, most of them have. Who is the target demographic of Fedora? I thought, we are talking about defaults/presets and not about disabling delta-rpms at all? Changing the default would not be much a problem to me, but not providing delta-rpms could likely become a problem. People with single-core machines and high speed broadband? What about people with slow connections? Is our response to them sucks to be you? Yes, not everyone can afford to buy a new machine Also keep in mind that we are in the age of mobile platforms, i.e. a user's situation isn't necessarily static. A user may prefer full rpms in one situation but may prefer delta rpms in another. E.g. you may have a high-speed broadband at your usual places (at home/in the office), but you may easily find yourself in situations with access to a slow, unreliable and costly internet connection, were using full rpms could be prohibitive. Ralf -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: No more deltarpms by default
On Mon, Oct 6, 2014 at 9:55 PM, Jonathan Dieter jdie...@lesbg.com wrote: On 10/06/2014 08:57 PM, Reindl Harald wrote: Am 06.10.2014 um 19:45 schrieb Florian Festi: The way of getting around all this unnecessary computation is establishing trust via the deltarpm itself and giving up the idea of reconstructing the originally rpm as a prove of everything worked out just fine. To save even more time the delta would need to be applied directly on disk without creating an package at all. This would require a deeper integration with rpm and requires some tricky algorithms to make sure everything is ok in advance and won't blow up during the rpm transaction. So if anyone needs a hobby... oh no - don't tie all together for reasons which did not destory the world over years - it is a damned good design that the part dealing with rpm packages don't need to know anything aboutt delta rpms because the normal packages are created before that step don't break the unix-way of work the current behavior follows for no good reason and there is none - otherwise deltarpm would not have been default over years the way it works now Ok, granted, this sounds pretty scary. But if we give rpm the ability to upgrade an installed package with a deltarpm, it wouldn't take away deltarpm's ability to generate a full rpm from a deltarpm. And it does have the advantage of cutting right through the knot. We already store checksums of the deltarpms in prestodelta.xml, as well as in the deltarpm itself. Probably the biggest weakness would be the chance that something would change on-disk between the check stage and actual install stage. We'd have to evaluate whether it's worth making a temporary copy of the old data during the check stage and then applying the deltarpm to that. All of this would require a lot of buy-in from the rpm guys, though (Florian, you're one of them, right?). If I recall correctly, when we first looked at deltarpms, one of the selling points was that rpm didn't have to change at all. Jonathan -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct Hi, I live in a country where 1 Mbps (128 KB/s) is more than the average (and considered GOOD), we really need deltarpm, it saves huge amounts of data and time. I think the problem is in compressing then decompressing the file, why we don't take checksum of the tar file before compression? (of course this must be done on the server too), this way we can use deltarpm without much CPU, this should provide the best of the two worlds. Mustafa Muhammad -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: No more deltarpms by default
Hello All! 2014-10-06 12:41 GMT+04:00 Rahul Sundaram methe...@gmail.com: Hi One of the long standing features that were enabled by default in yum is support for delta rpms. dnf developers have disabled this At last! -- With best regards, Peter Lemenkov. -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: No more deltarpms by default
On 6 October 2014 09:41, Rahul Sundaram methe...@gmail.com wrote: Hi One of the long standing features that were enabled by default in yum is support for delta rpms. dnf developers have disabled this and I think this change deserves a broader discussion https://bugzilla.redhat.com/show_bug.cgi?id=1148208 I have an internet flatrate at 150 mbs, and downloading the full rpms is ALOT faster than the the work that the delta rpms requires. Wow. Good to see normal users are taken into account. The main argument from a distro point of view is reducing load on servers, but I don't know many people on 150Mbs either. Heck, I've just tested my work janet connection and that's 100Mbs in our office. At home 8Mbps is a good day. (I'm assuming mbs is a typo for Mbps and not milli bit seconds, where the very slow transfer speed declines exponentially as the connection progresses.) -- imalone http://ibmalone.blogspot.co.uk -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: No more deltarpms by default
On Mon, Oct 06, 2014 at 04:41:07AM -0400, Rahul Sundaram wrote: Hi One of the long standing features that were enabled by default in yum is support for delta rpms. dnf developers have disabled this and I think this change deserves a broader discussion https://bugzilla.redhat.com/show_bug.cgi?id=1148208 The amount of time taken to rebuild rpms from delta rpms meant that they didn't seem to save anything for me. I had always assumed that this is because the rebuilding code was written in Python. In fact this is not so! Although the yum-presto plugin is written in Python, it just calls out to the applydeltarpm program written in C (assuming I'm looking at the right place: https://gitorious.org/deltarpm/deltarpm). Has anyone analyzed why the rebuild step is slow? Seems like the best thing to do would be to make it faster if possible. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-p2v converts physical machines to virtual machines. Boot with a live CD or over the network (PXE) and turn machines into KVM guests. http://libguestfs.org/virt-v2v -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: No more deltarpms by default
Am 06.10.2014 um 13:00 schrieb Richard W.M. Jones: On Mon, Oct 06, 2014 at 04:41:07AM -0400, Rahul Sundaram wrote: One of the long standing features that were enabled by default in yum is support for delta rpms. dnf developers have disabled this and I think this change deserves a broader discussion https://bugzilla.redhat.com/show_bug.cgi?id=1148208 The amount of time taken to rebuild rpms from delta rpms meant that they didn't seem to save anything for me. I had always assumed that this is because the rebuilding code was written in Python. In fact this is not so! Although the yum-presto plugin is written in Python, it just calls out to the applydeltarpm program written in C (assuming I'm looking at the right place: https://gitorious.org/deltarpm/deltarpm). Has anyone analyzed why the rebuild step is slow? Seems like the best thing to do would be to make it faster if possible because it needs to build the complete xz-compressed RPM there was a discussion here not that long ago signature.asc Description: OpenPGP digital signature -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: No more deltarpms by default
On Mon, 2014-10-06 at 10:54 +0100, Ian Malone wrote: On 6 October 2014 09:41, Rahul Sundaram methe...@gmail.com wrote: Hi One of the long standing features that were enabled by default in yum is support for delta rpms. dnf developers have disabled this and I think this change deserves a broader discussion https://bugzilla.redhat.com/show_bug.cgi?id=1148208 I have an internet flatrate at 150 mbs, and downloading the full rpms is ALOT faster than the the work that the delta rpms requires. Wow. Good to see normal users are taken into account. The main argument from a distro point of view is reducing load on servers, but I don't know many people on 150Mbs either. Heck, I've just tested my work janet connection and that's 100Mbs in our office. At home 8Mbps is a good day. (I'm assuming mbs is a typo for Mbps and not milli bit seconds, where the very slow transfer speed declines exponentially as the connection progresses.) The deltarpms were meant to serve two purposes 1) (lesser) Address the needs of users in developing countries (where Fedora is fairly popular) and bandwidth concerns are very considerable. Many of these users have connections that are either metered or extremely slow, so deltarpms provides a way to get the data to them more economically. This of course can be handled with a non-default option, so we can talk about making that more discoverable if we disable them by default. 2) (Major) Deltarpms significantly (Kevin, can you comment with numbers?) reduces the load on the Fedora update servers and mirrors, thereby reducing hosting costs as well as increasing the efficiency of the available bandwidth (since smaller downloads mean you will be tying up the line for a shorter amount of time). It would be nice to see if we can find ways to improve the performance of the deltarpm reconstruction instead. Much of the time is spent on compression/decompression tasks which *should* be massively parallel; we should be able to speed this up by taking advantage of additional cores (and hyperthreading) on the system, for example. Another option might be not to bother recompressing the RPMs but instead just install an uncompressed RPM (and possibly recompress it out of band if we needed to keep it in the cache). signature.asc Description: This is a digitally signed message part -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: No more deltarpms by default
On Mon, Oct 06, 2014 at 12:00:04PM +0100, Richard W.M. Jones wrote: The amount of time taken to rebuild rpms from delta rpms meant that they didn't seem to save anything for me. It's not about saving *time*; it's about reducing the amount of data sent over the wire -- this is particularly important when you have a slow/congested (and/or metered) internet connection. Personally, I leave deltarpms on unless I have a local repo mirror. - Solomon -- Solomon Peachy pizza at shaftnet dot org Delray Beach, FL ^^ (email/xmpp) ^^ Quidquid latine dictum sit, altum viditur. pgpF9Y005QRiT.pgp Description: PGP signature -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: No more deltarpms by default
On Mon, Oct 6, 2014 at 3:07 PM, Stephen Gallagher sgall...@redhat.com wrote: On Mon, 2014-10-06 at 10:54 +0100, Ian Malone wrote: On 6 October 2014 09:41, Rahul Sundaram methe...@gmail.com wrote: Hi One of the long standing features that were enabled by default in yum is support for delta rpms. dnf developers have disabled this and I think this change deserves a broader discussion https://bugzilla.redhat.com/show_bug.cgi?id=1148208 I have an internet flatrate at 150 mbs, and downloading the full rpms is ALOT faster than the the work that the delta rpms requires. Wow. Good to see normal users are taken into account. The main argument from a distro point of view is reducing load on servers, but I don't know many people on 150Mbs either. Heck, I've just tested my work janet connection and that's 100Mbs in our office. At home 8Mbps is a good day. (I'm assuming mbs is a typo for Mbps and not milli bit seconds, where the very slow transfer speed declines exponentially as the connection progresses.) The deltarpms were meant to serve two purposes 1) (lesser) Address the needs of users in developing countries (where Fedora is fairly popular) and bandwidth concerns are very considerable. Many of these users have connections that are either metered or extremely slow, so deltarpms provides a way to get the data to them more economically. This of course can be handled with a non-default option, so we can talk about making that more discoverable if we disable them by default. 2) (Major) Deltarpms significantly (Kevin, can you comment with numbers?) reduces the load on the Fedora update servers and mirrors, thereby reducing hosting costs as well as increasing the efficiency of the available bandwidth (since smaller downloads mean you will be tying up the line for a shorter amount of time). It would be nice to see if we can find ways to improve the performance of the deltarpm reconstruction instead. Much of the time is spent on compression/decompression tasks which *should* be massively parallel s/massively parallel/not done at all/ ... but we had this discussion each time this comes up. There is no point in compressing something just to uncompress it a few minutes later. -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: No more deltarpms by default
Hi, It would be nice to see if we can find ways to improve the performance of the deltarpm reconstruction instead. Much of the time is spent on compression/decompression tasks which *should* be massively parallel s/massively parallel/not done at all/ ... but we had this discussion each time this comes up. There is no point in compressing something just to uncompress it a few minutes later. IIRC the problem is that the _compressed_ rpm is signed, so things go compress - check sig - uncompress and you can't cut the compress +uncompress without also cutting the signature check :( cheers, Gerd -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: No more deltarpms by default
On Mon, Oct 6, 2014 at 3:48 PM, Gerd Hoffmann kra...@redhat.com wrote: Hi, It would be nice to see if we can find ways to improve the performance of the deltarpm reconstruction instead. Much of the time is spent on compression/decompression tasks which *should* be massively parallel s/massively parallel/not done at all/ ... but we had this discussion each time this comes up. There is no point in compressing something just to uncompress it a few minutes later. IIRC the problem is that the _compressed_ rpm is signed, so things go compress - check sig - uncompress and you can't cut the compress +uncompress without also cutting the signature check :( Is the payload actually signed? The conclusion from the last discussion was that only the header is signed (which in turn contains the checksums of the files). -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: No more deltarpms by default
- Original Message - On Mon, 2014-10-06 at 10:54 +0100, Ian Malone wrote: On 6 October 2014 09:41, Rahul Sundaram methe...@gmail.com wrote: Hi One of the long standing features that were enabled by default in yum is support for delta rpms. dnf developers have disabled this and I think this change deserves a broader discussion https://bugzilla.redhat.com/show_bug.cgi?id=1148208 I have an internet flatrate at 150 mbs, and downloading the full rpms is ALOT faster than the the work that the delta rpms requires. Wow. Good to see normal users are taken into account. The main argument from a distro point of view is reducing load on servers, but I don't know many people on 150Mbs either. Heck, I've just tested my work janet connection and that's 100Mbs in our office. At home 8Mbps is a good day. (I'm assuming mbs is a typo for Mbps and not milli bit seconds, where the very slow transfer speed declines exponentially as the connection progresses.) The deltarpms were meant to serve two purposes 1) (lesser) Address the needs of users in developing countries (where Fedora is fairly popular) and bandwidth concerns are very considerable. Many of these users have connections that are either metered or extremely slow, so deltarpms provides a way to get the data to them more economically. This of course can be handled with a non-default option, so we can talk about making that more discoverable if we disable them by default. This is a good point but even in developing countries internet access is getting better and better. A few years ago installation DVD was the only way how to access Fedora repo. It's not requested anymore. But yeah, I do not live there. 2) (Major) Deltarpms significantly (Kevin, can you comment with numbers?) reduces the load on the Fedora update servers and mirrors, thereby reducing hosting costs as well as increasing the efficiency of the available bandwidth (since smaller downloads mean you will be tying up the line for a shorter amount of time). It would be nice to see if we can find ways to improve the performance of the deltarpm reconstruction instead. Much of the time is spent on compression/decompression tasks which *should* be massively parallel; we should be able to speed this up by taking advantage of additional cores (and hyperthreading) on the system, for example. Another option might be not to bother recompressing the RPMs but instead just install an uncompressed RPM (and possibly recompress it out of band if we needed to keep it in the cache). One idea - could we do some measurements in presto code? Aka remember download speed for drpms, then try to rebuild a few drpms and compare speed. If internet connection is significantly faster, offer full rpms re-download. If not, continue with rebuild. I don't have really fast net connection, just VDSL limited to 16 mbit due to distance from DSLAM but still using older netbook - the first thing I had to do, was to disable drpms. Same my wife's laptop - any bigger update means she can't really work. That's one option. And I agree delta rpms optimization would be great first step. Jaroslav -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: No more deltarpms by default
On Mon, Oct 6, 2014 at 4:46 PM, Jaroslav Reznik jrez...@redhat.com wrote: - Original Message - On Mon, 2014-10-06 at 10:54 +0100, Ian Malone wrote: On 6 October 2014 09:41, Rahul Sundaram methe...@gmail.com wrote: Hi One of the long standing features that were enabled by default in yum is support for delta rpms. dnf developers have disabled this and I think this change deserves a broader discussion https://bugzilla.redhat.com/show_bug.cgi?id=1148208 I have an internet flatrate at 150 mbs, and downloading the full rpms is ALOT faster than the the work that the delta rpms requires. Wow. Good to see normal users are taken into account. The main argument from a distro point of view is reducing load on servers, but I don't know many people on 150Mbs either. Heck, I've just tested my work janet connection and that's 100Mbs in our office. At home 8Mbps is a good day. (I'm assuming mbs is a typo for Mbps and not milli bit seconds, where the very slow transfer speed declines exponentially as the connection progresses.) The deltarpms were meant to serve two purposes 1) (lesser) Address the needs of users in developing countries (where Fedora is fairly popular) and bandwidth concerns are very considerable. Many of these users have connections that are either metered or extremely slow, so deltarpms provides a way to get the data to them more economically. This of course can be handled with a non-default option, so we can talk about making that more discoverable if we disable them by default. This is a good point but even in developing countries internet access is getting better and better. A few years ago installation DVD was the only way how to access Fedora repo. It's not requested anymore. But yeah, I do not live there. 2) (Major) Deltarpms significantly (Kevin, can you comment with numbers?) reduces the load on the Fedora update servers and mirrors, thereby reducing hosting costs as well as increasing the efficiency of the available bandwidth (since smaller downloads mean you will be tying up the line for a shorter amount of time). It would be nice to see if we can find ways to improve the performance of the deltarpm reconstruction instead. Much of the time is spent on compression/decompression tasks which *should* be massively parallel; we should be able to speed this up by taking advantage of additional cores (and hyperthreading) on the system, for example. Another option might be not to bother recompressing the RPMs but instead just install an uncompressed RPM (and possibly recompress it out of band if we needed to keep it in the cache). One idea - could we do some measurements in presto code? Aka remember download speed for drpms, then try to rebuild a few drpms and compare speed. If internet connection is significantly faster, offer full rpms re-download. If not, continue with rebuild. I don't have really fast net connection, just VDSL limited to 16 mbit due to distance from DSLAM but still using older netbook - the first thing I had to do, was to disable drpms. Same my wife's laptop - any bigger update means she can't really work. That's one option. And I agree delta rpms optimization would be great first step. I am not convinced that being fast and download less are mutually exclusive when using deltas. So we should keep deltas *and* make them faster. -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: No more deltarpms by default
On Mon, Oct 6, 2014 at 8:00 AM, drago01 drag...@gmail.com wrote: I am not convinced that being fast and download less are mutually exclusive when using deltas. So we should keep deltas *and* make them faster. Exactly. The fact that some users have more bandwidth means exactly what? Most people also have faster processors and disks now. It is more efficient from a networking perspective to minimize unnecessary traffic and use local processing. That was behind the rationale when delta was introduced and made the default. It was valid then, and it is valid now. You don't change the default without following the proper process. -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: No more deltarpms by default
On Mon, 06 Oct 2014 09:07:33 -0400 Stephen Gallagher sgall...@redhat.com wrote: The deltarpms were meant to serve two purposes 1) (lesser) Address the needs of users in developing countries (where Fedora is fairly popular) and bandwidth concerns are very considerable. Many of these users have connections that are either metered or extremely slow, so deltarpms provides a way to get the data to them more economically. This of course can be handled with a non-default option, so we can talk about making that more discoverable if we disable them by default. 2) (Major) Deltarpms significantly (Kevin, can you comment with numbers?) reduces the load on the Fedora update servers and mirrors, thereby reducing hosting costs as well as increasing the efficiency of the available bandwidth (since smaller downloads mean you will be tying up the line for a shorter amount of time). I don't recall this ever being a reason. (I could be wrong). I think this actually is worse for mirrors in some ways. They have to mirror a bunch more files and take up space (which is very dear to mirrors). It does get them less BW used, but most mirrors are on big links and don't really notice. It's definitely worse for releng. Generating drpms takes a long time and delays things like updates pushes or composes. It would be nice to see if we can find ways to improve the performance of the deltarpm reconstruction instead. Much of the time is spent on compression/decompression tasks which *should* be massively parallel; we should be able to speed this up by taking advantage of additional cores (and hyperthreading) on the system, for example. Another option might be not to bother recompressing the RPMs but instead just install an uncompressed RPM (and possibly recompress it out of band if we needed to keep it in the cache). Folks wanting to do so are welcome to help out... Get to coding. ;) kevin signature.asc Description: PGP signature -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: No more deltarpms by default
FWIW, I wrote and maintained yum-presto before it was integrated into yum. I've commented inline: On 10/06/2014 06:31 PM, Kevin Fenzi wrote: On Mon, 06 Oct 2014 09:07:33 -0400 Stephen Gallagher sgall...@redhat.com wrote: The deltarpms were meant to serve two purposes 1) (lesser) Address the needs of users in developing countries (where Fedora is fairly popular) and bandwidth concerns are very considerable. Many of these users have connections that are either metered or extremely slow, so deltarpms provides a way to get the data to them more economically. This of course can be handled with a non-default option, so we can talk about making that more discoverable if we disable them by default. When I wrote yum-presto it was for this reason. We were on a lousy link and I couldn't run a local mirror without deltarpms. Having said that, with download caps now being a thing in certain developed countries, this is no longer just a developing countries problem. 2) (Major) Deltarpms significantly (Kevin, can you comment with numbers?) reduces the load on the Fedora update servers and mirrors, thereby reducing hosting costs as well as increasing the efficiency of the available bandwidth (since smaller downloads mean you will be tying up the line for a shorter amount of time). I don't recall this ever being a reason. (I could be wrong). I think this actually is worse for mirrors in some ways. They have to mirror a bunch more files and take up space (which is very dear to mirrors). It does get them less BW used, but most mirrors are on big links and don't really notice. It's definitely worse for releng. Generating drpms takes a long time and delays things like updates pushes or composes. +1 It would be nice to see if we can find ways to improve the performance of the deltarpm reconstruction instead. Much of the time is spent on compression/decompression tasks which *should* be massively parallel; we should be able to speed this up by taking advantage of additional cores (and hyperthreading) on the system, for example. Another option might be not to bother recompressing the RPMs but instead just install an uncompressed RPM (and possibly recompress it out of band if we needed to keep it in the cache). Folks wanting to do so are welcome to help out... Get to coding. ;) As mentioned elsewhere, the problem *is* signatures. yum (quite rightly) refuses to install an rpm whose signature doesn't match the one in the primary repodata. And I believe that the signature in the RPM is also over the whole compressed rpm. To make this work, we'd need to add an uncompressed signature for every package to the primary repodata as well as probably the rpms themselves. We have already written the code to have yum run applydeltarpm in parallel according to the number of processors on a system, but it remains a single-threaded application that spends most of its time recompressing the data. Finally, if we do want to stop using deltarpms by default, I think the easiest thing to do would be to set dnf to use deltarpms if deltarpm is not installed, remove the deltarpm requirement that dnf has, and remove deltarpm from default installs. Then upgrades don't get unexpected changes in behavior and new installs can be told that getting deltarpm support is as easy as dnf install deltarpm. Jonathan -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: No more deltarpms by default
On 10/06/2014 05:16 PM, Gerald B. Cox wrote: The fact that some users have more bandwidth means exactly what? Most people also have faster processors and disks now. It is more efficient from a networking perspective to minimize unnecessary traffic and use local processing. That was behind the rationale when delta was introduced and made the default. It was valid then, and it is valid now. This argument is not valid. While most parts of a computer got faster things are not growing at the same rate. So what might have made sense a few years ago might be completely useless now. One thing that makes deltarpm less useful nowadays are seek times in hard disks (although they are going away). They are still the same as in the nineties while the number of files have been growing. At the same time network bandwidth has been growing at a faster rate as everything else. So if you want to make an argument for deltarpm, please do so but do not try to convince us to buy into an outdated rational. Florian -- Red Hat GmbH, http://www.de.redhat.com/ Registered seat: Grasbrunn, Commercial register: Amtsgericht Muenchen, HRB 153243, Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, Charles Peters -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: No more deltarpms by default
Bandwidth may be growing faster, but it started way behind processing power. It hasn't caught up. The current definition from the FCC for broadband is 4Mb. They are working to increase it, but that hasn't happened. Carriers are looking for ways to throttle traffic. Your assumption that everyone has great amounts of bandwidth available is erroneous. You've also got it backwards. Deltarpm is the default. If you want to change it, you need to convince the Fedora community. On Mon, Oct 6, 2014 at 10:01 AM, Florian Festi ffe...@redhat.com wrote: On 10/06/2014 05:16 PM, Gerald B. Cox wrote: The fact that some users have more bandwidth means exactly what? Most people also have faster processors and disks now. It is more efficient from a networking perspective to minimize unnecessary traffic and use local processing. That was behind the rationale when delta was introduced and made the default. It was valid then, and it is valid now. This argument is not valid. While most parts of a computer got faster things are not growing at the same rate. So what might have made sense a few years ago might be completely useless now. One thing that makes deltarpm less useful nowadays are seek times in hard disks (although they are going away). They are still the same as in the nineties while the number of files have been growing. At the same time network bandwidth has been growing at a faster rate as everything else. So if you want to make an argument for deltarpm, please do so but do not try to convince us to buy into an outdated rational. Florian -- Red Hat GmbH, http://www.de.redhat.com/ Registered seat: Grasbrunn, Commercial register: Amtsgericht Muenchen, HRB 153243, Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, Charles Peters -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: No more deltarpms by default
because it needs to build the complete xz-compressed RPM there was a discussion here not that long ago Is the XZ the only option for RPMs now? Can't it do it uncompressed? Or at least gzip -1. Or Rich can add new feature to his ultra-blazing-fast multi-core XZ decompressor. Compression :-) -- Later, Lukas #lzap Zapletal -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: No more deltarpms by default
Am 06.10.2014 um 19:18 schrieb Lukas Zapletal: because it needs to build the complete xz-compressed RPM there was a discussion here not that long ago Is the XZ the only option for RPMs now? Can't it do it uncompressed? Or at least gzip -1. Or Rich can add new feature to his ultra-blazing-fast multi-core XZ decompressor. Compression :-) the last discussions suggested that the result needs to be *identical* to the full RPM downloaded for not break signatures for me the greatest thing about deltarpm is that finally you have a full RPM in /var/cache/yum and from there can populate your local caching repo and deploy to the other 10,20,50,100 machines which means finally you win on both sides: download/traffic and no other machine in the network needs to touch the internet or know about deltarpms signature.asc Description: OpenPGP digital signature -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: No more deltarpms by default
On Mon, Oct 6, 2014 at 7:01 PM, Florian Festi ffe...@redhat.com wrote: On 10/06/2014 05:16 PM, Gerald B. Cox wrote: The fact that some users have more bandwidth means exactly what? Most people also have faster processors and disks now. It is more efficient from a networking perspective to minimize unnecessary traffic and use local processing. That was behind the rationale when delta was introduced and made the default. It was valid then, and it is valid now. This argument is not valid. While most parts of a computer got faster things are not growing at the same rate. So what might have made sense a few years ago might be completely useless now. One thing that makes deltarpm less useful nowadays are seek times in hard disks (although they are going away). They are still the same as in the nineties while the number of files have been growing. No they aren't. Even on cheap SSDs there are at least one order of magnitude faster. -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: No more deltarpms by default
Ok I think the above thread explains it, the Jonathan's mail lists what would be needed and it looks like there are some blockers on the infra side. Disregard. -- Later, Lukas #lzap Zapletal -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: No more deltarpms by default
On 10/06/2014 06:53 PM, Jonathan Dieter wrote: Get to coding. ;) As mentioned elsewhere, the problem *is* signatures. yum (quite rightly) refuses to install an rpm whose signature doesn't match the one in the primary repodata. And I believe that the signature in the RPM is also over the whole compressed rpm. To make this work, we'd need to add an uncompressed signature for every package to the primary repodata as well as probably the rpms themselves. We have already written the code to have yum run applydeltarpm in parallel according to the number of processors on a system, but it remains a single-threaded application that spends most of its time recompressing the data. The way of getting around all this unnecessary computation is establishing trust via the deltarpm itself and giving up the idea of reconstructing the originally rpm as a prove of everything worked out just fine. To save even more time the delta would need to be applied directly on disk without creating an package at all. This would require a deeper integration with rpm and requires some tricky algorithms to make sure everything is ok in advance and won't blow up during the rpm transaction. So if anyone needs a hobby... Florian -- Red Hat GmbH, http://www.de.redhat.com/ Registered seat: Grasbrunn, Commercial register: Amtsgericht Muenchen, HRB 153243, Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, Charles Peters -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: No more deltarpms by default
Am 06.10.2014 um 19:45 schrieb Florian Festi: On 10/06/2014 06:53 PM, Jonathan Dieter wrote: Get to coding. ;) As mentioned elsewhere, the problem *is* signatures. yum (quite rightly) refuses to install an rpm whose signature doesn't match the one in the primary repodata. And I believe that the signature in the RPM is also over the whole compressed rpm. To make this work, we'd need to add an uncompressed signature for every package to the primary repodata as well as probably the rpms themselves. We have already written the code to have yum run applydeltarpm in parallel according to the number of processors on a system, but it remains a single-threaded application that spends most of its time recompressing the data. The way of getting around all this unnecessary computation is establishing trust via the deltarpm itself and giving up the idea of reconstructing the originally rpm as a prove of everything worked out just fine. To save even more time the delta would need to be applied directly on disk without creating an package at all. This would require a deeper integration with rpm and requires some tricky algorithms to make sure everything is ok in advance and won't blow up during the rpm transaction. So if anyone needs a hobby... oh no - don't tie all together for reasons which did not destory the world over years - it is a damned good design that the part dealing with rpm packages don't need to know anything aboutt delta rpms because the normal packages are created before that step don't break the unix-way of work the current behavior follows for no good reason and there is none - otherwise deltarpm would not have been default over years the way it works now signature.asc Description: OpenPGP digital signature -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: No more deltarpms by default
On 10/06/2014 07:53 PM, Jonathan Dieter wrote: As mentioned elsewhere, the problem *is* signatures. yum (quite rightly) refuses to install an rpm whose signature doesn't match the one in the primary repodata. And I believe that the signature in the RPM is also over the whole compressed rpm. To make this work, we'd need to add an uncompressed signature for every package to the primary repodata as well as probably the rpms themselves. IIRC repodata doesn't carry signatures, it caries a (sha256) checksum of its own on the entire package. Rpm signatures are a different beast: there's (sha1) checksum and a signature on the header, plus rpm v3 checksum and signature on header + payload. rpm -K style signature checking is the only thing that looks at the header + payload checksum and signature, otherwise rpm only uses the checksum/signature on header, which of course then has checksums of the individual files. Rpm can (and usually does) ignore the payload signature, file-level checksums get checked anyway (that too *can* be disabled but...) However it still requires the input data to be compressed in the format specified in the header. So to avoid having to compress tons of data only to decompress it shortly afterwards, there would have to be a way to tell librpm to expect a different payload compression (or specifically, that the payload is not compressed). Shouldn't be rocket science. - Panu - -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: No more deltarpms by default
On 10/06/2014 08:57 PM, Reindl Harald wrote: Am 06.10.2014 um 19:45 schrieb Florian Festi: The way of getting around all this unnecessary computation is establishing trust via the deltarpm itself and giving up the idea of reconstructing the originally rpm as a prove of everything worked out just fine. To save even more time the delta would need to be applied directly on disk without creating an package at all. This would require a deeper integration with rpm and requires some tricky algorithms to make sure everything is ok in advance and won't blow up during the rpm transaction. So if anyone needs a hobby... oh no - don't tie all together for reasons which did not destory the world over years - it is a damned good design that the part dealing with rpm packages don't need to know anything aboutt delta rpms because the normal packages are created before that step don't break the unix-way of work the current behavior follows for no good reason and there is none - otherwise deltarpm would not have been default over years the way it works now Ok, granted, this sounds pretty scary. But if we give rpm the ability to upgrade an installed package with a deltarpm, it wouldn't take away deltarpm's ability to generate a full rpm from a deltarpm. And it does have the advantage of cutting right through the knot. We already store checksums of the deltarpms in prestodelta.xml, as well as in the deltarpm itself. Probably the biggest weakness would be the chance that something would change on-disk between the check stage and actual install stage. We'd have to evaluate whether it's worth making a temporary copy of the old data during the check stage and then applying the deltarpm to that. All of this would require a lot of buy-in from the rpm guys, though (Florian, you're one of them, right?). If I recall correctly, when we first looked at deltarpms, one of the selling points was that rpm didn't have to change at all. Jonathan -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
