Re: [OE-core] Add package managers as a plugin
From: Paul Barker Sent: Thursday, July 2, 2020 10:28 PM To: Fredrik Gustafsson Cc: openembedded-core; tools-cfpbuild-internal Subject: Re: [OE-core] Add package managers as a plugin On Thu, 2 Jul 2020 at 20:29, Fredrik Gustafsson wrote: > > OE-core today has three different package managers, the well-known formats deb > and rpm is supported as well as ipkg that is good for embedded devices. > > When building and having a good cache hit, a significant amount of time is > spent in the phase of generating a rootfs, which is really about the > performance of the package manager. To save build time and also get a > package manager that is suitanle for use on targets where flash memory is a > concern, support for apk is suggested. > > However, it might or might not be what's wanted for OE-core since it increases > the test matrix. Therefore I will refactor the package management code to > allow a layer to add a new package manager without editing the meta layer. > > This refactor will be divided into multiple patch series to be easier to > review, > this is the second patch series. Sorry to be a pain but I think this series suffers from the opposite problem to the previous version - it's only a partial move of some of the code. I'd much rather review a series which does the complete refactor of the existing code into separate modules without making any functional changes. I'll let others give their opinions as well, I'm happy to review this one if we do want to look at this piecemeal. -- Paul Barker Konsulko Group No problems, as said before I will do my best to make this easier to review. I've sent a new patch series. /Fredrik-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#140279): https://lists.openembedded.org/g/openembedded-core/message/140279 Mute This Topic: https://lists.openembedded.org/mt/75057633/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] Add package managers as a plugin
On Thu, 2 Jul 2020 at 20:29, Fredrik Gustafsson wrote: > > OE-core today has three different package managers, the well-known formats deb > and rpm is supported as well as ipkg that is good for embedded devices. > > When building and having a good cache hit, a significant amount of time is > spent in the phase of generating a rootfs, which is really about the > performance of the package manager. To save build time and also get a > package manager that is suitanle for use on targets where flash memory is a > concern, support for apk is suggested. > > However, it might or might not be what's wanted for OE-core since it increases > the test matrix. Therefore I will refactor the package management code to > allow a layer to add a new package manager without editing the meta layer. > > This refactor will be divided into multiple patch series to be easier to > review, > this is the second patch series. Sorry to be a pain but I think this series suffers from the opposite problem to the previous version - it's only a partial move of some of the code. I'd much rather review a series which does the complete refactor of the existing code into separate modules without making any functional changes. I'll let others give their opinions as well, I'm happy to review this one if we do want to look at this piecemeal. -- Paul Barker Konsulko Group -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#140246): https://lists.openembedded.org/g/openembedded-core/message/140246 Mute This Topic: https://lists.openembedded.org/mt/75057633/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] Add package managers as a plugin
Hi all, Catching up on the conversation...impressive results coming out of apk. Haven't used apk enough to have an educated opinion, but by quickly glancing at the code, I believe it uses an ad hoc solver, which might struggle with corner cases (same issues we had with opkg, before switching to libsolv). For building images without package managers, its likely that this isn't a serious concern. Regarding opkg, I haven't focus on build time so I get the feeling there might be some low hanging fruit. I plan to spend some time investigating and will report back. The mechanism used by apk (1 read, 1 write) can be added to opkg too...in general the idea with opkg is that you set it up with the trade off that make sense for you. So I can envision having a config option that enables apk-like behavior. Regardless, its exciting to see changes that expand the ecosystem! Cheers, Alejandro On 6/30/20 5:12 PM, Fredrik Gustafsson wrote: Hi Martin, apk is quite different compared to other package managers. A traditional package manager download a package, saves it to a local cache, verifies it and then extract it. That's at least 3 reads and 2 writes of the data. Apk does this on the fly, so 1 read and 1 write of the data. I believe that it's a so fundamental design choice that opkg would have it very hard to compete for speed. The speed advantage is even bigger when installing over network since the extraction and verification is done wile the network is waiting for data. That means that apk has usually installed the package when an ordinary package manager just has downloaded it and should start to verify and install it. So there's no secret sauce or awesome optimizations, it's "just" an other architectural design. With that said, there's still some low hanging fruit for apk, for example package generation (my patch is doing 2 reads and 2 writes of all data) and index generation (duplicate reads and one extraction too much) that could have significant speedup. But let's make it possible to use apk at all before starting to tune it. More information: https://urldefense.com/v3/__https://www.youtube.com/watch?v=sIG2P9k6EjA__;!!FbZ0ZwI3Qg!4qK6Dao4S2RQqjMvNiVBB04AmB8DTgW7Ccfr39CwxKWkSMA8DjszgWpvvj5RVMCfLL158w$ BR Fredrik From: Martin Jansa Sent: Tuesday, June 30, 2020 11:54 PM To: Fredrik Gustafsson Cc: Ross Burton; OE-core; tools-cfpbuild-internal Subject: Re: [OE-core] Add package managers as a plugin On Tue, Jun 30, 2020 at 07:01:23PM +, Fredrik Gustafsson wrote: Hi Ross, those 5 seconds will increase to minutes for my use case and we build a lot hence I hope this will save us a lot of computer power and engineering time. For example I've sent numbers on building a bigger image (core-image-sato-sdk-ptest) to Alex and Alex: out.apk.1: 1:13.35 out.apk.2: 1:13.51 out.apk.3: 1:13.23 out.apk.4: 1:14.07 out.apk.5: 1:13.00 out.deb.1: 3:49.37 out.deb.2: 3:50.77 out.deb.3: 3:51.39 out.deb.4: 3:53.40 out.deb.5: 3:53.99 out.ipk.1: 2:38.99 out.ipk.2: 2:39.07 out.ipk.3: 2:35.34 out.ipk.4: 2:36.15 out.ipk.5: 2:34.55 out.rpm.1: 1:58.61 out.rpm.2: 1:59.42 out.rpm.3: 1:59.70 out.rpm.4: 1:58.96 out.rpm.5: 1:58.11 Also consider that if we've a target without python with limited space, rpm is out of the question. So the real comparison would be between ipk and apk. Let's say we can save 80 seconds in each build. Now multiply that with the number of builds, and we do a lot (I guess we might approach 100 000 builds/week in the next year). Then this will save 92.5 days build time each week. This is impressive. Have you tried to profile where opkg spends most of the time compared to apk? It would be nice to know if there is something fundamentally different in how they handle packages or if apk is just better optimized. Now with opkg better maintained (thanks Alejandro!) there might be some relatively low hanging optimalizations which could be implemented there as well (for people who didn't implement the rest to switch to apk yet :)). I know at LGE we have some patches (some related to performance as well) which unfortunately weren't ever upstreamed :/, some are at: https://urldefense.com/v3/__https://github.com/webosose/meta-webosose/tree/master/meta-webos/recipes-devtools/opkg/opkg__;!!FbZ0ZwI3Qg!4qK6Dao4S2RQqjMvNiVBB04AmB8DTgW7Ccfr39CwxKWkSMA8DjszgWpvvj5RVMDh3PHxpA$ Cheers, -- Cheers, Alejandro -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#140197): https://lists.openembedded.org/g/openembedded-core/message/140197 Mute This Topic: https://lists.openembedded.org/mt/75057633/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] Add package managers as a plugin
Yes, I thought we at @lge do a lot of builds, but after checking our statistics we average "only" around 30,000 builds per week in last 5 years. But on the other hand, our typical image is quite big (between 300MB and 1GB for enduser image and between 700MG and 1.5GB for developer images depending on the target MACHINE for tar.bz2 IMAGE_FSTYPE), but each build is producing at least 2 images (enduser and devel), but e.g. TVs are building 15 different rootfs for each MACHINE, so the number of do_rootfs tasks is easily over 100,000 per week and I'm seeing average do_rootfs times around 10-14 minutes for bigger images and 4 minutes for "small" images. I should definitely try apk to prevent global warming :). On Wed, Jul 1, 2020 at 1:15 AM Andre McCurdy wrote: > On Tue, Jun 30, 2020 at 2:54 PM Martin Jansa > wrote: > > On Tue, Jun 30, 2020 at 07:01:23PM +, Fredrik Gustafsson wrote: > > > Hi Ross, > > > those 5 seconds will increase to minutes for my use case and we build > a lot > > > hence I hope this will save us a lot of computer power and engineering > time. > > > For example I've sent numbers on building a bigger image > (core-image-sato-sdk-ptest) > > > to Alex and Alex: > > > out.apk.1: 1:13.35 > > > out.apk.2: 1:13.51 > > > out.apk.3: 1:13.23 > > > out.apk.4: 1:14.07 > > > out.apk.5: 1:13.00 > > > out.deb.1: 3:49.37 > > > out.deb.2: 3:50.77 > > > out.deb.3: 3:51.39 > > > out.deb.4: 3:53.40 > > > out.deb.5: 3:53.99 > > > out.ipk.1: 2:38.99 > > > out.ipk.2: 2:39.07 > > > out.ipk.3: 2:35.34 > > > out.ipk.4: 2:36.15 > > > out.ipk.5: 2:34.55 > > > out.rpm.1: 1:58.61 > > > out.rpm.2: 1:59.42 > > > out.rpm.3: 1:59.70 > > > out.rpm.4: 1:58.96 > > > out.rpm.5: 1:58.11 > > > > > > Also consider that if we've a target without python with limited > space, rpm is out > > > of the question. So the real comparison would be between ipk and apk. > Let's > > > say we can save 80 seconds in each build. Now multiply that with the > number of > > > builds, and we do a lot (I guess we might approach 100 000 builds/week > in the > > > next year). Then this will save 92.5 days build time each week. > > > > This is impressive. > > Indeed. 10 builds per minute, 24hrs a day. But not as impressive as > the scale of the development team able to create enough changes to > trigger so many builds and the scale of the test infrastructure able > to consume them! > > Assume one developer makes 10 changes a day and each change gets built > for 10 different targets... 100,000 builds per week implies a team of > over 140 developers. > -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#140156): https://lists.openembedded.org/g/openembedded-core/message/140156 Mute This Topic: https://lists.openembedded.org/mt/75057633/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] Add package managers as a plugin
On Tue, Jun 30, 2020 at 2:54 PM Martin Jansa wrote: > On Tue, Jun 30, 2020 at 07:01:23PM +, Fredrik Gustafsson wrote: > > Hi Ross, > > those 5 seconds will increase to minutes for my use case and we build a lot > > hence I hope this will save us a lot of computer power and engineering time. > > For example I've sent numbers on building a bigger image > > (core-image-sato-sdk-ptest) > > to Alex and Alex: > > out.apk.1: 1:13.35 > > out.apk.2: 1:13.51 > > out.apk.3: 1:13.23 > > out.apk.4: 1:14.07 > > out.apk.5: 1:13.00 > > out.deb.1: 3:49.37 > > out.deb.2: 3:50.77 > > out.deb.3: 3:51.39 > > out.deb.4: 3:53.40 > > out.deb.5: 3:53.99 > > out.ipk.1: 2:38.99 > > out.ipk.2: 2:39.07 > > out.ipk.3: 2:35.34 > > out.ipk.4: 2:36.15 > > out.ipk.5: 2:34.55 > > out.rpm.1: 1:58.61 > > out.rpm.2: 1:59.42 > > out.rpm.3: 1:59.70 > > out.rpm.4: 1:58.96 > > out.rpm.5: 1:58.11 > > > > Also consider that if we've a target without python with limited space, rpm > > is out > > of the question. So the real comparison would be between ipk and apk. Let's > > say we can save 80 seconds in each build. Now multiply that with the number > > of > > builds, and we do a lot (I guess we might approach 100 000 builds/week in > > the > > next year). Then this will save 92.5 days build time each week. > > This is impressive. Indeed. 10 builds per minute, 24hrs a day. But not as impressive as the scale of the development team able to create enough changes to trigger so many builds and the scale of the test infrastructure able to consume them! Assume one developer makes 10 changes a day and each change gets built for 10 different targets... 100,000 builds per week implies a team of over 140 developers. -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#140154): https://lists.openembedded.org/g/openembedded-core/message/140154 Mute This Topic: https://lists.openembedded.org/mt/75057633/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] Add package managers as a plugin
Hi Martin, apk is quite different compared to other package managers. A traditional package manager download a package, saves it to a local cache, verifies it and then extract it. That's at least 3 reads and 2 writes of the data. Apk does this on the fly, so 1 read and 1 write of the data. I believe that it's a so fundamental design choice that opkg would have it very hard to compete for speed. The speed advantage is even bigger when installing over network since the extraction and verification is done wile the network is waiting for data. That means that apk has usually installed the package when an ordinary package manager just has downloaded it and should start to verify and install it. So there's no secret sauce or awesome optimizations, it's "just" an other architectural design. With that said, there's still some low hanging fruit for apk, for example package generation (my patch is doing 2 reads and 2 writes of all data) and index generation (duplicate reads and one extraction too much) that could have significant speedup. But let's make it possible to use apk at all before starting to tune it. More information: https://www.youtube.com/watch?v=sIG2P9k6EjA BR Fredrik From: Martin Jansa Sent: Tuesday, June 30, 2020 11:54 PM To: Fredrik Gustafsson Cc: Ross Burton; OE-core; tools-cfpbuild-internal Subject: Re: [OE-core] Add package managers as a plugin On Tue, Jun 30, 2020 at 07:01:23PM +, Fredrik Gustafsson wrote: > Hi Ross, > those 5 seconds will increase to minutes for my use case and we build a lot > hence I hope this will save us a lot of computer power and engineering time. > For example I've sent numbers on building a bigger image > (core-image-sato-sdk-ptest) > to Alex and Alex: > out.apk.1: 1:13.35 > out.apk.2: 1:13.51 > out.apk.3: 1:13.23 > out.apk.4: 1:14.07 > out.apk.5: 1:13.00 > out.deb.1: 3:49.37 > out.deb.2: 3:50.77 > out.deb.3: 3:51.39 > out.deb.4: 3:53.40 > out.deb.5: 3:53.99 > out.ipk.1: 2:38.99 > out.ipk.2: 2:39.07 > out.ipk.3: 2:35.34 > out.ipk.4: 2:36.15 > out.ipk.5: 2:34.55 > out.rpm.1: 1:58.61 > out.rpm.2: 1:59.42 > out.rpm.3: 1:59.70 > out.rpm.4: 1:58.96 > out.rpm.5: 1:58.11 > > Also consider that if we've a target without python with limited space, rpm > is out > of the question. So the real comparison would be between ipk and apk. Let's > say we can save 80 seconds in each build. Now multiply that with the number of > builds, and we do a lot (I guess we might approach 100 000 builds/week in the > next year). Then this will save 92.5 days build time each week. This is impressive. Have you tried to profile where opkg spends most of the time compared to apk? It would be nice to know if there is something fundamentally different in how they handle packages or if apk is just better optimized. Now with opkg better maintained (thanks Alejandro!) there might be some relatively low hanging optimalizations which could be implemented there as well (for people who didn't implement the rest to switch to apk yet :)). I know at LGE we have some patches (some related to performance as well) which unfortunately weren't ever upstreamed :/, some are at: https://github.com/webosose/meta-webosose/tree/master/meta-webos/recipes-devtools/opkg/opkg Cheers, -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#140152): https://lists.openembedded.org/g/openembedded-core/message/140152 Mute This Topic: https://lists.openembedded.org/mt/75057633/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] Add package managers as a plugin
On Tue, Jun 30, 2020 at 07:01:23PM +, Fredrik Gustafsson wrote: > Hi Ross, > those 5 seconds will increase to minutes for my use case and we build a lot > hence I hope this will save us a lot of computer power and engineering time. > For example I've sent numbers on building a bigger image > (core-image-sato-sdk-ptest) > to Alex and Alex: > out.apk.1: 1:13.35 > out.apk.2: 1:13.51 > out.apk.3: 1:13.23 > out.apk.4: 1:14.07 > out.apk.5: 1:13.00 > out.deb.1: 3:49.37 > out.deb.2: 3:50.77 > out.deb.3: 3:51.39 > out.deb.4: 3:53.40 > out.deb.5: 3:53.99 > out.ipk.1: 2:38.99 > out.ipk.2: 2:39.07 > out.ipk.3: 2:35.34 > out.ipk.4: 2:36.15 > out.ipk.5: 2:34.55 > out.rpm.1: 1:58.61 > out.rpm.2: 1:59.42 > out.rpm.3: 1:59.70 > out.rpm.4: 1:58.96 > out.rpm.5: 1:58.11 > > Also consider that if we've a target without python with limited space, rpm > is out > of the question. So the real comparison would be between ipk and apk. Let's > say we can save 80 seconds in each build. Now multiply that with the number of > builds, and we do a lot (I guess we might approach 100 000 builds/week in the > next year). Then this will save 92.5 days build time each week. This is impressive. Have you tried to profile where opkg spends most of the time compared to apk? It would be nice to know if there is something fundamentally different in how they handle packages or if apk is just better optimized. Now with opkg better maintained (thanks Alejandro!) there might be some relatively low hanging optimalizations which could be implemented there as well (for people who didn't implement the rest to switch to apk yet :)). I know at LGE we have some patches (some related to performance as well) which unfortunately weren't ever upstreamed :/, some are at: https://github.com/webosose/meta-webosose/tree/master/meta-webos/recipes-devtools/opkg/opkg Cheers, signature.asc Description: PGP signature -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#140151): https://lists.openembedded.org/g/openembedded-core/message/140151 Mute This Topic: https://lists.openembedded.org/mt/75057633/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] Add package managers as a plugin
Hey Fredrik, On 6/30/20 1:38 PM, Fredrik Gustafsson wrote: I added apk to your script and then ran That is not supportive of my statement of ipk being the slowest package manager. That statement come from timing tests done with our proprietary version of poky that is somewhat extended when it comes to the do_rootfs task. We're generating a bit more artifacts than poky. Ah; good to know. Thanks for checking again! Sounds like it could be related to extant opkg bugs which degrade performance when you have many (read: thousands) of packages in the feed. https://bugzilla.yoctoproject.org/show_bug.cgi?id=13593 So I tried your script with core-image-sato-sdk-ptest instead that is a bigger image. The results where more mixed. There is a huge difference between different package managers: But yeah, I should probably rephrase my cover letter. You'll find a layer for APK that will apply on top of my patch serie attached to this email. Thanks for sending that; I'm interested to take a look at it. Those new performance numbers look impressive. Good luck with the patchset. -- Alex Stewart Software Engineer - LabVIEW Real-Time OS National Instruments alex.stew...@ni.com office: +1(512)683-8522 -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#140150): https://lists.openembedded.org/g/openembedded-core/message/140150 Mute This Topic: https://lists.openembedded.org/mt/75057633/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] Add package managers as a plugin
Hi Ross, those 5 seconds will increase to minutes for my use case and we build a lot hence I hope this will save us a lot of computer power and engineering time. For example I've sent numbers on building a bigger image (core-image-sato-sdk-ptest) to Alex and Alex: out.apk.1: 1:13.35 out.apk.2: 1:13.51 out.apk.3: 1:13.23 out.apk.4: 1:14.07 out.apk.5: 1:13.00 out.deb.1: 3:49.37 out.deb.2: 3:50.77 out.deb.3: 3:51.39 out.deb.4: 3:53.40 out.deb.5: 3:53.99 out.ipk.1: 2:38.99 out.ipk.2: 2:39.07 out.ipk.3: 2:35.34 out.ipk.4: 2:36.15 out.ipk.5: 2:34.55 out.rpm.1: 1:58.61 out.rpm.2: 1:59.42 out.rpm.3: 1:59.70 out.rpm.4: 1:58.96 out.rpm.5: 1:58.11 Also consider that if we've a target without python with limited space, rpm is out of the question. So the real comparison would be between ipk and apk. Let's say we can save 80 seconds in each build. Now multiply that with the number of builds, and we do a lot (I guess we might approach 100 000 builds/week in the next year). Then this will save 92.5 days build time each week. That's why I'm passionated about this. /Fredrik From: Ross Burton Sent: Friday, June 26, 2020 12:52 PM To: Fredrik Gustafsson Cc: OE-core; tools-cfpbuild-internal; Hugo Cedervall Subject: Re: [OE-core] Add package managers as a plugin On Thu, 25 Jun 2020 at 11:14, Fredrik Gustafsson wrote: > Poky today has three different package managers, the well-known formats deb > and rpm is supported as well as ipkg that is good for embedded devices. > > When building and having a good cache hit, a significant amount of time is > spent in the phase of generating a rootfs, which is really about the > performance of the package manager. ipkg is way slower than deb or rpm. To > save build time and also get a package manager that is suitanle for use on > targets where flash memory is a concern, support for apk is suggested. > > However, it might or might not be what's wanted for Poky since it increases > the test matrix. Therefore this patch series refactors the package > management code so that it's possible to add more package managers in other > own layer. I will send another patch serie that will add apk. As to the code review I mostly second what Paul said. However, obviously you actually have strong feelings on this as the series represents a fair amount of work. What are the compelling reasons to use apk? Five seconds on core-image-minimal doesn't really sell it to me, I don't think anyone ever claimed that opkg has been performance optimised, dpkg isn't really used that much: maybe we should re-evaluate what package managers are supported if apk has feature parity but is significantly faster or has other advantages. Ross -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#140148): https://lists.openembedded.org/g/openembedded-core/message/140148 Mute This Topic: https://lists.openembedded.org/mt/75057633/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] Add package managers as a plugin
Hi Alex, now I see what you mean. I got a script from Alex Stewart, here's the results for core-image-sato-sdk-ptest: out.apk.1: 1:13.35 out.apk.2: 1:13.51 out.apk.3: 1:13.23 out.apk.4: 1:14.07 out.apk.5: 1:13.00 out.deb.1: 3:49.37 out.deb.2: 3:50.77 out.deb.3: 3:51.39 out.deb.4: 3:53.40 out.deb.5: 3:53.99 out.ipk.1: 2:38.99 out.ipk.2: 2:39.07 out.ipk.3: 2:35.34 out.ipk.4: 2:36.15 out.ipk.5: 2:34.55 out.rpm.1: 1:58.61 out.rpm.2: 1:59.42 out.rpm.3: 1:59.70 out.rpm.4: 1:58.96 out.rpm.5: 1:58.11 BR Fredrik From: Alexander Kanavin Sent: Tuesday, June 30, 2020 5:43 PM To: Fredrik Gustafsson Cc: OE-core; tools-cfpbuild-internal; Hugo Cedervall Subject: Re: [OE-core] Add package managers as a plugin What I meant is that 15s vs 20s on a small image could have been 5 min vs 5min5s on a bigger image, so I wanted to see that the difference does scale up with image size. Can you to provide sdk-ptest numbers for deb and ipk as well? Alex On Tue, 30 Jun 2020 at 17:04, Fredrik Gustafsson mailto:fredrik.gustafs...@axis.com>> wrote: Hi Alexander, what are you finding strange with the numbers? Too fast or to big of a difference? Note that it's only the do_rootfs task that i measured. I've tried the core-image-sato-sdk-ptest as you suggested. I run the following command multiple times: bitbake core-image-sato-sdk-ptest -c cleansstate && bitbake core-image-sato-sdk-ptest -c clean && time bitbake core-image-sato-sdk-ptest -f -c do_rootfs APK 0,85s user 0,16s system 0% cpu 1:13,66 total 0,87s user 0,14s system 0% cpu 1:13,28 total 0,89s user 0,11s system 0% cpu 1:13,10 total RPM === 0,67s user 0,12s system 0% cpu 2:02,48 total 0,68s user 0,11s system 0% cpu 2:03,39 total 0,73s user 0,09s system 0% cpu 1:59,87 total 5 seconds doesn't sound that much but it's 25% speedup. Since building is mostly cached but generating rootfs isn't this is a huge time consumption for our use case and our image takes around 4 minutes to build. Saving one minutes times the many builds we do is a huge win. (Excuse my late reply, running core-image-sato-sdk-ptest uncovered a few bugs in the implementation of apk in OE. So thanks for having me testing this image) /Fredrik From: Alexander Kanavin mailto:alex.kana...@gmail.com>> Sent: Thursday, June 25, 2020 12:24 PM To: Fredrik Gustafsson Cc: OE-core; tools-cfpbuild-internal; Hugo Cedervall Subject: Re: [OE-core] Add package managers as a plugin Apologies, but these metrics do not look particularly convincing - 15s vs 20s? Can you try it with an image several gigabytes in size? Core-image-sato-sdk-ptest is a good option. Alex On Thu, 25 Jun 2020 at 12:14, Fredrik Gustafsson mailto:fredrik.gustafs...@axis.com>> wrote: Poky today has three different package managers, the well-known formats deb and rpm is supported as well as ipkg that is good for embedded devices. When building and having a good cache hit, a significant amount of time is spent in the phase of generating a rootfs, which is really about the performance of the package manager. ipkg is way slower than deb or rpm. To save build time and also get a package manager that is suitanle for use on targets where flash memory is a concern, support for apk is suggested. However, it might or might not be what's wanted for Poky since it increases the test matrix. Therefore this patch series refactors the package management code so that it's possible to add more package managers in other own layer. I will send another patch serie that will add apk. Perfomance metrics below, (note that this includes build times). APK === bitbake core-image-minimal 15.84s user 2.60s system 0% cpu 1:26:19.21 total 16.01s user 2.58s system 0% cpu 1:26:03.72 total 15.69s user 2.61s system 0% cpu 1:26:45.45 total bitbake core-image-minimal -c cleansstate bitbake core-image-minimal -c clean bitbake core-image-minimal -f -c do_rootfs 0.55s user 0.06s system 4% cpu 14.236 total 0.54s user 0.08s system 4% cpu 15.247 total 0.52s user 0.15s system 4% cpu 15.143 total RPM === bitbake core-image-minimal 18.57s user 3.09s system 0% cpu 1:31:29.09 total 18.58s user 3.08s system 0% cpu 1:30:53.80 total 18.20s user 3.31s system 0% cpu 1:31:06.69 total bitbake core-image-minimal -c cleansstate bitbake core-image-minimal -c clean bitbake core-image-minimal -c do_rootfs -f 0.58s user 0.10s system 3% cpu 19.470 total 0.57s user 0.09s system 3% cpu 19.480 total 0.60s user 0.07s system 3% cpu 20.381 total -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#140147): https://lists.openembedded.org/g/openembedded-core/message/140147 Mute This Topic: https://lists.openembedded.org/mt/75057633/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] Add package managers as a plugin
Hi Paul, thank you for your effort on this. It's appreciated! I'll go through ýour advice and see if I can improve this series. BR Fredrik From: Paul Barker Sent: Thursday, June 25, 2020 12:40 PM To: Fredrik Gustafsson Cc: openembedded-core; tools-cfpbuild-internal; Hugo Cedervall Subject: Re: [OE-core] Add package managers as a plugin On Thu, 25 Jun 2020 at 11:21, Fredrik Gustafsson wrote: > > Poky today has three different package managers, the well-known formats deb > and rpm is supported as well as ipkg that is good for embedded devices. > > When building and having a good cache hit, a significant amount of time is > spent in the phase of generating a rootfs, which is really about the > performance of the package manager. ipkg is way slower than deb or rpm. To > save build time and also get a package manager that is suitanle for use on > targets where flash memory is a concern, support for apk is suggested. > > However, it might or might not be what's wanted for Poky since it increases > the test matrix. Therefore this patch series refactors the package > management code so that it's possible to add more package managers in other > own layer. I will send another patch serie that will add apk. > > Perfomance metrics below, (note that this includes build times). > > APK > === > bitbake core-image-minimal > 15.84s user 2.60s system 0% cpu 1:26:19.21 total > 16.01s user 2.58s system 0% cpu 1:26:03.72 total > 15.69s user 2.61s system 0% cpu 1:26:45.45 total > > bitbake core-image-minimal -c cleansstate > bitbake core-image-minimal -c clean > bitbake core-image-minimal -f -c do_rootfs > 0.55s user 0.06s system 4% cpu 14.236 total > 0.54s user 0.08s system 4% cpu 15.247 total > 0.52s user 0.15s system 4% cpu 15.143 total > > RPM > === > bitbake core-image-minimal > 18.57s user 3.09s system 0% cpu 1:31:29.09 total > 18.58s user 3.08s system 0% cpu 1:30:53.80 total > 18.20s user 3.31s system 0% cpu 1:31:06.69 total > > bitbake core-image-minimal -c cleansstate > bitbake core-image-minimal -c clean > bitbake core-image-minimal -c do_rootfs -f > 0.58s user 0.10s system 3% cpu 19.470 total > 0.57s user 0.09s system 3% cpu 19.480 total > 0.60s user 0.07s system 3% cpu 20.381 total Some initial review comments incoming on all these patches. I've just done a surface level reading, I think the way you're submitting this still needs some work before it's in a state where we can properly review the changes in detail. Right now it's too big and too easy to get confused when looking through 26 large patches. I'm very supportive of tidying this up and making it more dynamic, we just need to make sure we don't break things along the way. Thanks, -- Paul Barker Konsulko Group -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#140146): https://lists.openembedded.org/g/openembedded-core/message/140146 Mute This Topic: https://lists.openembedded.org/mt/75057633/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] Add package managers as a plugin
Hi Alex,
Thanks for supplying a script!
I added apk to your script and then ran
```
for f in `ls`; do
printf "$f: "; cat $f | tail -2 | head -1 | awk -F ' ' '{print $3}' | sed -e
's/elapsed//'
done | sort
```
The result is here:
out.apk.1: 0:13.35
out.apk.2: 0:12.89
out.apk.3: 0:12.99
out.apk.4: 0:12.87
out.apk.5: 0:13.02
out.deb.1: 0:20.27
out.deb.2: 0:20.30
out.deb.3: 0:20.33
out.deb.4: 0:20.06
out.deb.5: 0:20.17
out.ipk.1: 0:18.51
out.ipk.2: 0:18.78
out.ipk.3: 0:18.80
out.ipk.4: 0:18.99
out.ipk.5: 0:18.97
out.rpm.1: 0:16.80
out.rpm.2: 0:16.83
out.rpm.3: 0:16.69
out.rpm.4: 0:16.78
out.rpm.5: 0:17.78
That is not supportive of my statement of ipk being the slowest package manager.
That statement come from timing tests done with our proprietary version of poky
that is somewhat extended when it comes to the do_rootfs task. We're generating
a bit more artifacts than poky. However apk is still ~22% faster than rpm and
~40%
faster than deb according to the data above.
So I tried your script with core-image-sato-sdk-ptest instead that is a bigger
image.
The results where more mixed. There is a huge difference between different
package
managers:
out.apk.1: 1:13.35
out.apk.2: 1:13.51
out.apk.3: 1:13.23
out.apk.4: 1:14.07
out.apk.5: 1:13.00
out.deb.1: 3:49.37
out.deb.2: 3:50.77
out.deb.3: 3:51.39
out.deb.4: 3:53.40
out.deb.5: 3:53.99
out.ipk.1: 2:38.99
out.ipk.2: 2:39.07
out.ipk.3: 2:35.34
out.ipk.4: 2:36.15
out.ipk.5: 2:34.55
out.rpm.1: 1:58.61
out.rpm.2: 1:59.42
out.rpm.3: 1:59.70
out.rpm.4: 1:58.96
out.rpm.5: 1:58.11
But yeah, I should probably rephrase my cover letter. You'll find a layer for
APK that will
apply on top of my patch serie attached to this email.
Also note that there's a difference between the time to generate a package.
Which
we completely ignore in this measurements. It's not as important since it's
being
cached, but for a fresh build it will be a difference between different package
managers.
BR
Fredrik
___
From: openembedded-core@lists.openembedded.org
on behalf of Alex Stewart
Sent: Wednesday, June 24, 2020 5:18 PM
To: Fredrik Gustafsson; openembedded-core@lists.openembedded.org
Cc: tools-cfpbuild-internal; Hugo Cedervall; Alejandro Del Castillo
Subject: Re: [OE-core] Add package managers as a plugin
On 6/23/20 6:13 AM, Fredrik Gustafsson wrote:
> When building and having a good cache hit, a significant amount of time is
> spent in the phase of generating a rootfs, which is really about the
> performance of the package manager. ipkg is way slower than deb or rpm.
I was interested enough in your comment here to collect a small sampling
of comparative do_rootfs runs on one of my dev machines.
I used a virgin poky clone and tried to replicate the steps you provided
in your OP, timing only the execution of `bitbake -c do_rootfs
core-image-minimal`. I used the debian `time` package for timing. Here's
what I got:
# Summary
PM Avg do_rootfs time (s)
-- --
deb33.89
ipk33.35
rpm31.85
# Raw Data
deb00:33.54
deb00:34.66
deb00:33.55
deb00:33.23
deb00:34.46
ipk00:33.96
ipk00:34.43
ipk00:33.99
ipk00:32.17
ipk00:32.22
rpm00:29.92
rpm00:32.31
rpm00:31.90
rpm00:33.31
rpm00:31.80
Based on those results, it seems like all the current PMs are similar in
terms of execution time. Are there major differences between my test
setup and yours? And do you already have a layer with APK available that
I could compare against on my hardware?
This is the crude script I used to automate the test runs:
```
#!/bin/bash
set -euxo pipefail
TIME=/usr/bin/time
recipe=core-image-minimal
# Options are:
# - 'package_deb' for debian style deb files
# - 'package_ipk' for ipk files are used by opkg (a debian style
embedded package manager)
# - 'package_rpm' for rpm style packages
function set_pkg_man() {
pkg_man=${1}
sed -i "s/\(^PACKAGE_CLASSES ?= \)\"\(.*\)\"/\1\"$pkg_man\"/"
conf/local.conf
}
function rebuild_recipe() {
bitbake -c cleanall $recipe
bitbake $recipe
}
function test_recipe() {
local name=${1}
bitbake -c cleansstate $recipe
bitbake -c clean $recipe
$TIME bitbake -c do_rootfs -f $recipe 2>&1 | tee outs/out.$name
}
# rpm
set_pkg_man package_rpm
rebuild_recipe
test_recipe rpm.1
test_recipe rpm.2
test_recipe rpm.3
test_recipe rpm.4
test_recipe rpm.5
# ipk
set_pkg_man package_ipk
rebuild_recipe
test_recipe ipk.1
test_recipe ipk.2
test_recipe ipk.3
test_recipe ipk.4
test_recipe ipk.5
# deb
set_pkg_man package_deb
rebuild_recipe
test_recipe deb.1
test_recipe deb.2
test_recipe deb.3
test_recipe deb.4
test_recipe deb.5
```
Thanks,
--
Alex Stewart
Software Engineer - LabVIEW Real-Time OS
National Instruments
alex.stew...@ni.com
office: +1(512)683-8522
meta-apk.tar.gz
Description: meta-apk.tar.gz
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages
Re: [OE-core] Add package managers as a plugin
What I meant is that 15s vs 20s on a small image could have been 5 min vs 5min5s on a bigger image, so I wanted to see that the difference does scale up with image size. Can you to provide sdk-ptest numbers for deb and ipk as well? Alex On Tue, 30 Jun 2020 at 17:04, Fredrik Gustafsson < fredrik.gustafs...@axis.com> wrote: > Hi Alexander, > > what are you finding strange with the numbers? Too fast or to big of a > difference? Note that it's only the do_rootfs task that i measured. > > I've tried the core-image-sato-sdk-ptest as you suggested. I run the > following command multiple times: > bitbake core-image-sato-sdk-ptest -c cleansstate && > > bitbake core-image-sato-sdk-ptest -c clean && > time bitbake core-image-sato-sdk-ptest -f -c do_rootfs > > > APK > > 0,85s user 0,16s system 0% cpu 1:13,66 total > 0,87s user 0,14s system 0% cpu 1:13,28 total > 0,89s user 0,11s system 0% cpu 1:13,10 total > > RPM > === > > 0,67s user 0,12s system 0% cpu 2:02,48 total > 0,68s user 0,11s system 0% cpu 2:03,39 total > 0,73s user 0,09s system 0% cpu 1:59,87 total > > 5 seconds doesn't sound that much but it's 25% speedup. Since building is > mostly cached but generating rootfs isn't this is a huge time consumption > for our use case and our image takes around 4 minutes to build. Saving one > minutes times the many builds we do is a huge win. > > (Excuse my late reply, running core-image-sato-sdk-ptest uncovered a few > bugs in the implementation of apk in OE. So thanks for having me testing > this image) > > > /Fredrik > > > -- > *From:* Alexander Kanavin > *Sent:* Thursday, June 25, 2020 12:24 PM > *To:* Fredrik Gustafsson > *Cc:* OE-core; tools-cfpbuild-internal; Hugo Cedervall > *Subject:* Re: [OE-core] Add package managers as a plugin > > Apologies, but these metrics do not look particularly convincing - 15s vs > 20s? Can you try it with an image several gigabytes in size? > Core-image-sato-sdk-ptest is a good option. > > Alex > > On Thu, 25 Jun 2020 at 12:14, Fredrik Gustafsson < > fredrik.gustafs...@axis.com> wrote: > >> Poky today has three different package managers, the well-known formats >> deb >> and rpm is supported as well as ipkg that is good for embedded devices. >> >> When building and having a good cache hit, a significant amount of time is >> spent in the phase of generating a rootfs, which is really about the >> performance of the package manager. ipkg is way slower than deb or rpm. To >> save build time and also get a package manager that is suitanle for use on >> targets where flash memory is a concern, support for apk is suggested. >> >> However, it might or might not be what's wanted for Poky since it >> increases >> the test matrix. Therefore this patch series refactors the package >> management code so that it's possible to add more package managers in >> other >> own layer. I will send another patch serie that will add apk. >> >> Perfomance metrics below, (note that this includes build times). >> >> APK >> === >> bitbake core-image-minimal >> 15.84s user 2.60s system 0% cpu 1:26:19.21 total >> 16.01s user 2.58s system 0% cpu 1:26:03.72 total >> 15.69s user 2.61s system 0% cpu 1:26:45.45 total >> >> bitbake core-image-minimal -c cleansstate >> bitbake core-image-minimal -c clean >> bitbake core-image-minimal -f -c do_rootfs >> 0.55s user 0.06s system 4% cpu 14.236 total >> 0.54s user 0.08s system 4% cpu 15.247 total >> 0.52s user 0.15s system 4% cpu 15.143 total >> >> RPM >> === >> bitbake core-image-minimal >> 18.57s user 3.09s system 0% cpu 1:31:29.09 total >> 18.58s user 3.08s system 0% cpu 1:30:53.80 total >> 18.20s user 3.31s system 0% cpu 1:31:06.69 total >> >> bitbake core-image-minimal -c cleansstate >> bitbake core-image-minimal -c clean >> bitbake core-image-minimal -c do_rootfs -f >> 0.58s user 0.10s system 3% cpu 19.470 total >> 0.57s user 0.09s system 3% cpu 19.480 total >> 0.60s user 0.07s system 3% cpu 20.381 total >> >> > -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#140139): https://lists.openembedded.org/g/openembedded-core/message/140139 Mute This Topic: https://lists.openembedded.org/mt/75057633/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] Add package managers as a plugin
On 6/26/20 3:52 AM, Ross Burton wrote: On Thu, 25 Jun 2020 at 11:14, Fredrik Gustafsson wrote: Poky today has three different package managers, the well-known formats deb and rpm is supported as well as ipkg that is good for embedded devices. When building and having a good cache hit, a significant amount of time is spent in the phase of generating a rootfs, which is really about the performance of the package manager. ipkg is way slower than deb or rpm. To save build time and also get a package manager that is suitanle for use on targets where flash memory is a concern, support for apk is suggested. However, it might or might not be what's wanted for Poky since it increases the test matrix. Therefore this patch series refactors the package management code so that it's possible to add more package managers in other own layer. I will send another patch serie that will add apk. As to the code review I mostly second what Paul said. However, obviously you actually have strong feelings on this as the series represents a fair amount of work. What are the compelling reasons to use apk? Five seconds on core-image-minimal doesn't really sell it to me, I don't think anyone ever claimed that opkg has been performance optimised, dpkg isn't really used that much: maybe we should re-evaluate what package managers are supported if apk has feature parity but is significantly faster or has other advantages. 25% is a lot, real world images are way more complex than reference images in reference distribution, we have images which use 7 to 10 mins in rootfs, I have seen a desire by many developers in optimizing build time so this could be a real help, especially where we dont care about package management at runtime, we have done many micro optimizations and rejected patches since they added to build time because we want to be sensitive for increasing build time. rootfs is one task where sstate caching does not help as well, while other package builds we can benefit from sstate, rootfs not so much hence such an optimizations, if we test it regularly perhaps can be debated, but it certainly seems interesting option to have. Ross -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#140138): https://lists.openembedded.org/g/openembedded-core/message/140138 Mute This Topic: https://lists.openembedded.org/mt/75057633/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] Add package managers as a plugin
Hi Denys, you're right. I was confusing poky and OE-core. I'll change the cover letter. Thanks! I haven't tested it with other distros but I see no reason that it wouldn't work. BR Fredrik From: Denys Dmytriyenko Sent: Tuesday, June 23, 2020 8:50 PM To: Fredrik Gustafsson Cc: openembedded-core@lists.openembedded.org; tools-cfpbuild-internal; Hugo Cedervall Subject: Re: [OE-core] Add package managers as a plugin You mentioned Poky few times below - why is it Poky-specific? Since you are not submitting this change to Poky, but to OE-Core instead, can this be used by other OE distros? -- Denys On Tue, Jun 23, 2020 at 01:13:26PM +0200, Fredrik Gustafsson wrote: > Poky today has three different package managers, the well-known formats deb > and rpm is supported as well as ipkg that is good for embedded devices. > > When building and having a good cache hit, a significant amount of time is > spent in the phase of generating a rootfs, which is really about the > performance of the package manager. ipkg is way slower than deb or rpm. To > save build time and also get a package manager that is suitanle for use on > targets where flash memory is a concern, support for apk is suggested. > > However, it might or might not be what's wanted for Poky since it increases > the test matrix. Therefore this patch series refactors the package > management code so that it's possible to add more package managers in other > own layer. I will send another patch serie that will add apk. > > Perfomance metrics below, (note that this includes build times). > > APK > === > bitbake core-image-minimal > 15.84s user 2.60s system 0% cpu 1:26:19.21 total > 16.01s user 2.58s system 0% cpu 1:26:03.72 total > 15.69s user 2.61s system 0% cpu 1:26:45.45 total > > bitbake core-image-minimal -c cleansstate > bitbake core-image-minimal -c clean > bitbake core-image-minimal -f -c do_rootfs > 0.55s user 0.06s system 4% cpu 14.236 total > 0.54s user 0.08s system 4% cpu 15.247 total > 0.52s user 0.15s system 4% cpu 15.143 total > > RPM > === > bitbake core-image-minimal > 18.57s user 3.09s system 0% cpu 1:31:29.09 total > 18.58s user 3.08s system 0% cpu 1:30:53.80 total > 18.20s user 3.31s system 0% cpu 1:31:06.69 total > > bitbake core-image-minimal -c cleansstate > bitbake core-image-minimal -c clean > bitbake core-image-minimal -c do_rootfs -f > 0.58s user 0.10s system 3% cpu 19.470 total > 0.57s user 0.09s system 3% cpu 19.480 total > 0.60s user 0.07s system 3% cpu 20.381 total > -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#140137): https://lists.openembedded.org/g/openembedded-core/message/140137 Mute This Topic: https://lists.openembedded.org/mt/75057633/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] Add package managers as a plugin
Hi Alexander, what are you finding strange with the numbers? Too fast or to big of a difference? Note that it's only the do_rootfs task that i measured. I've tried the core-image-sato-sdk-ptest as you suggested. I run the following command multiple times: bitbake core-image-sato-sdk-ptest -c cleansstate && bitbake core-image-sato-sdk-ptest -c clean && time bitbake core-image-sato-sdk-ptest -f -c do_rootfs APK 0,85s user 0,16s system 0% cpu 1:13,66 total 0,87s user 0,14s system 0% cpu 1:13,28 total 0,89s user 0,11s system 0% cpu 1:13,10 total RPM === 0,67s user 0,12s system 0% cpu 2:02,48 total 0,68s user 0,11s system 0% cpu 2:03,39 total 0,73s user 0,09s system 0% cpu 1:59,87 total 5 seconds doesn't sound that much but it's 25% speedup. Since building is mostly cached but generating rootfs isn't this is a huge time consumption for our use case and our image takes around 4 minutes to build. Saving one minutes times the many builds we do is a huge win. (Excuse my late reply, running core-image-sato-sdk-ptest uncovered a few bugs in the implementation of apk in OE. So thanks for having me testing this image) /Fredrik From: Alexander Kanavin Sent: Thursday, June 25, 2020 12:24 PM To: Fredrik Gustafsson Cc: OE-core; tools-cfpbuild-internal; Hugo Cedervall Subject: Re: [OE-core] Add package managers as a plugin Apologies, but these metrics do not look particularly convincing - 15s vs 20s? Can you try it with an image several gigabytes in size? Core-image-sato-sdk-ptest is a good option. Alex On Thu, 25 Jun 2020 at 12:14, Fredrik Gustafsson mailto:fredrik.gustafs...@axis.com>> wrote: Poky today has three different package managers, the well-known formats deb and rpm is supported as well as ipkg that is good for embedded devices. When building and having a good cache hit, a significant amount of time is spent in the phase of generating a rootfs, which is really about the performance of the package manager. ipkg is way slower than deb or rpm. To save build time and also get a package manager that is suitanle for use on targets where flash memory is a concern, support for apk is suggested. However, it might or might not be what's wanted for Poky since it increases the test matrix. Therefore this patch series refactors the package management code so that it's possible to add more package managers in other own layer. I will send another patch serie that will add apk. Perfomance metrics below, (note that this includes build times). APK === bitbake core-image-minimal 15.84s user 2.60s system 0% cpu 1:26:19.21 total 16.01s user 2.58s system 0% cpu 1:26:03.72 total 15.69s user 2.61s system 0% cpu 1:26:45.45 total bitbake core-image-minimal -c cleansstate bitbake core-image-minimal -c clean bitbake core-image-minimal -f -c do_rootfs 0.55s user 0.06s system 4% cpu 14.236 total 0.54s user 0.08s system 4% cpu 15.247 total 0.52s user 0.15s system 4% cpu 15.143 total RPM === bitbake core-image-minimal 18.57s user 3.09s system 0% cpu 1:31:29.09 total 18.58s user 3.08s system 0% cpu 1:30:53.80 total 18.20s user 3.31s system 0% cpu 1:31:06.69 total bitbake core-image-minimal -c cleansstate bitbake core-image-minimal -c clean bitbake core-image-minimal -c do_rootfs -f 0.58s user 0.10s system 3% cpu 19.470 total 0.57s user 0.09s system 3% cpu 19.480 total 0.60s user 0.07s system 3% cpu 20.381 total -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#140136): https://lists.openembedded.org/g/openembedded-core/message/140136 Mute This Topic: https://lists.openembedded.org/mt/75057633/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] Add package managers as a plugin
On Fri, Jun 26, 2020 at 3:53 AM Ross Burton wrote: > On Thu, 25 Jun 2020 at 11:14, Fredrik Gustafsson > wrote: > > Poky today has three different package managers, the well-known formats > deb > > and rpm is supported as well as ipkg that is good for embedded devices. > > > > When building and having a good cache hit, a significant amount of time > is > > spent in the phase of generating a rootfs, which is really about the > > performance of the package manager. ipkg is way slower than deb or rpm. > To > > save build time and also get a package manager that is suitanle for use > on > > targets where flash memory is a concern, support for apk is suggested. > > > > However, it might or might not be what's wanted for Poky since it > increases > > the test matrix. Therefore this patch series refactors the package > > management code so that it's possible to add more package managers in > other > > own layer. I will send another patch serie that will add apk. > > As to the code review I mostly second what Paul said. > > However, obviously you actually have strong feelings on this as the > series represents a fair amount of work. What are the compelling > reasons to use apk? Five seconds on core-image-minimal doesn't really > sell it to me, I don't think anyone ever claimed that opkg has been > performance optimised, dpkg isn't really used that much: maybe we > should re-evaluate what package managers are supported if apk has > feature parity but is significantly faster or has other advantages. > FWIW, I was contemplating apk packaging because of heavy usage in Docker containers based on Alpine Linux. I haven’t reviewed the patch series, but the theme of adding apk as another viable package type is welcome. > Ross > > -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#140038): https://lists.openembedded.org/g/openembedded-core/message/140038 Mute This Topic: https://lists.openembedded.org/mt/75057633/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] Add package managers as a plugin
On Thu, 25 Jun 2020 at 11:14, Fredrik Gustafsson wrote: > Poky today has three different package managers, the well-known formats deb > and rpm is supported as well as ipkg that is good for embedded devices. > > When building and having a good cache hit, a significant amount of time is > spent in the phase of generating a rootfs, which is really about the > performance of the package manager. ipkg is way slower than deb or rpm. To > save build time and also get a package manager that is suitanle for use on > targets where flash memory is a concern, support for apk is suggested. > > However, it might or might not be what's wanted for Poky since it increases > the test matrix. Therefore this patch series refactors the package > management code so that it's possible to add more package managers in other > own layer. I will send another patch serie that will add apk. As to the code review I mostly second what Paul said. However, obviously you actually have strong feelings on this as the series represents a fair amount of work. What are the compelling reasons to use apk? Five seconds on core-image-minimal doesn't really sell it to me, I don't think anyone ever claimed that opkg has been performance optimised, dpkg isn't really used that much: maybe we should re-evaluate what package managers are supported if apk has feature parity but is significantly faster or has other advantages. Ross -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#140010): https://lists.openembedded.org/g/openembedded-core/message/140010 Mute This Topic: https://lists.openembedded.org/mt/75057633/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] Add package managers as a plugin
On Thu, 25 Jun 2020 at 11:21, Fredrik Gustafsson wrote: > > Poky today has three different package managers, the well-known formats deb > and rpm is supported as well as ipkg that is good for embedded devices. > > When building and having a good cache hit, a significant amount of time is > spent in the phase of generating a rootfs, which is really about the > performance of the package manager. ipkg is way slower than deb or rpm. To > save build time and also get a package manager that is suitanle for use on > targets where flash memory is a concern, support for apk is suggested. > > However, it might or might not be what's wanted for Poky since it increases > the test matrix. Therefore this patch series refactors the package > management code so that it's possible to add more package managers in other > own layer. I will send another patch serie that will add apk. > > Perfomance metrics below, (note that this includes build times). > > APK > === > bitbake core-image-minimal > 15.84s user 2.60s system 0% cpu 1:26:19.21 total > 16.01s user 2.58s system 0% cpu 1:26:03.72 total > 15.69s user 2.61s system 0% cpu 1:26:45.45 total > > bitbake core-image-minimal -c cleansstate > bitbake core-image-minimal -c clean > bitbake core-image-minimal -f -c do_rootfs > 0.55s user 0.06s system 4% cpu 14.236 total > 0.54s user 0.08s system 4% cpu 15.247 total > 0.52s user 0.15s system 4% cpu 15.143 total > > RPM > === > bitbake core-image-minimal > 18.57s user 3.09s system 0% cpu 1:31:29.09 total > 18.58s user 3.08s system 0% cpu 1:30:53.80 total > 18.20s user 3.31s system 0% cpu 1:31:06.69 total > > bitbake core-image-minimal -c cleansstate > bitbake core-image-minimal -c clean > bitbake core-image-minimal -c do_rootfs -f > 0.58s user 0.10s system 3% cpu 19.470 total > 0.57s user 0.09s system 3% cpu 19.480 total > 0.60s user 0.07s system 3% cpu 20.381 total Some initial review comments incoming on all these patches. I've just done a surface level reading, I think the way you're submitting this still needs some work before it's in a state where we can properly review the changes in detail. Right now it's too big and too easy to get confused when looking through 26 large patches. I'm very supportive of tidying this up and making it more dynamic, we just need to make sure we don't break things along the way. Thanks, -- Paul Barker Konsulko Group -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#139938): https://lists.openembedded.org/g/openembedded-core/message/139938 Mute This Topic: https://lists.openembedded.org/mt/75057633/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] Add package managers as a plugin
Apologies, but these metrics do not look particularly convincing - 15s vs 20s? Can you try it with an image several gigabytes in size? Core-image-sato-sdk-ptest is a good option. Alex On Thu, 25 Jun 2020 at 12:14, Fredrik Gustafsson < fredrik.gustafs...@axis.com> wrote: > Poky today has three different package managers, the well-known formats deb > and rpm is supported as well as ipkg that is good for embedded devices. > > When building and having a good cache hit, a significant amount of time is > spent in the phase of generating a rootfs, which is really about the > performance of the package manager. ipkg is way slower than deb or rpm. To > save build time and also get a package manager that is suitanle for use on > targets where flash memory is a concern, support for apk is suggested. > > However, it might or might not be what's wanted for Poky since it increases > the test matrix. Therefore this patch series refactors the package > management code so that it's possible to add more package managers in other > own layer. I will send another patch serie that will add apk. > > Perfomance metrics below, (note that this includes build times). > > APK > === > bitbake core-image-minimal > 15.84s user 2.60s system 0% cpu 1:26:19.21 total > 16.01s user 2.58s system 0% cpu 1:26:03.72 total > 15.69s user 2.61s system 0% cpu 1:26:45.45 total > > bitbake core-image-minimal -c cleansstate > bitbake core-image-minimal -c clean > bitbake core-image-minimal -f -c do_rootfs > 0.55s user 0.06s system 4% cpu 14.236 total > 0.54s user 0.08s system 4% cpu 15.247 total > 0.52s user 0.15s system 4% cpu 15.143 total > > RPM > === > bitbake core-image-minimal > 18.57s user 3.09s system 0% cpu 1:31:29.09 total > 18.58s user 3.08s system 0% cpu 1:30:53.80 total > 18.20s user 3.31s system 0% cpu 1:31:06.69 total > > bitbake core-image-minimal -c cleansstate > bitbake core-image-minimal -c clean > bitbake core-image-minimal -c do_rootfs -f > 0.58s user 0.10s system 3% cpu 19.470 total > 0.57s user 0.09s system 3% cpu 19.480 total > 0.60s user 0.07s system 3% cpu 20.381 total > > -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#139937): https://lists.openembedded.org/g/openembedded-core/message/139937 Mute This Topic: https://lists.openembedded.org/mt/75057633/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] Add package managers as a plugin
On 6/23/20 6:13 AM, Fredrik Gustafsson wrote:
When building and having a good cache hit, a significant amount of time is
spent in the phase of generating a rootfs, which is really about the
performance of the package manager. ipkg is way slower than deb or rpm.
I was interested enough in your comment here to collect a small sampling
of comparative do_rootfs runs on one of my dev machines.
I used a virgin poky clone and tried to replicate the steps you provided
in your OP, timing only the execution of `bitbake -c do_rootfs
core-image-minimal`. I used the debian `time` package for timing. Here's
what I got:
# Summary
PM Avg do_rootfs time (s)
-- --
deb 33.89
ipk 33.35
rpm 31.85
# Raw Data
deb 00:33.54
deb 00:34.66
deb 00:33.55
deb 00:33.23
deb 00:34.46
ipk 00:33.96
ipk 00:34.43
ipk 00:33.99
ipk 00:32.17
ipk 00:32.22
rpm 00:29.92
rpm 00:32.31
rpm 00:31.90
rpm 00:33.31
rpm 00:31.80
Based on those results, it seems like all the current PMs are similar in
terms of execution time. Are there major differences between my test
setup and yours? And do you already have a layer with APK available that
I could compare against on my hardware?
This is the crude script I used to automate the test runs:
```
#!/bin/bash
set -euxo pipefail
TIME=/usr/bin/time
recipe=core-image-minimal
# Options are:
# - 'package_deb' for debian style deb files
# - 'package_ipk' for ipk files are used by opkg (a debian style
embedded package manager)
# - 'package_rpm' for rpm style packages
function set_pkg_man() {
pkg_man=${1}
sed -i "s/\(^PACKAGE_CLASSES ?= \)\"\(.*\)\"/\1\"$pkg_man\"/"
conf/local.conf
}
function rebuild_recipe() {
bitbake -c cleanall $recipe
bitbake $recipe
}
function test_recipe() {
local name=${1}
bitbake -c cleansstate $recipe
bitbake -c clean $recipe
$TIME bitbake -c do_rootfs -f $recipe 2>&1 | tee outs/out.$name
}
# rpm
set_pkg_man package_rpm
rebuild_recipe
test_recipe rpm.1
test_recipe rpm.2
test_recipe rpm.3
test_recipe rpm.4
test_recipe rpm.5
# ipk
set_pkg_man package_ipk
rebuild_recipe
test_recipe ipk.1
test_recipe ipk.2
test_recipe ipk.3
test_recipe ipk.4
test_recipe ipk.5
# deb
set_pkg_man package_deb
rebuild_recipe
test_recipe deb.1
test_recipe deb.2
test_recipe deb.3
test_recipe deb.4
test_recipe deb.5
```
Thanks,
--
Alex Stewart
Software Engineer - LabVIEW Real-Time OS
National Instruments
alex.stew...@ni.com
office: +1(512)683-8522
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#139891):
https://lists.openembedded.org/g/openembedded-core/message/139891
Mute This Topic: https://lists.openembedded.org/mt/75057633/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] Add package managers as a plugin
You mentioned Poky few times below - why is it Poky-specific? Since you are not submitting this change to Poky, but to OE-Core instead, can this be used by other OE distros? -- Denys On Tue, Jun 23, 2020 at 01:13:26PM +0200, Fredrik Gustafsson wrote: > Poky today has three different package managers, the well-known formats deb > and rpm is supported as well as ipkg that is good for embedded devices. > > When building and having a good cache hit, a significant amount of time is > spent in the phase of generating a rootfs, which is really about the > performance of the package manager. ipkg is way slower than deb or rpm. To > save build time and also get a package manager that is suitanle for use on > targets where flash memory is a concern, support for apk is suggested. > > However, it might or might not be what's wanted for Poky since it increases > the test matrix. Therefore this patch series refactors the package > management code so that it's possible to add more package managers in other > own layer. I will send another patch serie that will add apk. > > Perfomance metrics below, (note that this includes build times). > > APK > === > bitbake core-image-minimal > 15.84s user 2.60s system 0% cpu 1:26:19.21 total > 16.01s user 2.58s system 0% cpu 1:26:03.72 total > 15.69s user 2.61s system 0% cpu 1:26:45.45 total > > bitbake core-image-minimal -c cleansstate > bitbake core-image-minimal -c clean > bitbake core-image-minimal -f -c do_rootfs > 0.55s user 0.06s system 4% cpu 14.236 total > 0.54s user 0.08s system 4% cpu 15.247 total > 0.52s user 0.15s system 4% cpu 15.143 total > > RPM > === > bitbake core-image-minimal > 18.57s user 3.09s system 0% cpu 1:31:29.09 total > 18.58s user 3.08s system 0% cpu 1:30:53.80 total > 18.20s user 3.31s system 0% cpu 1:31:06.69 total > > bitbake core-image-minimal -c cleansstate > bitbake core-image-minimal -c clean > bitbake core-image-minimal -c do_rootfs -f > 0.58s user 0.10s system 3% cpu 19.470 total > 0.57s user 0.09s system 3% cpu 19.480 total > 0.60s user 0.07s system 3% cpu 20.381 total > -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#139846): https://lists.openembedded.org/g/openembedded-core/message/139846 Mute This Topic: https://lists.openembedded.org/mt/75057633/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
