Re: [oe] Simultaneous build of two intree packages
> Fixing the packages, is probably the way to go. > You could also create pseudo package "D" which is available in several > versions. > > Package D-1.0c: Package D is depending on nothing > Package D-1.0bc: Package D is depending on B > > Package C is always dependent on Package D and A. > Package B is always dependent on Package A > > If you set PREFERRED_VERSION_D = "1.0c" > C is depending on D, which is depending on nothing, and C is depending on A. > No problem, A and C is built. > > If you set PREFERRED_VERSION_D = "1.0c" > C is depending on D, which is depending on B. B and C is depending on A. > A is built, then B is built, and then D finalizing with C. > > If you only want B, don't include C in the image. > > Not tested, but it looks like it would work. Thank you for very inspirational suggestion. This may actually work. I was already considering using distro features for defining which recipes should be included and than making chain of dependency based on them. Also as Ciprian suggested putting everything into one recipe and building multiple packages may be the way to go. Thank you both for good suggestions! Radek ___ Openembedded-devel mailing list Openembedded-devel@lists.openembedded.org http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel
Re: [oe] Simultaneous build of two intree packages
Not sure if it applies, but if all build dirs share the same configuration step, you could create a single recipe and split the results into more packages. On Mon, 2012-05-07 at 23:23 +0200, Radek Dostal wrote: > On Mon, 2012-05-07 at 15:08 -0600, Gary Thomas wrote: > > I think that your recipes are not well behaved - two recipes probably > > should not be creating the same file in such a way. > > > > Can you post the recipes, or at least the code snippets that break? > > I agree usually it should not be a case, but since both of the recipes > are in-tree recipes within the same tree, than obviously tree is the > shared resource. > > Pasting the recipes would probably not be very helpful. Only thing they > do is that they call the cmake bellow and actual error occurs within > cmake scripts. > > Just in order to give overview: First whole structure is configured. > Configuration is shared between all the recipes. Second actual make is > called on different subdirectories. Cmake in its nature is not designed > to be able to started multiple time over same tread (with exception of > starting it via -jX). > > Thanks, > Radek > > > ___ > Openembedded-devel mailing list > Openembedded-devel@lists.openembedded.org > http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel ___ Openembedded-devel mailing list Openembedded-devel@lists.openembedded.org http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel
Re: [oe] Simultaneous build of two intree packages
On 2012-05-07 14:33, Radek Dostal wrote: On Mon, 2012-05-07 at 06:02 -0600, Gary Thomas wrote: On 2012-05-07 05:54, Radek Dostal wrote: Dear All, I have three intree recipes A, B and C which are part of the same layer. => B depends on A => C depends on A. There is no dependency between B and C. Only important thing is that B and C are not build simultaneously. This is not a problem unless I use configure BB_NUMBER_THREADS> 1. Than builds B and C starts to collide and produce mess because they are both intree builds. What are you doing that makes these clash? usually it crashes during "ln -sf target dest" which is executed simultaneously with same parameters for both builds. There are other issues as well in addition to this ln issue. Is there something like a mutex/lock mechanism in openembedded-core, which would forbid two recipes to be build simultaneously? Just make C depend on B I have already considered this, but unfortunately I also have configurations where only B is included or where only C is included :( Thanks, Radek ___ Openembedded-devel mailing list Openembedded-devel@lists.openembedded.org http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel Fixing the packages, is probably the way to go. You could also create pseudo package "D" which is available in several versions. Package D-1.0c: Package D is depending on nothing Package D-1.0bc:Package D is depending on B Package C is always dependent on Package D and A. Package B is always dependent on Package A If you set PREFERRED_VERSION_D = "1.0c" C is depending on D, which is depending on nothing, and C is depending on A. No problem, A and C is built. If you set PREFERRED_VERSION_D = "1.0c" C is depending on D, which is depending on B. B and C is depending on A. A is built, then B is built, and then D finalizing with C. If you only want B, don't include C in the image. Not tested, but it looks like it would work. -- Best Regards Ulf Samuelsson u...@emagii.com +46 722 427437 ___ Openembedded-devel mailing list Openembedded-devel@lists.openembedded.org http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel
Re: [oe] Simultaneous build of two intree packages
On Mon, May 7, 2012 at 2:01 PM, Radek Dostal wrote: > Because ln -sf first do symlink(1) unlink(2) and symlink(3) again. In > case link is created by another process between 2 and 3 than ln fails. > > It is quite well described here: > http://rcrowley.org/2010/01/06/things-unix-can-do-atomically.html > Do you think I don't know how ln works ? all I was asking why are same symlinks being created by two different packages that seems broken to me. Fix the packages ___ Openembedded-devel mailing list Openembedded-devel@lists.openembedded.org http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel
Re: [oe] Simultaneous build of two intree packages
On Mon, May 7, 2012 at 2:01 PM, Radek Dostal wrote: > Because ln -sf first do symlink(1) unlink(2) and symlink(3) again. In > case link is created by another process between 2 and 3 than ln fails. > > It is quite well described here: > http://rcrowley.org/2010/01/06/things-unix-can-do-atomically.html > Do you think I don't know how ln works ? all I was asking why are same symlinks being created by two different packages that seems broken to me. Fix the packages ___ Openembedded-devel mailing list Openembedded-devel@lists.openembedded.org http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel
Re: [oe] Simultaneous build of two intree packages
On Mon, 2012-05-07 at 15:08 -0600, Gary Thomas wrote: > I think that your recipes are not well behaved - two recipes probably > should not be creating the same file in such a way. > > Can you post the recipes, or at least the code snippets that break? I agree usually it should not be a case, but since both of the recipes are in-tree recipes within the same tree, than obviously tree is the shared resource. Pasting the recipes would probably not be very helpful. Only thing they do is that they call the cmake bellow and actual error occurs within cmake scripts. Just in order to give overview: First whole structure is configured. Configuration is shared between all the recipes. Second actual make is called on different subdirectories. Cmake in its nature is not designed to be able to started multiple time over same tread (with exception of starting it via -jX). Thanks, Radek ___ Openembedded-devel mailing list Openembedded-devel@lists.openembedded.org http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel
Re: [oe] Simultaneous build of two intree packages
On 2012-05-07 15:01, Radek Dostal wrote: On Mon, 2012-05-07 at 08:46 -0700, Khem Raj wrote: usually it crashes during "ln -sf target dest" which is executed simultaneously with same parameters for both builds. There are other issues as well in addition to this ln issue. This is confusing why does this happen at all Because ln -sf first do symlink(1) unlink(2) and symlink(3) again. In case link is created by another process between 2 and 3 than ln fails. It is quite well described here: http://rcrowley.org/2010/01/06/things-unix-can-do-atomically.html I was also able to verify my finding by running two ln -sf in two for loops in two different terminals and they were failing constantly. So any idea on implementing locks/mutexes inside OE. If somebody can point me in the right direction I should be able to implement and feed back. I think that your recipes are not well behaved - two recipes probably should not be creating the same file in such a way. Can you post the recipes, or at least the code snippets that break? -- Gary Thomas | Consulting for the MLB Associates |Embedded world ___ Openembedded-devel mailing list Openembedded-devel@lists.openembedded.org http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel
Re: [oe] Simultaneous build of two intree packages
On Mon, 2012-05-07 at 08:46 -0700, Khem Raj wrote: > > usually it crashes during "ln -sf target dest" which is executed > > simultaneously with same parameters for both builds. There are other > > issues as well in addition to this ln issue. > > This is confusing why does this happen at all Because ln -sf first do symlink(1) unlink(2) and symlink(3) again. In case link is created by another process between 2 and 3 than ln fails. It is quite well described here: http://rcrowley.org/2010/01/06/things-unix-can-do-atomically.html I was also able to verify my finding by running two ln -sf in two for loops in two different terminals and they were failing constantly. So any idea on implementing locks/mutexes inside OE. If somebody can point me in the right direction I should be able to implement and feed back. Thanks, Radek ___ Openembedded-devel mailing list Openembedded-devel@lists.openembedded.org http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel
Re: [oe] Simultaneous build of two intree packages
On Monday, May 7, 2012, Radek Dostal wrote: > On Mon, 2012-05-07 at 06:02 -0600, Gary Thomas wrote: >> On 2012-05-07 05:54, Radek Dostal wrote: >> > Dear All, >> > >> > I have three intree recipes A, B and C which are part of the same layer. >> > => B depends on A >> > => C depends on A. >> > >> > There is no dependency between B and C. Only important thing is that B >> > and C are not build simultaneously. This is not a problem unless I use >> > configure BB_NUMBER_THREADS> 1. Than builds B and C starts to collide >> > and produce mess because they are both intree builds. >> >> What are you doing that makes these clash? > > usually it crashes during "ln -sf target dest" which is executed > simultaneously with same parameters for both builds. There are other > issues as well in addition to this ln issue. This is confusing why does this happen at all > >> > >> > Is there something like a mutex/lock mechanism in openembedded-core, >> > which would forbid two recipes to be build simultaneously? >> >> Just make C depend on B > > I have already considered this, but unfortunately I also have > configurations where only B is included or where only C is included :( > > Thanks, > Radek > > > ___ > Openembedded-devel mailing list > Openembedded-devel@lists.openembedded.org > http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel > ___ Openembedded-devel mailing list Openembedded-devel@lists.openembedded.org http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel
Re: [oe] Simultaneous build of two intree packages
On Mon, 2012-05-07 at 06:02 -0600, Gary Thomas wrote: > On 2012-05-07 05:54, Radek Dostal wrote: > > Dear All, > > > > I have three intree recipes A, B and C which are part of the same layer. > > => B depends on A > > => C depends on A. > > > > There is no dependency between B and C. Only important thing is that B > > and C are not build simultaneously. This is not a problem unless I use > > configure BB_NUMBER_THREADS> 1. Than builds B and C starts to collide > > and produce mess because they are both intree builds. > > What are you doing that makes these clash? usually it crashes during "ln -sf target dest" which is executed simultaneously with same parameters for both builds. There are other issues as well in addition to this ln issue. > > > > Is there something like a mutex/lock mechanism in openembedded-core, > > which would forbid two recipes to be build simultaneously? > > Just make C depend on B I have already considered this, but unfortunately I also have configurations where only B is included or where only C is included :( Thanks, Radek ___ Openembedded-devel mailing list Openembedded-devel@lists.openembedded.org http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel
Re: [oe] Simultaneous build of two intree packages
On 2012-05-07 05:54, Radek Dostal wrote: Dear All, I have three intree recipes A, B and C which are part of the same layer. => B depends on A => C depends on A. There is no dependency between B and C. Only important thing is that B and C are not build simultaneously. This is not a problem unless I use configure BB_NUMBER_THREADS> 1. Than builds B and C starts to collide and produce mess because they are both intree builds. What are you doing that makes these clash? Is there something like a mutex/lock mechanism in openembedded-core, which would forbid two recipes to be build simultaneously? Just make C depend on B -- Gary Thomas | Consulting for the MLB Associates |Embedded world ___ Openembedded-devel mailing list Openembedded-devel@lists.openembedded.org http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel
[oe] Simultaneous build of two intree packages
Dear All, I have three intree recipes A, B and C which are part of the same layer. => B depends on A => C depends on A. There is no dependency between B and C. Only important thing is that B and C are not build simultaneously. This is not a problem unless I use configure BB_NUMBER_THREADS > 1. Than builds B and C starts to collide and produce mess because they are both intree builds. Is there something like a mutex/lock mechanism in openembedded-core, which would forbid two recipes to be build simultaneously? Thanks, Radek ___ Openembedded-devel mailing list Openembedded-devel@lists.openembedded.org http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel
