Re: [Openembedded-architecture] layers.openembedded.org - suggestion to remove obsolete layers

[Paul Eggleton]

On Tuesday, 3 December 2019 3:46:00 AM NZDT Mark Hatle wrote:
> On 12/2/19 8:30 AM, Martin Jansa wrote:
> > Even if the original URL isn't available anymore there might be some 
> > existing
> > forks on github or elsewhere.
> 
> But how would someone find it, and if they did find a fork how would they know
> it's 'authentic', i.e. not maliciously tampered with?
>
> It's one thing if someone forks it and maintains it, I'd hope/expect in that
> case they would try to take ownership on the layerindex.  (This has certainly
> happened in the past.)

Indeed, and I'd like to encourage more of this in future. If unmaintained 
layers could be marked in some way, perhaps with a link to suggested next steps 
for those interested in picking up the reins, I think that process would be 
facilitated.

> Yes, I'm only saying it is removed from master, not all of the index.
> 
> In the current layer index version there is a switch.  One is simply to stop
> indexing a particular layer.  This is not what I am suggesting.

(As an aside, in case anyone wonders, we have not used this switch in the 
public index - it was added to the code for use in private instances.)
 
> Instead what we could do is stop indexing/publishing new layer branches when 
> the
> LAYERCOMPAT doesn't match oe-core.
>
> This would allow a layer that still exists but is no longer functional to be
> disabled from master.. but if master and/or a new release branch is created it
> can validate and then publish.  (Once published it wouldn't be 'unpublished'
> from a release branch.  Also it wouldn't be unpublished/removed from master
> unless the suggested policy happens, i.e. URL goes away or it hasn't been
> updated in in 3 releases.)

I think this seems reasonable. Additionally we can easily add flags with 
comments to layers that are considered unmaintained, that would show up in the 
web interface but could also be shown by Toaster, bitbake-layers and other 
places. (We do already have layer notes that can be used for this in a 
free-form manner, but an explicit flag would allow for easier filtering, 
possibly excluding such layers by default in certain contexts.)

FYI, relating to this thread at Mark's suggestion I am already working on 
adding some fields to record when a layer was last successfully fetched, and 
when we last attempted to fetch it, so we can produce report on and/or 
automatically flag layers where the repo is gone for a long time (if we can't 
fetch a layer for e.g. 30 days, assuming it wasn't prevented from happening by 
some local issue I think we can safely assume it's gone).

Cheers,
Paul

-- 

Paul Eggleton
Intel System Software Products


___
Openembedded-architecture mailing list
Openembedded-architecture@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-architecture

Re: [Openembedded-architecture] Heterogeneous System Proposal

[Paul Eggleton]

Taking just this part of the discussion for now (since it is a small familiar 
aspect to me)

On Tuesday, 3 December 2019 1:08:54 PM NZDT Mark Hatle wrote:
> > You may want to consider this now. Adding system with the "nomachine"
> > mode you mention is a nightmare scenario for parsing. Bitbake has a
> > problem with knowing when to parse or not to parse since to 'skip'
> > parsing a recipe, it needs to parse enough to know it should skip it.
> > COMPATIBLE_MACHINE is implemented with the skip mechanism for example.
> > If parsing is a concern (and I agree it is), you could probably get
> > much further with a BBMASK approach to just the system-images recipe as
> > then bitbake would know what it needs to parse without needing to read
> > the recipes.
> 
> The problem I have with BBMASK is that when a user tries to do something with
> it, they don't get a reasonable error message.
>
> For instance if I BBMASK out bash, and the user does bitbake bash they get an
> error.  Instead of a 'bash is unavailable because ...'.

The only practical way for that to work is the existing skip mechanism, since 
PN isn't always completely defined by the .bb filename, we need to actually 
parse to determine its value. gcc-cross and other parts of the toolchain are 
examples. 

BBMASK is such a usability pain - both in your scenario but also in the reverse 
where you have to hack it until it only masks exactly what you do want masked - 
that I think we would be best advised not to make any more use of it than we 
currently do, especially if it's only to save a bit of time parsing.

Cheers
Paul

-- 

Paul Eggleton
Intel System Software Products


___
Openembedded-architecture mailing list
Openembedded-architecture@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-architecture

Re: [Openembedded-architecture] Heterogeneous System Proposal

[Mark Hatle]

On 12/2/19 5:53 PM, Richard Purdie wrote:
> On Mon, 2019-12-02 at 17:19 -0600, Mark Hatle wrote:
>>
>> On 12/2/19 3:58 PM, Richard Purdie wrote:
>>> On Mon, 2019-12-02 at 15:37 -0600, Mark Hatle wrote:
 In all of the above examples, the user has manually configured
 the
 multiconfig within their project.  There is a simple way to move
 that
 configuration to a layer, simply place it in a conf/multiconfig
 directory within that layer.

 This ability suggests to be that there should be a standard way
 to
 specify a layer above that of machine, which defines the overall
 characteristics of the system.

 I'm proposing calling this new layer type as a system layer and
 it's
 configuration variable "SYSTEM".  It will be used instead of
 MACHINE
 when there is no single machine to quantify the contents of the
 produced system image.  When implementing a system, we do not
 want to
 make major changes to any other components.  Due to the existing
 implementation requiring MACHINE and certain TUNE parameters,
 this
 will require us to provide a special MACHINE value that can be
 used
 for a heterogeneous system.  I suggest we create a new
 'nomachine'
 system that only defines/uses an equivalent noarch style
 tune.  This
 will instruct the system that this configuration can be used to
 noarch software and create images (including with wic), but it is
 not
 able to compile specific applications.  Each of these
 applications or
 images must come from a defined MACHINE.
>>>
>>> I'm not sure I 'buy' this explanation and I think a "nomachine"
>>> target
>>> will be horrible in practice. In your example below, why can't
>>> mysystem.conf just set MACHINE = "zcu_cortex-a72" and the other
>>> bits of
>>> linux.conf and BBMULTICONFIG = "bootloader fpga" ? 
>>
>> This assumes the linux side of things will load the others.  In the
>> systems I am thinking of Linux may not be a primary system.
>>
>> I.e.  a baremetal system based on a microblaze may be the core
>> system, which then bootstraps and launches the linux system.  The
>> core 'machine' should it be defined as baremetal and
>> zcu102_microblaze then?
> 
> Sure, why not? I picked on 'linux' but I could have equally chosen any
> one of the targets, there wasn't any specific reason.
> 
>> Problem is baremetal isn't really an operating system, so how do we
>> handle that?
> 
> I'm not sure it matters, MACHINE doesn't define an operating system.

Correct distro does, I was confusing my statements.  Both MACHINE and DISTRO
have to work together to define the system.  My assumption was that a system
would define reasonable defaults, like a multiconfig does.  (Maybe this is an
error on my part though.)

>> Introduce nomachine and notune, and adjust system components to
 know
 how to deal with these properly.  I.e. recipes that are NOT
 noarch
 would be be rejected in this configuration.
>>>
>>> I definitely do not like this at all. I'm not sure why its needed.
>>
>> I want a way to say that this configuration is a sum of it's parts,
>> but by itself can't do anything.  It's not arm, it's not ppc, it's
>> not x86, etc..
> 
> The more I read and think about this, the more I'm concluding it still
> *is* a MACHINE. A MACHINE represents a target piece of hardware, not an
> OS, or an architecture. Those are configurations of that hardware
> combined with software.
> 
>> This is common on FPGA systems that may have to load from special
>> configurations, the hardware itself may be loading a bitstream (not a
>> cpu instruction set) that in turn may be a first stage loader for one
>> or multiple on system CPUs, which may in turn instruct other aspects
>> of the system to load.
>>
>> I don't want the proposal to be FPGA specific, as I expect it will be
>> used in other heterogeneous cases.  Primarily ones that have a single
>> host CPU (that boot) and multiple plugin cards.  In those cases, it
>> -CAN- have both a SYSTEM and a MACHINE defined, or just simply a
>> MACHINE with multiconfigs that the MACHINE has a way of
>> understanding.
> 
> I think you need a convention for MACHINE that works for these systems.
> I'm not sure we need a "new" level specially for this.

Agreed.  The only reason to add a SYSTEM would be to distinguish it in the
local.conf, but from a practical reason it and MACHINE are equivalent in
hierarchy in my mind.  SYSTEM is 1..n multiconfigs, while MACHINE requires a
DISTRO to provide a working system.

>>> I'd also put the idea out there that in many ways what you're
>>> describing is actually a configuration or system configuration. Its
>>> configurations like this we actually test on the project
>>> autobuilder
>>> and its what the code in yocto-autobuilder-helper generates from
>>> json
>>> files.
>>
>> I would expect in my particular use-case that I do need to generate
>> configurations based on certain 

Re: [Openembedded-architecture] Heterogeneous System Proposal

[Mark Hatle]

On 12/2/19 5:47 PM, Richard Purdie wrote:
> On Mon, 2019-12-02 at 15:27 -0800, Alejandro Enedino Hernandez
> Samaniego wrote:
>> So I have tried something similar to this, and while it does work,
>> due to some parsing issue (I still have not figured out exactly what)
>> the multiconfig dependencies between multiconfigs are not being taken
>> into account, which defeats the whole purpose.
> 
> It likely makes sense to understand why its breaking since wouldn't
> those issues affect this regardless of which direction things go in?
> 
>> While I understand that introducing a new variable might create some
>> confusion, at the same time I do believe that it would be
>> better if we do it that way, and in some way separate workflows when
>> we are building a "full" heterogeneous system vs a single
>> architecture system, the difference lies in the fact that we dont set
>> what the "master" or core MACHINE is, they are all equal.
>>
>> I have been working on the implementation of this proposal and here
>> is what I currently have for which of course I am open to change.
>>
>> Like I said, I had issues with defining a placeholder machine for the
>> multiconfig dependencies, so I did go with defining
>>
>> SYSTEM = "dummy-system"
>>
>> on local.conf (also not a huge fan of the name since to me its a bit
>> too generic,but as RP said, the name is probably the least of our
>> problems)
>>
>> then
>> include conf/system/${SYSTEM}.conf
>>
>> and I have the following set up on a layer:
>>
>> meta-heterogeneous
>>   conf/layer.conf
>>   conf/system/dummy-system.conf
>>   conf/multiconfig/dummy-arm64.conf
>>   conf/multiconfig/dummy-x86-64.conf
>>   recipes-heterogeneous/heterogeneous-example_1.0.bb
>>
>> Explanation:
>> layer.conf  - normal layer.conf, nothing out of the ordinary
>> dummy-system.conf - Defines BBMULTICONFIG as dummy-arm64 and dummy-
>> x86-64, AND defines the multiconfig dependencies between them
>> dummy-*64.conf - Define each of the multiconfigs, in them there are
>> only 3 variables, DISTRO, MACHINE (qemuarm64 and qemux86-64
>> respectively), and TMPDIR which is simpy defined as tmp-${MACHINE}
>> heterogeneous-example_1.0.bb : This was a placeholder recipe, since
>> to my knowledge bitbake required a target, and it simply contained a
>> line that set up a multiconfig dependency that would allow:
>>
>> $ bitbake heterogeneous-example
> 
> I'm going to try and make a concise reply to both you and Mark.
> 
> What worries me is that in this "dummy-system" context there is only
> one valid recipe which is the heterogeneous-example, everything else is
> meaningless.

My assumption was that, like a baremetal system, recipes would be limited to a
small number of things -- primarily different image recipes.  But this is out of
convention rather then actually modifying things to say 'only images can be 
used'.

> If we go this route we'll start having to add special corner casing for
> "nomachine" and "notune" and everywhere, then special case those values
> since there is code they will need to pass through. I don't think these
> concepts make sense to the majority of users and I think the damage
> we'd do to the metadata trying to retrofit them will be significant and
> not in the interests of the project.

(I choose the names nomachine and notune to match the existing nodistro option.
 Provide reasonable defaults that do the 'right thing' within that context.)

I was expecting the opposite here.  We wouldn't modify any existing code (I
think), so hashes, metadata etc wouldn't need to be modified.  Since nomachine
would presumably load notune.. Then the notune could define a blank tune that
doesn't "do antyhing".

The only place we may need some code is in an anon python chunk that if notune
is defined and the recipe isn't noarch/all.. it would raise the skipRecipe
exception with an appropriate message.

> Looking at things a different way, the proposal makes little difference
> to adding a new machine above called "dummy-system" which sets up the
> multiconfigs, it just gives it a different name, SYSTEM instead of
> MACHINE. I think you can achieve much of (all?) the same things with a
> naming convention rather than needing a new namespace. If multiconfig
> has bugs with that, we should fix them?

Agreed.  SYSTEM, MACHINE, MACHINE w/ a convention.. it's all fine, as long as we
have a standard way to refer to the item and define it.

--Mark

> Cheers,
> 
> Richard
> 
> 
> 
___
Openembedded-architecture mailing list
Openembedded-architecture@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-architecture

Re: [Openembedded-architecture] Heterogeneous System Proposal

[Richard Purdie]

On Mon, 2019-12-02 at 17:19 -0600, Mark Hatle wrote:
> 
> On 12/2/19 3:58 PM, Richard Purdie wrote:
> > On Mon, 2019-12-02 at 15:37 -0600, Mark Hatle wrote:
> > > In all of the above examples, the user has manually configured
> > > the
> > > multiconfig within their project.  There is a simple way to move
> > > that
> > > configuration to a layer, simply place it in a conf/multiconfig
> > > directory within that layer.
> > > 
> > > This ability suggests to be that there should be a standard way
> > > to
> > > specify a layer above that of machine, which defines the overall
> > > characteristics of the system.
> > > 
> > > I'm proposing calling this new layer type as a system layer and
> > > it's
> > > configuration variable "SYSTEM".  It will be used instead of
> > > MACHINE
> > > when there is no single machine to quantify the contents of the
> > > produced system image.  When implementing a system, we do not
> > > want to
> > > make major changes to any other components.  Due to the existing
> > > implementation requiring MACHINE and certain TUNE parameters,
> > > this
> > > will require us to provide a special MACHINE value that can be
> > > used
> > > for a heterogeneous system.  I suggest we create a new
> > > 'nomachine'
> > > system that only defines/uses an equivalent noarch style
> > > tune.  This
> > > will instruct the system that this configuration can be used to
> > > noarch software and create images (including with wic), but it is
> > > not
> > > able to compile specific applications.  Each of these
> > > applications or
> > > images must come from a defined MACHINE.
> > 
> > I'm not sure I 'buy' this explanation and I think a "nomachine"
> > target
> > will be horrible in practice. In your example below, why can't
> > mysystem.conf just set MACHINE = "zcu_cortex-a72" and the other
> > bits of
> > linux.conf and BBMULTICONFIG = "bootloader fpga" ? 
> 
> This assumes the linux side of things will load the others.  In the
> systems I am thinking of Linux may not be a primary system.
> 
> I.e.  a baremetal system based on a microblaze may be the core
> system, which then bootstraps and launches the linux system.  The
> core 'machine' should it be defined as baremetal and
> zcu102_microblaze then?

Sure, why not? I picked on 'linux' but I could have equally chosen any
one of the targets, there wasn't any specific reason.

> Problem is baremetal isn't really an operating system, so how do we
> handle that?

I'm not sure it matters, MACHINE doesn't define an operating system.

> Introduce nomachine and notune, and adjust system components to
> > > know
> > > how to deal with these properly.  I.e. recipes that are NOT
> > > noarch
> > > would be be rejected in this configuration.
> > 
> > I definitely do not like this at all. I'm not sure why its needed.
> 
> I want a way to say that this configuration is a sum of it's parts,
> but by itself can't do anything.  It's not arm, it's not ppc, it's
> not x86, etc..

The more I read and think about this, the more I'm concluding it still
*is* a MACHINE. A MACHINE represents a target piece of hardware, not an
OS, or an architecture. Those are configurations of that hardware
combined with software.

> This is common on FPGA systems that may have to load from special
> configurations, the hardware itself may be loading a bitstream (not a
> cpu instruction set) that in turn may be a first stage loader for one
> or multiple on system CPUs, which may in turn instruct other aspects
> of the system to load.
> 
> I don't want the proposal to be FPGA specific, as I expect it will be
> used in other heterogeneous cases.  Primarily ones that have a single
> host CPU (that boot) and multiple plugin cards.  In those cases, it
> -CAN- have both a SYSTEM and a MACHINE defined, or just simply a
> MACHINE with multiconfigs that the MACHINE has a way of
> understanding.

I think you need a convention for MACHINE that works for these systems.
I'm not sure we need a "new" level specially for this.

> > I'd also put the idea out there that in many ways what you're
> > describing is actually a configuration or system configuration. Its
> > configurations like this we actually test on the project
> > autobuilder
> > and its what the code in yocto-autobuilder-helper generates from
> > json
> > files.
> 
> I would expect in my particular use-case that I do need to generate
> configurations based on certain parameters (specifically a system
> device tree).  However, other configurations are going to be very
> static and then predefined 'systems' can be provided by layer that
> give the user the ability to modify (as they would any other part of
> the system today.)
> 
> > There is a little known/used piece of bitbake which is BBTARGETS.
> > If
> > your myssystem.conf sets:
> > 
> > BBTARGETS = "system-images:do_image"
> 
> I wasn't aware this was a thing, but it could very well be used by a
> system configuration to define the right combination of things to
> build.  I didn't 

Re: [Openembedded-architecture] Heterogeneous System Proposal

[Richard Purdie]

On Mon, 2019-12-02 at 15:27 -0800, Alejandro Enedino Hernandez
Samaniego wrote:
> So I have tried something similar to this, and while it does work,
> due to some parsing issue (I still have not figured out exactly what)
> the multiconfig dependencies between multiconfigs are not being taken
> into account, which defeats the whole purpose.

It likely makes sense to understand why its breaking since wouldn't
those issues affect this regardless of which direction things go in?

> While I understand that introducing a new variable might create some
> confusion, at the same time I do believe that it would be
> better if we do it that way, and in some way separate workflows when
> we are building a "full" heterogeneous system vs a single
> architecture system, the difference lies in the fact that we dont set
> what the "master" or core MACHINE is, they are all equal.
> 
> I have been working on the implementation of this proposal and here
> is what I currently have for which of course I am open to change.
> 
> Like I said, I had issues with defining a placeholder machine for the
> multiconfig dependencies, so I did go with defining
> 
> SYSTEM = "dummy-system"
> 
> on local.conf (also not a huge fan of the name since to me its a bit
> too generic,but as RP said, the name is probably the least of our
> problems)
> 
> then
> include conf/system/${SYSTEM}.conf
> 
> and I have the following set up on a layer:
> 
> meta-heterogeneous
>   conf/layer.conf
>   conf/system/dummy-system.conf
>   conf/multiconfig/dummy-arm64.conf
>   conf/multiconfig/dummy-x86-64.conf
>   recipes-heterogeneous/heterogeneous-example_1.0.bb
> 
> Explanation:
> layer.conf  - normal layer.conf, nothing out of the ordinary
> dummy-system.conf - Defines BBMULTICONFIG as dummy-arm64 and dummy-
> x86-64, AND defines the multiconfig dependencies between them
> dummy-*64.conf - Define each of the multiconfigs, in them there are
> only 3 variables, DISTRO, MACHINE (qemuarm64 and qemux86-64
> respectively), and TMPDIR which is simpy defined as tmp-${MACHINE}
> heterogeneous-example_1.0.bb : This was a placeholder recipe, since
> to my knowledge bitbake required a target, and it simply contained a
> line that set up a multiconfig dependency that would allow:
> 
> $ bitbake heterogeneous-example

I'm going to try and make a concise reply to both you and Mark.

What worries me is that in this "dummy-system" context there is only
one valid recipe which is the heterogeneous-example, everything else is
meaningless.

If we go this route we'll start having to add special corner casing for
"nomachine" and "notune" and everywhere, then special case those values
since there is code they will need to pass through. I don't think these
concepts make sense to the majority of users and I think the damage
we'd do to the metadata trying to retrofit them will be significant and
not in the interests of the project.

Looking at things a different way, the proposal makes little difference
to adding a new machine above called "dummy-system" which sets up the
multiconfigs, it just gives it a different name, SYSTEM instead of
MACHINE. I think you can achieve much of (all?) the same things with a
naming convention rather than needing a new namespace. If multiconfig
has bugs with that, we should fix them?

Cheers,

Richard




___
Openembedded-architecture mailing list
Openembedded-architecture@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-architecture

Re: [Openembedded-architecture] Heterogeneous System Proposal

[Alejandro Enedino Hernandez Samaniego]

Hey guys,

On Mon, Dec 2, 2019 at 3:24 PM Mark Hatle 
wrote:

>
>
> On 12/2/19 4:09 PM, Richard Purdie wrote:
> > On Mon, 2019-12-02 at 21:58 +, Richard Purdie wrote:
> >> On Mon, 2019-12-02 at 15:37 -0600, Mark Hatle wrote:
> >>> In all of the above examples, the user has manually configured the
> >>> multiconfig within their project.  There is a simple way to move
> >>> that
> >>> configuration to a layer, simply place it in a conf/multiconfig
> >>> directory within that layer.
> >>>
> >>> This ability suggests to be that there should be a standard way to
> >>> specify a layer above that of machine, which defines the overall
> >>> characteristics of the system.
> >>>
> >>> I'm proposing calling this new layer type as a system layer and
> >>> it's
> >>> configuration variable "SYSTEM".  It will be used instead of
> >>> MACHINE
> >>> when there is no single machine to quantify the contents of the
> >>> produced system image.  When implementing a system, we do not want
> >>> to
> >>> make major changes to any other components.  Due to the existing
> >>> implementation requiring MACHINE and certain TUNE parameters, this
> >>> will require us to provide a special MACHINE value that can be used
> >>> for a heterogeneous system.  I suggest we create a new 'nomachine'
> >>> system that only defines/uses an equivalent noarch style
> >>> tune.  This
> >>> will instruct the system that this configuration can be used to
> >>> noarch software and create images (including with wic), but it is
> >>> not
> >>> able to compile specific applications.  Each of these applications
> >>> or
> >>> images must come from a defined MACHINE.
> >>
> >> I'm not sure I 'buy' this explanation and I think a "nomachine"
> >> target will be horrible in practice. In your example below, why can't
> >> mysystem.conf just set MACHINE = "zcu_cortex-a72" and the other bits
> >> of linux.conf and BBMULTICONFIG = "bootloader fpga" ?
> >
> > In fact, if you create a conf/machine/system-zcu_cortex-a72.conf and
> > have it set something like:
> >
> > BBMULTICONFIG = "bootloader fpga"
> > MACHINE = "zcu_cortex-a72"
> > require conf/machine/${MACHINE}.conf
> >
> > I think what I/you describe may even work today without a new variable
> > and simply a new convention of a special set of new machine name
> > targets in layers...
>

So I have tried something similar to this, and while it does work, due to
some parsing issue (I still have not figured out exactly what)
the multiconfig dependencies between multiconfigs are not being taken into
account, which defeats the whole purpose.

While I understand that introducing a new variable might create some
confusion, at the same time I do believe that it would be
better if we do it that way, and in some way separate workflows when we are
building a "full" heterogeneous system vs a single
architecture system, the difference lies in the fact that we dont set what
the "master" or core MACHINE is, they are all equal.

I have been working on the implementation of this proposal and here is what
I currently have for which of course I am open to change.

Like I said, I had issues with defining a placeholder machine for the
multiconfig dependencies, so I did go with defining

SYSTEM = "dummy-system"

on local.conf (also not a huge fan of the name since to me its a bit too
generic,but as RP said, the name is probably the least of our problems)

then
include conf/system/${SYSTEM}.conf

and I have the following set up on a layer:

meta-heterogeneous
  conf/layer.conf
  conf/system/dummy-system.conf
  conf/multiconfig/dummy-arm64.conf
  conf/multiconfig/dummy-x86-64.conf
  recipes-heterogeneous/heterogeneous-example_1.0.bb

Explanation:
layer.conf  - normal layer.conf, nothing out of the ordinary
dummy-system.conf - Defines BBMULTICONFIG as dummy-arm64 and dummy-x86-64,
AND defines the multiconfig dependencies between them
dummy-*64.conf - Define each of the multiconfigs, in them there are only 3
variables, DISTRO, MACHINE (qemuarm64 and qemux86-64 respectively), and
TMPDIR which is simpy defined as tmp-${MACHINE}
heterogeneous-example_1.0.bb : This was a placeholder recipe, since to my
knowledge bitbake required a target, and it simply contained a line that
set up a multiconfig dependency that would allow:

$ bitbake heterogeneous-example

to start a buld for core-image-minimal (for example) for all archs in the
SYSTEM.

Now that I read RP's email about BBTARGETS I believe this can be improved
and at the very least delete the example recipe, let me know what yo guys
think

Cheers,

Alejandro




>
> Yes, SYSTEM could be implemented as something like:
>
> MACHINE ??= ${@'system-${SYSTEM}' if d.getVar('SYSTEM')}
>
> (I'm not really sure if that will work on not...)
>
> The key though, I want a way to specify a system that by itself can't build
> binaries -- but can only include binaries from other architectures.  (It
> -CAN-
> however build noarch/all [text file] components to be included in the
> image.)
>
> So you 

Re: [Openembedded-architecture] Heterogeneous System Proposal

[Mark Hatle]

On 12/2/19 4:09 PM, Richard Purdie wrote:
> On Mon, 2019-12-02 at 21:58 +, Richard Purdie wrote:
>> On Mon, 2019-12-02 at 15:37 -0600, Mark Hatle wrote:
>>> In all of the above examples, the user has manually configured the
>>> multiconfig within their project.  There is a simple way to move
>>> that
>>> configuration to a layer, simply place it in a conf/multiconfig
>>> directory within that layer.
>>>
>>> This ability suggests to be that there should be a standard way to
>>> specify a layer above that of machine, which defines the overall
>>> characteristics of the system.
>>>
>>> I'm proposing calling this new layer type as a system layer and
>>> it's
>>> configuration variable "SYSTEM".  It will be used instead of
>>> MACHINE
>>> when there is no single machine to quantify the contents of the
>>> produced system image.  When implementing a system, we do not want
>>> to
>>> make major changes to any other components.  Due to the existing
>>> implementation requiring MACHINE and certain TUNE parameters, this
>>> will require us to provide a special MACHINE value that can be used
>>> for a heterogeneous system.  I suggest we create a new 'nomachine'
>>> system that only defines/uses an equivalent noarch style
>>> tune.  This
>>> will instruct the system that this configuration can be used to
>>> noarch software and create images (including with wic), but it is
>>> not
>>> able to compile specific applications.  Each of these applications
>>> or
>>> images must come from a defined MACHINE.
>>
>> I'm not sure I 'buy' this explanation and I think a "nomachine"
>> target will be horrible in practice. In your example below, why can't
>> mysystem.conf just set MACHINE = "zcu_cortex-a72" and the other bits
>> of linux.conf and BBMULTICONFIG = "bootloader fpga" ? 
> 
> In fact, if you create a conf/machine/system-zcu_cortex-a72.conf and
> have it set something like:
> 
> BBMULTICONFIG = "bootloader fpga"
> MACHINE = "zcu_cortex-a72"
> require conf/machine/${MACHINE}.conf
> 
> I think what I/you describe may even work today without a new variable
> and simply a new convention of a special set of new machine name
> targets in layers...

Yes, SYSTEM could be implemented as something like:

MACHINE ??= ${@'system-${SYSTEM}' if d.getVar('SYSTEM')}

(I'm not really sure if that will work on not...)

The key though, I want a way to specify a system that by itself can't build
binaries -- but can only include binaries from other architectures.  (It -CAN-
however build noarch/all [text file] components to be included in the image.)

So you end up with:

system-zcu.conf
zcu-cortex-a72.conf
zcu-r5.conf
zcu-microblaze.conf

In the end the users should be able to build each of the components
independently WITHOUT ever using system, if they choose the 'old' way of
combining parts and pieces post build outside of the build system.

If they want a pre-defined multiconfig build, then they could use the system-zcu
to select all of the defaults for them.

--Mark

> Cheers,
> 
> Richard
> 
___
Openembedded-architecture mailing list
Openembedded-architecture@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-architecture

Re: [Openembedded-architecture] Heterogeneous System Proposal

[Mark Hatle]

On 12/2/19 3:58 PM, Richard Purdie wrote:
> On Mon, 2019-12-02 at 15:37 -0600, Mark Hatle wrote:
>> In all of the above examples, the user has manually configured the
>> multiconfig within their project.  There is a simple way to move that
>> configuration to a layer, simply place it in a conf/multiconfig
>> directory within that layer.
>>
>> This ability suggests to be that there should be a standard way to
>> specify a layer above that of machine, which defines the overall
>> characteristics of the system.
>>
>> I'm proposing calling this new layer type as a system layer and it's
>> configuration variable "SYSTEM".  It will be used instead of MACHINE
>> when there is no single machine to quantify the contents of the
>> produced system image.  When implementing a system, we do not want to
>> make major changes to any other components.  Due to the existing
>> implementation requiring MACHINE and certain TUNE parameters, this
>> will require us to provide a special MACHINE value that can be used
>> for a heterogeneous system.  I suggest we create a new 'nomachine'
>> system that only defines/uses an equivalent noarch style tune.  This
>> will instruct the system that this configuration can be used to
>> noarch software and create images (including with wic), but it is not
>> able to compile specific applications.  Each of these applications or
>> images must come from a defined MACHINE.
> 
> I'm not sure I 'buy' this explanation and I think a "nomachine" target
> will be horrible in practice. In your example below, why can't
> mysystem.conf just set MACHINE = "zcu_cortex-a72" and the other bits of
> linux.conf and BBMULTICONFIG = "bootloader fpga" ? 

This assumes the linux side of things will load the others.  In the systems I am
thinking of Linux may not be a primary system.

I.e.  a baremetal system based on a microblaze may be the core system, which
then bootstraps and launches the linux system.  The core 'machine' should it be
defined as baremetal and zcu102_microblaze then?

Problem is baremetal isn't really an operating system, so how do we handle that?

>> The SYSTEM level multiconfig could be used to combine any homogeneous
>> or heterogeneous configuration.  For example:
>>
>> build
>>   conf
>>local.conf:
>> SYSTEM = "mysystem"
>> layers
>>   meta-
>>conf
>> system
>>  mysystem.conf
>>   MACHINE = "nomachine"
>>   BBMULTICONFIG = "bootloader fpga linux"
>>  mysystem.wks
>> multiconfig
>>  bootloader.conf:
>>   MACHINE = "zcu102_microblaze"
>>   DISTRO = "baremetal"
>>   TMPDIR = "${TOPDIR}/tmp/multi/bootloader"
>>  fpga.conf:
>>   MACHINE = "zcu_fpga"
>>   DISTRO = "baremetal"
>>   TMPDIR = "${TOPDIR}/tmp/multi/fpga"
>>  linux.conf:
>>   MACHINE = "zcu_cortex-a72"
>>   DISTRO = "poky"
>>   TMPDIR = "${TOPDIR}/tmp/multi/linux"
>>recipes
>> images
>>  system-images.bb
>>   do_image[mcdepends] = "mc:bootloader:application:do_deploy \
>>  mc:fpga:application:do_deploy \
>>  mc:linux:core-image-minimal:do_rootfs"
>>   do_image() { ... instructions for combining stuff ... }
>>   meta-
>>conf
>> machine
>>  zcu_microblaze.conf
>>  zcu_fpga.conf
>>  zcu_cortext-a72.conf
>>   meta-mybootloader
>>conf
>> distro
>>  baremetal.conf
>>recipes
>> newlib
>>  newlib_ver.bb
>> first_stage
>>  first_stage.bb
>> second_stage
>>  second_stage.bb
>> bootloader
>>  bootloader.bb
>>   DEPENDS = "newlib first_stage second_stage"
>>   meta-fpga
>>conf
>> distro
>>  baremetal.conf
>>recipes-baremetal
>> newlib
>>  newlib_ver.bb
>> library
>>  library.bb
>> application
>>  application.bb
>>   DEPENDS = "newlib library application"
>>
>> So proposal:
>>
>> Introduce a new OPTIONAL "SYSTEM" variable.  Which, like machine,
>> would automatically "include conf/${SYSTEM}.conf", if defined.
> 
> I know we talked and I think I mentioned SYSTEM. I'm still not entirely
> sold on that name, however the name is only one piece of the puzzle so
> I'll defer that for now.
> 
>> Introduce nomachine and notune, and adjust system components to know
>> how to deal with these properly.  I.e. recipes that are NOT noarch
>> would be be rejected in this configuration.
> 
> I definitely do not like this at all. I'm not sure why its needed.

I want a way to say that this configuration is a sum of it's parts, but by
itself can't do anything.  It's not arm, it's not ppc, it's not x86, etc..

This is common on FPGA systems that may have to load from special
configurations, the hardware itself may be loading a bitstream (not a cpu
instruction set) that in turn may be a first stage loader for one or multiple on
system CPUs, which may in turn instruct other aspects of the system to load.

I don't want the proposal to be FPGA specific, as I expect it will be used in
other 

Re: [Openembedded-architecture] Heterogeneous System Proposal

[Richard Purdie]

On Mon, 2019-12-02 at 21:58 +, Richard Purdie wrote:
> On Mon, 2019-12-02 at 15:37 -0600, Mark Hatle wrote:
> > In all of the above examples, the user has manually configured the
> > multiconfig within their project.  There is a simple way to move
> > that
> > configuration to a layer, simply place it in a conf/multiconfig
> > directory within that layer.
> > 
> > This ability suggests to be that there should be a standard way to
> > specify a layer above that of machine, which defines the overall
> > characteristics of the system.
> > 
> > I'm proposing calling this new layer type as a system layer and
> > it's
> > configuration variable "SYSTEM".  It will be used instead of
> > MACHINE
> > when there is no single machine to quantify the contents of the
> > produced system image.  When implementing a system, we do not want
> > to
> > make major changes to any other components.  Due to the existing
> > implementation requiring MACHINE and certain TUNE parameters, this
> > will require us to provide a special MACHINE value that can be used
> > for a heterogeneous system.  I suggest we create a new 'nomachine'
> > system that only defines/uses an equivalent noarch style
> > tune.  This
> > will instruct the system that this configuration can be used to
> > noarch software and create images (including with wic), but it is
> > not
> > able to compile specific applications.  Each of these applications
> > or
> > images must come from a defined MACHINE.
> 
> I'm not sure I 'buy' this explanation and I think a "nomachine"
> target will be horrible in practice. In your example below, why can't
> mysystem.conf just set MACHINE = "zcu_cortex-a72" and the other bits
> of linux.conf and BBMULTICONFIG = "bootloader fpga" ? 

In fact, if you create a conf/machine/system-zcu_cortex-a72.conf and
have it set something like:

BBMULTICONFIG = "bootloader fpga"
MACHINE = "zcu_cortex-a72"
require conf/machine/${MACHINE}.conf

I think what I/you describe may even work today without a new variable
and simply a new convention of a special set of new machine name
targets in layers...

Cheers,

Richard

___
Openembedded-architecture mailing list
Openembedded-architecture@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-architecture

[Openembedded-architecture] Heterogeneous System Proposal

[Mark Hatle]

Problem Statement
-
In the current world there is an increasing number of heterogeneous
systems being developed.  Currently these components can be built
independently of each other, and then combined later.  For ease of use,
it would be nice to be able to build these systems with a single build,
including generating a final bootable image.  A heterogeneous system may
simply be different configurations for different components, different
operating systems for different components, or a system made up of
diverse processor architectures.  Recently the Yocto Project has added
multiconfig in order to enable these types of configurations, but
suggested workflows for various configurations are needed to avoid
confusion and to avoid developers implementing their own schemes.

Proposal

During the course of discussing this with other people a number of terms
have been involved, and after numerous discussions it is clear that
people have a different definition for the various components.  The
purpose of this document is to explain the various pieces of this
proposal, so that we can all be using the same items for the various
components necessary to build a heterogeneous system.

For a homogeneous build, the traditional Open Embedded/Yocto Project
components that are used include:

* Build Configuration (local.conf)
* MACHINE
   * Specifies target device information, including hardware
 capabilities, console settings, boot image configurations, etc.
 These settings are used by MACHINE packages, as well as image
 generation.
   * Define MACHINE in the conf/local.conf file in the Build Directory.
* SOC_FAMILY (optional, but implied by the machine)
   * Way to group together machines based upon common System On Chip
 components.
   * A SOC_FAMILY by itself is not a fully configured and bootable
 machine, but may be used by a series of machines to specify the
 common components.
* TUNE (typically specified implicitly by the machine)
   * Specifies the CPU (instruction set), and ABI settings available to
 the user
   * DEFAULTTUNE then selects which of the available tunes is to be used.
 This is usually set by the machine.
* DISTRO (Distribution Configuration)
   * Specifies cross recipe configurations that together result in an
 overall distribution configuration.
   * Define DISTRO in the conf/local.conf file in the Build Directory.
 If not specified, a default "nodistro" distribution is used.
* Recipes (.bb files)
   * Specify how to generically build individual items.  These recipes
 will inherit the system wide settings from the distribution
 configuration, and tune.  For MACHINE specific packages they can
 also inherit machine specific settings as well.
* Image Recipes
   * Specify a series of dependencies that cause recipes to be built and
 a list of resulting packages to be installed into a target image.
   * Image recipes are responsible for constructing a filesystem image.
 Further the system can extend these into a bootable disc image
 format.

In any sort of heterogeneous configuration we want to use and build upon
the existing homogeneous components.  A heterogeneous solution is really
comprised of a number of homogeneous configurations that when deployed
together result in a fully functional device.  In other words, each of
individual parts of the heterogeneous build are standalone and not tied
to the assembly of a specific system.

Based on this, we want to avoid any changes that complicate the existing
homogeneous build components or even adding additional levels of
configuration as this will complicate existing and future uses.  For
example, in the past there have been suggestions for SUBMACHINE or other
levels of hierarchy between SOC_FAMILY and MACHINE.  Adding this level
of indirection can make it more difficult to combine different
configurations into new heterogeneous solutions.  For example, if
someone has already defined a heterogeneous solution using a MACHINE,
SUBMACHINE, and SOC_FAMILY heirachy and you wish to extend it the
existing MACHINE / SUBMACHINE may interfear with your own systems unique
configuration.

There are a few types of heterogenous systems that I have seen.  Each of
them can be constructed by combining the output of homogeneous
configurations.  The most basic heterogenous systems I have seen include
either a collection of containers or different OSes, a primary CPU +
co-processors, or all of the CPUs are independent of each other but
share resources.  It is also possible to combine these heterogenous
configurations such as a multiple CPU system, with some CPUs having
co-processors with one or more CPU running containers.

In the case of the container based system, you really want a master
homogeneous machine configuration along with a few additional
configurations that can be incorporated into that image.  Using a
multilib configuration you would have something like:

build (build 

Re: [Openembedded-architecture] layers.openembedded.org - suggestion to remove obsolete layers

[Mark Hatle]

On 12/2/19 8:30 AM, Martin Jansa wrote:
> Even if the original URL isn't available anymore there might be some existing
> forks on github or elsewhere.

But how would someone find it, and if they did find a fork how would they know
it's 'authentic', i.e. not maliciously tampered with?

It's one thing if someone forks it and maintains it, I'd hope/expect in that
case they would try to take ownership on the layerindex.  (This has certainly
happened in the past.)

> When searching for strange recipes, even finding the name of the layer where 
> it
> was before is sometimes useful to find it somewhere else - so I would prefer 
> to
> still be able to find it in the index even with clear warning that the 
> original
> repo is gone or seems unmaintained by some rules.
> 
> I agree that including whole unmaintained layer to build from master or latest
> release is often problematic, but many projects are still using old releases
> (this might happen even more often when LTS is available) where it still might
> work fine even with last commit in pyro branch 2 years ago and importing just 
> a
> few recipes from even older less maintained layer is still generally easy and
> useful starting point to start maintaining them somewhere else.

As long as the URL is valid, I would not ever suggest removing a layer from a
released branch.  Updating (or not) doesn't matter as the layer itself claims to
be compatible with that release branch.

> If some layer is actively used in project based on e.g. pyro, then I 
> understand
> that maintainer might be updating only pyro and if nobody steps in to maintain
> (and more importantly use and test the layer in project using newer release),
> then it looks unmaintained, but still useful for some people (and IMHO should 
> be
> discoverable on the index).

Yes, I'm only saying it is removed from master, not all of the index.

In the current layer index version there is a switch.  One is simply to stop
indexing a particular layer.  This is not what I am suggesting.

Instead what we could do is stop indexing/publishing new layer branches when the
LAYERCOMPAT doesn't match oe-core.

This would allow a layer that still exists but is no longer functional to be
disabled from master.. but if master and/or a new release branch is created it
can validate and then publish.  (Once published it wouldn't be 'unpublished'
from a release branch.  Also it wouldn't be unpublished/removed from master
unless the suggested policy happens, i.e. URL goes away or it hasn't been
updated in in 3 releases.)

--Mark

> On Mon, Dec 2, 2019 at 12:44 AM Mark Hatle  > wrote:
> 
> 
> 
> On 12/1/19 5:20 PM, Rich Persaud wrote:
> >> On Dec 1, 2019, at 16:47, Mark Hatle  > wrote:
> >>
> >>
> >> I've been looking through the layer index (master primarly), and I 
> think
> we need
> >> to figure out a policy of when to remove a layer from master indexing.
> >>
> >> So I'd like to suggest the following policy:
> >>
> >> - For released branches, we do not remove layers from the index unless
> the layer
> >> URL is no longer valid.  If it is not valid for more then 90 days, we 
> should
> >> remove it.
> >>
> >> - For master branch, we use a series of tests to determine if the layer
> is still
> >> actively maintained and useful to users:
> >>
> >> 1. Is the layer URL valid?  If it has not been valid for more then 90 
> days,
> >> it should be removed.
> >>
> >> 2. Does the layer claim to support any of the last three releases: the
> >> current (or planned release) or prior two releases?
> >>    i.e. LAYERSERIES_COMPAT does not have thud, warrior or zeus in it.
> >>
> >>    This means the layer has not been updated within the last 18 
> months, and
> >> should be removed.
> >
> > How about 90 days after a release date which crosses the threshold, or 
> 90
> days after a 1 week grace period for migration of the site hosting the
> layer?  That would give the maintainer a consistent 90-day notice for the
> two conditions which could trigger moving the layer to an "unmaintained"
> state. 
> 
> If the URL is not valid, then nobody can use it.  The maintainer could be
> notified, and the URL updated if it's moved.  If it has just been 
> removed, then
> it is useless to index it.
> 
> > As Adrian said, unmaintained layers could be later adopted or forked by 
> a
> new maintainer.  We may also want to archive layers for the historical
> record, e.g. at the time they are added to the index and/or periodically.
> 
> Current layer index I don't think there is any way to do this.  As long 
> as the
> URL is valid, this is possible -- but it would require us to have a way 
> to list
> obsolete layers without impacting layerindex users.  (Users include those 
> who
> connect 

Re: [Openembedded-architecture] layers.openembedded.org - suggestion to remove obsolete layersHow

[Mark Hatle]

On 12/1/19 10:52 PM, akuster808 wrote:
> 
> 
> On 12/1/19 1:47 PM, Mark Hatle wrote:
>> I've been looking through the layer index (master primarly), and I think we 
>> need
>> to figure out a policy of when to remove a layer from master indexing.
>>
>> So I'd like to suggest the following policy:
>>
>> - For released branches, we do not remove layers from the index unless the 
>> layer
>> URL is no longer valid.  If it is not valid for more then 90 days, we should
>> remove it.
>>
>> - For master branch, we use a series of tests to determine if the layer is 
>> still
>> actively maintained and useful to users:
>>
>>1. Is the layer URL valid?  If it has not been valid for more then 90 
>> days,
>> it should be removed.
>>
>>2. Does the layer claim to support any of the last three releases: the
>> current (or planned release) or prior two releases?
>>   i.e. LAYERSERIES_COMPAT does not have thud, warrior or zeus in it.
>>
>>   This means the layer has not been updated within the last 18 months, 
>> and
>> should be removed.
>>
>>
>> So why the suggestion above?
>>
>> The URL one should be obvious, if the layer can no longer be downloaded then 
>> it
>> should be removed.  We should be able to detect when the index hasn't been
>> updated in 90 days.  (We could consider dropping this to 45 or even 30 days.)
>>
>> Looking for the last 3 releases give people the opportunity to update their
>> layers, or other people to find layers that may be of interest to them and
>> submit updates.  After 18 months, the layer is clearly no longer being
>> maintained and will cause more confusion then actually helping people find
>> something useful.  (Also after 18 months of inactivity, there is a much 
>> higher
>> likelihood of security vulnerabilities in the code.)
> 
> Can we get a "Delete " button so I take that action myself for my Layers?
> 
> Why put that burden on you and Paul?

I thought if you were the owner of the layer you could already remove it
yourself.  (You just have to login, otherwise contact Paul or I and we can do it
pretty quickly.)

>>
>> So if the TSC agrees that this is a good proposal, 
> Which TSC, there are two now : )

OpenEmbedded since they own the layer index.

--Mark

> thanks for sending that out.
> 
> -Armin
>> I can work with Paul to
>> implement it (most likely manually at first, and then eventually via 
>> automation.)
>>
>> --Mark
>> ___
>> Openembedded-architecture mailing list
>> Openembedded-architecture@lists.openembedded.org
>> http://lists.openembedded.org/mailman/listinfo/openembedded-architecture
> 
___
Openembedded-architecture mailing list
Openembedded-architecture@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-architecture

Re: [Openembedded-architecture] layers.openembedded.org - suggestion to remove obsolete layers

[Adrian Bunk]

On Sun, Dec 01, 2019 at 04:20:45PM -0600, Mark Hatle wrote:
> 
> 
> On 12/1/19 4:01 PM, Adrian Bunk wrote:
> > On Sun, Dec 01, 2019 at 03:47:43PM -0600, Mark Hatle wrote:
> >> ...
> >> After 18 months, the layer is clearly no longer being
> >> maintained and will cause more confusion then actually helping people find
> >> something useful.
> >> ...
> > 
> > If it is not too much extra work, it would be useful if you could keep
> > a non-default way to show master including obsolete layers.
> > 
> > There are fringe usecases like "has anyone ever packaged this before".
> 
> There are ways to add comments within the Index itself.  However, there is no
> way to do this for users of the index, such as:
> 
> bitbake-layers layerindex-fetch 
> 
> I was doing this earlier and pulled in a couple of layers that clear do not
> work, and haven't been updated in a long time..  So it's a real issue that I
> think we need to address.

My usecase was searching for recipes at http://layers.openembedded.org

"fringe usecases" and "If it is not too much extra work" apply,
I cannot judge whether adding "master (including obsolete)" to
the web UI would be trivial or difficult.

> --Mark

cu
Adrian
___
Openembedded-architecture mailing list
Openembedded-architecture@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-architecture