Re: [RTEMS Project] #3351: libbsd: Add support for conditional compilation of modules and compiler flags

[RTEMS trac]

#3351: libbsd: Add support for conditional compilation of modules and compiler
flags
+-
 Reporter:  Christian Mauderer  |   Owner:  Christian Mauderer
 Type:  enhancement |  Status:  assigned
 Priority:  normal  |   Milestone:  Indefinite
Component:  network/libbsd  | Version:  5
 Severity:  normal  |  Resolution:
 Keywords:  libbsd waf IPSec|  Blocked By:
 Blocking:  |
+-

Comment (by Christian Mauderer ):

 In [changeset:"854427bec4dce5cf7a88eb68a806d64f89da99dc/rtems-libbsd"
 854427b/rtems-libbsd]:
 {{{
 #!CommitTicketReference repository="rtems-libbsd"
 revision="854427bec4dce5cf7a88eb68a806d64f89da99dc"
 waf: Add configurations with different modules.

 Update #3351
 }}}

--
Ticket URL: 
RTEMS Project 
RTEMS Project
___
bugs mailing list
bugs@rtems.org
http://lists.rtems.org/mailman/listinfo/bugs

Re: [RTEMS Project] #3351: libbsd: Add support for conditional compilation of modules and compiler flags

[RTEMS trac]

#3351: libbsd: Add support for conditional compilation of modules and compiler
flags
+-
 Reporter:  Christian Mauderer  |   Owner:  Christian Mauderer
 Type:  enhancement |  Status:  assigned
 Priority:  normal  |   Milestone:  Indefinite
Component:  network/libbsd  | Version:  5
 Severity:  normal  |  Resolution:
 Keywords:  libbsd waf IPSec|  Blocked By:
 Blocking:  |
+-

Comment (by Christian Mauderer):

 Thanks for the work. I had started with a similar patch on Friday and most
 likely would have posted a first version today if you wouldn't have been
 faster. So I'm quite OK with that direction. I'll take a more detailed
 look at your patches and write some more feedback on the mailing list in a
 few hours.

--
Ticket URL: 
RTEMS Project 
RTEMS Project
___
bugs mailing list
bugs@rtems.org
http://lists.rtems.org/mailman/listinfo/bugs

Re: [RTEMS Project] #3351: libbsd: Add support for conditional compilation of modules and compiler flags

[RTEMS trac]

#3351: libbsd: Add support for conditional compilation of modules and compiler
flags
+-
 Reporter:  Christian Mauderer  |   Owner:  Christian Mauderer
 Type:  enhancement |  Status:  assigned
 Priority:  normal  |   Milestone:  Indefinite
Component:  network/libbsd  | Version:  5
 Severity:  normal  |  Resolution:
 Keywords:  libbsd waf IPSec|  Blocked By:
 Blocking:  |
+-

Comment (by Chris Johns):

 Enable control requires all the functions in `libbsd.py` are converted to
 classes.

--
Ticket URL: 
RTEMS Project 
RTEMS Project
___
bugs mailing list
bugs@rtems.org
http://lists.rtems.org/mailman/listinfo/bugs

Re: [RTEMS Project] #3351: libbsd: Add support for conditional compilation of modules and compiler flags

[RTEMS trac]

#3351: libbsd: Add support for conditional compilation of modules and compiler
flags
+-
 Reporter:  Christian Mauderer  |   Owner:  Christian Mauderer
 Type:  enhancement |  Status:  assigned
 Priority:  normal  |   Milestone:  Indefinite
Component:  network/libbsd  | Version:  5
 Severity:  normal  |  Resolution:
 Keywords:  libbsd waf IPSec|  Blocked By:
 Blocking:  |
+-

Comment (by Chris Johns):

 I have posted a patch to devel for review. It has a couple of issues:

 1. No changes to `libbsd.txt`
 2. The headers need to all be parse for `@CPU@` as `bld.env['RTEMS_CPU']`
 was taken out and replaced with `@CPU@`.

 If you are happy with the direction I can clean up these two issues and
 push to master.

 Note, I have moved a range of configuration pieces from `builder.py` to
 `waf_libbsd.py._defaults` to form the basis of the configuration data.
 After loading the configuration is updated to reflect the variant being
 built.

 The next step is to add a module level enable control and to default it to
 `True` then un-comment the modules which are commented defaulting them to
 disabled. This will give us 2 more entries in the configuration dict,
 `modules` and `modules-enabled` and by default `modules-enabled` is copied
 to `modules-build` and that list is the modules to build. If we add
 'archive-name` and `defines` you are close to supporting build variants.
 It becomes a simple matter or looping over a config creating a `builder`
 instance for each variant to build a number of variants. All this is done
 before the first build job is run.

--
Ticket URL: 
RTEMS Project 
RTEMS Project
___
bugs mailing list
bugs@rtems.org
http://lists.rtems.org/mailman/listinfo/bugs

Re: [RTEMS Project] #3351: libbsd: Add support for conditional compilation of modules and compiler flags

[RTEMS trac]

#3351: libbsd: Add support for conditional compilation of modules and compiler
flags
+-
 Reporter:  Christian Mauderer  |   Owner:  Christian Mauderer
 Type:  enhancement |  Status:  assigned
 Priority:  normal  |   Milestone:  Indefinite
Component:  network/libbsd  | Version:  5
 Severity:  normal  |  Resolution:
 Keywords:  libbsd waf IPSec|  Blocked By:
 Blocking:  |
+-

Comment (by Christian Mauderer):

 Thanks for the feedback. I had a look at that yesterday and had a similar
 course of action in mind. Good to know that you have about the same idea
 about it. Also I didn't expect that on my initial estimation for that
 work, removing the generated script will simplify the task a lot so I
 agree that it is a good idea. It will also make it simpler to understand
 the build system for anyone looking at it and future adaptions will be
 easier.

--
Ticket URL: 
RTEMS Project 
RTEMS Project
___
bugs mailing list
bugs@rtems.org
http://lists.rtems.org/mailman/listinfo/bugs

Re: [RTEMS Project] #3351: libbsd: Add support for conditional compilation of modules and compiler flags

[RTEMS trac]

#3351: libbsd: Add support for conditional compilation of modules and compiler
flags
+-
 Reporter:  Christian Mauderer  |   Owner:  Christian Mauderer
 Type:  enhancement |  Status:  assigned
 Priority:  normal  |   Milestone:  Indefinite
Component:  network/libbsd  | Version:  5
 Severity:  normal  |  Resolution:
 Keywords:  libbsd waf IPSec|  Blocked By:
 Blocking:  |
+-

Comment (by Chris Johns):

 I think the simplest solution is to:

 1. Leave `libbsd.py` as is for now. I feel this file can be split into
 parts after an initial refactoring has been completed.

 2. Create `waf_libbsd.py` which is a copy of `libbsd_waf.py` and convert
 all the `self.add('')` calls into actual code. I thnk `freebsd-to-
 rtems.py` can use the `builder.py` base class because all it needs to call
 is `processSource()` which is all in `builder.py`.

 3. Add class methods to match the required `waf` functions:
  a. `init`
  b. `options`
  c. `bsp_configure`
  d. `configure`
  e. `build`

 4. Update `wscript` to create a `Modules` instance and hold globally. It
 then uses it for each of it's calls.

 5. Update `freebsd-to-rtems.py` to use `waf_libbsd.py` and remove the
 generate code.

 This conversion is not too difficult. The INI configuration support can be
 built on opt of this. The result is:

 a. List of modules to build
 b. Defines
 c. Compile flags

--
Ticket URL: 
RTEMS Project 
RTEMS Project
___
bugs mailing list
bugs@rtems.org
http://lists.rtems.org/mailman/listinfo/bugs

Re: [RTEMS Project] #3351: libbsd: Add support for conditional compilation of modules and compiler flags

[RTEMS trac]

#3351: libbsd: Add support for conditional compilation of modules and compiler
flags
+-
 Reporter:  Christian Mauderer  |   Owner:  Christian Mauderer
 Type:  enhancement |  Status:  assigned
 Priority:  normal  |   Milestone:  Indefinite
Component:  network/libbsd  | Version:  5
 Severity:  normal  |  Resolution:
 Keywords:  libbsd waf IPSec|  Blocked By:
 Blocking:  |
+-

Comment (by Chris Johns):

 Should `libbsd.py` we split up into multiple files?

--
Ticket URL: 
RTEMS Project 
RTEMS Project
___
bugs mailing list
bugs@rtems.org
http://lists.rtems.org/mailman/listinfo/bugs

Re: [RTEMS Project] #3351: libbsd: Add support for conditional compilation of modules and compiler flags

[RTEMS trac]

#3351: libbsd: Add support for conditional compilation of modules and compiler
flags
+-
 Reporter:  Christian Mauderer  |   Owner:  Christian Mauderer
 Type:  enhancement |  Status:  assigned
 Priority:  normal  |   Milestone:  Indefinite
Component:  network/libbsd  | Version:  5
 Severity:  normal  |  Resolution:
 Keywords:  libbsd waf IPSec|  Blocked By:
 Blocking:  |
+-

Comment (by Chris Johns):

 Nice summary, thank you.

 Yes to 1, 2, and 3. The variants will evolve and I suspect will be driven
 by user demands. Your initial list is really nice. I really like the first
 one as an easy step for those migrating from the legacy stack.

 I think the 4, 5, and 6 sum up what I was thinking. I am currently
 swinging between the build details being in an INI file or in a waf script
 and so Python. I am concerned moving all the build to INI format creates a
 new build system based on INI files which I would consider a mistake. The
 configuration management of the parts can be an INI file.

 I will take a look at moving the current build to just waf and to remove
 the generate stage and then report back on how this looks and what I find.
 I feel time invested in doing this will benefit how we proceed.

--
Ticket URL: 
RTEMS Project 
RTEMS Project
___
bugs mailing list
bugs@rtems.org
http://lists.rtems.org/mailman/listinfo/bugs

Re: [RTEMS Project] #3351: libbsd: Add support for conditional compilation of modules and compiler flags

[RTEMS trac]

#3351: libbsd: Add support for conditional compilation of modules and compiler
flags
+-
 Reporter:  Christian Mauderer  |   Owner:  Christian Mauderer
 Type:  enhancement |  Status:  assigned
 Priority:  normal  |   Milestone:  Indefinite
Component:  network/libbsd  | Version:  5
 Severity:  normal  |  Resolution:
 Keywords:  libbsd waf IPSec|  Blocked By:
 Blocking:  |
+-

Comment (by Christian Mauderer):

 Thanks for the input. Let me summarize how I understood you.

 1. We try to find an officially supported set of build variants for the
 library. That list should ideally not grow to more than 20 to 30 variants
 over time. At the moment I would see the following three (of course we can
 discuss the exact variants but maybe we should keep that for later (see
 also note at the end regarding SSL and WiFi)):
   - Size optimized with about the same functions as the classic stack: no
 IPv6, no IPsec, maybe no ; `-Os`
   - Normal functionality that should cover most use cases: basically
 everything that is available now; `-O2`
   - IPSec: Normal + IPsec; `-O2`
   - Debug build: Everything; `-Og` or `-O0`

 2. We want to use ini files for configuring the variants.

 3. The system should be build in a way that if the user really wants to
 build an unsupported variant, he can use some own configuration file.

 If that's what you meant I would be OK with this direction. Otherwise
 please correct me.

 Now some more guesswork about the implementation:

 4. The top configuration files would include the following information:
   - The modules to include into the build. That would replace the part
 that is currently done in libbsd.py in the `def sources(mm)` with the
 `mm.addModule(...)` lines.
   - Extra compiler flags (like `-DINET6` or `-Og`) that should be added to
 the ones already used for the modules. Some of them would have to be
 removed from the modules and put here instead (`INET6`, ...)

 5. Later the rest of the `libbsd.py` should be migrated to a ini too. But
 that is not necessarily in the scope of this ticket.

 6. When I'm already busy with changing the build process, it would be good
 to remove the generated `libbsd_waf.py` completely (which most likely
 needs quite some thought).

 Is that about what you have thought?

 How should we deliver the variants? Install them all and tell the user to
 use `-lbsd_small` or `-lbsd_ipsec` when he links?

 ''Note for the build variants:''

 Regarding SSL: I hope that this does not influence how other code is built
 in libbsd. So it should be OK to put it into the base set. It would be
 only linked in if some application uses it. Of course the installed
 headers might have an influence on the user application.

 Regarding WiFi: Maybe it's possible to deactivate it by default without an
 extra build variant. I have already removed quite some big parts by re-
 defining mainly the `_bsd_ifconfig_ieee80211_ctor()` as an empty function.
 If that doesn't work another variant might could be useful.

--
Ticket URL: 
RTEMS Project 
RTEMS Project
___
bugs mailing list
bugs@rtems.org
http://lists.rtems.org/mailman/listinfo/bugs

Re: [RTEMS Project] #3351: libbsd: Add support for conditional compilation of modules and compiler flags

[RTEMS trac]

#3351: libbsd: Add support for conditional compilation of modules and compiler
flags
+-
 Reporter:  Christian Mauderer  |   Owner:  Christian Mauderer
 Type:  enhancement |  Status:  assigned
 Priority:  normal  |   Milestone:  Indefinite
Component:  network/libbsd  | Version:  5
 Severity:  normal  |  Resolution:
 Keywords:  libbsd waf IPSec|  Blocked By:
 Blocking:  |
+-

Comment (by Chris Johns):

 Replying to [comment:2 Christian Mauderer]:
 > If we build multiple variants in one go we might have a lot of different
 variants soon because every option would double the variants.

 Yes. A way to configure and build does not diminish the obligation on all
 developers making changes to build all possible variants or at least an
 agreed number. Over the years we have added options to RTEMS's `configure`
 and has become a problem. If you add on to the possible configure options
 the possible `BSP_OPTS` we have something we can never ever test build not
 even once. I am attempting to control the issue with the `rtems-bsp-
 builder` and it's 1600+ lines of Python. If you review a recent build from
 Joel (1) you will see the minimal set of 13 build variants I could come up
 with generates over 1500 build jobs or separate builds of RTEMS. I would
 like to avoid this as much as possible but recognize it is difficult or
 impossible to do without limiting the features we want.

 (1) https://lists.rtems.org/pipermail/build/2018-March/000553.html

 > The IPSec is one option but we already had an application where we
 disabled IPv6 for one specific use case due to size reasons. Another
 option could be WiFi or no WiFi (which adds quite some code which isn't
 used by many applications). So it would be already eight different
 versions.

 Yes, and while we have used options for this I feel that method may not
 scale. It has been useful up to now but I think we need to consider
 something else.

 > My long-time direction would be to provide options via a configuration
 file (for example some ini). With that it would be possible to use
 different default configuration for some BSPs (for example the ones with /
 without WiFi). A user could provide his own config that matches his
 application. I still have to have a more detailed look over the option
 parser used in waf but I would expect that it can be somehow convinced to
 read options via a file too.

 I would prefer this be a near-term direction and we develop Python code
 and support so we can define a set of agreed "variant builds" . I see a
 "variant" consisting of compile options, ie `-Os` etc, specific defines
 and specific sets of files. The base build would be a bare stack, for
 example no IPv6, SSL, etc, and an IPV6 with SSL would be `IPV6-SSL = base
 + IPV6 + SSL`.

 I am OK with INI files because it is built into Python. YAML would be nice
 but it requires extra host packages and that creates user support issues.
 Please take a look at `tester/rt/check.py` and
 `rtemstoolkit/configuration.py` in the `rtems-tools` repo. It is fine to
 copy down to this project the configuration file support from the
 rtemstoolkit.

 I think it would be best to leave `libbsd.py` as it is for now. I would
 like to move that configuration to an INI format and remove the
 intermediate step of generating `libbsd_waf.py`. There is no real need to
 generate the file other than the former support of `Makefile` builds. This
 means we work on a top level INI design and we can flesh out the migration
 from `libbsd.py` later. The `rtemstoolkit/configuration.py` supports
 includes which allows a tree structure to be formed letting us clip below
 the needed file sets when that change happens.

 The key requirement I would like to have is being able to configuring a
 build for libbsd that builds all variants we agree are important before
 submitting a patch. I do not want to have to write another `rtems-bsp-
 builder`, I would like that functionality built in.

--
Ticket URL: 
RTEMS Project 
RTEMS Project
___
bugs mailing list
bugs@rtems.org
http://lists.rtems.org/mailman/listinfo/bugs

Re: [RTEMS Project] #3351: libbsd: Add support for conditional compilation of modules and compiler flags

[RTEMS trac]

#3351: libbsd: Add support for conditional compilation of modules and compiler
flags
+-
 Reporter:  Christian Mauderer  |   Owner:  Christian Mauderer
 Type:  enhancement |  Status:  assigned
 Priority:  normal  |   Milestone:  Indefinite
Component:  network/libbsd  | Version:  5
 Severity:  normal  |  Resolution:
 Keywords:  libbsd waf IPSec|  Blocked By:
 Blocking:  |
+-

Comment (by Christian Mauderer):

 Just noted: A partial support for setting defines via a waf-option is
 already there: The `--freebsd-option` switch for the waf configure call.
 That one can already set a define. Most likely it would be useful to
 extend or replace that.

--
Ticket URL: 
RTEMS Project 
RTEMS Project
___
bugs mailing list
bugs@rtems.org
http://lists.rtems.org/mailman/listinfo/bugs

Re: [RTEMS Project] #3351: libbsd: Add support for conditional compilation of modules and compiler flags

[RTEMS trac]

#3351: libbsd: Add support for conditional compilation of modules and compiler
flags
+-
 Reporter:  Christian Mauderer  |   Owner:  Christian Mauderer
 Type:  enhancement |  Status:  assigned
 Priority:  normal  |   Milestone:  Indefinite
Component:  network/libbsd  | Version:  5
 Severity:  normal  |  Resolution:
 Keywords:  libbsd waf IPSec|  Blocked By:
 Blocking:  |
+-

Comment (by Christian Mauderer):

 If we build multiple variants in one go we might have a lot of different
 variants soon because every option would double the variants.

 The IPSec is one option but we already had an application where we
 disabled IPv6 for one specific use case due to size reasons. Another
 option could be WiFi or no WiFi (which adds quite some code which isn't
 used by many applications). So it would be already eight different
 versions.

 My long-time direction would be to provide options via a configuration
 file (for example some ini). With that it would be possible to use
 different default configuration for some BSPs (for example the ones with /
 without WiFi). A user could provide his own config that matches his
 application. I still have to have a more detailed look over the option
 parser used in waf but I would expect that it can be somehow convinced to
 read options via a file too.

--
Ticket URL: 
RTEMS Project 
RTEMS Project
___
bugs mailing list
bugs@rtems.org
http://lists.rtems.org/mailman/listinfo/bugs

Re: [RTEMS Project] #3351: libbsd: Add support for conditional compilation of modules and compiler flags

[RTEMS trac]

#3351: libbsd: Add support for conditional compilation of modules and compiler
flags
+-
 Reporter:  Christian Mauderer  |   Owner:  Christian Mauderer
 Type:  enhancement |  Status:  assigned
 Priority:  normal  |   Milestone:  Indefinite
Component:  network/libbsd  | Version:  5
 Severity:  normal  |  Resolution:
 Keywords:  libbsd waf IPSec|  Blocked By:
 Blocking:  |
+-

Comment (by Chris Johns):

 Do we have a single build which creates variant libraries and a user
 selects which variant they want or do we have a single library built for a
 specific purpose?

 I feel building variant libraries is better solution because every build
 by a developer checks all variants, and it aids deployment because you do
 not have users needing to manage specific build instances.

 If we can establish how we want to package libbsd in light of this change
 and the need to have a difference the details will be must easier to
 manage.

--
Ticket URL: 
RTEMS Project 
RTEMS Project
___
bugs mailing list
bugs@rtems.org
http://lists.rtems.org/mailman/listinfo/bugs