Thanks, applied as 9a631d7435ef7d012bdccc4354d37c446873f4e4. Michael
[sent from post-receive hook] On Tue, 29 Jun 2021 07:09:16 +0200, Roland Hieber <[email protected]> wrote: > The "Adding new packages" section has become quite long already, to the > point that some headings have reached level five and no longer make it > into the table of contents. The subsections "Advanced rule files" and > "Patching packages" are also not only relevant for new packages, so > propagate them to their own sections. > > Signed-off-by: Roland Hieber <[email protected]> > Message-Id: <[email protected]> > Signed-off-by: Michael Olbrich <[email protected]> > > diff --git a/doc/dev_add_new_pkgs.rst b/doc/dev_add_new_pkgs.rst > index 0dbc7a266ea8..3506436d78ec 100644 > --- a/doc/dev_add_new_pkgs.rst > +++ b/doc/dev_add_new_pkgs.rst > @@ -648,692 +648,3 @@ repository and its content. It contains a list of known > licenses based on their > SPDX identifier. The content is without formatting to simplify text search. > > https://github.com/spdx/license-list-data/tree/master/text > - > -Advanced Rule Files > -~~~~~~~~~~~~~~~~~~~ > - > -The previous example on how to create a rule file sometimes works as > -shown above. But most of the time source archives are not that simple. > -In this section we want to give the user a more detailed selection how > -the package will be built. > - > -Adding Static Configure Parameters > -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > - > -The ``configure`` scripts of various source archives provide additional > -parameters to enable or disable features, or to configure them in a > -specific way. > - > -We assume the ``configure`` script of our ``foo`` example (refer to > -section :ref:`foo_example`) supports two additional parameters: > - > -- **--enable-debug**: Make the program more noisy. It’s disabled by > - default. > - > -- **--with-bar**: Also build the special executable **bar**. Building > - this executable is also disabled by default. > - > -We now want to forward these options to the ``configure`` script when it > -runs in the *prepare* stage. To do so, we must again open the rule file > -with our favourite editor and navigate to the *prepare* stage entry. > - > -PTXdist uses the variable ``FOO_CONF_OPT`` as the list of parameters to > -be given to ``configure``. > - > -Currently this variable is commented out and defined to: > - > -.. code-block:: make > - > - # FOO_CONF_OPT := $(CROSS_AUTOCONF_USR) > - > -The variable ``CROSS_AUTOCONF_USR`` is predefined by PTXdist and > -contains all basic parameters to instruct ``configure`` to prepare for a > -**cross** compile environment. > - > -To use the two additional mentioned ``configure`` parameters, we comment > -in this line and supplement this expression as follows: > - > -.. code-block:: make > - > - FOO_CONF_OPT := \ > - $(CROSS_AUTOCONF_USR) \ > - --enable-debug \ > - --with-bar > - > -.. note:: We recommend to use this format with each parameter on a line of > - its own. This format is easier to read and a diff shows more exactly any > - change. > - > -To do a fast check if this addition was successful, we run: > - > -.. code-block:: text > - > - $ ptxdist print FOO_CONF_OPT > - --prefix=/usr --sysconfdir=/etc --host=|ptxdistCompilerName| > --build=i686-host-linux-gnu --enable-debug --with-bar > - > -.. note:: It depends on the currently selected platform and its architecture > - what content this variable will have. The content shown above is an > - example for a target. > - > -Or re-build the package with the new settings: > - > -.. code-block:: text > - > - $ ptxdist drop foo prepare > - $ ptxdist targetinstall foo > - > -Adding Dynamic Configure Parameters > -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > - > -Sometimes it makes sense to add this kind of parameters on demand only; > -especially a parameter like ``--enable-debug``. To let the user decide > -if this parameter is to be used or not, we must add a menu entry. So, > -let’s expand our menu. Here is its current content: > - > -.. code-block:: kconfig > - > - ## SECTION=project_specific > - > - config FOO > - tristate > - prompt "foo" > - help > - FIXME > - > -We’ll add two menu entries, one for each optional parameter we want to > -add on demand to the ``configure`` parameters: > - > -.. code-block:: kconfig > - > - ## SECTION=project_specific > - > - config FOO > - tristate > - prompt "foo" > - help > - FIXME > - > - if FOO > - config FOO_DEBUG > - bool > - prompt "add debug noise" > - > - config FOO_BAR > - bool > - prompt "build bar" > - > - endif > - > -.. important:: Always follow the rule to extend the base name by a suboption > - name as the trailing part of the variable name. This gives PTXdist the > ability > - to detect a change in the package’s settings (via menuconfig) to force its > - rebuild on demand. > - > -To make usage of the new menu entries, we must check them in the rule > -file and add the correct parameters: > - > -.. code-block:: make > - > - # > - # autoconf > - # > - FOO_CONF_OPT := \ > - $(CROSS_AUTOCONF_USR) \ > - --$(call ptx/endis, PTXCONF_FOO_DEBUG)-debug \ > - --$(call ptx/wwo, PTXCONF_FOO_BAR)-bar > - > -.. important:: Please note the leading ``PTXCONF_`` for each define. While > Kconfig is > - using ``FOO_BAR``, the rule file must use ``PTXCONF_FOO_BAR`` instead. > - > -.. note:: Refer :ref:`Rule File Macro Reference <param_macros>` for further > - details about these special kind of option macros (e.g. ``ptx/...``). > - > -It is a good practice to always add both settings, e.g. ``--disable-debug`` > -even if this is the default case. Sometimes ``configure`` tries to guess > -something and the binary result might differ depending on the build > -order. For example some kind of package would also build some X related > -tools, if X libraries are found. In this case it depends on the build > -order, if the X related tools are built or not. All the autocheck > -features are problematic here. So, if we do not want ``configure`` to > -guess its settings we **must disable everything we do not want**. > - > -To support this process, PTXdist supplies a helper script, located at > -``/path/to/ptxdist/scripts/configure-helper.py`` that compares the configure > -output with the settings from ``FOO_CONF_OPT``: > - > -.. code-block:: text > - > - $ /opt/ptxdist-2017.06.0/scripts/configure-helper.py -p libsigrok > - --- rules/libsigrok.make > - +++ libsigrok-0.5.0 > - @@ -4,3 +4,74 @@ > - --libdir=/usr/lib > - --build=x86_64-host-linux-gnu > - --host=arm-v7a-linux-gnueabihf > - + --enable-warnings=min|max|fatal|no > - + --disable-largefile > - + --enable-all-drivers > - + --enable-agilent-dmm > - [...] > - + --enable-ruby > - + --enable-java > - + --without-libserialport > - + --without-libftdi > - + --without-libusb > - + --without-librevisa > - + --without-libgpib > - + --without-libieee1284 > - + --with-jni-include-path=DIR-LIST > - > -In this example, many configure options from libsigrok (marked with ``+``) > -are not yet present in ``LIBSIGROK_CONF_OPT`` and must be added, possibly > also > -by providing more dynamic options in the package definition. > - > -If some parts of a package are built on demand only, they must also be > -installed on demand only. Besides the *prepare* stage, we also must > -modify our *targetinstall* stage: > - > -.. code-block:: make > - > - @$(call install_copy, foo, 0, 0, 0755, $(FOO_DIR)/foo, /usr/bin/foo) > - > - ifdef PTXCONF_FOO_BAR > - @$(call install_copy, foo, 0, 0, 0755, $(FOO_DIR)/bar, /usr/bin/bar) > - endif > - > - @$(call install_finish, foo) > - @$(call touch) > - > -Now we can play with our new menu entries and check if they are working > -as expected: > - > -.. code-block:: text > - > - $ ptxdist menuconfig > - $ ptxdist targetinstall foo > - > -Whenever we change a *FOO* related menu entry, PTXdist should detect it > -and re-build the package when a new build is started. > - > -.. _external_dependencies: > - > -Managing External Compile Time Dependencies > -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > - > -While running the prepare stage, it could happen that it fails due to a > -missing external dependency. > - > -For example: > - > -.. code-block:: text > - > - checking whether zlib exists....failed > - > -In this example, our new package depends on the compression library > -*zlib*. PTXdist comes with a target *zlib*. All we need to do in this > -case is to declare that our new package *foo* depends on *zlib*. This > -kind of dependency is managed in the menu file of our new package by > -simply adding the ``select ZLIB`` line. After this addition our menu > -file looks like: > - > -.. code-block:: kconfig > - > - ## SECTION=project_specific > - > - config FOO > - tristate > - select ZLIB > - prompt "foo" > - help > - FIXME > - > - if FOO > - config FOO_DEBUG > - bool > - prompt "add debug noise" > - > - config FOO_BAR > - bool > - prompt "build bar" > - > - endif > - > -PTXdist now builds the *zlib* first and our new package thereafter. > - > -Refer :ref:`external_dependencies_variants` for more specific dependency > -description. > - > -Managing External Compile Time Dependencies on Demand > -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > - > -It is good practice to add only those dependencies that are really > -required for the current configuration of the package. If the package > -provides the features *foo* and *bar* and its ``configure`` provides > -switches to enable/disable them independently, we can also add > -dependencies on demand. Let’s assume feature *foo* needs the compression > -library *libz* and *bar* needs the XML2 library *libxml2*. These > -libraries are only required at run-time if the corresponding feature is > -enabled. To add these dependencies on demand, the menu file looks like: > - > -.. code-block:: kconfig > - > - ## SECTION=project_specific > - > - config FOO > - tristate > - select ZLIB if FOO_FOO > - select LIBXML2 if FOO_BAR > - prompt "foo" > - help > - FIXME > - > - if FOO > - config FOO_DEBUG > - bool > - prompt "add debug noise" > - > - config FOO_FOO > - bool > - prompt "build foo" > - > - config FOO_BAR > - bool > - prompt "build bar" > - > - endif > - > -.. important:: Do not add these ``select`` statements to the corresponding > menu entry. > - They must belong to the main menu entry of the package to ensure that > - the calculation of the dependencies between the packages is done in a > - correct manner. > - > -Managing External Runtime Dependencies > -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > - > -Some packages are building all of their components and also installing > -them into the target’s sysroot. But only their *targetinstall* stage > -decides which parts are copied to the root filesystem. So, compiling and > -linking of our package will work, because everything required is found > -in the target’s sysroot. > - > -In our example there is a hidden dependency to the math library > -``libm``. Our new package was built successfully, because the linker was > -able to link our binaries against the ``libm`` from the toolchain. But > -in this case the ``libm`` must also be available in the target’s root > -filesystem to fulfill the run-time dependency: We have to force PTXdist to > -install ``libm``. ``libm`` is part of the *glibc* package, but is not > -installed by default (to keep the root filesystem small). So, it **does > -not** help to select the ``GLIBC`` symbol, to get a ``libm`` at run-time. > - > -The correct solution here is to add a ``select LIBC_M`` to our menu > -file. With all the additions above it now looks like: > - > -.. code-block:: kconfig > - > - ## SECTION=project_specific > - > - config FOO > - tristate > - select ZLIB if FOO_FOO > - select LIBXML2 if FOO_BAR > - select LIBC_M > - prompt "foo" > - help > - FIXME > - > - if FOO > - config FOO_DEBUG > - bool > - prompt "add debug noise" > - > - config FOO_FOO > - bool > - prompt "build foo" > - > - config FOO_BAR > - bool > - prompt "build bar" > - > - endif > - > -.. note:: There are other packages around, that do not install everything by > - default. If our new package needs something special, we must take a look > - into the menu of the other package how to force the required components > - to be installed and add the corresponding ``selects`` to our own menu > - file. In this case it does not help to enable the required parts in our > - project configuration, because this has no effect on the build order! > - > -Managing Plain Makefile Packages > -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > - > -Many packages are still coming with a plain ``Makefile``. The user has > -to adapt it to make it work in a cross compile environment as well. > -PTXdist can also handle this kind of packages. We only have to specify > -a special *prepare* and *compile* stage. > - > -Such packages often have no special need for any kind of preparation. In > -this we must instruct PTXdist to do nothing in the *prepare* stage: > - > -.. code-block:: make > - > - FOO_CONF_TOOL := NO > - > -To compile the package, we can use ``make``\ ’s feature to overwrite > -variables used in the ``Makefile``. With this feature we can still use > -the original ``Makefile`` but with our own (cross compile) settings. > - > -Most of the time the generic compile rule can be used, only a few > -settings are required. For a well defined ``Makefile`` it is sufficient to > -set up the correct cross compile environment for the *compile* stage: > - > -.. code-block:: make > - > - FOO_MAKE_ENV := $(CROSS_ENV) > - > -``make`` will be called in this case with: > - > -``$(FOO_MAKE_ENV) $(MAKE) -C $(FOO_DIR) $(FOO_MAKE_OPT)`` > - > -So, in the rule file only the two variables ``FOO_MAKE_ENV`` and > -``FOO_MAKE_OPT`` must be set, to forward the required settings to the > -package’s buildsystem. If the package cannot be built in parallel, we > -can also add the ``FOO_MAKE_PAR := NO``. ``YES`` is the default. > - > -Managing CMake/QMake/Meson Packages > -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > - > -Building packages that use ``cmake``, ``qmake`` or ``meson`` is much like > -building packages with an autotools based buildsystem. We need to specify > -the configuration tool: > - > -.. code-block:: make > - > - FOO_CONF_TOOL := cmake > - > -or > - > -.. code-block:: make > - > - FOO_CONF_TOOL := qmake > - > -or respectively > - > -.. code-block:: make > - > - FOO_CONF_TOOL := meson > - > -And provide the correct configuration options. The syntax is different so > -PTXdist provides additional macros to simplify configurable features. > -For ``cmake`` the configuration options typically look like this: > - > -.. code-block:: make > - > - FOO_CONF_OPT := \ > - $(CROSS_CMAKE_USR) \ > - -DBUILD_TESTS:BOOL=OFF \ > - -DENABLE_BAR:BOOL=$(call ptx/onoff, PTXCONF_FOO_BAR) > - > -For ``qmake`` the configuration options typically look like this: > - > -.. code-block:: make > - > - FOO_CONF_OPT := \ > - $(CROSS_QMAKE_OPT) \ > - PREFIX=/usr > - > -And for ``meson`` the configuration options typically look like this: > - > -.. code-block:: make > - > - FOO_CONF_OPT := \ > - $(CROSS_MESON_USR) \ > - -Dbar=$(call ptx/truefalse,PTXCONF_FOO_BAR) > - > -Please note that currently only host and target ``cmake``\/``meson`` packages > -and only target ``qmake`` packages are supported. > - > -Managing Python Packages > -^^^^^^^^^^^^^^^^^^^^^^^^ > - > -As with any other package, the correct configuration tool must be selected > -for Python packages: > - > -.. code-block:: make > - > - FOO_CONF_TOOL := python > - > -.. note:: For Python3 packages the value must be ``python3``. > - > -No Makefiles are used when building Python packages so the usual ``make`` > -and ``make install`` for the *compile* and *install* stages cannot be used. > -PTXdist will call ``python setup.py build`` and ``python setup.py install`` > -instead. > - > -.. note:: *FOO* is still the name of our example package. It must be > - replaced by the real package name. > - > - > -.. _patching_packages: > - > -Patching Packages > -~~~~~~~~~~~~~~~~~ > - > -There can be various reasons why a package must be patched: > - > -- Package is broken for cross compile environments > - > -- Package is broken within a specific feature > - > -- Package is vulnerable and needs some fixes > - > -- or anything else (this case is the most common one) > - > -Ideally, those problems should be addressed in the original project, > -so any patches you add to your BSP or to PTXdist should also be submitted > upstream. > -The upstream project can often provide better feedback, they can integrate > your > -patch into a new release, and also maintain your changes as part of the > project. > -This way we make sure that all advantages of the open source idea work for > us; > -and your patch can be removed again later when a new release of the project > is > -integrated into your BSP or into PTXdist. > - > -PTXdist handles patching automatically. > -After extracting the archive of a package, PTXdist checks for the existence > of > -a patch directory named like its ``<PKG>_PATCHES`` variable, or, if this > variable > -is not set, like its ``<PKG>`` variable. > -The patch directory is then searched in all locations listed by the > -``PTXDIST_PATH_PATCHES`` variable, and the first one found is used. > -Take an exemplary package ``foo`` with version ``1.1.0``: > -The variable ``FOO`` will have the value ``foo-1.1.0``, so PTXdist will look > for > -a patch directory named ``foo-1.1.0`` in the following locations: > - > -#. the current layer: > - > - a. project (``./patches/foo-1.1.0``) > - b. platform (``./configs/|ptxdistPlatformConfigDir|/patches/foo-1.1.0``) > - > -#. any :ref:`base layers <layers-in-ptxdist>`, > - applying the same search order as above for each layer recursively > - > -#. ptxdist (``<ptxdist/installation/path>/patches/foo-1.1.0``) > - > -The patches from the first location found are used. Note: Due to this > -search order, a PTXdist project can replace global patches from the > -PTXdist installation. This can be useful if a project sticks to a > -specific PTXdist revision but fixes from a more recent revision of > -PTXdist should be used. > - > -PTXdist uses the utilities *git*, *patch* or *quilt* to work with > -patches or patch series. We recommend *git*, as it can manage patch > -series in a very easy way. > - > -Creating a Patch Series for a Package > -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > - > -To create a patch series for the first time, we can run the following > -steps. We are still using our *foo-1.1.0* example package here: > - > -Using Quilt > -""""""""""" > - > -We create a special directory for the patch series in the local project > -directory: > - > -.. code-block:: text > - > - $ mkdir -p patches/foo-1.1.0 > - > -PTXdist expects a ``series`` file in the patch directory and at least > -one patch. Otherwise it fails. Due to the fact that we do not have any > -patch content yet, we’ll start with a dummy entry in the ``series`` file > -and an empty ``patch`` file. > - > -.. code-block:: text > - > - $ touch patches/foo-1.1.0/dummy > - $ echo dummy > patches/foo-1.1.0/series > - > -Next is to extract the package (if already done, we must remove it > -first): > - > -.. code-block:: text > - > - $ ptxdist extract foo > - > -This will extract the archive and create a symbolic link in the build > -directory pointing to our local patch directory. Working this way will > -ensure that we do not lose our created patches if we enter > -``ptxdist clean foo`` by accident. In our case the patches are still > -present in ``patches/foo-1.1.0`` and can be used the next time we > -extract the package again. > - > -All we have to do now is to do the modification we need to make the > -package work. We change into the build directory and use quilt_ to > -create new patches, add files to respective patches, modify these files > -and refresh the patches to save our changes. > -See the *quilt* documentation (``man 1 quilt``) for more information. > - > -.. note:: For patches that are intended for PTXdist upstream use the git > - workflow described below to get proper patch headers. > - > -.. _quilt: http://savannah.nongnu.org/projects/quilt > - > -Using Git > -""""""""" > - > -Create the patch directory like above for *quilt*, > -but only add an empty series file: > - > -.. code-block:: text > - > - $ mkdir -p patches/foo-1.1.0 > - $ touch patches/foo-1.1.0/series > - > -Then extract the package with an additional command line switch: > - > -.. code-block:: text > - > - $ ptxdist --git extract foo > - > -The empty series file makes PTXdist create a Git repository in the > -respective package build directory, > -and import the package source as the first commit. > - > -.. note:: Optionally, you can enable the setting *Developer Options → > - use git to apply patches* in `ptxdist setup` to get this behaviour > - as a default for every package. > - However, note that this setting is meant for development only, and can lead > - to failures – some packages try to determine if they are being compiled > from > - a Git source tree, and behave differently in that case. > - > -Then you can change into the package build directory > -(``platform-<name>/build-target/foo-1.1.0``), > -patch the required source files, > -and make Git commits on the way. > -The Git history should now look something like this: > - > -.. code-block:: text > - > - $ git log --oneline --decorate > - * df343e821851 (HEAD -> master) Makefile: don't build the tests > - * 65a360c2bd60 strfry.c: frobnicate the excusator > - * fdc315f6844c (tag: foobar-1.1.0, tag: base) initial commit > - > -Finally, call ``git ptx-patches`` to transform those Git commits into the > patch > -series in the ``patches/foo-1.1.0`` folder. > -This way they don't get lost when cleaning the package. > - > -.. note:: PTXdist will only create a Git repository for packages with > - patches. To use Git to generate the first patch, create an empty series > - file ``patches/foobar-1.1.0/series`` before extracting the packages. This > - will tell PTXdist to use Git anyways and ``git ptx-patches`` will put the > - patches there. > - > -Both approaches (Git and quilt) are not suitable for modifying files > -that are autogenerated in autotools-based buildsystems. > -Refer to the section :ref:`configure_rebuild` on how PTXdist can > -handle this special task. > - > -Adding More Patches to a Package > -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > - > -If we want to add more patches to an already patched package, we can use > -nearly the same way as creating patches for the first time. But if the > -patch series comes from the PTXdist main installation, we do not have > -write permissions to these directories (do NEVER work on the main > -installation directories, NEVER, NEVER, NEVER). Due to the search order > -in which PTXdist searches for patches for a specific package, we can > -copy the global patch series to our local project directory. Now we have > -the permissions to add more patches or modify the existing ones. Also > -*quilt* and *git* are our friends here to manage the patch series. > - > -If we think that our new patches are valuable also for others, or they > -fix an error, it could be a good idea to send these patches to PTXdist > -mainline, and to the upstream project too. > - > - > -.. _configure_rebuild: > - > -Modifying Autotoolized Packages > -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > - > -Autotoolized packages are very picky when automatically generated files > -get patched. The patch order is very important in this case and > -sometimes it even fails and nobody knows why. > - > -To improve a package’s autotools-based build system, PTXdist comes with > -its own project local autotools to regenerate the autotools template > -files, instead of patching them. With this feature, only the template > -files must be patched, the required ``configure`` script and the > -``Makefile.in`` files are regenerated in the final stages of the > -*prepare* step. > - > -This feature works like the regular patching mechanism. The only > -difference is the additional ``autogen.sh`` file in the patch directory. > -If it exists and has execution permissions, it will be called after the > -package was patched (while the *extract* stage is running). > - > -Its content depends on developer needs; for the most simple case the > -content can be: > - > -.. code-block:: bash > - > - #!/bin/bash > - > - aclocal $ACLOCAL_FLAGS > - > - libtoolize \ > - --force \ > - --copy > - > - autoreconf \ > - --force \ > - --install \ > - --warnings=cross \ > - --warnings=syntax \ > - --warnings=obsolete \ > - --warnings=unsupported > - > -.. note:: In this way not yet autotoolized package can be autotoolized. We > - just have to add the common autotool template files (``configure.ac`` > - and ``Makefile.am`` for example) via a patch series to the package > - source and the ``autogen.sh`` to the patch directory. > diff --git a/doc/dev_advanced_rule_files.rst b/doc/dev_advanced_rule_files.rst > new file mode 100644 > index 000000000000..794bd148cbae > --- /dev/null > +++ b/doc/dev_advanced_rule_files.rst > @@ -0,0 +1,463 @@ > +******************* > +Advanced Rule Files > +******************* > + > +The previous example on how to create a rule file sometimes works as > +shown above. But most of the time source archives are not that simple. > +In this section we want to give the user a more detailed selection how > +the package will be built. > + > +Adding Static Configure Parameters > +================================== > + > +The ``configure`` scripts of various source archives provide additional > +parameters to enable or disable features, or to configure them in a > +specific way. > + > +We assume the ``configure`` script of our ``foo`` example (refer to > +section :ref:`foo_example`) supports two additional parameters: > + > +- **--enable-debug**: Make the program more noisy. It’s disabled by > + default. > + > +- **--with-bar**: Also build the special executable **bar**. Building > + this executable is also disabled by default. > + > +We now want to forward these options to the ``configure`` script when it > +runs in the *prepare* stage. To do so, we must again open the rule file > +with our favourite editor and navigate to the *prepare* stage entry. > + > +PTXdist uses the variable ``FOO_CONF_OPT`` as the list of parameters to > +be given to ``configure``. > + > +Currently this variable is commented out and defined to: > + > +.. code-block:: make > + > + # FOO_CONF_OPT := $(CROSS_AUTOCONF_USR) > + > +The variable ``CROSS_AUTOCONF_USR`` is predefined by PTXdist and > +contains all basic parameters to instruct ``configure`` to prepare for a > +**cross** compile environment. > + > +To use the two additional mentioned ``configure`` parameters, we comment > +in this line and supplement this expression as follows: > + > +.. code-block:: make > + > + FOO_CONF_OPT := \ > + $(CROSS_AUTOCONF_USR) \ > + --enable-debug \ > + --with-bar > + > +.. note:: We recommend to use this format with each parameter on a line of > + its own. This format is easier to read and a diff shows more exactly any > + change. > + > +To do a fast check if this addition was successful, we run: > + > +.. code-block:: text > + > + $ ptxdist print FOO_CONF_OPT > + --prefix=/usr --sysconfdir=/etc --host=|ptxdistCompilerName| > --build=i686-host-linux-gnu --enable-debug --with-bar > + > +.. note:: It depends on the currently selected platform and its architecture > + what content this variable will have. The content shown above is an > + example for a target. > + > +Or re-build the package with the new settings: > + > +.. code-block:: text > + > + $ ptxdist drop foo prepare > + $ ptxdist targetinstall foo > + > +Adding Dynamic Configure Parameters > +=================================== > + > +Sometimes it makes sense to add this kind of parameters on demand only; > +especially a parameter like ``--enable-debug``. To let the user decide > +if this parameter is to be used or not, we must add a menu entry. So, > +let’s expand our menu. Here is its current content: > + > +.. code-block:: kconfig > + > + ## SECTION=project_specific > + > + config FOO > + tristate > + prompt "foo" > + help > + FIXME > + > +We’ll add two menu entries, one for each optional parameter we want to > +add on demand to the ``configure`` parameters: > + > +.. code-block:: kconfig > + > + ## SECTION=project_specific > + > + config FOO > + tristate > + prompt "foo" > + help > + FIXME > + > + if FOO > + config FOO_DEBUG > + bool > + prompt "add debug noise" > + > + config FOO_BAR > + bool > + prompt "build bar" > + > + endif > + > +.. important:: Always follow the rule to extend the base name by a suboption > + name as the trailing part of the variable name. This gives PTXdist the > ability > + to detect a change in the package’s settings (via menuconfig) to force its > + rebuild on demand. > + > +To make usage of the new menu entries, we must check them in the rule > +file and add the correct parameters: > + > +.. code-block:: make > + > + # > + # autoconf > + # > + FOO_CONF_OPT := \ > + $(CROSS_AUTOCONF_USR) \ > + --$(call ptx/endis, PTXCONF_FOO_DEBUG)-debug \ > + --$(call ptx/wwo, PTXCONF_FOO_BAR)-bar > + > +.. important:: Please note the leading ``PTXCONF_`` for each define. While > Kconfig is > + using ``FOO_BAR``, the rule file must use ``PTXCONF_FOO_BAR`` instead. > + > +.. note:: Refer :ref:`Rule File Macro Reference <param_macros>` for further > + details about these special kind of option macros (e.g. ``ptx/...``). > + > +It is a good practice to always add both settings, e.g. ``--disable-debug`` > +even if this is the default case. Sometimes ``configure`` tries to guess > +something and the binary result might differ depending on the build > +order. For example some kind of package would also build some X related > +tools, if X libraries are found. In this case it depends on the build > +order, if the X related tools are built or not. All the autocheck > +features are problematic here. So, if we do not want ``configure`` to > +guess its settings we **must disable everything we do not want**. > + > +To support this process, PTXdist supplies a helper script, located at > +``/path/to/ptxdist/scripts/configure-helper.py`` that compares the configure > +output with the settings from ``FOO_CONF_OPT``: > + > +.. code-block:: text > + > + $ /opt/ptxdist-2017.06.0/scripts/configure-helper.py -p libsigrok > + --- rules/libsigrok.make > + +++ libsigrok-0.5.0 > + @@ -4,3 +4,74 @@ > + --libdir=/usr/lib > + --build=x86_64-host-linux-gnu > + --host=arm-v7a-linux-gnueabihf > + + --enable-warnings=min|max|fatal|no > + + --disable-largefile > + + --enable-all-drivers > + + --enable-agilent-dmm > + [...] > + + --enable-ruby > + + --enable-java > + + --without-libserialport > + + --without-libftdi > + + --without-libusb > + + --without-librevisa > + + --without-libgpib > + + --without-libieee1284 > + + --with-jni-include-path=DIR-LIST > + > +In this example, many configure options from libsigrok (marked with ``+``) > +are not yet present in ``LIBSIGROK_CONF_OPT`` and must be added, possibly > also > +by providing more dynamic options in the package definition. > + > +If some parts of a package are built on demand only, they must also be > +installed on demand only. Besides the *prepare* stage, we also must > +modify our *targetinstall* stage: > + > +.. code-block:: make > + > + @$(call install_copy, foo, 0, 0, 0755, $(FOO_DIR)/foo, /usr/bin/foo) > + > + ifdef PTXCONF_FOO_BAR > + @$(call install_copy, foo, 0, 0, 0755, $(FOO_DIR)/bar, /usr/bin/bar) > + endif > + > + @$(call install_finish, foo) > + @$(call touch) > + > +Now we can play with our new menu entries and check if they are working > +as expected: > + > +.. code-block:: text > + > + $ ptxdist menuconfig > + $ ptxdist targetinstall foo > + > +Whenever we change a *FOO* related menu entry, PTXdist should detect it > +and re-build the package when a new build is started. > + > +.. _external_dependencies: > + > +Managing External Compile Time Dependencies > +=========================================== > + > +While running the prepare stage, it could happen that it fails due to a > +missing external dependency. > + > +For example: > + > +.. code-block:: text > + > + checking whether zlib exists....failed > + > +In this example, our new package depends on the compression library > +*zlib*. PTXdist comes with a target *zlib*. All we need to do in this > +case is to declare that our new package *foo* depends on *zlib*. This > +kind of dependency is managed in the menu file of our new package by > +simply adding the ``select ZLIB`` line. After this addition our menu > +file looks like: > + > +.. code-block:: kconfig > + > + ## SECTION=project_specific > + > + config FOO > + tristate > + select ZLIB > + prompt "foo" > + help > + FIXME > + > + if FOO > + config FOO_DEBUG > + bool > + prompt "add debug noise" > + > + config FOO_BAR > + bool > + prompt "build bar" > + > + endif > + > +PTXdist now builds the *zlib* first and our new package thereafter. > + > +Refer :ref:`external_dependencies_variants` for more specific dependency > +description. > + > +Managing External Compile Time Dependencies on Demand > +===================================================== > + > +It is good practice to add only those dependencies that are really > +required for the current configuration of the package. If the package > +provides the features *foo* and *bar* and its ``configure`` provides > +switches to enable/disable them independently, we can also add > +dependencies on demand. Let’s assume feature *foo* needs the compression > +library *libz* and *bar* needs the XML2 library *libxml2*. These > +libraries are only required at run-time if the corresponding feature is > +enabled. To add these dependencies on demand, the menu file looks like: > + > +.. code-block:: kconfig > + > + ## SECTION=project_specific > + > + config FOO > + tristate > + select ZLIB if FOO_FOO > + select LIBXML2 if FOO_BAR > + prompt "foo" > + help > + FIXME > + > + if FOO > + config FOO_DEBUG > + bool > + prompt "add debug noise" > + > + config FOO_FOO > + bool > + prompt "build foo" > + > + config FOO_BAR > + bool > + prompt "build bar" > + > + endif > + > +.. important:: Do not add these ``select`` statements to the corresponding > menu entry. > + They must belong to the main menu entry of the package to ensure that > + the calculation of the dependencies between the packages is done in a > + correct manner. > + > +Managing External Runtime Dependencies > +====================================== > + > +Some packages are building all of their components and also installing > +them into the target’s sysroot. But only their *targetinstall* stage > +decides which parts are copied to the root filesystem. So, compiling and > +linking of our package will work, because everything required is found > +in the target’s sysroot. > + > +In our example there is a hidden dependency to the math library > +``libm``. Our new package was built successfully, because the linker was > +able to link our binaries against the ``libm`` from the toolchain. But > +in this case the ``libm`` must also be available in the target’s root > +filesystem to fulfill the run-time dependency: We have to force PTXdist to > +install ``libm``. ``libm`` is part of the *glibc* package, but is not > +installed by default (to keep the root filesystem small). So, it **does > +not** help to select the ``GLIBC`` symbol, to get a ``libm`` at run-time. > + > +The correct solution here is to add a ``select LIBC_M`` to our menu > +file. With all the additions above it now looks like: > + > +.. code-block:: kconfig > + > + ## SECTION=project_specific > + > + config FOO > + tristate > + select ZLIB if FOO_FOO > + select LIBXML2 if FOO_BAR > + select LIBC_M > + prompt "foo" > + help > + FIXME > + > + if FOO > + config FOO_DEBUG > + bool > + prompt "add debug noise" > + > + config FOO_FOO > + bool > + prompt "build foo" > + > + config FOO_BAR > + bool > + prompt "build bar" > + > + endif > + > +.. note:: There are other packages around, that do not install everything by > + default. If our new package needs something special, we must take a look > + into the menu of the other package how to force the required components > + to be installed and add the corresponding ``selects`` to our own menu > + file. In this case it does not help to enable the required parts in our > + project configuration, because this has no effect on the build order! > + > +Managing Plain Makefile Packages > +================================ > + > +Many packages are still coming with a plain ``Makefile``. The user has > +to adapt it to make it work in a cross compile environment as well. > +PTXdist can also handle this kind of packages. We only have to specify > +a special *prepare* and *compile* stage. > + > +Such packages often have no special need for any kind of preparation. In > +this we must instruct PTXdist to do nothing in the *prepare* stage: > + > +.. code-block:: make > + > + FOO_CONF_TOOL := NO > + > +To compile the package, we can use ``make``\ ’s feature to overwrite > +variables used in the ``Makefile``. With this feature we can still use > +the original ``Makefile`` but with our own (cross compile) settings. > + > +Most of the time the generic compile rule can be used, only a few > +settings are required. For a well defined ``Makefile`` it is sufficient to > +set up the correct cross compile environment for the *compile* stage: > + > +.. code-block:: make > + > + FOO_MAKE_ENV := $(CROSS_ENV) > + > +``make`` will be called in this case with: > + > +``$(FOO_MAKE_ENV) $(MAKE) -C $(FOO_DIR) $(FOO_MAKE_OPT)`` > + > +So, in the rule file only the two variables ``FOO_MAKE_ENV`` and > +``FOO_MAKE_OPT`` must be set, to forward the required settings to the > +package’s buildsystem. If the package cannot be built in parallel, we > +can also add the ``FOO_MAKE_PAR := NO``. ``YES`` is the default. > + > +Managing CMake/QMake/Meson Packages > +=================================== > + > +Building packages that use ``cmake``, ``qmake`` or ``meson`` is much like > +building packages with an autotools based buildsystem. We need to specify > +the configuration tool: > + > +.. code-block:: make > + > + FOO_CONF_TOOL := cmake > + > +or > + > +.. code-block:: make > + > + FOO_CONF_TOOL := qmake > + > +or respectively > + > +.. code-block:: make > + > + FOO_CONF_TOOL := meson > + > +And provide the correct configuration options. The syntax is different so > +PTXdist provides additional macros to simplify configurable features. > +For ``cmake`` the configuration options typically look like this: > + > +.. code-block:: make > + > + FOO_CONF_OPT := \ > + $(CROSS_CMAKE_USR) \ > + -DBUILD_TESTS:BOOL=OFF \ > + -DENABLE_BAR:BOOL=$(call ptx/onoff, PTXCONF_FOO_BAR) > + > +For ``qmake`` the configuration options typically look like this: > + > +.. code-block:: make > + > + FOO_CONF_OPT := \ > + $(CROSS_QMAKE_OPT) \ > + PREFIX=/usr > + > +And for ``meson`` the configuration options typically look like this: > + > +.. code-block:: make > + > + FOO_CONF_OPT := \ > + $(CROSS_MESON_USR) \ > + -Dbar=$(call ptx/truefalse,PTXCONF_FOO_BAR) > + > +Please note that currently only host and target ``cmake``\/``meson`` packages > +and only target ``qmake`` packages are supported. > + > +Managing Python Packages > +======================== > + > +As with any other package, the correct configuration tool must be selected > +for Python packages: > + > +.. code-block:: make > + > + FOO_CONF_TOOL := python > + > +.. note:: For Python3 packages the value must be ``python3``. > + > +No Makefiles are used when building Python packages so the usual ``make`` > +and ``make install`` for the *compile* and *install* stages cannot be used. > +PTXdist will call ``python setup.py build`` and ``python setup.py install`` > +instead. > + > +.. note:: *FOO* is still the name of our example package. It must be > + replaced by the real package name. > diff --git a/doc/dev_manual.rst b/doc/dev_manual.rst > index c232cc91428a..e9a88c1a97f5 100644 > --- a/doc/dev_manual.rst > +++ b/doc/dev_manual.rst > @@ -11,6 +11,8 @@ This chapter shows all (or most) of the details of how > PTXdist works. > > dev_dir_hierarchy > dev_add_new_pkgs > + dev_advanced_rule_files > + dev_patching > dev_add_bin_only_files > dev_create_new_pkg_templates > dev_layers_in_ptxdist > diff --git a/doc/dev_patching.rst b/doc/dev_patching.rst > new file mode 100644 > index 000000000000..e3690dfe1b48 > --- /dev/null > +++ b/doc/dev_patching.rst > @@ -0,0 +1,226 @@ > +.. _patching_packages: > + > +***************** > +Patching Packages > +***************** > + > +There can be various reasons why a package must be patched: > + > +- Package is broken for cross compile environments > + > +- Package is broken within a specific feature > + > +- Package is vulnerable and needs some fixes > + > +- or anything else (this case is the most common one) > + > +Ideally, those problems should be addressed in the original project, > +so any patches you add to your BSP or to PTXdist should also be submitted > upstream. > +The upstream project can often provide better feedback, they can integrate > your > +patch into a new release, and also maintain your changes as part of the > project. > +This way we make sure that all advantages of the open source idea work for > us; > +and your patch can be removed again later when a new release of the project > is > +integrated into your BSP or into PTXdist. > + > +PTXdist handles patching automatically. > +After extracting the archive of a package, PTXdist checks for the existence > of > +a patch directory named like its ``<PKG>_PATCHES`` variable, or, if this > variable > +is not set, like its ``<PKG>`` variable. > +The patch directory is then searched in all locations listed by the > +``PTXDIST_PATH_PATCHES`` variable, and the first one found is used. > +Take an exemplary package ``foo`` with version ``1.1.0``: > +The variable ``FOO`` will have the value ``foo-1.1.0``, so PTXdist will look > for > +a patch directory named ``foo-1.1.0`` in the following locations: > + > +#. the current layer: > + > + a. project (``./patches/foo-1.1.0``) > + b. platform (``./configs/|ptxdistPlatformConfigDir|/patches/foo-1.1.0``) > + > +#. any :ref:`base layers <layers-in-ptxdist>`, > + applying the same search order as above for each layer recursively > + > +#. ptxdist (``<ptxdist/installation/path>/patches/foo-1.1.0``) > + > +The patches from the first location found are used. Note: Due to this > +search order, a PTXdist project can replace global patches from the > +PTXdist installation. This can be useful if a project sticks to a > +specific PTXdist revision but fixes from a more recent revision of > +PTXdist should be used. > + > +PTXdist uses the utilities *git*, *patch* or *quilt* to work with > +patches or patch series. We recommend *git*, as it can manage patch > +series in a very easy way. > + > +Creating a Patch Series for a Package > +===================================== > + > +To create a patch series for the first time, we can run the following > +steps. We are still using our *foo-1.1.0* example package here: > + > +Using Quilt > +----------- > + > +We create a special directory for the patch series in the local project > +directory: > + > +.. code-block:: text > + > + $ mkdir -p patches/foo-1.1.0 > + > +PTXdist expects a ``series`` file in the patch directory and at least > +one patch. Otherwise it fails. Due to the fact that we do not have any > +patch content yet, we’ll start with a dummy entry in the ``series`` file > +and an empty ``patch`` file. > + > +.. code-block:: text > + > + $ touch patches/foo-1.1.0/dummy > + $ echo dummy > patches/foo-1.1.0/series > + > +Next is to extract the package (if already done, we must remove it > +first): > + > +.. code-block:: text > + > + $ ptxdist extract foo > + > +This will extract the archive and create a symbolic link in the build > +directory pointing to our local patch directory. Working this way will > +ensure that we do not lose our created patches if we enter > +``ptxdist clean foo`` by accident. In our case the patches are still > +present in ``patches/foo-1.1.0`` and can be used the next time we > +extract the package again. > + > +All we have to do now is to do the modification we need to make the > +package work. We change into the build directory and use quilt_ to > +create new patches, add files to respective patches, modify these files > +and refresh the patches to save our changes. > +See the *quilt* documentation (``man 1 quilt``) for more information. > + > +.. note:: For patches that are intended for PTXdist upstream use the git > + workflow described below to get proper patch headers. > + > +.. _quilt: http://savannah.nongnu.org/projects/quilt > + > +Using Git > +--------- > + > +Create the patch directory like above for *quilt*, > +but only add an empty series file: > + > +.. code-block:: text > + > + $ mkdir -p patches/foo-1.1.0 > + $ touch patches/foo-1.1.0/series > + > +Then extract the package with an additional command line switch: > + > +.. code-block:: text > + > + $ ptxdist --git extract foo > + > +The empty series file makes PTXdist create a Git repository in the > +respective package build directory, > +and import the package source as the first commit. > + > +.. note:: Optionally, you can enable the setting *Developer Options → > + use git to apply patches* in `ptxdist setup` to get this behaviour > + as a default for every package. > + However, note that this setting is meant for development only, and can lead > + to failures – some packages try to determine if they are being compiled > from > + a Git source tree, and behave differently in that case. > + > +Then you can change into the package build directory > +(``platform-<name>/build-target/foo-1.1.0``), > +patch the required source files, > +and make Git commits on the way. > +The Git history should now look something like this: > + > +.. code-block:: text > + > + $ git log --oneline --decorate > + * df343e821851 (HEAD -> master) Makefile: don't build the tests > + * 65a360c2bd60 strfry.c: frobnicate the excusator > + * fdc315f6844c (tag: foobar-1.1.0, tag: base) initial commit > + > +Finally, call ``git ptx-patches`` to transform those Git commits into the > patch > +series in the ``patches/foo-1.1.0`` folder. > +This way they don't get lost when cleaning the package. > + > +.. note:: PTXdist will only create a Git repository for packages with > + patches. To use Git to generate the first patch, create an empty series > + file ``patches/foobar-1.1.0/series`` before extracting the packages. This > + will tell PTXdist to use Git anyways and ``git ptx-patches`` will put the > + patches there. > + > +Both approaches (Git and quilt) are not suitable for modifying files > +that are autogenerated in autotools-based buildsystems. > +Refer to the section :ref:`configure_rebuild` on how PTXdist can > +handle this special task. > + > + > +Adding More Patches to a Package > +================================ > + > +If we want to add more patches to an already patched package, we can use > +nearly the same way as creating patches for the first time. But if the > +patch series comes from the PTXdist main installation, we do not have > +write permissions to these directories (do NEVER work on the main > +installation directories, NEVER, NEVER, NEVER). Due to the search order > +in which PTXdist searches for patches for a specific package, we can > +copy the global patch series to our local project directory. Now we have > +the permissions to add more patches or modify the existing ones. Also > +*quilt* and *git* are our friends here to manage the patch series. > + > +If we think that our new patches are valuable also for others, or they > +fix an error, it could be a good idea to send these patches to PTXdist > +mainline, and to the upstream project too. > + > + > +.. _configure_rebuild: > + > +Modifying Autotoolized Packages > +=============================== > + > +Autotoolized packages are very picky when automatically generated files > +get patched. The patch order is very important in this case and > +sometimes it even fails and nobody knows why. > + > +To improve a package’s autotools-based build system, PTXdist comes with > +its own project local autotools to regenerate the autotools template > +files, instead of patching them. With this feature, only the template > +files must be patched, the required ``configure`` script and the > +``Makefile.in`` files are regenerated in the final stages of the > +*prepare* step. > + > +This feature works like the regular patching mechanism. The only > +difference is the additional ``autogen.sh`` file in the patch directory. > +If it exists and has execution permissions, it will be called after the > +package was patched (while the *extract* stage is running). > + > +Its content depends on developer needs; for the most simple case the > +content can be: > + > +.. code-block:: bash > + > + #!/bin/bash > + > + aclocal $ACLOCAL_FLAGS > + > + libtoolize \ > + --force \ > + --copy > + > + autoreconf \ > + --force \ > + --install \ > + --warnings=cross \ > + --warnings=syntax \ > + --warnings=obsolete \ > + --warnings=unsupported > + > +.. note:: In this way not yet autotoolized package can be autotoolized. We > + just have to add the common autotool template files (``configure.ac`` > + and ``Makefile.am`` for example) via a patch series to the package > + source and the ``autogen.sh`` to the patch directory. _______________________________________________ ptxdist mailing list [email protected] To unsubscribe, send a mail with subject "unsubscribe" to [email protected]
