A little context before the actual questions. As a project that I have worked
on grew, the number of repositories used for the application we're developing
grew as well. To the point that it's gotten unbearable to keep making
modifications to the codebase: when something is changed in the core library in
one repository we need to make adjustments to other Python middleware projects
that use this library. Whereas this sounds not that terrifying, managing this
in Yocto has become a burden: on every version bump of the Python library we
need to bump all of the dependant projects as well. After some thinking, we
decided to try a monorepo approach to deal with the complexity. I'm omitting
the logic behind choosing this approach but I can delve into it if you think
this is wrong.
1) Phase 1 was easy. Just bring every component of our middleware to one
repository. Right now we have a big repository with more than git 20
submodules. Submodules will be gone once we finish the transition but as the
project doesn't stop while we're doing the transition, submodules were chosen
to track the changes and keep the monorepo up to date. Every submodule is a
Python code with a setup.py/Pipfile et al. that does the bootstrapping.
2) Phase 2 is to integrate everything in Yocto. It has turned out to be more
difficult than I had anticipated.
Right now we have this:
*Application monorepo*
Pipfile
python - library1 / setup. py
python - library1 / Makefile
python - library1 / python - library1 /< code >
python - library2 / setup. py
python - library2 / Makefile
python - library2 / python - library2 /< code >
...
python - libraryN / setup. py
python - libraryN / Makefile
python - libraryN / python - libraryN /< code >
Naturally, right now we have python-library1_<some_version>.bb ,
python-library2_<some_other_version>.bb in Yocto. Now we want to get rid of
this version hell and stick to monorepo versions so that we could have just
monorepo_1.0.bb that is to be updated when anything changes in any part.
Right now I have the layout described below.
*monorepo.inc* is as follows:
SRC_URI =< gitsm :// the location of the monorepo , branch =...,>
S = "${WORKDIR}/git"
SRCREV = <hash>
*python-libraryN.inc:*
require monorepo.inc
S = "${WORKDIR}/git/python-libraryN"
inherit setuptools
#some libraries also use entry points and the recipe needs to inherit systemd
to enable some services.
This approach works. But it has a very significant drawback I can't ignore.
Every time SRCREV (or PV/tag in the future) is changed every recipe downloads
the whole repository that is quite big.
So, well, the question is how to structure the Yocto recipes for Python
monorepo manipulation? I'm looking for a way to trick do_unpack and do_fetch to
their thing on the monorepo basis.
This (
https://www.yoctoproject.org/docs/current/bitbake-user-manual/bitbake-user-manual.html#ref-bitbake-tasklist
) section of the documentation makes me think that I'm approaching this task
wrong. I was trying to decipher gcc-source.inc and gcc-shared-source.inc files
in the poky but they are out of my depth for now and look a little bit
overcomplicated for my use-case. So basically my question boils down to how to
deal with multiple recipes referencing the same source. Any structural change
to either the monorepo or the recipe structure can be done now, so I would be
very grateful for any feedback.
P.S. 1) The monorepo structure is not set in stone so if you have any
suggestions, I'm more than open to any criticism 2) The.inc file structure
might not be suitable for another reason. This stack is deployed across a dozen
of different devices and some of those python-library_N.bb have.bbappends in
other layers that are custom for the devices. I mean that some of them might
require different components of the system installed or some configs and files/
modifications.
Any suggestion on how to deal with the complexity of a big Python application
in Yocto will be welcome. Thanks in advance.
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#47584): https://lists.yoctoproject.org/g/yocto/message/47584
Mute This Topic: https://lists.yoctoproject.org/mt/67468660/21656
Mute #yocto: https://lists.yoctoproject.org/mk?hashtag=yocto&subid=6691583
Mute #python: https://lists.yoctoproject.org/mk?hashtag=python&subid=6691583
Mute #monorepo: https://lists.yoctoproject.org/mk?hashtag=monorepo&subid=6691583
Group Owner: [email protected]
Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub
[[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
