Hi David,
These are very good questions, and, fortunately, there are some very simple
answers to these questions. I'm gonna go with a bit of explanation of how
things work below.
In short, you only have to modify the way that toaster starts, and where
the database is. When you do "source toaster start", three things happen:
- a bitbake server is started
- a toaster UI is started that connects to the bitbake server on one side
and to the SQL database on the other side
- the web server is started, that reads the database and shows the Web user
interface
This is done, for convenience, in a single place, but there is no reason
that the web server is attached to the bitbake server, because the SQL
database acts as a separation layer. Ie. a Toaster instance is not
necessarely a set of these three components, but it minimally requires at
least one of these components to run. So we are free to manually start as
many components as we need instead of using the "toaster" script.
So you raise two problems:
- *multiple bitbake directories; *what we need here is a bitbake instance
per build directory, and a toaster UI instance per build directory to log
the build; yet we use a single web server instance that will show the data
from _all_ the builds. As for the SQL separation layer, this is solved by
setting a SQL server (eg. MySQL) outside the build directories.
How this will work ? each bitbake build will log the results in the common
SQL database, and the Web server will show the data from that database.
- *database persistence;* this is again solved by setting the logging
database as a external DB server, (e.g. MySQL). Really, the choice of using
sqlite file in the build directory was made just to make it easier for
adopters, but it's totally not ok if you want any kind of serious
deployment with persistance.
NOTE: it is not needed to even run the bitbake servers, the SQL server and
the Web server on the same machine ! Each component can be run on it's own
machine.
As for concrete steps to set this up:
*1). Set up the SQL logging server and the Web server*
- set up a SQL server; anything out of box (e.g. apt-get install
mysql-server) would work well enough for demo purposes; serious things
would need hand-tuned sql servers for performance reasons; make sure you
set proper username/password access for the server; one needs admin right -
mysql root account (not the same thing as root on the machine!) would fit
just as good.
- you'll need a separate tree just for running the web server; you'll not
do builds on this tree - just clone toaster/master somewhere out of your
normal build locations
edit bitbake/lib/toaster/toastermain/settings.py in this web tree to
alter the "DATABASES" value to point to the previously created database
server; use the username/password you've set up.
run the database sync scripts to create needed tables:
python bitbake/lib/toaster/manage.py syncdb
python bitbake/lib/toaster/manage.py migrate orm
- start the web server
python bitbake/lib/toaster/manage.py runserver
*2). Enable build logging to the common SQL server for each build directory
that you're using*
edit _build local_ bitbake/lib/toaster/toastermain/settings.py to
alter the "DATABASES" value to point to the common SQL logging server
start the bitbake server
bitbake --postread conf/toaster.conf --server-only -t xmlrpc -B
localhost:0 && export BBSERVER=localhost:-1
start the logging UI
nohup bitbake --observe-only -u toasterui &
NOTE: there are no hardcoded ports used in here; we have enough code to run
autodiscovery of ports to prevent collisions.
now you can run builds as usuall:
bitbake core-image-minimal
when you're done, kill the server:
bitbake -m
NOTE: there are outstanding patches that solve some problems with stopping,
but not yet merged; I think you may see toasterui remaining hanged
sometimes :(
*3). Verify build logging works, persistance works, multiple builds from
different directories are shown, etc.*
and let us know if you run into problems :).
Why isn't this all automated ?
Well, we can't force our world view on how things work for different users,
so we cannot automate complex setups. You have to set your environment as
suited to your needs.
What I'm working on, as part of the work needed for triggering builds, is
to make the bitbake start automatically all components needed for toaster
on environment setup;
so the 2nd step sequence from above should be just:
export TOASTERDB=[address of common SQL server, like
"dbtype:username@password:ip:port"]
bitbake -u toasterui # start server and
logging in a single step
export BBSERVER=localhost:-1 # enable server autodetection
bitbake core-image-minimal
eliminating the need to edit settings file in the build bitbake trees,
manually start the server and the UI separately, and other unpleasant tasks.
Hope this helps,
Alex
On Wed, Feb 5, 2014 at 10:21 AM, Reyna, David <[email protected]>wrote:
> Hi Alex,
>
> I have a few questions about the Toaster backend support and its
> connection to the Toaster web server. I would like to demo the Toaster
> internally to Wind River, and I need to understand a few things.
>
> * In the example Toaster web pages, we show builds for multiple and
> separate machines, for example “atom-pc” and “qemuarm”.
>
> I think that to create these different builds, you are doing to following.
>
> 1) Create the build directory and edit the “conf” file(s) for the first
> machine type.
> 2) Build the needed images for that machine type.
> 3) Re-edit the “conf” files for the next machine type.
> 4) Repeat #2 and #3 as needed.
>
> Since the generated image files and package work directories are name-safe
> per machine, there are no file collisions.
>
> Since all of the work is in the same project directory, Toaster has no
> problem because it can work off of a single SQL database and attach that to
> the Toaster via port 8080.
>
> * But what it the customer wants separate physical directories for each of
> their builds (as is the case with for example Wind River customers)?
>
> We can certainly setup the project directories and bitbake environments
> per build, but each project would then have its own separate database, and
> I am guessing that each “source toaster start” would vie for the same 8080
> port. And if we instead assign separate port numbers to each of these
> Toaster instances, then the user would need to then access each separate
> URLs and thus have an unconsolidated view?
>
> Did you have a design that supports multiple build directories?
>
> * Did you have ideas about persistence of the project’s database? This
> question came up in a Wind River call.
>
> For example, in our nightly builds we constantly create/build projects and
> then delete them to make room for the next cycle. It would therefore be
> good to have a way to preserve the SQL database somewhere (without the
> original project) and be able to connect to it via Toaster.
>
> Thanks,
> David
>
>
>
--
Alex Damian
Yocto Project
SSG / OTC
_______________________________________________
toaster mailing list
[email protected]
https://lists.yoctoproject.org/listinfo/toaster
