Hi there,
I've been writing up the work that I'm doing on the Apache-based depot,
and have posted the project proposal in a few different formats at:
https://cr.opensolaris.org/action/browse/pkg/timf/apache-depot-proposal/
For convenience, I've inlined the rst source in this mail, but it's best
to look at either the html or the pdf, or at least reference the diagram
in the link above.
The project is still in its early stages - I have a working utility that
creates an Apache configuration, and have verified that clients can
install packages from an Apache instance that uses that config, but
there's more work to do. I haven't started on the mod_wsgi application
to serve search/0 operations, or the service-management parts.
This is just the initial direction of the project, I'm open to other
ideas and approaches we could take, and comments would be most welcome.
cheers,
tim
=========================
pkg(5) Apache-based depot
=========================
Version: |version|
.. contents:: :depth: 2
Proposal
--------
IPS currently uses a CherryPy-based application, |pkg.depotd| to serve
content to |pkg5| clients.
In order to provide scalability and ease of administration, we have
recommended that users employ an Apache-based reverse-proxy in front of
multiple |pkg.depotd| processes, taking advantage of the caching and
load-balancing functionality that Apache provides.
This project allows us to to serve |pkg5| repository content directly to
clients using Apache, which means that administrators will no longer
necessarily need a reverse-proxy configuration to serve large numbers of
clients efficiently.
Package publication to an Apache-based depot is not supported.
Implementation
--------------
SMF service model
.................
To make Apache-based depots as easy as possible to use, we propose to
reuse instances of the existing |svc-pkg-depot| service to hold the
repository configuration.
Preserving existing behaviour, a new boolean property is available for
this service, ``pkg/standalone``. When set to ``true``, a
CherryPy daemon is used as it is today.
When ``pkg/standalone`` is set to ``false`` *and* when ``pkg/readonly``
is set to ``true``, the Apache-based depot is used, and the
|svc-pkg-depot| services simply become containers for configuration,
they are not associated with a running process.
Any |svc-pkg-depot| instances that do not associate with a CherryPy
daemon will use the ``$SMF_EXIT_TEMP_TRANSIENT`` exit-code to declare
themselves as transient services.
We have not yet decided the default for the ``pkg/standalone`` property.
Rather than spawning multiple Apache instances, one per |svc-pkg-server|
instance, we will create a single service |svc-apache-depot| to serve
content of behalf of all instances of |svc-pkg-server|.
A diagram of how this would look is shown below:
.. image:: smf-service-model-diagram.png
:alt: A diagram showing the proposed SMF service model
When starting the Apache-based depot service, the method-script looks
for enabled instances of |svc-pkg-server|, checking for those marked as
``pkg/readonly = true`` and ``pkg/standalone = false``.
If any two instances that satisfy those criteria serve the same |pkg5|
publisher prefix, the Apache depot will offset the url namespace for
those publishers with an SHA-1 hash of the repository paths, updating
the 'publisher/0' response for the Apache-based depot accordingly. If
older configurations used multiple readonly instances of
|svc-pkg-server| to serve files from the same repository, we recommend
that only a single instance is used when the Apache-based depot is
running.
When enabling or disabling a |svc-pkg-server| instance, if applicable,
the method script will invoke a ``refresh`` operation on the
|svc-apache-depot| service to cause it to rebuild its configuration
to account for the newly enabled or disabled ``pkg/server`` instance.
The method script for |svc-apache-depot| will build an Apache httpd.conf
configuration file and start a ``httpd.worker`` process using that
configuration. All logging for the Apache instance will go to log files
in ``/var/log/pkg/depot``
We need to investigate how to deal with cases where an Apache-based
depot service drops to maintenance, and how that failure should be
reflected in the corresponding |svc-pkg-server| instances.
Apache configuration
....................
The Apache configuration is built dynamically using
``/usr/lib/pkg.apache-depot`` as part of the start-method script for
|svc-apache-depot|, using a Mako template stored in
``/etc/pkg/depot``.
A series of ``RewriteRule`` directives are used to serve the file
content directly from the repository. This means that any repositories
served by the Apache-based depot must be readable by the SMF service.
Best performance will result when the repositories are local to the
machine running the Apache-based depot, rather than having it access
the repositories over the network, via eg. NFS.
Apache logs are written to /var/log/pkg/depot
The configuration includes a mod_wsgi instance that serves
``'search/0'`` operations against the repositories configured. The
|svc-apache-depot| allows the user to define a writable area, similar to
``pkg/writable_root``, where indices for all repositories are stored.
A subset of ``'admin/0'`` operations are supported by the
|svc-apache-depot| instance (similar to readonly |pkg.depotd| instances)
using a mod_wsgi application.
Related work
............
The indexing mod_wsgi application created as part of this project is
also used to serve ``'search/0'`` operations for file repositories
served to non-global zones though the system-repository,
|svc-system-repository|.
.. |pkg| replace:: ``pkg(1)``
.. |pkg.depotd| replace:: ``pkg.depotd(1M)``
.. |apache-depotd| replace:: ``pkg.apache-depotd(1M)``
.. |pkg5| replace:: ``pkg(5)``
.. |svc-pkg-server| replace:: ``svc:/application/pkg/server``
.. |svc-apache-depot| replace:: ``svc:/application/pkg/apache-depot``
.. |svc-system-repository| replace::
``svc:/application/pkg/system-repository``
.. |svc-pkg-depot| replace:: ``svc:/application/pkg/server``
.. |version| replace:: 2012-08-15:15:37:08-NZST 7696ad9d823f tip
_______________________________________________
pkg-discuss mailing list
[email protected]
http://mail.opensolaris.org/mailman/listinfo/pkg-discuss
