Hi Henning,
My team also face such fast-pace evolution that sometime impose quick
modifications to APIs. The real problem is not the lack of thinking we put in
designing these extended API, but the fact that these API extensions might be
required at a time when it would be unacceptable to force immediate upgrade of
all related bundles. Of course, simply not allowing unordered upgrades is, from
a technical point of view, the best and simplest approach. But sometimes, the
pressure to quickly provide a "low risk" upgrade to existing feature is simply
too high, be it the "time to market" factor, or whatever situation we may have
with a specific customer. In our case, being able to do so is actually a
business requirement.
Let me share with you an overview of how some development practices we have
established in order to better to deal with these situations. The key
strategies are:
Heavily split bundles in API vs Implementations bundles;
Design "set-of-features" add-on interfaces whenever a fundamental interface
need to be upgraded;
Fork projects when a bundle is subject to external evolution factors (or has to
be versionized simultaneously on two axis);
Testing API is even more important than testing implementations.
The first point states that, in most case, a bundle should either offer an API
or an implementation, but not both at the same time. I say in most cases,
though, because "utilities" bundles and some clearly delimited bundles might
not need such precautions. But not splitting enough tends to hurt more than
splitting too much, so in case of doubts, better split.
API bundles (and packages…) are semantically versionnized. API classes and
interfaces grow from a minor to the next one, and can eventually be
consolidated at major, for example to fix up any mistakes that might have been
introduced over time, or to remove methods that have become deprecated. Note
that given the semantic versioning rules, API bundles can almost never be
altered at micro level.
The main down side of isolating API is that it becomes impossible for a bundle
to directly instantiate (or extends) classes provided by implementation
bundles. This is easily fixed however by defining
factory/repository/manager/whatever else interfaces in the API, and having the
implementation bundle register its implementation of that interface as a
service. As for inheritance, it is considered a bad practice anyway.
Let's move on to the second point: we must be very strict in how API interfaces
and classes evolve. Basically, we consider that most API interfaces are
unalterable within the scope of a minor upgrade. This is a common best practice
whenever an interface is intended to be implemented by several external
bundles, because it may be impossible to track with certainty all
implementations of the interface, which therefore present risk of binary
incompatibilities at runtime.
However, we also consider as unalterable many interfaces not intended to be
publicly implemented, such as interfaces of services exposed through OSGi.
Instead, when new methods are required, we define a new interface that extends
the original interface (in general, we define those within the scope of the
original interface, simply to make it easier to track all extensions that have
been developed). For example, if we defined interface "IProductStore" in the
API bundle c.x.productstores v1.0.0, then we shall not alter that interface
before v2.0.0 of the API. If we eventually need to add more methods in order to
support a new feature, then we would define them in a new interface, say
IManageableProductStore (that extends IProductStore). That would be API bundle
c.x.productstores v.1.1.0. Then in v1.2.0, we might have
ISearcheableProductStore (also extending IProductStore), which would introduce
two more methods.
The real gain in doing so is that we might independently delay upgrading to a
specific feature, on both side of the API (that is, implementers and users).
For example, if for some reason, we consider that one implementation of
IProductStore can't easily be transitioned to support the
IManageableProductStore, then it may still support the ISearcheableProductStore
interface. Obviously, this push some of the difficulties of upgrades back to
the code using the API, since programmers must now be careful to check if a
given object implements the extended interface before invoking some extended
feature. However, this is generally a very descent trade off: first, there are
in general few users of those extended API (otherwise, it is probable that the
methods would have been identified early during development). Second, this is
new code anyway, so all places where checks has to be performed are known
anyway. And third, there are tons of very nice idioms to efficiently deal with
these "test before acting" sequences.
Note that this solution can trivially be implemented using the "instance of"
operator, followed by casting objects to the extended class. However, we prefer
to define a method "adapt(Class<X> clazz) : X" in our base interfaces, which
allow more dynamic decisions on either a given feature should be supported or
not. In the previous example, a specific implementation of IProductStore might
accept to be adapted to IManageableProductStore only if the connected user
possess administration rights. Or the code might look at company-level settings
to decide either a given feature is enabled or not. This is both cleaner and
safer than having the code using the API deal with permissions and
"company-wide" settings.
Now up to the third point… As we described earlier, mapping evolution of an API
bundle in a semantic versioning scheme is straight forward. In general,
implementation bundles can also be trivially mapped into semantic versioning
scheme (I won't elaborate here). However, we note that in some situations, the
versioning of a bundle may be affected not only by internal corrections and API
additions, but also by external factors.
Such external factors are common when a bundle ensure compatibility with an
external subsystem: there might be several major version of a key external
application, the customer database may be frozen to an older schema version,
and so on. In simple case, these external factors might easily be dealt by "if"
statements and dynamic checks at runtime, or possibly by having a few distinct
classes inside a same bundle. However, in more complex situations, it may be
tempting to produce distinct versions of the bundle. To illustrate this case,
let assume that bundle c.x.fileimport.appa provides a connector to read files
produced by an external program named "Application A - 1.0". Now if that
application file format changes in release 2.0, one might think that the bundle
should equivalently be upgraded to "2.0". Yet, assuming that support for file
format 1 and file format 2 can't reasonably be implemented in the same bundle,
then changing the bundle version number would be inappropriate: doing so would
prevent the bundle for format 1 from later being upgraded to a new major API,
since that would require that the implementation bundle's version be also
upgraded to the next major number, which is no longer available. In these
cases, the correct approach (still assuming that both formats can't reasonably
be implemented in a same bundle) would be to fork the original bundle into a
second one, with a distinct id and a distinct version number. That could be for
example c.x.fileimport.appa.format2 or c.x.fileimport.appa_v2.
Hopefully, this example is easily understood and accepted. However, we have
found that the same logic holds in many cases that are much less trivial. For
example, for reasons that I won't elaborate here, our database schemas
evolution is not coordinated with our main software releases. This means that
at the time the software connects to the database, the schema version is
checked, and then the bundle specific to the schema version is used thereafter.
However, because evolution of the implementation bundles happens on two axes
(specifically, the API version and the schema version), it would be problematic
to express semantic evolution through the classical version numbers. Therefore,
we create distinct bundles for each schema version, named something like
c.x.persistence.ourdatabase._3_141 4.2.5, where the schema version is 3.141,
the implemented API is 4.2, and there has been 5 corrections to the code since
the last upgrade to the API.
This strategy can be extended to other cases where external factors make it
impossible to allow a bundle evolution by following a simpler path. For
example, we sometimes judge that, for exceptional reasons, it is preferable for
a specific bundle to be temporarily forked for the needs of a specific
customer. Doing this produce a totally distinct bundle, let's say
c.x.somebundle.customizations.custumerabc, with distinct version numbers. This
is possible because anyway, no one has direct requirements for the original
bundle (c.x.somebundle.impl).
Obviously, systematic forking has some drawback, most notably that it makes the
code harder to maintain; fortunately, these forks are generally very short
lived. For example, we rarely have to support more than 2 or 3 schema versions
backward, and we only allow a customer specific branch to live until the end of
a "customer production cycle" (which rarely span more than 3 months, during
which we have to minimize risks for that specific customer). As soon as that
customer's production cycle is completed, we transition him back to the main
branch.
Together, the previous strategies ensure that we always have a few possible
solutions to deal with customer specific evolutions. We may for example fork
bundles (usually not the preferred path, but it is sometimes unavoidable), or
introduce new configuration items (which may be a good approach when we believe
that several customers might be interested in that same customization, and it
appears that the customization will remain pertinent in the long term), or
define a cleaner customization end point (if the customization appears
legitimate in the long term, and we believe that similar customizations will be
required for other customers), and so on. Also, even if we choose to introduce
a new, customer specific bundle, we might prefer to do so as a fragment, or to
avoid code duplication by having the replacement service (registered with an
higher ranking) capturing the service exported by the original implementation
bundle, and then delegate or decorate method calls to the original service
(rather than reimplement all the code).
In all case, we must carefully weight the pros and cons of each solution for
any specific issue. Maintainability and correctness are both majors issues with
these customizations. For this reason, I always question the time to live of a
customization and the probability that both the original code and the
customized code will have to evolve again during that time span.
The last point basically states that we should always have
implementation-independent tests, that validate that implementations do indeed
honour the API contract. It is obviously not always possible to test all aspect
of an implementation bundle in this way, but these implementation-independant
tests are particularly important once we deal with multiple forks of a base
implementation. Indeed, back porting tests between each customer-specific
branches of a base implementation is cumbersome. Testing to API definitely
reduce the maintenance cost of customer specific branches.
I hope this helps,
James Watkins-Harvey
On 2013-01-25, at 09h54, Henning Andersen wrote:
> Hi OSGi developers,
>
> we have been using OSGi for 3 years now, however without semantic versioning.
> We now want to take our versioning strategy to the next level by introducing
> semantic versioning and allowing individual release cycles for different
> components in our system.
>
> One of the issues we are struggling with is how to handle branches and
> versioning, especially in relation to API changes.
>
> Consider a package p with a single class C, with a single method m:
>
> package p;
>
> public class C {
> public void m();
> }
>
> Now say, we released a bundle B in version 1.0, with this package in version
> 1.0. To allow us to bug fix, the bundle B is branched into a 1.0 branch too
> in our source code repository.
>
> Another bundle X has Import-Package: p; version="[1.0;2)" as per semantic
> versioning and is released as version 1.0.
>
> Development of class C continues and a new method is added:
>
> public class C {
> public void m();
> public void n();
> }
>
> This is released in a new version of B, bundle-version 1.1, package version
> 1.1.
>
> Now a customer has X 1.0 and B 1.0. They request a new feature to X, but
> reject to upgrade B. The feature in X requires an API extension to the class
> C. So we add this in the 1.0 branch:
>
> public class C {
> public void m();
> public void o();
> }
>
> If we were to follow semantic versioning, we should release this as B version
> 1.1, p version 1.1. However, these numbers are already occupied.
>
> So we have to break the principle and release them as B version 1.0.1 and p
> version 1.0.1.
>
> The developer of X now needs to import the package p in the right version.
> However, using: Import-Package: p; version="[1.0.1;2)" is not entirely right,
> since his code is not compatible with the 1.1 version.
>
> So far we have 5 solutions:
>
> 1. Do not allow it. Only allow extending the API in a new minor version
> following the latest released version.
> 2. Add an attribute to the export 'Export-Package: a; version=1.0.1; p.C.o =
> true' and do the same on 'Import-Package: a; version="[1.0.1,2)"; p.C.o =
> true'. Requires that we identify the dependency to method level.
> 3. Fix the version of a in X: 'Import-Package: a; version="[1.0.1,1.1)"'.
> This however has the implication that X need upgrade when we upgrade B.
> 4. Build two bundles out of X, one with 'Import-Package: a;
> version="[1.0.1,1.1)"' and one with 'Import-Package: a; version="[1.1.1,2)"'
> (assuming the method was also added in 1.1.1).
> 5. Branch X into two versions (much like 3), with different import versions
>
> We think some of these could work, but do wonder if others have run into the
> same issues or similar branching/versioning issues and how they solved it?
> Any input is highly appreciated.
>
> Thanks,
>
> Henning
>
>
> _______________________________________________
> OSGi Developer Mail List
> [email protected]
> https://mail.osgi.org/mailman/listinfo/osgi-dev
_______________________________________________
OSGi Developer Mail List
[email protected]
https://mail.osgi.org/mailman/listinfo/osgi-dev
