Hi all, thanks for all your comments, suggestions and experience, it is highly appreciated! A few follow-up questions:
@James: The issues you are facing sound very familiar to ours. I like the fact that you have several options and I think you are right in that there is no single solution to the issues. Thanks for the elaborate response and I hope you do not mind a few questions: Re item 2, "set-of-features" add-on interfaces: do you also do this in branches? If yes, am I correct in that you then must put the new interface in a new package to be able to give it a proper version? Re item 3, forking bundles: when you fork a bundle for a specific customer, into: c.x.somebundle.customizations.custumerabc, do you only do so for pure implementation bundles or also for bundles with APIs? If also for APIs, how do you handle the version numbers of package exports? @Neil: Thanks for the tip on bndtools/bnd. I see the new commandline options (baseline and diff plus some I did not try out) in bnd and it does have a bit of help built-in. I wonder if there is more documentation available anywhere (seems the aqute.biz site does not contain any info on this and neither does the "bnd-book"). Specifically I miss info on: 1. How to indicate that an interface is consumer implemented (ie. any extension is also a major change)? 2. Whether it is possible to add an indication to the package that a new minor or major version is necessary due to changing semantics of implementation/javadoc specification (so a CI server can later see it). 3. Any plugin capabilities available to override some of the behaviour? I could of course read the code to find out! Best regards, Henning On Tue, Jan 29, 2013 at 11:11 PM, James Watkins-Harvey < [email protected]> wrote: > Hi Henning, > > > My team also face such fast-pace evolution that sometime impose quick > modifications to APIs. The real problem is not the lack of thinking we put > in designing these extended API, but the fact that these API extensions > might be required at a time when it would be unacceptable to force > immediate upgrade of all related bundles. Of course, simply not allowing > unordered upgrades is, from a technical point of view, the best and > simplest approach. But sometimes, the pressure to quickly provide a "low > risk" upgrade to existing feature is simply too high, be it the "time to > market" factor, or whatever situation we may have with a specific customer. > In our case, being able to do so is actually a business requirement. > > Let me share with you an overview of how some development practices we > have established in order to better to deal with these situations. The key > strategies are: > > - Heavily split bundles in API vs Implementations bundles; > - Design "set-of-features" add-on interfaces whenever a fundamental > interface need to be upgraded; > - Fork projects when a bundle is subject to external evolution factors > (or has to be versionized simultaneously on two axis); > - Testing API is even more important than testing implementations. > > The first point states that, in most case, a bundle should either offer an > API or an implementation, but not both at the same time. I say in most > cases, though, because "utilities" bundles and some clearly delimited > bundles might not need such precautions. But not splitting enough tends to > hurt more than splitting too much, so in case of doubts, better split. > > API bundles (and packages…) are semantically versionnized. API classes and > interfaces grow from a minor to the next one, and can eventually be > consolidated at major, for example to fix up any mistakes that might have > been introduced over time, or to remove methods that have become > deprecated. Note that given the semantic versioning rules, API bundles can > almost never be altered at micro level. > > The main down side of isolating API is that it becomes impossible for a > bundle to directly instantiate (or extends) classes provided by > implementation bundles. This is easily fixed however by defining > factory/repository/manager/whatever else interfaces in the API, and having > the implementation bundle register its implementation of that interface as > a service. As for inheritance, it is considered a bad practice anyway. > > > Let's move on to the second point: we must be very strict in how API > interfaces and classes evolve. Basically, we consider that most API > interfaces are unalterable within the scope of a minor upgrade. This is a > common best practice whenever an interface is intended to be implemented by > several external bundles, because it may be impossible to track with > certainty all implementations of the interface, which therefore present > risk of binary incompatibilities at runtime. > > However, we also consider as unalterable many interfaces not intended to > be publicly implemented, such as interfaces of services exposed through > OSGi. Instead, when new methods are required, we define a new interface > that extends the original interface (in general, we define those within the > scope of the original interface, simply to make it easier to track all > extensions that have been developed). For example, if we defined interface > "IProductStore" in the API bundle c.x.productstores v1.0.0, then we shall > not alter that interface before v2.0.0 of the API. If we eventually need to > add more methods in order to support a new feature, then we would define > them in a new interface, say IManageableProductStore (that extends > IProductStore). That would be API bundle c.x.productstores v.1.1.0. Then in > v1.2.0, we might have ISearcheableProductStore (also extending > IProductStore), which would introduce two more methods. > > The real gain in doing so is that we might independently delay upgrading > to a specific feature, on both side of the API (that is, implementers and > users). For example, if for some reason, we consider that one > implementation of IProductStore can't easily be transitioned to support the > IManageableProductStore, then it may still support > the ISearcheableProductStore interface. Obviously, this push some of the > difficulties of upgrades back to the code using the API, since programmers > must now be careful to check if a given object implements the extended > interface before invoking some extended feature. However, this is generally > a very descent trade off: first, there are in general few users of those > extended API (otherwise, it is probable that the methods would have been > identified early during development). Second, this is new code anyway, so > all places where checks has to be performed are known anyway. And third, > there are tons of very nice idioms to efficiently deal with these "test > before acting" sequences. > > Note that this solution can trivially be implemented using the "instance > of" operator, followed by casting objects to the extended class. However, > we prefer to define a method "adapt(Class<X> clazz) : X" in our base > interfaces, which allow more dynamic decisions on either a given feature > should be supported or not. In the previous example, a specific > implementation of IProductStore might accept to be adapted to > IManageableProductStore only if the connected user possess administration > rights. Or the code might look at company-level settings to decide either a > given feature is enabled or not. This is both cleaner and safer than having > the code using the API deal with permissions and "company-wide" settings. > > > Now up to the third point… As we described earlier, mapping evolution of > an API bundle in a semantic versioning scheme is straight forward. In > general, implementation bundles can also be trivially mapped into semantic > versioning scheme (I won't elaborate here). However, we note that in some > situations, the versioning of a bundle may be affected not only by internal > corrections and API additions, but also by external factors. > > Such external factors are common when a bundle ensure compatibility with > an external subsystem: there might be several major version of a key > external application, the customer database may be frozen to an older > schema version, and so on. In simple case, these external factors might > easily be dealt by "if" statements and dynamic checks at runtime, or > possibly by having a few distinct classes inside a same bundle. However, in > more complex situations, it may be tempting to produce distinct versions of > the bundle. To illustrate this case, let assume that bundle > c.x.fileimport.appa provides a connector to read files produced by an > external program named "Application A - 1.0". Now if that application file > format changes in release 2.0, one might think that the bundle should > equivalently be upgraded to "2.0". Yet, assuming that support for file > format 1 and file format 2 can't reasonably be implemented in the same > bundle, then changing the bundle version number would be inappropriate: > doing so would prevent the bundle for format 1 from later being upgraded to > a new major API, since that would require that the implementation bundle's > version be also upgraded to the next major number, which is no longer > available. In these cases, the correct approach (still assuming that both > formats can't reasonably be implemented in a same bundle) would be to fork > the original bundle into a second one, with a distinct id and a distinct > version number. That could be for example c.x.fileimport.appa.format2 or > c.x.fileimport.appa_v2. > > Hopefully, this example is easily understood and accepted. However, we > have found that the same logic holds in many cases that are much less > trivial. For example, for reasons that I won't elaborate here, our database > schemas evolution is not coordinated with our main software releases. This > means that at the time the software connects to the database, the schema > version is checked, and then the bundle specific to the schema version is > used thereafter. However, because evolution of the implementation bundles > happens on two axes (specifically, the API version and the schema version), > it would be problematic to express semantic evolution through the classical > version numbers. Therefore, we create distinct bundles for each schema > version, named something like c.x.persistence.ourdatabase._3_141 4.2.5, > where the schema version is 3.141, the implemented API is 4.2, and there > has been 5 corrections to the code since the last upgrade to the API. > > This strategy can be extended to other cases where external factors make > it impossible to allow a bundle evolution by following a simpler path. For > example, we sometimes judge that, for exceptional reasons, it is preferable > for a specific bundle to be temporarily forked for the needs of a specific > customer. Doing this produce a totally distinct bundle, let's say > c.x.somebundle.customizations.custumerabc, with distinct version numbers. > This is possible because anyway, no one has direct requirements for the > original bundle (c.x.somebundle.impl). > > Obviously, systematic forking has some drawback, most notably that it > makes the code harder to maintain; fortunately, these forks are generally > very short lived. For example, we rarely have to support more than 2 or 3 > schema versions backward, and we only allow a customer specific branch to > live until the end of a "customer production cycle" (which rarely span more > than 3 months, during which we have to minimize risks for that specific > customer). As soon as that customer's production cycle is completed, we > transition him back to the main branch. > > > Together, the previous strategies ensure that we always have a few > possible solutions to deal with customer specific evolutions. We may for > example fork bundles (usually not the preferred path, but it is sometimes > unavoidable), or introduce new configuration items (which may be a good > approach when we believe that several customers might be interested in that > same customization, and it appears that the customization will remain > pertinent in the long term), or define a cleaner customization end point > (if the customization appears legitimate in the long term, and we believe > that similar customizations will be required for other customers), and so > on. Also, even if we choose to introduce a new, customer specific bundle, > we might prefer to do so as a fragment, or to avoid code duplication by > having the replacement service (registered with an higher ranking) > capturing the service exported by the original implementation bundle, and > then delegate or decorate method calls to the original service (rather than > reimplement all the code). > > In all case, we must carefully weight the pros and cons of each solution > for any specific issue. Maintainability and correctness are both majors > issues with these customizations. For this reason, I always question the > time to live of a customization and the probability that both the original > code and the customized code will have to evolve again during that time > span. > > > The last point basically states that we should always have > implementation-independent tests, that validate that implementations do > indeed honour the API contract. It is obviously not always possible to test > all aspect of an implementation bundle in this way, but these > implementation-independant tests are particularly important once we deal > with multiple forks of a base implementation. Indeed, back porting tests > between each customer-specific branches of a base implementation is > cumbersome. Testing to API definitely reduce the maintenance cost of > customer specific branches. > > > > I hope this helps, > > James Watkins-Harvey > > > > > On 2013-01-25, at 09h54, Henning Andersen wrote: > > Hi OSGi developers, > > we have been using OSGi for 3 years now, however without semantic > versioning. We now want to take our versioning strategy to the next level > by introducing semantic versioning and allowing individual release cycles > for different components in our system. > > One of the issues we are struggling with is how to handle branches and > versioning, especially in relation to API changes. > > Consider a package p with a single class C, with a single method m: > > package p; > > public class C { > public void m(); > } > > Now say, we released a bundle B in version 1.0, with this package in > version 1.0. To allow us to bug fix, the bundle B is branched into a 1.0 > branch too in our source code repository. > > Another bundle X has Import-Package: p; version="[1.0;2)" as per semantic > versioning and is released as version 1.0. > > Development of class C continues and a new method is added: > > public class C { > public void m(); > public void n(); > } > > This is released in a new version of B, bundle-version 1.1, package > version 1.1. > > Now a customer has X 1.0 and B 1.0. They request a new feature to X, but > reject to upgrade B. The feature in X requires an API extension to the > class C. So we add this in the 1.0 branch: > > public class C { > public void m(); > public void o(); > } > > If we were to follow semantic versioning, we should release this as B > version 1.1, p version 1.1. However, these numbers are already occupied. > > So we have to break the principle and release them as B version 1.0.1 and > p version 1.0.1. > > The developer of X now needs to import the package p in the right version. > However, using: Import-Package: p; version="[1.0.1;2)" is not entirely > right, since his code is not compatible with the 1.1 version. > > So far we have 5 solutions: > > 1. Do not allow it. Only allow extending the API in a new minor version > following the latest released version. > 2. Add an attribute to the export 'Export-Package: a; version=1.0.1; p.C.o > = true' and do the same on 'Import-Package: a; version="[1.0.1,2)"; p.C.o = > true'. Requires that we identify the dependency to method level. > 3. Fix the version of a in X: 'Import-Package: a; version="[1.0.1,1.1)"'. > This however has the implication that X need upgrade when we upgrade B. > 4. Build two bundles out of X, one with 'Import-Package: a; > version="[1.0.1,1.1)"' and one with 'Import-Package: a; > version="[1.1.1,2)"' (assuming the method was also added in 1.1.1). > 5. Branch X into two versions (much like 3), with different import versions > > We think some of these could work, but do wonder if others have run into > the same issues or similar branching/versioning issues and how they solved > it? Any input is highly appreciated. > > Thanks, > > Henning > > > _______________________________________________ > OSGi Developer Mail List > [email protected] > https://mail.osgi.org/mailman/listinfo/osgi-dev > > > > _______________________________________________ > OSGi Developer Mail List > [email protected] > https://mail.osgi.org/mailman/listinfo/osgi-dev >
_______________________________________________ OSGi Developer Mail List [email protected] https://mail.osgi.org/mailman/listinfo/osgi-dev
