Bryan--
Given that the arrival of some new build infrastructure is coming shortly, it'd be great if we could wait on this patch so it can be updated for the new setup.
One thing I didn't mention previously is that there will be a /distribution.xml file which encapsulates the process of building the distribution (runtime / samples / docs / archives).
Once you're sync'ed up, could you propose a library distribution structure and re-work the patch?
Apologies for having stomped on this!
Thanks,
Eddie
Bryan Che wrote:
Hi, this is a patch to the beehive build system for creating a library distribution. I'll hold off on making other build system changes until Eddie's current refactoring is done. But, I thought I'd throw this out there for now.
From the README:
This is a patch to Beehive's build system. It seperates distribution-building out from build.xml and into a new file, distribution.xml. Furthermore, it adds the ability to build a library distribution and refactors some code for re-use.
To apply: - add distribution.xml to trunk/ - apply the patches in distribution.patch
Bryan
Bryan Che wrote:
Just responses at the top to make things easier to read. =)
Regarding removing third party libraries: if people would prefer to keep them with Beehive for the sake of an easy initial setup, that's fine and I won't push hard for it. I can see the value in keeping the libraries as well.
I am also fine with just shipping build scripts and no deploy scripts with the distribution. Most people likely would not use the deploy scripts. And, yes, I was making a distinction between application deployers and developers.
As for app-server scripts in SVN: I agree that we don't need to ship them in the distribution. But, having them there in the tree for testing purposes would be useful. We wouldn't need to add in support for various app servers all at once. Hopefully, as the Beehive community grows, various people interested in different app servers could contribute and maintain different app-server-specific config files. But, we could still refactor the build files in SVN so that the capability for testing different app servers based on config files like I proposed is there.
I think we pretty much agree on everything else, which is great. =) So, next steps:
- other people comment (hopefully) - you finish your current svn build system work - I submit another proposal reflecting the new state of the build system - people comment and concur (hopefully) - I get to work on the new system
Sound good?
Bryan
Eddie O'Neil wrote:
More comments in line. At some point, we might consider moving this to a wiki.
:)
Would be great to have anyone else interested chime in here.
Bryan Che wrote:
Hi Eddie, thanks for the feedback. Comments are inline. First, some extra background, though: Part of my packaging ideas are driven by the prevalant Linux/Java packaging philsophies at http://jpackage.org. Red Hat also follows jpackage.org guidelines for packaging its own Java software.
In the JPackage world, individual jars are versioned and separately installed via RPM's. These jar RPM's can have dependencies on other jar RPM's as well. All jars are installed into a central repository on a system. Java applications then link to these jars. For example, for its JOnAS distribution, Red Hat does not actually place any required third-party jars under JONAS_ROOT/lib. Rather, it installs jpackage versions of log4j, struts, and so in in a central location and links to them from JONAS_ROOT/lib.
So, in a world where users version and centrally manage their jars, minimizing extra jars in frameworks like Beehive makes things easier.
I actually think that this makes using Beehive from a distribution pretty difficult. It means that users need to go through an assembly step before they can get something that actually runs. In the old days of the Tomcat CVS build, this was a manual process and required developers to download bunches of JARs before being able to get started.
Personally, I'd advocate a model like Struts where the commons-* JARs are packaged with the Struts Library (and Binary) install. They don't just ship you struts.jar -- you get something that can be used to build an application out of the box.
As things like Maven (and JPackage, etc) take hold, we should definitely look at it, but in the near term, the distribution seems best when it's complete.
Log4j seems to be one negotiable part of that, but even there, it cripples a distribution if you can't just download Beehive and start building an application.
Now, on to your comments:
Eddie O'Neil wrote:
Bryan--
Hey; good proposal. I definitely agree that we need to revisit the structure of the distribution in the coming releases to simplify the Ant we provide and to address shipping zips for doc, samples, libraries, and the whole thing.
Couple of general comments:
1) It'd be great to understand more specifics about some of the changes you propose to both SVN and the distribution. Asked some questions about this below. :)
2) I'm still not convinced that Beehive should be in the business of having Ant in SVN and the distribution that is aware of various application containers.
I agree that Beehive should not distribute Ant or even Tomcat in SVN--sorry if that wasn't clear.
You were clear about Ant and Tomcat installers in SVN. I was talking here specifically about Ant build files that are distributed / maintained by Beehive and handle build / deploy for various application containers.
In SVN:
We definitely need some container support so that we can run tests on a running server. But, the number of these SVN-only integrations would presumably be small.
I would vote for including support for the main open-source app servers: tomcat, jonas, geronimo, and jboss. As Beehive gains adoption, we can probably get contributors who work with these various app servers to contribute specific config files if we provide the framework for supporting the config files.
We could definitely look at this, but I'm concerned about what it would mean to track both released and in-development versions of all of these containers. If people can get signed up to maintain those, then we should look at it, but it still seems debatable whether we'd ship them in the distribution.
In the distribution:
For builds, it seems like they should be customizable to adapt to various containers (more on this below). For deployment, this seems like an exercise best left to the developer. Two solutions would be to make container integrations available in a Beehive contrib/ directory (that doesn't ship but is checked in) or to have the community put this type of Ant on the Beehive wiki.
Thoughts from anyone else on this?
I like documentation.
Yeah, documentation for how to do a build and deploy against other containers seems like a good way to go.
For full disclosure, there are a couple of us that are doing work to revamp the entire SVN build system to remove the use of the OS BEEHIVE_HOME variable, to reduce the dependence on hard-coded classpaths and JAR locations, and to remove Tomcat and Ant from SVN. Xibin and I have been doing this on and off for a couple of weeks now. Should be done in another week or two.
That's great. Sounds like making these changes should wait until after your current work is committed.
Probably a good idea. :) We'll try to be quick, but I think these changes will be well worth it.
Other comments are in-line below.
Eddie
Bryan Che wrote:
Hi, this is a proposal for an updated Beehive build system that more broadly addresses support for various application servers (http://issues.apache.org/jira/browse/BEEHIVE-102) and other issues.
Please look it over and comment. If we get consensus on proceeding with something like this, then I will go ahead and do the implementation.
Bryan
------------------------------------------------------------------------
Proposal For a New Apache Beehive Build System
by Bryan Che ([EMAIL PROTECTED]
2005-01-06
************************************************ Contents
1 Introduction 2 Background 3 New Build System Use Cases And Requirements 4 Proposed New Beehive Distributions 5 Proposed New Beehive Source Tree Development Environment 6 Conclusion
************************************************
1 Introduction
This is a proposal for modifying Apache Beehive's current build/test/distribution system so that it will support broader user needs, increase support for various application servers, and improve Beehive's maintainability.
2 Background
Beehive has two main build environments: the distribution environment
and the source-tree environment. The distribution environment is the
environment that Beehive users setup when downloading Beehive for
developing and/or running Beehive-based applications. The source-tree
environment is the environment where developers develop on, build, and
test Beehive itself.
2.1 Distribution Environment
Currently, the Beehive distribution is available as one large download
which includes Beehive jars, third-party jars, sample applications, a
set of build tools for building Beehive applications, and
documentation. The build tools include ant tasks for deploying Beehive
applications to specific application servers and starting/stopping those
application servers. Additionally, the build tools have extension
points for adding support for other application servers.
The Beehive distribution seems targeted at developers who want to
develop applications using Beehive and to work principally with
Beehive's build tools and environment. Because it uncompresses directly
into a usable development environment, it makes starting to develop a
Beehive application straightforward. However, this distribution format
has several deficiencies:
- It does not support users who want to install Beehive but not develop
with it. This includes users who want to deploy their own
applications or users who want to install other applications that
depend upon Beehive
- It does not support developers who want to use Beehive in the context
of a different or larger development environment
- It does not well support multiple developers working on the same
machine
If I understand this and the solution below, this is facilitated by having the "Beehive development tools" distribution installable by each user, correct?
Yes, although that is not the only purpose of the dev tools package. It is also to keep a clean separation between people who want to use/deploy Beehive vs people who want to develop and build with it.
I think I see what you're getting at here -- you're making the distinction between an application deployer and an application developer. The deployer deploys the app and the dev builds it. Is that right?
Assuming this is the case, I'm actually advocating that we not ship any deploy tools as part of the Beehive distribution. Even if we did ship some, they'd probably not be generally useful for someone in the deployer role in an enterprise just because the app deploy process always seems to be more complicated than what we could capture in one Ant file that we provide.
Let me know if I'm missing something here...
- It only supports the specific application servers for which it
includes configuration files
- It includes copies of third-party libraries, which could conflict with
other versions of those libraries elsewhere on a user's system
The 3rd party libraries that Beehive ships in the distribution are those which are *required* for the runtime. For example, Beehive doesn't work with the v1 of Apache XMLBeans, which is why we ship v2 in the distribution. Classes in a Beehive webapp are application scoped, and hopefully don't conflict with libraries elsewhere in a user's application. It may be possible to swap out things like Log4J 1.2.8 for a 1.2.9 or 1.3.0, but this depends on the stability of those APIs.
What currently distributed libraries would you like to see removed?
This is negotiable. Log4J and Axis seem like candidates. So would XMLBeans once v2 is finalized. There are two ways to address requiremenhts. One is to document dependencies and then point the user to where to get them. For example, http://struts.apache.org/userGuide/installation.html#Prerequisites. Another is to put the jars in svn and re-distribute them, which is what Beehive currently does. The former is a bit cleaner and works better for JPackage-style use cases. It also makes upgrading one individual library easier (for example, if there is a critical bug fix to a jar that doesn't change functionality). The latter makes it easier for a new user to get started with Beehive.
Yeah, as above, I'd still argue for shipping a complete Beehive distribution. The Struts link above talks about both JARs that the developer provides and those that are provided in the distribution (commons-*, for example).
Any of the Struts committers reading this want to comment on how / if you'd change your distribution relative to shipping dependencies?
Thus, the current Beehive distribution works great for a specific set of
users--those who are primarily developing Beehive on a clean system and
with an application server that Beehive supports. It does not work well
for many other users.
2.2 Source-Tree Environment
Beehive's source-tree environment is easy to configure and setup for
developers who want to work on Beehive itself. It provides ant tasks
for setting up an work environment, and it also includes automated unit
tests. These tests include mechanisms for testing Beehive inside of the
Tomcat application server. This environment, though, also has a couple
deficiencies:
- It only supports Tomcat and no other application servers
- It includes a large set of third party software that would be better
left for the user to obtain and install
Adding support for other application servers and minimizing the set of
included software would make Beehive more portable and easier to
maintain.
3 New Build System Use Cases And Requirements
3.1 Use Cases
An improved Beehive build system should support the following use cases:
It should support two main groups of users: people who want to obtain
a Beehive distribution, and people who want to work on Beehive itself.
3.1.1 Distribution Use Cases
For people who want to work on a Beehive distribution, the build system and environment should enable them to work in several manners: - As developers of Beehive-based applications - As developers of large applications that depend upon Beehive - As users of Beehive applications
Furthermore, multiple users should be able with Beehive on the same
machine, and they should be able to work with the application servers of
their choice.
3.1.2 Source-Tree Use Cases
Developers working on the Beehive source-tree should be able to setup a
work environment, develop on Beehive, build Beehive, and test Beehive.
Furthermore, they should be able to differentiate Beehive itself from
other software like application servers.
3.2 Requirements
Based upon the preceding use cases and also experience with the current
Beehive build system, a new Beehive build system should have the
following requirements:
3.2.1 Distribution Requirements
- The distribution should support two different sets of users equally
well: those who just want to use Beehive, and those who want to
develop Beehive applications
- The distribution should be able to setup a proper development
environment for those users who want one
- The distribution should support multiple users per box
- The distribution should support any application server
- The distribution should support mechanisms for improving integration
with specific application servers
Seems to me that there are two major issues for app container integration -- application build and application deployment.
On the application build side, we need two things:
1) decompose the application build infrastructure so that the various steps (XMLBean build, controls build, JPF build, etc) are separated and can be composed or worked into an existing app build by a Beehive user to build Beehive artifacts. Ant 1.6 provides several good mechanisms for this.
2) make it possible to customize application classpaths to include container specific JARs used during a build. For example, on Tomcat, the JSP / Servlet APIs are located in common/lib, and on WebLogic, these are in weblogic.jar. This user-defined <path> would be used by the "compile JPF" Ant snippet provided in (1).
For deployment, Beehive would not support any application server OOTB -- users need to use their own Ant to manage the application (deploy / undeploy / redeploy).
:)
Again, perhaps we can add these to a contrib/ or to the wiki.
Yes, I would advocate documenting what is required in a classpath in order to build and then having the user specify the location of those classes. Documenting common app server examples seems good to me.
Definitely.
3.2.2 Source Tree Requirements
- The source-tree environment should support developers working on Beehive - The environment should support automated unit testing on various application servers - The environment should minimize bundling of third-party libraries
Specifically, which ones?
- The environment should be easy to setup for working
4 Proposed New Beehive Distributions
The Beehive distribution should be available in multiple binary packages: - Beehive Libraries - Beehive Developer Tools - Beehive Documentation - Full Beehive
4.1 Beehive Libraries
Beehive libraries would consist of just jars of Beehive code and nothing
else. This distribution would be for users who have no intention of
developing with Beehive--they just need it as a dependency. Also, it
would benefit packagers who want to re-distribute Beehive with other
software.
Definitely! We need to have a library-only distribution a la the Struts Library one.
4.2 Beehive Developer Tools
The Beehive Developer Tools would come with a set of scripts to setup
and support development environments for working with Beehive. This
environment would be similar to the current Beehive distribution
environment. However, it would have several differences:
- It would not include jars in itself. Rather it would depend upon the
Beehive Libraries distribution for providing Beehive jars and require
installation of third-party software (just like the current Beehive
distribution requires the user to install ant, Tomcat and Java).
- It could be installed in multiple places on the same machine. It
would have configuration files pointing to where canonical Beehive and
other third-party libraries are located
- It would build and deploy wars of Beehive-based applications.
Currently, deploying is contained in application-server-specific build
files. The new developer tools would have an
application-server-neutral deploy task that builds a war and copies it
to a user-specified location. Extra, application-server-specific
build files would add further functionality for popular servers
- It would provide tools to verify that the development environment is
properly setup and that necessary libraries are present
If this is something that the community would be interested in, I think this would be great!
One question, though -- is this meant for devs to customize their "Beehive Developer Tools" environment individually? If several users are sharing a Beehive installation, would it be the case that it's easier to customize the one install for a particular container?
The separate dev tools package isn't just for letting multiple users work on the same box. It's necessarily its own package if we ship the beehive libraries separately. But, because it's its own package, different users can install their own dev environments in various locations and configure them as they wish.
Got it, thanks.
4.3 Beehive Documentation
Comprehensive Beehive documentation would be available as its own download.
Agreed.
4.4 Full Beehive
A full Beehive download would include all the other Beehive distributions: the libraries, developer tools, and documentation.
5 Proposed New Beehive Source Tree Development Environment
The new Beehive source tree development environment would remain similar
to the current one. However, it would have the following changes:
- The build files would be refactored to support multiple application
servers rather than just Tomcat. The end product would be similar to
the build files in the current (not new) Beehive distribution build
files: there would be application-server-specific files for
running/deploying servers and a configuration switch for changing
between application servers. Since Beehive is an open-source project,
it could rely to some extent on its community to contribute
application-server files for servers that people want to see supported
As with the distribution, I'm concerned about having container-specific integrations in trunk/. We need at least one container (presumably Tomcat) in trunk/ for testing. But, I'd vote for making it possible to *enable* cleanly plugging in other containers rather than trying to to *solve* it for various containers.
The patch I submitted for supporting JOnAS in the distribution lets you plugin various app servers by writing one ant build file for each app server you want to support. We could do something similar here by including a minimal set of config files in the trunk and keeping a separate repository of other app server config files. But, testing with more than one app server may reveal bugs in Beehive masked by a buggy app server.
Agreed. And, Tomcat / JOnAS seem like two good containers to support in SVN for the sake of testing. Though, it seems hard to justify drawing a line between OSS containers and commercial containers for supporting application deployment in the distribution.
That's why I'd just leave it as an exercise for the container user. In general, distribution users are going to already have app deployment Ant they're using for existing applications, or these deployment commands are already provided with the container itself.
That being said, we may have to make some changes to the way the tests run to make it easier to run test webapps on other servers.
- The tree would not include Tomcat or Ant and would also remove other
third-party libraries that are common and easy to obtain. The build
scripts would be updated to verify that this software is present
Definitely. We're working on this now, and it should be done in a week or two.
- The source tree build tools would be able to build the various types
of new Beehive distributions
Yep. I'd love to see a distribution.xml Ant file in trunk/ that knows how to build all of these things. Want to take a crack at this?
;)
I'm happy to work on this as well as all the other changes I proposed (or whatever ends up being the consensus).
6 Conclusion
By making these changes to Beehive's build system, Beehive would be able
to suit a broader audience beyond just developers using Tomcat. In
particular, the distribution changes would make things much simpler for
Beehive users who do not develop. And, they would bring Beehive
packaging more in line with that of other software like Struts.
Providing specific support for various application servers in the
source-tree environment would allow people to test that Beehive works on
servers other than Tomcat. Conversely, it would also allow
application-server providers to certify Beehive on their own application
server.
Absolutely -- the container specific testing is important. We just need to figure out how to get there.
