[this announcement is available online at https://s.apache.org/yFgQ ]

By Nick Kew
I started working on a range of Web applications in the 1990s, the first of 
them internal to my (then) workplace where it provided an operator interface to 
the daily processing, archiving and distribution of satellite image data; the 
second a forerunner of what is now called social media, and my first use of an 
Apache server. The release of Apache HTTPD 2.0 drew me from server user to 
developer: in part because I needed to re-implement some existing functions, 
but more excitingly because I saw tremendous potential for the server itself to 
become a powerful applications platform. This led me to working on the core 
software and interacting with the Apache community alongside releasing my own 
modules and documentation. In 2003 I gave my first ApacheCon presentation, and 
sometime after that was invited into the Foundation first as a Committer, and 
became a Member in 2005. Since then my interests have encompassed not just the 
Web server and related projects, but also the Apache community and its 
dynamics. I’ve been involved in mentoring several projects through the 
Incubator. If you were to ask me today about the single goal I’d most like to 
accomplish, it’s a framework for Identity management that is not merely 
cryptographically strong, but sufficiently straightforward for the world to 
use, and robust against social engineering attacks such as phishing, while at 
the same time free of any centralised authority (such as government) whose 
motives might come under suspicion. An end to identity fraud, and to password 
management nightmares.

Much has been said and written about The Apache Way [1]. It typically focusses 
on the importance of community, and on the democratic and meritocratic elements 
of project governance. And on the role of an Apache project's formal governing 
body, the Project Management Committee (PMC), comprising contributors elected 
by the community on the basis of their track record of contributions and 
constructive engagement.

In practice, the role of the PMC is largely reactive. The big, interesting 
questions like "what direction does our project go from here" are discussed in 
public, where inputs from the wider community are welcomed. A lot of the 
nitty-gritty detail is determined by the needs of the community as measured by 
feedback received and seen, and by the needs of individual developers. The 
latter is often described as "scratch your own itch".

Setting aside the purely reactive, the detail of what actually happens is often 
determined as much by what a developer is prepared to work on than by any grand 
plan. If you do the work, then you control what happens. If you need something, 
you go ahead and make it happen. This has occasionally been described by a 
clumsy English-Greek hybrid word "do-ocracy". In the hope that it's not too 
late to nip that in the bud before it becomes as ubiquitous as the Greek/Latin 
hybrid of which CP Scott memorably told us no good would come [2], I shall 
instead call it Pratocracy. Rule of the Makers. It applies both within an 
Apache core team and in the broader community.

So how does a Pratocratic project work?  How does it avoid the chaos of a Tower 
of Babel?  How does it remain cohesive and focussed? And if an Apache project 
looks like a good basis for your company's needs, how best can you work with 
the Pratocracy towards your goals?

Taking the first question first, how does the Pratocracy work? Eric Raymond 
proposed the contrast of the Cathedral vs the Bazaar: the autocracy of the 
traditional project vs a free ecosystem. The Bazaar ecosystem is essentially 
Darwinian, in that the strongest projects prosper and evolve, while a Long Tail 
go nowhere very much and are forgotten as their authors move on to new things.

The Apache processes are all about ensuring our projects are fit to be among 
the winners in that ecosystem. An Apache project is a graduate of an incubation 
process [3] that requires it to build a broad development community, with 
sufficient diversity to survive the loss not merely of a key individual, but 
even of a key company team that might have been the originators of the project. 
The key principle is that if the community is healthy, the software will follow.

Thus the context of the Pratocracy is selective: it is the well-known 
Meritocracy element of Apache. Members of the core community are elected based 
on demonstrated commitment to the project. That means they understand the 
distinction between Apache work and company work, and are comfortable holding 
multiple allegiances and avoiding or resolving potential conflicts between them.

So, to the crucial question. Suppose I have identified an Apache project that 
doesn't fully meet my company's needs, but is the ideal starting point for our 
project. How do I best work with the Pratocracy to make it happen?

Let's consider some possible approaches:
1. Ask via the development list and issues database.
2. Approach developers individually to discuss the project.
3. Just go ahead with the company's in-house developers.

Each of these approaches has merit in its own right. But in many cases, the 
ideal approach will encompass all the above. In an Apache community, the whole 
is greater than the sum of the parts. The closer you can work with the 
community, the more benefits you stand to derive, particularly if your project 
finds itself wanting to integrate more deeply with the Apache project than is 
provided for in stable public interfaces, when the alternative to working 
within the Community is a patched core and the risk of a maintenance nightmare. 
On the other hand, there are good and bad ways to engage with an Apache 
community: the right approach can make all the difference to the benefits you 
can derive from it.

First, it's always worth asking about your needs. Post a question about your 
requirements to the project's lists, or issues database. Engage in discussion. 
It may be that someone somewhere has already done the work, or that it's 
closely aligned with someone's work-in-progress. Even if you draw a blank, it's 
a start to engaging with the community.

Approaching individuals is most likely to be appropriate if you're looking to 
hire in expertise. General questions are likely to be redirected to public 
support fora, but some Apache developers work as consultants and may welcome a 
professional approach. They can help you figure out the best way to proceed, 
both technically and in terms of working with the Apache community. But bear in 
mind the potential for conflicts of interest. If you want your work to happen 
within the Apache project, your consultant will want to be satisfied that you 
and your project are indeed a good fit, and won't cause trouble such as 
unmaintainable software down the line. And of course, any NDA should make it 
clear you're not laying claim to the consultant's regular Apache work!

If you have the in-house developers for it, you might just task them to go 
ahead with the work. That's easiest when you can work with public, stable 
interfaces of the Apache project, harder and messy if you have to hack into the 
core. When you start that work, the Apache community becomes your team's best 
resource, so if your developers are any good (and not cowed into keeping their 
heads down and never talking to outsiders) they'll be interacting with the 
Apache community as they go along.

Now you're participating in the Pratocracy, albeit (usually) outside the core 
community. You have something to offer. If you share it with Apache, members of 
the Apache community will take a look, and may get involved. At best this can 
lead to a virtuous circle where your developers not merely benefit from the 
Apache team, but contribute to it, and earn themselves the merit to be invited 
to join. But this is by no means automatic: both developers and their managers 
need to be able to see outside corporate control structures. Crucially, 
individual developers have to be comfortable separating out their Apache hats 
from their professional allegiances, which means managers have to allow that to 
happen. But don't try to manage such a process from above: that's herding cats. 
Just facilitate!

That probably means that at some stage you'll have to sit down with the team 
and fix boundaries: what belongs in the Apache project vs what belongs in your 
company's own space. It helps immensely with developers' confidence if they're 
not forever looking over their shoulders and wrestling with ambiguity. This may 
be obvious if your project isn't destined to be shared openly with the world, 
but it's a discussion worth having even if everything is Apache-licensed. In 
the latter case, the boundaries will focus on what your company expects to sell 
and/or support commercially and wants to maintain full control on, vs what 
passes out of your direct control.

Even if your company merely clones or mirrors the Apache project and 
contributes everything, you'll want to separate out matters of your business 
model, and customer-facing vs Apache-facing matters.

Note that it's easy to get this kind of thing wrong. Something we sometimes see 
at Apache is developers who've been told by their managers to join an Apache 
community, but have no specific tasks and no motivation. That becomes painfully 
obvious, and leads nowhere useful.
The right approach is to give your developers real-world tasks and let them 
come to the Apache community - if their course takes them that way - in their 
own time. If close interaction with the Apache project is an important goal, 
you may want to approach and maybe hire individual developers having a strong 
track record with Apache or comparable Open Source environments.

To conclude, let's consider a real-world example from my own experience with 
HTTPD (the Web server still commonly called "Apache") and APR projects. The 
history of SQL database integration into the Web server spans many project 
teams, both within and outside Apache, both corporate and community-driven. 
Each developer or team just went ahead with something appropriate to its time, 
and (at least those you or I have heard of) achieved success in their 
respective contexts.

In the early days, there was a Web server with no SQL support. So any 
application that required it had to manage its own database connections. A 
range of third-party modules were developed for purposes such as authentication 
by SQL, and logging to SQL, both with popular databases such as MySQL and 
PostgreSQL and in commercially-supported products with commercial databases. 
Generalised frameworks for Web/SQL applications were provided by scripting 
modules, pioneered by mod_perl, giving developers a language's own database 
interfaces, together with a persistent database connection. 

In time, the server and its applications outgrew this ad-hoc approach, with 
multi-threaded processing and multiple applications in a server. Building on a 
range of ideas, a native SQL database connection framework was proposed and 
developed, offering some major efficiency gains to applications through dynamic 
connection pooling and sharing. It also promoted rationalisation, so in an area 
like authentication, a whole raft of different database-driven modules could be 
replaced by a single module for generic SQL authentication. Different SQL 
databases are handled by per-database drivers which serve all kinds of 
application in the server.
This work took place within the core Apache team, and propagated more widely as 
others such as the scripting languages incorporated it as an option for their 
users. It evolved from minimalist beginnings to provide more capabilities as 
required by developers "scratching an itch", and has benefited from 
contributions of database drivers from a range of different sources.
Having said that, it remains intentionally fairly small, and won’t meet the 
needs of every application. Some projects continue to use SQL their own way. 
The fact there is now a Right Way to access SQL doesn't mean that other ways 
are wrong! There are still third-party projects that, for their own reasons, 
use other ways to access SQL. The fact that someone has done the work and met a 
need gives them an entirely valid niche in a Pratocratic ecosystem.

The bottom line? Many Apache users can just use our software off-the-shelf 
(perhaps after asking around for solutions to problems), others may need to add 
significantly to it to meet their needs. If you fall into the latter camp, just 
go ahead and do it: you can discuss and/or share your work at any point you 
choose! The basic management decision is the tradeoff between control of your 
work and community input. Interaction with the project's core people is down to 
how your developers work best. If leaving that to chance is not sufficient 
--perhaps because you would like to influence the project's core software or 
introduce new APIs-- that's the point when you’ll need to consider your 
developers’ track records in community-driven Open Source, and make sure you 
have that expertise to hand.

[1] https://www.apache.org/foundation/how-it-works.html
[2] https://en.wikiquote.org/wiki/C._P._Scott
[3] http://incubator.apache.org/incubation/Process_Description.html


# # #

"Success at Apache" is a new monthly blog series that focuses on the processes 
behind why the ASF "just works":
1) Project Independence https://s.apache.org/CE0V
2) All Carrot and No Stick https://s.apache.org/ykoG
3) Asynchronous Decision Making https://s.apache.org/PMvk

Reply via email to