On Tuesday, March 11, 2003, at 05:32 AM, Pier Fumagalli wrote:
I suggest linking to http://xml.apache.org/cocoon/mirror.cgi#nightly but I guess that's your intention ;-)
Yes, but at that point we'll have to re-build the site once again...
Ok, we shouldn't be so limited in rebuilding the site as often as necessary.
The wiki shows pretty evidently that our documentation publishing system is currently too hard to use and to slow to work with. This is evident when people are even afraid to touch it.
We must fix this.
Here's why it has been tedious and time consuming in the past, at least for me.
Ok, let's start planning something better.
1. Many, many committers weren't updating release and head branches with their doc updates. It took time to scrutinize differences in the branches, to make sure all relevant docs were in the release branch, which is what is used to generate the web site.
Agreed this is a problem.
Hopefully, now that we have two clearly separated repositories, people will document only in the appropriate one.
2. Updating the live site repository is time consuming, at least for me, on a slow dial-up connection (I live in a rural area of the US with no broadband option). The api docs directory is the time killer here. I spent eight hours, one night, simply performing a cvs update followed by a cvs commit. The most recent update wasn't so bad. The commit/update took only 2.5 hours.
Oh, god. I didn't know that. This is a shame. I'm sorry Diana. I know you don't pay dial-up per-minute fees as we normally do here in europe, but still.
we must make a better system.
3. I was really excited about Forrest transition, thinking the automation would save me all of the above time which I could devote to docs content. Unfortunately:
- only a few committers participated in the trial run, so it seemed to me, interest/support is not that great.
I would like to know the issues that are still left on the table to solve and work on them. Forrest is clearly the way to go and the site transition give us the opportunity to think about it.
- Forresters seemed to suggest, and I could be wrong, that the live site cvs update would **still** be required even with Forrest.
No, not necessarely, but a totally different system must be setup in place.
Thus, I failed to see how the transition would make my volunteer committer life any more liberated, since this time killing step was still necessary.
This *must* go away.
I'm happy to help with updating the site based on the revised cvs mirror links discussed in this thread. However, I can't do it until later this week. In the future, I think it's better if more committers would share the burden of updating the live site cvs every now and then, particularly those with greater bandwidth connections. In the hopes that this will happen, I'll post detailed instructions on how this can be done on wiki. (I've posted email instructions on two separate occasions in the past which I will now fine tune.)
Please do, those will help, but for now, let's clear the whiteboard and start outlining the best publishing system.
- o -
Apache has very high security standards. If our web sites get hacked, the ASF image of quality and security is damaged. Having easier to install docs, but lower security is not an option.
This means that every solution must be *designed* around security.
IMO, the metapattern of IoC gives us a lot of security. So, the best publishing system would be something like:
repository -(generation)-> staging -(publishing)-> production
where
repository is used for storing our documents
stating is the location of the staged documentation
production is the location of the final docs
From a security analysis, a compromise of the staging area is not a big risk, this means that staging can be automated without major political issues.
While the above arrows indicate the flow of data, the flow of control must be inverted to provide complete vertical security:
repository <-(reads from)- staging <-(reads from)- production
- o -
So, here is the plan I propose:
1) repository is CVS on icarus. as it is today. no changes required in the editing/authoring process (for now, at least)
2) automated staging server is moof (or nagoya)
I'd suggest to install it on moof (or nagoya) [moof is a macosx server donated to the ASF by apple and located in their campus, lots of bandwidth and support for final java 1.4.1 as for yesterday, administred by the apache instrastructure]
Checkout is done over anoncvs, so no possible compromise from the staging server to the repository.
3) the staging server should work for docs as gump works for nightly builds, nagging the appropriate mail list if:
- docs aren't valid - links are broken
Note that javadocs and idldocs must be automated as well.
4) the staging server will regenerate results automatically grabbing the changes out of CVS directly. this operation will perform unassisted, just like gump. in fact, forrest was born to be the gump alter-ego for documentation.
5) when publishing on production is needed, a person with an account on icarus will simply log in and perform a remote rsync between the staging area and the server. A simple script with a readme will do the job. I estimate it might take less than 60 seconds to update the web site this way since the thruput between moof and daedalus is several Mbits.
- o -
From a usability point of view we gain:
- no need for broadband. This means that people can update the site even on a GPRS cellphone, or on a slow link in africa.
- we are nagged if things go wrong: continous integration for documents.
- the site update frequency will hopefully improve, thus improving our quality of service.
From a security point of view:
- we use existing ASF-proven security infrastructure (everything is done over SSH)
- by keeping the stating server on a different machine and with no information on how to access the others, we don't create more points of failure
- o -
NOTE: from an operativity point of view, Pier has enough karma to setup everything we need on moof or nagoya, as well as providing accounts for those who want to help running the staging server (I would suspect Jeff and Steven to be interested in helping out, hopefully others as well). We might need to post our plan of action to infrastructure@ once we decide what to do, but since there are no security issues they shouldn't be concerned about it (I've already discussed this architecture and people didn't have objections).
Comments and suggestions will be very appreciated.
Thanks.
S.
