Matt, thanks for pulling this together. I completely agree that we need to go all in on either cwiki or the README.md's. I think the wiki is poorly updated and can cause confusion for new users and devs. My preference is certainly for the README.md's.
I like your approach but also agree that we shouldn't need to roll our own here. I really like the Spark documentation that Mike pointed out. Any way we can duplicate/adapt their approach? -Kyle On Thu, Jan 12, 2017 at 7:19 PM, Michael Miklavcic < [email protected]> wrote: > Casey, Matt - These guys are using doxia > https://github.com/apache/falcon/tree/master/docs > > Honestly, I kind of like Spark's approach - > https://github.com/apache/spark/tree/master/docs > > Mike > > On Thu, Jan 12, 2017 at 4:48 PM, Matt Foley <[email protected]> wrote: > > > I’m ambivalent; I think we’d end up tied to the doxia processing > pipeline, > > which is “yet another arcane toolset” to learn. Using .md as the input > > format decreases the dependency, but we’d still be dependent on it. > > > > I had anticipated that the web page would be a write-once thing that > would > > be only a couple days for an experienced Web developer. But I was going > to > > get an estimate from some co-workers before actually trying to get it > > implemented. And the script is a few hours of work with find and awk. > > > > On the other hand, doxia is certainly an expectable solution. Is setting > > up that infrastructure less work than developing the web page? Or is it > > actually just a matter of a few lines in pom.xml? > > > > > > On 1/12/17, 3:24 PM, "Casey Stella" <[email protected]> wrote: > > > > Just a followup thought that's a bit more constructive, maybe we > could > > migrate the README.md's into a site directory and use doxia markdown > > (example here <https://github.com/larrycai/doxia-markdown-demo>) to > > generate the site as part of the build to resolve 1 through 3? > > > > On Thu, Jan 12, 2017 at 6:19 PM, Casey Stella <[email protected]> > > wrote: > > > > > So, I do think this would be better than what we currently do. I > > like a > > > few things in particular: > > > > > > - I don't like the wiki one bit. > > > - We have a LOT of documentation in the README.md's and it's > > sometimes > > > poorly organized > > > - I like a documentation preprocessing pipeline to be present. > > For > > > instance, a major ask is all of the stellar functions in one > > place. That's > > > solved by updating an index manually in the READMEs and keeping > > it in sync > > > with the annotation. I'd like to make a stellar annotation -> > > markdown > > > generator as part of the build and this would be nice for such a > > task. > > > > > > My only concern is that the html generation/viewer seems like a > fair > > > amount of engineering. Are you sure there isn't something easier > > that we > > > could conform to? I'm sure we aren't the only project in the world > > that > > > has this particular issue. Is there something like a maven site > > plugin or > > > something? Just a thought. I'll come back with more :) > > > > > > Great ideas! Keep them coming! > > > > > > Casey > > > > > > On Thu, Jan 12, 2017 at 6:05 PM, Matt Foley <[email protected]> > > wrote: > > > > > >> We currently have three forms of documentation, with the following > > >> advantages and disadvantages: > > >> > > >> || Docs || Pro || Con || > > >> | CWiki | > > >> Easy to edit, no special tools required, don't have to be a > > >> developer to contribute, google and wiki search | > > >> Not versioned, no review process, distant from the code, obsolete > > content > > >> tends to accumulate | > > >> | Site | > > >> Versioned and reviewed, only committers can edit, google > > search | > > >> Yet another arcane toolset must be learned, only web > > programmers > > >> feel comfortable contributing, "asf-site" branch not related to > code > > >> versions, distant from the code, tends to go obsolete due to > > >> non-maintenance | > > >> | README.md | > > >> Versioned and reviewed, only committers can edit, tied to > code > > >> versions, highly local to the code being documented | > > >> Non-developers don't know about them, may be scared by > > github, poor > > >> scoring in google search, no high-level presentation | > > >> > > >> Various discussion threads indicate the developer community likes > > >> README-based docs, and it's easy to see why from the above. I > > propose this > > >> extension to the README-based documentation, to address their > > disadvantages: > > >> > > >> 1. Produce a script that gathers the README.md files from all code > > >> subdirectories into a hierarchical list. The script would have an > > >> exclusion list for non-user-content, which at this point would > > consist of > > >> [site/*, build_utils/*]. The hierarchy would be sorted > > depth-first. The > > >> resulting hierarchical list at this time (with six added README > > files to > > >> complete the hierarchy) would be: > > >> > > >> ./README.md > > >> ./metron-analytics/README.md <== (need file here) > > >> ./metron-analytics/metron-maas-service/README.md > > >> ./metron-analytics/metron-profiler/README.md > > >> ./metron-analytics/metron-profiler-client/README.md > > >> ./metron-analytics/metron-statistics/README.md > > >> ./metron-deployment/README.md > > >> ./metron-deployment/amazon-ec2/README.md > > >> ./metron-deployment/packaging/README.md <== (need file here) > > >> ./metron-deployment/packaging/ambari/README.md <== (need file > here) > > >> ./metron-deployment/packaging/docker/ansible-docker/README.md > > >> ./metron-deployment/packaging/docker/rpm-docker/README.md > > >> ./metron-deployment/packer-build/README.md > > >> ./metron-deployment/roles/ <== (need file here) > > >> ./metron-deployment/roles/kibana/README.md > > >> ./metron-deployment/roles/monit/README.md > > >> ./metron-deployment/roles/opentaxii/README.md > > >> ./metron-deployment/roles/pcap_replay/README.md > > >> ./metron-deployment/roles/sensor-test-mode/README.md > > >> ./metron-deployment/vagrant/README.md <== (need file here) > > >> ./metron-deployment/vagrant/codelab-platform/README.md > > >> ./metron-deployment/vagrant/fastcapa-test-platform/README.md > > >> ./metron-deployment/vagrant/full-dev-platform/README.md > > >> ./metron-deployment/vagrant/quick-dev-platform/README.md > > >> ./metron-platform/README.md > > >> ./metron-platform/metron-api/README.md > > >> ./metron-platform/metron-common/README.md > > >> ./metron-platform/metron-data-management/README.md > > >> ./metron-platform/metron-enrichment/README.md > > >> ./metron-platform/metron-indexing/README.md > > >> ./metron-platform/metron-management/README.md > > >> ./metron-platform/metron-parsers/README.md > > >> ./metron-platform/metron-pcap-backend/README.md > > >> ./metron-sensors/README.md <== (need file here) > > >> ./metron-sensors/fastcapa/README.md > > >> ./metron-sensors/pycapa/README.md > > >> > > >> 2. Arrange to run this script as part of the build process, and > > commit > > >> the resulting hierarchy list to someplace in the versioned and > > branched > > >> ./site/ subdirectory. > > >> > > >> 3. Produce a "doc reader" web page that takes in this hierarchy of > > .md > > >> pages, and presents a LHS doc tree of links, and a main display > > area for a > > >> currently selected file. If we want to get fancy, this page would > > also > > >> provide: (a) telescoping (collapse/expand) of the doc tree; (b) > > floating > > >> next/prev/up/home buttons in the display area. > > >> > > >> #4. Add to this web page a pull-down menu that selects among all > the > > >> release versions of Metron, and (if not running in the Apache > site) > > a > > >> SNAPSHOT version for the current filesystem version (for developer > > >> preview). Let it re-write the file paths per release version to > > the proper > > >> release tag in github. This web page will therefore be > > >> version-independent. Put it in the asf-site branch of the Apache > > site, as > > >> the new "docs" sub-site from the home web page. Update the list > of > > >> releases at each release, or if we want to get fancy, teach it to > > read the > > >> release tags from github. > > >> > > >> 5. As part of the release process, the release manager (a) assures > > the > > >> release is tagged in github with a consistent naming convention, > > and (b) > > >> submits the new hierarchy of links to google search (there's an > api > > for > > >> that). > > >> > > >> 6. Deprecate the use of cwiki for anything but long-lived > > >> demonstrations/tutorials that are unlikely to go obsolete. > > >> > > >> > > >> Do folks feel this would be a good contribution to the visibility, > > >> timeliness, and usability of our docs? > > >> Is this an adequate solution for the current problems? > > >> > > >> Thanks, > > >> --Matt > > >> > > >> > > >> > > > > > > > > > > > >
