Oh wow, I really like the looks of that. I was skeptical before, but if you got that far in a couple of days, I think this is a worth-while endeavor! Thanks so much Matt!
Casey On Thu, Jan 19, 2017 at 12:39 PM, Matt Foley <[email protected]> wrote: > Thanks, Jon! I’m working on characterizing exactly how to fix the two > main issues. I think I’ve got a script that will auto-fix the > triple-backtick problem. The bullet list problem will require > hand-editing, so I want to make sure I’ve got the right recommendation. > > The larger issue is going thru and making the doc content better and more > usable. > But that can occur over time, and will be motivated by having the book > there to gripe about :-) > > On 1/19/17, 9:05 AM, "[email protected]" <[email protected]> wrote: > > Looking at the screenshot, that would be an incredible improvement on > what > we currently have. I'd be happy to help out with any markdown > modifications and documentation cleanup, if necessary, to fix the > problems > you outlined above. > > Jon > > On Thu, Jan 19, 2017 at 11:22 AM Matt Foley <[email protected]> wrote: > > > Sorry, I forgot text-only messages won’t accept attachments. Please > see > > > > https://issues.apache.org/jira/secure/attachment/ > 12848335/Metron-book-screenshot.png > > > > Thanks, > > --Matt > > > > > > On 1/19/17, 6:03 AM, "Otto Fowler" <[email protected]> wrote: > > > > Not seeing the attachment, is it attached to a jira? > > > > > > On January 19, 2017 at 04:19:02, Matt Foley ([email protected]) > wrote: > > > > Here’s a screen shot, attached :-) > > > > On 1/19/17, 1:04 AM, "Matt Foley" <[email protected]> wrote: > > > > Hi all, > > I’ve put together a prototype doc book, along the lines we > discussed, > > and > > it looks pretty worthwhile. > > Many thanks to Mike M. who whipped the pom.xml file into shape, > and > > helped > > me find the right site.xml file to imitate. > > > > If you’re interested, please do a single-branch clone as follows: > > git clone --single-branch -b METRON-660 > > https://github.com/mattf-horton/incubator-metron.git [clonename] > > (or whatever git command pleases you :-) > > > > In this branch, there’s a new top-level subdirectory named > site-book/. > > This > > is not necessarily how we want to integrate stuff, it was just > > convenient > > to do it separate from the existing site/ directory for now. To > build > > the > > book, do these three commands: > > cd site-book/ > > bin/generate-md.sh #This gathers all the *.md files into > > site-book/src/site/markdown/**, and generates the menu tree into > > site-book/src/site/site.xml > > mvn clean site:site #This builds the book with the maven site > plugin > > and > > doxia-markdown plugin > > > > If both those steps are successful, you can then go to a browser > and > > open > > > > file:///Users/yourname/yourpath/clonename/site-book/ > target/site/index.html > > and see the book, with the nav menu on the LHS. > > It’s important to note that a very usable (not perfect) nav > hierarchy > > has > > been auto-generated; this is not hardwired nor hand-edited. > > I do plan to add some overrides that allow individual items in > the > > menu to > > be tweaked. > > > > While it already looks fairly nice, and clearly illustrates the > value > > of > > building a book, there are two glaring issues. > > • Doxia-markdown doesn’t process the triple back-tick (```) the > same > > way as > > Github Markdown. It seems to color-code it as <code>, but doesn’t > > preserve > > line breaks, which is really bad. > > • Similarly, it only processes bullet lists in isolation, and it > > doesn’t > > correctly combine bullet lists subordinate to a numbered list. > > > > The upshot is that > > • both code and bullet lists often lose their linebreaks, and get > > mushed > > into run-on paragraphs, usually combined with the preceding > paragraph, > > and > > • bullet lists interrupt numbered lists and make them start over > at #1. > > > > Perhaps 80-90% of these issues can be fixed by editing the > markdown > > files > > to put blank lines around the list formats. I started doing > this, but I > > didn’t want to obscure the proto by editing tons of .md files. > As of > > this > > proto, only the half dozen actually broken files (that caused > maven > > site > > build errors) have been fixed. > > The other 10-20% will just require simplification of the > markdown used, > > unless we can get an updated version of the plugins. > > > > Anyway, please take a look and share your thoughts. > > > > Thanks, > > --Matt > > > > > > On 1/16/17, 1:02 PM, "Michael Miklavcic" < > [email protected]> > > wrote: > > > > Hey Matt, feel free to ping me. > > > > On Mon, Jan 16, 2017 at 1:39 PM, Matt Foley <[email protected]> > wrote: > > > > > I looked into the Falcon website and doxia over the weekend, > and I’m > > > convinced that using the doxia-markdown plugin should make it > dirt > > simple > > > to do what’s been discussed in this thread, with no overhead > on the > > part > > of > > > people writing the README.md files. > > > > > > I fiddled with trying to do a POC, and unfortunately concluded > > (again) > > > that I don’t really know maven very well :-) > > > Are there any maven experts out there who would be willing to > give me > > some > > > pointers (offline) on how to make use of this apparently > simple maven > > > plug-in? > > > > > > I can do the bit of scripting needed to gather the docs. I’ve > opened > > > https://issues.apache.org/jira/browse/METRON-660 with some > > sub-tasks for > > > this work. > > > --Matt > > > > > > On 1/13/17, 12:04 PM, "[email protected]" <[email protected]> > wrote: > > > > > > +1 on any improvement to documentation and more consistency. > At this > > > point, I think getting rid of or hiding some of the pages on > the wiki > > > (at > > > least for the short term) would be better than leaving them > around > > > because > > > there's a lot of misinformation. > > > > > > Jon > > > > > > On Fri, Jan 13, 2017 at 10:13 AM Nick Allen < > [email protected]> > > > wrote: > > > > > > > +1 I think it is sorely needed. > > > > > > > > If we can come up with a really slick solution like Spark, > then > > > great. I am > > > > also not against a half-baked solution that can later evolve > into > > > something > > > > else. For example, create an index README.md that links > together > > > all the > > > > existing READMEs and run Pandoc on it. Not ideal, but way > better > > > than what > > > > we have. > > > > > > > > > > > > > > > > On Fri, Jan 13, 2017 at 9:53 AM, Otto Fowler < > > > [email protected]> > > > > wrote: > > > > > > > > > I think something that does what you have laid out here, no > > matter > > > the > > > > > implementation details would be ideal > > > > > > > > > > > > > > > On January 12, 2017 at 18:05:24, Matt Foley ( > [email protected]) > > > wrote: > > > > > > > > > > We currently have three forms of documentation, with the > > following > > > > > advantages and disadvantages: > > > > > > > > > > || Docs || Pro || Con || > > > > > | CWiki | > > > > > Easy to edit, no special tools required, don't have to be a > > > developer to > > > > > contribute, google and wiki search | > > > > > Not versioned, no review process, distant from the code, > obsolete > > > content > > > > > tends to accumulate | > > > > > | Site | > > > > > Versioned and reviewed, only committers can edit, google > search | > > > > > Yet another arcane toolset must be learned, only web > programmers > > > feel > > > > > comfortable contributing, "asf-site" branch not related to > code > > > versions, > > > > > distant from the code, tends to go obsolete due to > > non-maintenance > > > | > > > > > | README.md | > > > > > Versioned and reviewed, only committers can edit, tied to > code > > > versions, > > > > > highly local to the code being documented | > > > > > Non-developers don't know about them, may be scared by > github, > > poor > > > > scoring > > > > > in google search, no high-level presentation | > > > > > > > > > > Various discussion threads indicate the developer > community likes > > > > > README-based docs, and it's easy to see why from the > above. I > > > propose > > > > this > > > > > extension to the README-based documentation, to address > their > > > > > disadvantages: > > > > > > > > > > 1. Produce a script that gathers the README.md files from > all > > code > > > > > subdirectories into a hierarchical list. The script would > have an > > > > exclusion > > > > > list for non-user-content, which at this point would > consist of > > > [site/*, > > > > > build_utils/*]. The hierarchy would be sorted depth-first. > The > > > resulting > > > > > hierarchical list at this time (with six added README > files to > > > complete > > > > the > > > > > hierarchy) would be: > > > > > > > > > > ./README.md > > > > > ./metron-analytics/README.md <== (need file here) > > > > > ./metron-analytics/metron-maas-service/README.md > > > > > ./metron-analytics/metron-profiler/README.md > > > > > ./metron-analytics/metron-profiler-client/README.md > > > > > ./metron-analytics/metron-statistics/README.md > > > > > ./metron-deployment/README.md > > > > > ./metron-deployment/amazon-ec2/README.md > > > > > ./metron-deployment/packaging/README.md <== (need file > here) > > > > > ./metron-deployment/packaging/ambari/README.md <== (need > file > > > here) > > > > > ./metron-deployment/packaging/ > docker/ansible-docker/README.md > > > > > ./metron-deployment/packaging/docker/rpm-docker/README.md > > > > > ./metron-deployment/packer-build/README.md > > > > > ./metron-deployment/roles/ <== (need file here) > > > > > ./metron-deployment/roles/kibana/README.md > > > > > ./metron-deployment/roles/monit/README.md > > > > > ./metron-deployment/roles/opentaxii/README.md > > > > > ./metron-deployment/roles/pcap_replay/README.md > > > > > ./metron-deployment/roles/sensor-test-mode/README.md > > > > > ./metron-deployment/vagrant/README.md <== (need file here) > > > > > ./metron-deployment/vagrant/codelab-platform/README.md > > > > > ./metron-deployment/vagrant/fastcapa-test-platform/README. > md > > > > > ./metron-deployment/vagrant/full-dev-platform/README.md > > > > > ./metron-deployment/vagrant/quick-dev-platform/README.md > > > > > ./metron-platform/README.md > > > > > ./metron-platform/metron-api/README.md > > > > > ./metron-platform/metron-common/README.md > > > > > ./metron-platform/metron-data-management/README.md > > > > > ./metron-platform/metron-enrichment/README.md > > > > > ./metron-platform/metron-indexing/README.md > > > > > ./metron-platform/metron-management/README.md > > > > > ./metron-platform/metron-parsers/README.md > > > > > ./metron-platform/metron-pcap-backend/README.md > > > > > ./metron-sensors/README.md <== (need file here) > > > > > ./metron-sensors/fastcapa/README.md > > > > > ./metron-sensors/pycapa/README.md > > > > > > > > > > 2. Arrange to run this script as part of the build > process, and > > > commit > > > > the > > > > > resulting hierarchy list to someplace in the versioned and > > branched > > > > ./site/ > > > > > subdirectory. > > > > > > > > > > 3. Produce a "doc reader" web page that takes in this > hierarchy > > of > > > .md > > > > > pages, and presents a LHS doc tree of links, and a main > display > > > area for > > > > a > > > > > currently selected file. If we want to get fancy, this > page would > > > also > > > > > provide: (a) telescoping (collapse/expand) of the doc > tree; (b) > > > floating > > > > > next/prev/up/home buttons in the display area. > > > > > > > > > > #4. Add to this web page a pull-down menu that selects > among all > > > the > > > > > release versions of Metron, and (if not running in the > Apache > > > site) a > > > > > SNAPSHOT version for the current filesystem version (for > > developer > > > > > preview). Let it re-write the file paths per release > version to > > the > > > > proper > > > > > release tag in github. This web page will therefore be > > > > version-independent. > > > > > Put it in the asf-site branch of the Apache site, as the > new > > "docs" > > > > > sub-site from the home web page. Update the list of > releases at > > > each > > > > > release, or if we want to get fancy, teach it to read the > release > > > tags > > > > from > > > > > github. > > > > > > > > > > 5. As part of the release process, the release manager (a) > > assures > > > the > > > > > release is tagged in github with a consistent naming > convention, > > > and (b) > > > > > submits the new hierarchy of links to google search > (there's an > > > api for > > > > > that). > > > > > > > > > > 6. Deprecate the use of cwiki for anything but long-lived > > > > > demonstrations/tutorials that are unlikely to go obsolete. > > > > > > > > > > > > > > > Do folks feel this would be a good contribution to the > > visibility, > > > > > timeliness, and usability of our docs? > > > > > Is this an adequate solution for the current problems? > > > > > > > > > > Thanks, > > > > > --Matt > > > > > > > > > > > > > > > > > > > > > -- > > > > Nick Allen <[email protected]> > > > > > > > -- > > > > > > Jon > > > > > > Sent from my mobile device > > > > > > > > > > > > > > > > > > > > > > > -- > > Jon > > Sent from my mobile device > > > >
