Yeah, this looks like a huge leap forward, and I'm thrilled that you made such good progress. Great job, Matt.
On Thu, Jan 19, 2017 at 1:55 PM, Michael Miklavcic < [email protected]> wrote: > Agreed! Matt, thanks for taking this on and glad I could help. > > M > > On Thu, Jan 19, 2017 at 11:42 AM, Casey Stella <[email protected]> wrote: > > > Oh wow, I really like the looks of that. I was skeptical before, but if > > you got that far in a couple of days, I think this is a worth-while > > endeavor! Thanks so much Matt! > > > > Casey > > > > On Thu, Jan 19, 2017 at 12:39 PM, Matt Foley <[email protected]> wrote: > > > > > Thanks, Jon! I’m working on characterizing exactly how to fix the two > > > main issues. I think I’ve got a script that will auto-fix the > > > triple-backtick problem. The bullet list problem will require > > > hand-editing, so I want to make sure I’ve got the right recommendation. > > > > > > The larger issue is going thru and making the doc content better and > more > > > usable. > > > But that can occur over time, and will be motivated by having the book > > > there to gripe about :-) > > > > > > On 1/19/17, 9:05 AM, "[email protected]" <[email protected]> wrote: > > > > > > Looking at the screenshot, that would be an incredible improvement > on > > > what > > > we currently have. I'd be happy to help out with any markdown > > > modifications and documentation cleanup, if necessary, to fix the > > > problems > > > you outlined above. > > > > > > Jon > > > > > > On Thu, Jan 19, 2017 at 11:22 AM Matt Foley <[email protected]> > > wrote: > > > > > > > Sorry, I forgot text-only messages won’t accept attachments. > > Please > > > see > > > > > > > > https://issues.apache.org/jira/secure/attachment/ > > > 12848335/Metron-book-screenshot.png > > > > > > > > Thanks, > > > > --Matt > > > > > > > > > > > > On 1/19/17, 6:03 AM, "Otto Fowler" <[email protected]> > > wrote: > > > > > > > > Not seeing the attachment, is it attached to a jira? > > > > > > > > > > > > On January 19, 2017 at 04:19:02, Matt Foley ( > [email protected]) > > > wrote: > > > > > > > > Here’s a screen shot, attached :-) > > > > > > > > On 1/19/17, 1:04 AM, "Matt Foley" <[email protected]> wrote: > > > > > > > > Hi all, > > > > I’ve put together a prototype doc book, along the lines we > > > discussed, > > > > and > > > > it looks pretty worthwhile. > > > > Many thanks to Mike M. who whipped the pom.xml file into > shape, > > > and > > > > helped > > > > me find the right site.xml file to imitate. > > > > > > > > If you’re interested, please do a single-branch clone as > > follows: > > > > git clone --single-branch -b METRON-660 > > > > https://github.com/mattf-horton/incubator-metron.git > > [clonename] > > > > (or whatever git command pleases you :-) > > > > > > > > In this branch, there’s a new top-level subdirectory named > > > site-book/. > > > > This > > > > is not necessarily how we want to integrate stuff, it was > just > > > > convenient > > > > to do it separate from the existing site/ directory for now. > To > > > build > > > > the > > > > book, do these three commands: > > > > cd site-book/ > > > > bin/generate-md.sh #This gathers all the *.md files into > > > > site-book/src/site/markdown/**, and generates the menu tree > > into > > > > site-book/src/site/site.xml > > > > mvn clean site:site #This builds the book with the maven site > > > plugin > > > > and > > > > doxia-markdown plugin > > > > > > > > If both those steps are successful, you can then go to a > > browser > > > and > > > > open > > > > > > > > file:///Users/yourname/yourpath/clonename/site-book/ > > > target/site/index.html > > > > and see the book, with the nav menu on the LHS. > > > > It’s important to note that a very usable (not perfect) nav > > > hierarchy > > > > has > > > > been auto-generated; this is not hardwired nor hand-edited. > > > > I do plan to add some overrides that allow individual items > in > > > the > > > > menu to > > > > be tweaked. > > > > > > > > While it already looks fairly nice, and clearly illustrates > the > > > value > > > > of > > > > building a book, there are two glaring issues. > > > > • Doxia-markdown doesn’t process the triple back-tick (```) > the > > > same > > > > way as > > > > Github Markdown. It seems to color-code it as <code>, but > > doesn’t > > > > preserve > > > > line breaks, which is really bad. > > > > • Similarly, it only processes bullet lists in isolation, and > > it > > > > doesn’t > > > > correctly combine bullet lists subordinate to a numbered > list. > > > > > > > > The upshot is that > > > > • both code and bullet lists often lose their linebreaks, and > > get > > > > mushed > > > > into run-on paragraphs, usually combined with the preceding > > > paragraph, > > > > and > > > > • bullet lists interrupt numbered lists and make them start > > over > > > at #1. > > > > > > > > Perhaps 80-90% of these issues can be fixed by editing the > > > markdown > > > > files > > > > to put blank lines around the list formats. I started doing > > > this, but I > > > > didn’t want to obscure the proto by editing tons of .md > files. > > > As of > > > > this > > > > proto, only the half dozen actually broken files (that caused > > > maven > > > > site > > > > build errors) have been fixed. > > > > The other 10-20% will just require simplification of the > > > markdown used, > > > > unless we can get an updated version of the plugins. > > > > > > > > Anyway, please take a look and share your thoughts. > > > > > > > > Thanks, > > > > --Matt > > > > > > > > > > > > On 1/16/17, 1:02 PM, "Michael Miklavcic" < > > > [email protected]> > > > > wrote: > > > > > > > > Hey Matt, feel free to ping me. > > > > > > > > On Mon, Jan 16, 2017 at 1:39 PM, Matt Foley < > [email protected]> > > > wrote: > > > > > > > > > I looked into the Falcon website and doxia over the > weekend, > > > and I’m > > > > > convinced that using the doxia-markdown plugin should make > it > > > dirt > > > > simple > > > > > to do what’s been discussed in this thread, with no > overhead > > > on the > > > > part > > > > of > > > > > people writing the README.md files. > > > > > > > > > > I fiddled with trying to do a POC, and unfortunately > > concluded > > > > (again) > > > > > that I don’t really know maven very well :-) > > > > > Are there any maven experts out there who would be willing > to > > > give me > > > > some > > > > > pointers (offline) on how to make use of this apparently > > > simple maven > > > > > plug-in? > > > > > > > > > > I can do the bit of scripting needed to gather the docs. > I’ve > > > opened > > > > > https://issues.apache.org/jira/browse/METRON-660 with some > > > > sub-tasks for > > > > > this work. > > > > > --Matt > > > > > > > > > > On 1/13/17, 12:04 PM, "[email protected]" <[email protected] > > > > > wrote: > > > > > > > > > > +1 on any improvement to documentation and more > consistency. > > > At this > > > > > point, I think getting rid of or hiding some of the pages > on > > > the wiki > > > > > (at > > > > > least for the short term) would be better than leaving them > > > around > > > > > because > > > > > there's a lot of misinformation. > > > > > > > > > > Jon > > > > > > > > > > On Fri, Jan 13, 2017 at 10:13 AM Nick Allen < > > > [email protected]> > > > > > wrote: > > > > > > > > > > > +1 I think it is sorely needed. > > > > > > > > > > > > If we can come up with a really slick solution like > Spark, > > > then > > > > > great. I am > > > > > > also not against a half-baked solution that can later > > evolve > > > into > > > > > something > > > > > > else. For example, create an index README.md that links > > > together > > > > > all the > > > > > > existing READMEs and run Pandoc on it. Not ideal, but way > > > better > > > > > than what > > > > > > we have. > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Jan 13, 2017 at 9:53 AM, Otto Fowler < > > > > > [email protected]> > > > > > > wrote: > > > > > > > > > > > > > I think something that does what you have laid out > here, > > no > > > > matter > > > > > the > > > > > > > implementation details would be ideal > > > > > > > > > > > > > > > > > > > > > On January 12, 2017 at 18:05:24, Matt Foley ( > > > [email protected]) > > > > > wrote: > > > > > > > > > > > > > > We currently have three forms of documentation, with > the > > > > following > > > > > > > advantages and disadvantages: > > > > > > > > > > > > > > || Docs || Pro || Con || > > > > > > > | CWiki | > > > > > > > Easy to edit, no special tools required, don't have to > > be a > > > > > developer to > > > > > > > contribute, google and wiki search | > > > > > > > Not versioned, no review process, distant from the > code, > > > obsolete > > > > > content > > > > > > > tends to accumulate | > > > > > > > | Site | > > > > > > > Versioned and reviewed, only committers can edit, > google > > > search | > > > > > > > Yet another arcane toolset must be learned, only web > > > programmers > > > > > feel > > > > > > > comfortable contributing, "asf-site" branch not related > > to > > > code > > > > > versions, > > > > > > > distant from the code, tends to go obsolete due to > > > > non-maintenance > > > > > | > > > > > > > | README.md | > > > > > > > Versioned and reviewed, only committers can edit, tied > to > > > code > > > > > versions, > > > > > > > highly local to the code being documented | > > > > > > > Non-developers don't know about them, may be scared by > > > github, > > > > poor > > > > > > scoring > > > > > > > in google search, no high-level presentation | > > > > > > > > > > > > > > Various discussion threads indicate the developer > > > community likes > > > > > > > README-based docs, and it's easy to see why from the > > > above. I > > > > > propose > > > > > > this > > > > > > > extension to the README-based documentation, to address > > > their > > > > > > > disadvantages: > > > > > > > > > > > > > > 1. Produce a script that gathers the README.md files > from > > > all > > > > code > > > > > > > subdirectories into a hierarchical list. The script > would > > > have an > > > > > > exclusion > > > > > > > list for non-user-content, which at this point would > > > consist of > > > > > [site/*, > > > > > > > build_utils/*]. The hierarchy would be sorted > > depth-first. > > > The > > > > > resulting > > > > > > > hierarchical list at this time (with six added README > > > files to > > > > > complete > > > > > > the > > > > > > > hierarchy) would be: > > > > > > > > > > > > > > ./README.md > > > > > > > ./metron-analytics/README.md <== (need file here) > > > > > > > ./metron-analytics/metron-maas-service/README.md > > > > > > > ./metron-analytics/metron-profiler/README.md > > > > > > > ./metron-analytics/metron-profiler-client/README.md > > > > > > > ./metron-analytics/metron-statistics/README.md > > > > > > > ./metron-deployment/README.md > > > > > > > ./metron-deployment/amazon-ec2/README.md > > > > > > > ./metron-deployment/packaging/README.md <== (need file > > > here) > > > > > > > ./metron-deployment/packaging/ambari/README.md <== > (need > > > file > > > > > here) > > > > > > > ./metron-deployment/packaging/ > > > docker/ansible-docker/README.md > > > > > > > ./metron-deployment/packaging/ > > docker/rpm-docker/README.md > > > > > > > ./metron-deployment/packer-build/README.md > > > > > > > ./metron-deployment/roles/ <== (need file here) > > > > > > > ./metron-deployment/roles/kibana/README.md > > > > > > > ./metron-deployment/roles/monit/README.md > > > > > > > ./metron-deployment/roles/opentaxii/README.md > > > > > > > ./metron-deployment/roles/pcap_replay/README.md > > > > > > > ./metron-deployment/roles/sensor-test-mode/README.md > > > > > > > ./metron-deployment/vagrant/README.md <== (need file > > here) > > > > > > > ./metron-deployment/vagrant/codelab-platform/README.md > > > > > > > ./metron-deployment/vagrant/ > > fastcapa-test-platform/README. > > > md > > > > > > > ./metron-deployment/vagrant/ > full-dev-platform/README.md > > > > > > > ./metron-deployment/vagrant/ > quick-dev-platform/README.md > > > > > > > ./metron-platform/README.md > > > > > > > ./metron-platform/metron-api/README.md > > > > > > > ./metron-platform/metron-common/README.md > > > > > > > ./metron-platform/metron-data-management/README.md > > > > > > > ./metron-platform/metron-enrichment/README.md > > > > > > > ./metron-platform/metron-indexing/README.md > > > > > > > ./metron-platform/metron-management/README.md > > > > > > > ./metron-platform/metron-parsers/README.md > > > > > > > ./metron-platform/metron-pcap-backend/README.md > > > > > > > ./metron-sensors/README.md <== (need file here) > > > > > > > ./metron-sensors/fastcapa/README.md > > > > > > > ./metron-sensors/pycapa/README.md > > > > > > > > > > > > > > 2. Arrange to run this script as part of the build > > > process, and > > > > > commit > > > > > > the > > > > > > > resulting hierarchy list to someplace in the versioned > > and > > > > branched > > > > > > ./site/ > > > > > > > subdirectory. > > > > > > > > > > > > > > 3. Produce a "doc reader" web page that takes in this > > > hierarchy > > > > of > > > > > .md > > > > > > > pages, and presents a LHS doc tree of links, and a main > > > display > > > > > area for > > > > > > a > > > > > > > currently selected file. If we want to get fancy, this > > > page would > > > > > also > > > > > > > provide: (a) telescoping (collapse/expand) of the doc > > > tree; (b) > > > > > floating > > > > > > > next/prev/up/home buttons in the display area. > > > > > > > > > > > > > > #4. Add to this web page a pull-down menu that selects > > > among all > > > > > the > > > > > > > release versions of Metron, and (if not running in the > > > Apache > > > > > site) a > > > > > > > SNAPSHOT version for the current filesystem version > (for > > > > developer > > > > > > > preview). Let it re-write the file paths per release > > > version to > > > > the > > > > > > proper > > > > > > > release tag in github. This web page will therefore be > > > > > > version-independent. > > > > > > > Put it in the asf-site branch of the Apache site, as > the > > > new > > > > "docs" > > > > > > > sub-site from the home web page. Update the list of > > > releases at > > > > > each > > > > > > > release, or if we want to get fancy, teach it to read > the > > > release > > > > > tags > > > > > > from > > > > > > > github. > > > > > > > > > > > > > > 5. As part of the release process, the release manager > > (a) > > > > assures > > > > > the > > > > > > > release is tagged in github with a consistent naming > > > convention, > > > > > and (b) > > > > > > > submits the new hierarchy of links to google search > > > (there's an > > > > > api for > > > > > > > that). > > > > > > > > > > > > > > 6. Deprecate the use of cwiki for anything but > long-lived > > > > > > > demonstrations/tutorials that are unlikely to go > > obsolete. > > > > > > > > > > > > > > > > > > > > > Do folks feel this would be a good contribution to the > > > > visibility, > > > > > > > timeliness, and usability of our docs? > > > > > > > Is this an adequate solution for the current problems? > > > > > > > > > > > > > > Thanks, > > > > > > > --Matt > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > Nick Allen <[email protected]> > > > > > > > > > > > -- > > > > > > > > > > Jon > > > > > > > > > > Sent from my mobile device > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > Jon > > > > > > Sent from my mobile device > > > > > > > > > > > > > > >
