This is fantastic. Great job, Matt 29.01.2017, 22:13, "Matt Foley" <[email protected]>: > Hi all, please take a look at > https://github.com/apache/incubator-metron/pull/429 > I think all major issues are resolved, as best I can tell from an eyeball > scan of the result. > Thanks, > --Matt > > On 1/19/17, 12:06 PM, "Justin Leet" <[email protected]> wrote: > > Yeah, this looks like a huge leap forward, and I'm thrilled that you made > such good progress. Great job, Matt. > > On Thu, Jan 19, 2017 at 1:55 PM, Michael Miklavcic < > [email protected]> wrote: > > > Agreed! Matt, thanks for taking this on and glad I could help. > > > > M > > > > On Thu, Jan 19, 2017 at 11:42 AM, Casey Stella <[email protected]> > wrote: > > > > > Oh wow, I really like the looks of that. I was skeptical before, but > if > > > you got that far in a couple of days, I think this is a worth-while > > > endeavor! Thanks so much Matt! > > > > > > Casey > > > > > > On Thu, Jan 19, 2017 at 12:39 PM, Matt Foley <[email protected]> wrote: > > > > > > > Thanks, Jon! I’m working on characterizing exactly how to fix the > two > > > > main issues. I think I’ve got a script that will auto-fix the > > > > triple-backtick problem. The bullet list problem will require > > > > hand-editing, so I want to make sure I’ve got the right > recommendation. > > > > > > > > The larger issue is going thru and making the doc content better and > > more > > > > usable. > > > > But that can occur over time, and will be motivated by having the > book > > > > there to gripe about :-) > > > > > > > > On 1/19/17, 9:05 AM, "[email protected]" <[email protected]> wrote: > > > > > > > > Looking at the screenshot, that would be an incredible improvement > > on > > > > what > > > > we currently have. I'd be happy to help out with any markdown > > > > modifications and documentation cleanup, if necessary, to fix the > > > > problems > > > > you outlined above. > > > > > > > > Jon > > > > > > > > On Thu, Jan 19, 2017 at 11:22 AM Matt Foley <[email protected]> > > > wrote: > > > > > > > > > Sorry, I forgot text-only messages won’t accept attachments. > > > Please > > > > see > > > > > > > > > > https://issues.apache.org/jira/secure/attachment/ > > > > 12848335/Metron-book-screenshot.png > > > > > > > > > > Thanks, > > > > > --Matt > > > > > > > > > > > > > > > On 1/19/17, 6:03 AM, "Otto Fowler" <[email protected]> > > > wrote: > > > > > > > > > > Not seeing the attachment, is it attached to a jira? > > > > > > > > > > > > > > > On January 19, 2017 at 04:19:02, Matt Foley ( > > [email protected]) > > > > wrote: > > > > > > > > > > Here’s a screen shot, attached :-) > > > > > > > > > > On 1/19/17, 1:04 AM, "Matt Foley" <[email protected]> wrote: > > > > > > > > > > Hi all, > > > > > I’ve put together a prototype doc book, along the lines we > > > > discussed, > > > > > and > > > > > it looks pretty worthwhile. > > > > > Many thanks to Mike M. who whipped the pom.xml file into > > shape, > > > > and > > > > > helped > > > > > me find the right site.xml file to imitate. > > > > > > > > > > If you’re interested, please do a single-branch clone as > > > follows: > > > > > git clone --single-branch -b METRON-660 > > > > > https://github.com/mattf-horton/incubator-metron.git > > > [clonename] > > > > > (or whatever git command pleases you :-) > > > > > > > > > > In this branch, there’s a new top-level subdirectory named > > > > site-book/. > > > > > This > > > > > is not necessarily how we want to integrate stuff, it was > > just > > > > > convenient > > > > > to do it separate from the existing site/ directory for now. > > To > > > > build > > > > > the > > > > > book, do these three commands: > > > > > cd site-book/ > > > > > bin/generate-md.sh #This gathers all the *.md files into > > > > > site-book/src/site/markdown/**, and generates the menu tree > > > into > > > > > site-book/src/site/site.xml > > > > > mvn clean site:site #This builds the book with the maven site > > > > plugin > > > > > and > > > > > doxia-markdown plugin > > > > > > > > > > If both those steps are successful, you can then go to a > > > browser > > > > and > > > > > open > > > > > > > > > > file:///Users/yourname/yourpath/clonename/site-book/ > > > > target/site/index.html > > > > > and see the book, with the nav menu on the LHS. > > > > > It’s important to note that a very usable (not perfect) nav > > > > hierarchy > > > > > has > > > > > been auto-generated; this is not hardwired nor hand-edited. > > > > > I do plan to add some overrides that allow individual items > > in > > > > the > > > > > menu to > > > > > be tweaked. > > > > > > > > > > While it already looks fairly nice, and clearly illustrates > > the > > > > value > > > > > of > > > > > building a book, there are two glaring issues. > > > > > • Doxia-markdown doesn’t process the triple back-tick (```) > > the > > > > same > > > > > way as > > > > > Github Markdown. It seems to color-code it as <code>, but > > > doesn’t > > > > > preserve > > > > > line breaks, which is really bad. > > > > > • Similarly, it only processes bullet lists in isolation, and > > > it > > > > > doesn’t > > > > > correctly combine bullet lists subordinate to a numbered > > list. > > > > > > > > > > The upshot is that > > > > > • both code and bullet lists often lose their linebreaks, and > > > get > > > > > mushed > > > > > into run-on paragraphs, usually combined with the preceding > > > > paragraph, > > > > > and > > > > > • bullet lists interrupt numbered lists and make them start > > > over > > > > at #1. > > > > > > > > > > Perhaps 80-90% of these issues can be fixed by editing the > > > > markdown > > > > > files > > > > > to put blank lines around the list formats. I started doing > > > > this, but I > > > > > didn’t want to obscure the proto by editing tons of .md > > files. > > > > As of > > > > > this > > > > > proto, only the half dozen actually broken files (that caused > > > > maven > > > > > site > > > > > build errors) have been fixed. > > > > > The other 10-20% will just require simplification of the > > > > markdown used, > > > > > unless we can get an updated version of the plugins. > > > > > > > > > > Anyway, please take a look and share your thoughts. > > > > > > > > > > Thanks, > > > > > --Matt > > > > > > > > > > > > > > > On 1/16/17, 1:02 PM, "Michael Miklavcic" < > > > > [email protected]> > > > > > wrote: > > > > > > > > > > Hey Matt, feel free to ping me. > > > > > > > > > > On Mon, Jan 16, 2017 at 1:39 PM, Matt Foley < > > [email protected]> > > > > wrote: > > > > > > > > > > > I looked into the Falcon website and doxia over the > > weekend, > > > > and I’m > > > > > > convinced that using the doxia-markdown plugin should make > > it > > > > dirt > > > > > simple > > > > > > to do what’s been discussed in this thread, with no > > overhead > > > > on the > > > > > part > > > > > of > > > > > > people writing the README.md files. > > > > > > > > > > > > I fiddled with trying to do a POC, and unfortunately > > > concluded > > > > > (again) > > > > > > that I don’t really know maven very well :-) > > > > > > Are there any maven experts out there who would be willing > > to > > > > give me > > > > > some > > > > > > pointers (offline) on how to make use of this apparently > > > > simple maven > > > > > > plug-in? > > > > > > > > > > > > I can do the bit of scripting needed to gather the docs. > > I’ve > > > > opened > > > > > > https://issues.apache.org/jira/browse/METRON-660 with some > > > > > sub-tasks for > > > > > > this work. > > > > > > --Matt > > > > > > > > > > > > On 1/13/17, 12:04 PM, "[email protected]" <[email protected] > > > > > > > wrote: > > > > > > > > > > > > +1 on any improvement to documentation and more > > consistency. > > > > At this > > > > > > point, I think getting rid of or hiding some of the pages > > on > > > > the wiki > > > > > > (at > > > > > > least for the short term) would be better than leaving them > > > > around > > > > > > because > > > > > > there's a lot of misinformation. > > > > > > > > > > > > Jon > > > > > > > > > > > > On Fri, Jan 13, 2017 at 10:13 AM Nick Allen < > > > > [email protected]> > > > > > > wrote: > > > > > > > > > > > > > +1 I think it is sorely needed. > > > > > > > > > > > > > > If we can come up with a really slick solution like > > Spark, > > > > then > > > > > > great. I am > > > > > > > also not against a half-baked solution that can later > > > evolve > > > > into > > > > > > something > > > > > > > else. For example, create an index README.md that links > > > > together > > > > > > all the > > > > > > > existing READMEs and run Pandoc on it. Not ideal, but way > > > > better > > > > > > than what > > > > > > > we have. > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Jan 13, 2017 at 9:53 AM, Otto Fowler < > > > > > > [email protected]> > > > > > > > wrote: > > > > > > > > > > > > > > > I think something that does what you have laid out > > here, > > > no > > > > > matter > > > > > > the > > > > > > > > implementation details would be ideal > > > > > > > > > > > > > > > > > > > > > > > > On January 12, 2017 at 18:05:24, Matt Foley ( > > > > [email protected]) > > > > > > wrote: > > > > > > > > > > > > > > > > We currently have three forms of documentation, with > > the > > > > > following > > > > > > > > advantages and disadvantages: > > > > > > > > > > > > > > > > || Docs || Pro || Con || > > > > > > > > | CWiki | > > > > > > > > Easy to edit, no special tools required, don't have to > > > be a > > > > > > developer to > > > > > > > > contribute, google and wiki search | > > > > > > > > Not versioned, no review process, distant from the > > code, > > > > obsolete > > > > > > content > > > > > > > > tends to accumulate | > > > > > > > > | Site | > > > > > > > > Versioned and reviewed, only committers can edit, > > google > > > > search | > > > > > > > > Yet another arcane toolset must be learned, only web > > > > programmers > > > > > > feel > > > > > > > > comfortable contributing, "asf-site" branch not related > > > to > > > > code > > > > > > versions, > > > > > > > > distant from the code, tends to go obsolete due to > > > > > non-maintenance > > > > > > | > > > > > > > > | README.md | > > > > > > > > Versioned and reviewed, only committers can edit, tied > > to > > > > code > > > > > > versions, > > > > > > > > highly local to the code being documented | > > > > > > > > Non-developers don't know about them, may be scared by > > > > github, > > > > > poor > > > > > > > scoring > > > > > > > > in google search, no high-level presentation | > > > > > > > > > > > > > > > > Various discussion threads indicate the developer > > > > community likes > > > > > > > > README-based docs, and it's easy to see why from the > > > > above. I > > > > > > propose > > > > > > > this > > > > > > > > extension to the README-based documentation, to address > > > > their > > > > > > > > disadvantages: > > > > > > > > > > > > > > > > 1. Produce a script that gathers the README.md files > > from > > > > all > > > > > code > > > > > > > > subdirectories into a hierarchical list. The script > > would > > > > have an > > > > > > > exclusion > > > > > > > > list for non-user-content, which at this point would > > > > consist of > > > > > > [site/*, > > > > > > > > build_utils/*]. The hierarchy would be sorted > > > depth-first. > > > > The > > > > > > resulting > > > > > > > > hierarchical list at this time (with six added README > > > > files to > > > > > > complete > > > > > > > the > > > > > > > > hierarchy) would be: > > > > > > > > > > > > > > > > ./README.md > > > > > > > > ./metron-analytics/README.md <== (need file here) > > > > > > > > ./metron-analytics/metron-maas-service/README.md > > > > > > > > ./metron-analytics/metron-profiler/README.md > > > > > > > > ./metron-analytics/metron-profiler-client/README.md > > > > > > > > ./metron-analytics/metron-statistics/README.md > > > > > > > > ./metron-deployment/README.md > > > > > > > > ./metron-deployment/amazon-ec2/README.md > > > > > > > > ./metron-deployment/packaging/README.md <== (need file > > > > here) > > > > > > > > ./metron-deployment/packaging/ambari/README.md <== > > (need > > > > file > > > > > > here) > > > > > > > > ./metron-deployment/packaging/ > > > > docker/ansible-docker/README.md > > > > > > > > ./metron-deployment/packaging/ > > > docker/rpm-docker/README.md > > > > > > > > ./metron-deployment/packer-build/README.md > > > > > > > > ./metron-deployment/roles/ <== (need file here) > > > > > > > > ./metron-deployment/roles/kibana/README.md > > > > > > > > ./metron-deployment/roles/monit/README.md > > > > > > > > ./metron-deployment/roles/opentaxii/README.md > > > > > > > > ./metron-deployment/roles/pcap_replay/README.md > > > > > > > > ./metron-deployment/roles/sensor-test-mode/README.md > > > > > > > > ./metron-deployment/vagrant/README.md <== (need file > > > here) > > > > > > > > ./metron-deployment/vagrant/codelab-platform/README.md > > > > > > > > ./metron-deployment/vagrant/ > > > fastcapa-test-platform/README. > > > > md > > > > > > > > ./metron-deployment/vagrant/ > > full-dev-platform/README.md > > > > > > > > ./metron-deployment/vagrant/ > > quick-dev-platform/README.md > > > > > > > > ./metron-platform/README.md > > > > > > > > ./metron-platform/metron-api/README.md > > > > > > > > ./metron-platform/metron-common/README.md > > > > > > > > ./metron-platform/metron-data-management/README.md > > > > > > > > ./metron-platform/metron-enrichment/README.md > > > > > > > > ./metron-platform/metron-indexing/README.md > > > > > > > > ./metron-platform/metron-management/README.md > > > > > > > > ./metron-platform/metron-parsers/README.md > > > > > > > > ./metron-platform/metron-pcap-backend/README.md > > > > > > > > ./metron-sensors/README.md <== (need file here) > > > > > > > > ./metron-sensors/fastcapa/README.md > > > > > > > > ./metron-sensors/pycapa/README.md > > > > > > > > > > > > > > > > 2. Arrange to run this script as part of the build > > > > process, and > > > > > > commit > > > > > > > the > > > > > > > > resulting hierarchy list to someplace in the versioned > > > and > > > > > branched > > > > > > > ./site/ > > > > > > > > subdirectory. > > > > > > > > > > > > > > > > 3. Produce a "doc reader" web page that takes in this > > > > hierarchy > > > > > of > > > > > > .md > > > > > > > > pages, and presents a LHS doc tree of links, and a main > > > > display > > > > > > area for > > > > > > > a > > > > > > > > currently selected file. If we want to get fancy, this > > > > page would > > > > > > also > > > > > > > > provide: (a) telescoping (collapse/expand) of the doc > > > > tree; (b) > > > > > > floating > > > > > > > > next/prev/up/home buttons in the display area. > > > > > > > > > > > > > > > > #4. Add to this web page a pull-down menu that selects > > > > among all > > > > > > the > > > > > > > > release versions of Metron, and (if not running in the > > > > Apache > > > > > > site) a > > > > > > > > SNAPSHOT version for the current filesystem version > > (for > > > > > developer > > > > > > > > preview). Let it re-write the file paths per release > > > > version to > > > > > the > > > > > > > proper > > > > > > > > release tag in github. This web page will therefore be > > > > > > > version-independent. > > > > > > > > Put it in the asf-site branch of the Apache site, as > > the > > > > new > > > > > "docs" > > > > > > > > sub-site from the home web page. Update the list of > > > > releases at > > > > > > each > > > > > > > > release, or if we want to get fancy, teach it to read > > the > > > > release > > > > > > tags > > > > > > > from > > > > > > > > github. > > > > > > > > > > > > > > > > 5. As part of the release process, the release manager > > > (a) > > > > > assures > > > > > > the > > > > > > > > release is tagged in github with a consistent naming > > > > convention, > > > > > > and (b) > > > > > > > > submits the new hierarchy of links to google search > > > > (there's an > > > > > > api for > > > > > > > > that). > > > > > > > > > > > > > > > > 6. Deprecate the use of cwiki for anything but > > long-lived > > > > > > > > demonstrations/tutorials that are unlikely to go > > > obsolete. > > > > > > > > > > > > > > > > > > > > > > > > Do folks feel this would be a good contribution to the > > > > > visibility, > > > > > > > > timeliness, and usability of our docs? > > > > > > > > Is this an adequate solution for the current problems? > > > > > > > > > > > > > > > > Thanks, > > > > > > > > --Matt > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > Nick Allen <[email protected]> > > > > > > > > > > > > > -- > > > > > > > > > > > > Jon > > > > > > > > > > > > Sent from my mobile device > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > Jon > > > > > > > > Sent from my mobile device > > > > > > > > > > > > > > > > > > > > >
------------------- Thank you, James Sirota PPMC- Apache Metron (Incubating) jsirota AT apache DOT org
