I spent a bit of time getting this to run. I currently get a host of 404's and am going to need to dig in a bit and see just what they are. In the meantime, I'm going to commit https://github.com/apache/metron/pull/1310 and have created a Jira to track adding site-book link validation to our Travis checks - https://issues.apache.org/jira/browse/METRON-1952. For the curious, I landed on the following modified commands to get this running:
nohup python -m SimpleHTTPServer 2>&1 & wget -Dlocalhost --spider -e robots=off -w 1 -r -p http://localhost:8000 2>&1 > /dev/null | grep -C 2 -E "404" if [ $? -eq 0 ]; then echo FAILED; else echo PASSED; fi http.server is the impl for Python 3. Simple tweak using SimpleHTTPServer for Python 2. Cheers, Mike On Thu, Dec 20, 2018 at 1:24 PM Casey Stella <ceste...@gmail.com> wrote: > You will want to add a -Dlocalhost to the wget to ensure you're not > checking domains linked from our docs and turn travis into google. ;) > > > On Thu, Dec 20, 2018 at 3:19 PM Michael Miklavcic < > michael.miklav...@gmail.com> wrote: > > > Well golly, I love this. I'll give that a whirl! > > > > On Thu, Dec 20, 2018 at 1:08 PM Casey Stella <ceste...@gmail.com> wrote: > > > > > I definitely agree with option 3; that's a no-brainer IMO. I thought > for > > > sure this was already happening, honestly. > > > > > > As for 2, we could even script the broken link check by: > > > > > > - Serving up the site locally via python with `python -m > http.server` > > > from the site-book output directory > > > - Looking at the output of wget --spider -e robots=off -w 1 -r -p > > > http://localhost:8000 for 404s > > > > > > > > > As for 1, I'm fine with it if that's where we want to go. I'm a +0 > > > > > > On Thu, Dec 20, 2018 at 2:47 PM Michael Miklavcic < > > > michael.miklav...@gmail.com> wrote: > > > > > > > We recently had our site-book doc generation break due to our not > > > including > > > > it in the Travis build. The fix for a broken build seems simple > enough > > - > > > > add it to our build process and assuming it doesn't cause build > timeout > > > > issues, we should be good to go. > > > > > > > > Beyond that, there are additional issues with the existing process. > We > > > have > > > > a step in our PR review for validating that the docs are rendering > > > > properly. I know I've gone back and corrected issues with broken > images > > > or > > > > incorrectly rendering pages at least a few times now. On one hand, we > > > might > > > > say this is simply a matter of being better about validating > > > documentation > > > > during the review process. That may be true, but rather than fight > > > upstream > > > > like a salmon, I would prefer to simplify things, automate what we > can, > > > and > > > > use technology to work with us. Based on this conversation on > > > METRON-1950 - > > > > ( > > > > > > > > > > > > > > https://lists.apache.org/thread.html/e2acf91efc5f51ba0e26d76b00ca02415d3c6ee0adee74a037ab2beb@%3Cdev.metron.apache.org%3E > > > > ), > > > > I'd like to open up a general convo about improvements to our > > > documentation > > > > generation. > > > > > > > > *Current Issues:* > > > > > > > > 1. Duplicated effort - have to check pages render in Github and > the > > > > Doxia-generated site-book > > > > 2. Inconsistent model - what works for Github markdown may not > work > > > for > > > > Doxia, and vice versa > > > > 3. Github is part of our workflow and easy to check, Doxia > requires > > an > > > > extra separate step - suffers unintentional bugs due to #2. > > > > 4. Images have to be manually added to the site rendering code for > > > > copying to the "images" folder, and explicit src ref replacements > > have > > > > to > > > > be included for all affected pages/links as well. > > > > 5. Page links and images are not validated - this currently > requires > > > > manual review and intervention during PR review and whenever we > > > create a > > > > new Metron release. > > > > 6. Failed site-book build is not validated. Broken build does not > > fail > > > > Travis > > > > > > > > *Options and Solutions:* > > > > > > > > 1. Otto has already brought up using Ascii doc as one option for > > > solving > > > > a number of these issues. > > > > 2. For issue #5, we can write a scraper that validates links or > use > > > > tooling like Cypress for this. > > > > 3. For issue #6, we can add site-book building to our Travis runs. > > > It's > > > > pretty quick to generate and will catch the more egregious > rendering > > > > bugs. > > > > I plan to look at this presently. > > > > > > > > Mike > > > > > > > > > >
