I just pencilled in snippet preprocessing support into the thrift builds. Once Jake and I have settled on the feature set of ASF::Value::Snippet, and we have ported the snippet calling text within the markdown pages to the new format, things will just work (knock on wood).
> On Wednesday, March 19, 2014 6:48 AM, Joseph Schaefer > <[email protected]> wrote: > > Sure I understand that need. If there's a way of pulling the document > sources directly from the web I can parse the content to look for snippet > markers and make those available to the template system. That's pretty much > how www.apache.org is generated. > > Sent from my iPhone > > >> On Mar 19, 2014, at 5:28 AM, Henrique Mendonça <[email protected]> > wrote: >> >> Hi Joe, >> >> Thanks for your effort. I just wanted to add that the code snippets are a >> crucial feature for this project, which has way more target languages than >> active contributors. In our case, reducing manual maintenance costs is much >> more important than the total site build time. Perhaps we can also solve >> this in the CMS, though. >> >> - Henrique >> >> >>> On 19 March 2014 03:17, Joe Schaefer <[email protected]> > wrote: >>> >>> A couple more thoughts regarding the background on the CMS... >>> the bulk of the focus of the CMS up until now has been to >>> support really big sites like maven and openoffice, both of >>> which have websites well over 5GB worth of text files. The >>> build system was hollowed out and revamped to support external >>> builds in other programming languages, as well as scaling up >>> the perl builds to provide about 8 child processes that build >>> the site in parallel by default. Full site build times for >>> openoffice went from over 30 minutes to between 5 to 10 minutes >>> depending on disk activity, which made experimentation with site-wide >>> changes feasible on a reasonable timeframe. At first the project was > in >>> the habit of running the builds locally before checking in their > changes >>> because the delay in feedback post-commit was unreasonable, but with > smart >>> use of SSI the were able to cut down on the number of changes that > required >>> full site builds and hence no longer bother with the whole pre-build > mumbo >>> jumbo before checking in changes. The CMS is designed to only build > pages >>> that were altered, and if you just edit a page or two only those pages > and >>> the pages that depend on them will be built. >>> >>> Now that I've turned my attention to thrift, there are several > aspects >>> that are better off being retuned based on the modest size of the site. >>> First of all you have a sitemap url that lists all of your > markdown-based >>> pages, which would be a nonstarter for a site like openoffice. The >>> implications of the sitemap url are that every page needs to be built > in >>> memory just to generate that page, which leads to a lot of redundancy > in a >>> build system designed to run as forked child processes. We can do > better... >>> >>> I've done two things to make this work well- first I made the > number of >>> child processes "aka runners" tunable on a per-project basis, > and since the >>> sitemap is going to build every page anyway as the slowest page to > build, I >>> memoized the main view that builds the markdown pages for thrift. The > best >>> results are when there's only one runner so we get the full benefit > of >>> memoization: typical site build times are consistently under 2 seconds > now. >>> >>> You'll appreciate why I am such a performance stickler for the CMS > when >>> you start using the bookmarklet, since builds are the only asynchronous >>> part of the process from making a change to a page and publishing the >>> results. Ideally when the system is working well the builds happen > before >>> you have time to go looking for them, but you do need to be mindful of > not >>> publishing before the built pages have been committed back to svn by >>> buildbot. >>> >>> >>> >>>> On Monday, March 17, 2014 11:59 PM, Joe Schaefer > <[email protected]> >>> wrote: >>>>> Ok there's enough of an idea about how to port the rest of > the site's >>>> pages over based on the porting work I've already done that > it's time >>>> for me to step back and let you guys catch up. Basically the idea > is >>> that while >>>> there is templating support in the markdown sources, it's > fairly limited >>>> compared to what nanoc offers, but instead of writing a lot of > complex >>> template >>>> code into your sources, with the cms you just add code to > lib/view.pmand/or >>>> lib/path.pm to keep the document sources simple and just add more >>> template >>>> variables. What I just finished in lib/view.pm and lib/path.pm is >>> directory >>>> listing support for /docs.html but it can be carried over to > similar >>> pages. >>>> >>>> >>>> >>>> >>>>> On Monday, March 17, 2014 9:16 PM, Joe Schaefer >>>> <[email protected]> wrote: >>>>>> Interesting ideas, thanks. FWIW I think that sort >>>>> of thing might be best supported as a periodic cron >>>>> if we can write something stable enough, because pulling >>>>> stuff out of the git repo is something the CMS isn't going >>>>> to do all that well being an svn application. >>>>> >>>>> The CMS build results are available at >>> http://thrift.staging.apache.org/ >>>>> feel free to play with the sources in > repos/asf/thrift/cms-site/trunk. >>>>> It's very fast, probably an order of magnitude faster than > your current >>>> >>>>> tech. >>>>> Whole site builds in two seconds; fractions of a second for > typical >>> mods to >>>> >>>>> content/ markdown pages. >>>>> >>>>> The CMS bookmarklet is compatible with the staging site FWIW. >>>>> >>>>> >>>>> >>>>> >>>>>> On Monday, March 17, 2014 7:40 PM, Roger Meier >>>>> <[email protected]> wrote: >>>>>>> Hi Joe & Co >>>>>> >>>>>> thanks for this test run with Apache CMS! >>>>>> >>>>>> What I would love to see is this: >>>>>> >>>>>> Source file: => Web Site URL: >>>>>> test/README.md => thrift.apache.org/test >>>>>> test/STATUS.md => thrift.apache.org/test/status >>>>>> test/keys/README.md => thrift.apache.org/test/keys >>>>>> lib/<lang>/README.md => > thrift.apache.org/lib/<lang> >>>>>> lib/<lang>/test/README.md => >>>>> thrift.apache.org/lib/<lang>/test >>>>>> lib/<lang>/examples/README.md => >>>>>> thrift.apache.org/lib/<lang>/examples >>>>>> compiler/cpp/README.md => thrift.apache.org/compiler >>>>>> debian/README.md => thrift.apache.org/debian >>>>>> contrib/<topic>/README.md => > thrift.apache.org/<topic> >>>>>> >>>>>> just made the following patch to rename all file to .md > within our >>>> source >>>>>> tree: https://issues.apache.org/jira/browse/THRIFT-2407 >>>>>> ... we have *41* md's within the source tree and > that's the >>>> place >>>>> where >>>>>> people look for documentation first. => simple to > manage >>>>>> Would be user friendly to have all of this online with > each release >>>>>> with the URL layout mentioned above. >>>>>> >>>>>> just received the buildboot... >>>>>> Do you have kind of a preview of your prototype? >>>>>> >>>>>> all the best! >>>>>> -roger >>>>>> ;-r >>>>>> >>>>>> >>>>>> Quoting Jake Farrell <[email protected]>: >>>>>> >>>>>>> No objections from me >>>>>>> >>>>>>> -Jake >>>>>>> >>>>>>> >>>>>>> On Mon, Mar 17, 2014 at 12:04 PM, Joe Schaefer >>>>>> <[email protected]>wrote: >>>>>>> >>>>>>>> Any objections to me making a copy of the site >>>>>>>> tree alongside it called cms-site? I'd like > to >>>>>>>> get cracking on knocking up the supporting perl >>>>>>>> for this so we can continue to brainstorm... >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> On Sunday, March 16, 2014 1:53 PM, Joe Schaefer >>>>>> <[email protected]> >>>>>>>> wrote: >>>>>>>>>> On Sunday, March 16, 2014 1:40 PM, Jake > Farrell >>>>>> <[email protected]> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>> Hey Joe >>>>>>>>>> Thanks for reaching out, we chose to go > with a site >>>>> generator >>>>>> over the >>>>>>>>>> Apache CMS due to it initially not meeting > all of >>>> our >>>>> needs. >>>>>> Our current >>>>>>>>>> needs for the site are >>>>>>>>>> >>>>>>>>>> - Markdown to html conversion >>>>>>>>> >>>>>>>>> >>>>>>>>> That one's easy- its the default for the > cms. >>>>>>>>> >>>>>>>>> >>>>>>>>>> - Global variables/config usable throughout > in a >>>> template >>>>> >>>>>> based layout >>>>>>>>> >>>>>>>>> >>>>>>>>> Easy too- just create a perl hash somewhere and > make it >>>>> available >>>>>> to >>>>>>>> your views. >>>>>>>>> Code your views to pass that hash along to your >>>> templates >>>>> and you >>>>>> are >>>>>>>> all set. >>>>>>>>> Note if your hash contains objects you can make > method >>>> calls >>>>> on >>>>>> them in >>>>>>>> your >>>>>>>>> templates. Note: while I wouldn't > recommend this in >>>> >>>>> general, >>>>>> for smaller >>>>>>>>> projects like thrift that prefer convenience > over >>>> separation >>>>> of >>>>>> concerns >>>>>>>> you can >>>>>>>>> have the django template engine do a pass over > your >>>> markdown >>>>>> before >>>>>>>> passing it >>>>>>>>> along to your actual template page, just as is > seemingly >>>> >>>>> common to >>>>>> other >>>>>>>> popular >>>>>>>>> site generation software. >>>>>>>>> >>>>>>>>> >>>>>>>>>> - Syntax highlighting >>>>>>>>> >>>>>>>>> >>>>>>>>> Comes with python's markdown support, but > some >>>> people >>>>> prefer >>>>>> the look of >>>>>>>>> javascript highlighters. >>>>>>>>> >>>>>>>>>> - Easily include code snippets from our > source code >>>> base >>>>>>>>> >>>>>>>>> >>>>>>>>> Statically I hope. No idea how to pull > snippets out of >>>> your >>>>> git >>>>>> repo >>>>>>>> via the >>>>>>>>> cms. >>>>>>>>> >>>>>>>>>> - Ability to locally test before committing >>>>>>>>> >>>>>>>>> >>>>>>>>> Belt and suspenders with the cms- you can build > your >>>> changes >>>>>> before >>>>>>>> committing >>>>>>>>> with the build scripts, browse your changes > before >>>> committing >>>>> via >>>>>> the cms >>>>>>>>> webgui, or simply commit your changes and view > the >>>> staging >>>>> site >>>>>> prior to >>>>>>>> live >>>>>>>>> publication (which is a separate step). >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> >>>>>>>>>> -Jake >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Sun, Mar 16, 2014 at 12:44 PM, Joe > Schaefer >>>>>>>>> <[email protected]>wrote: >>>>>>>>>> >>>>>>>>>>> As it so happens I am interested in > working on >>>>> supporting >>>>>>>>>>> thrift's use case as a potential > user of >>>> the >>>>> Apache >>>>>> CMS. >>>>>>>>>>> While the CMS works well for massive > projects >>>> there >>>>> are >>>>>>>>>>> things it could do better at supporting > for >>>> more >>>>> moderate >>>>>>>>>>> sized sites like thrift's. >>>>>>>>>>> >>>>>>>>>>> I'd therefore like to engage you > folks in a >>>> >>>>>> brainstorming session >>>>>>>>>>> about what sorts of features you want > for your >>>> site >>>>> and >>>>>> to find >>>>>>>>>>> simple ways of supporting those > features for >>>> you. >>>>>>>>>>> >>>>>>>>>>> Thanks. >>> >
