I’ve been talking abut this issue since before we were in the incubator, probably because I maintain one of the most used templates. My vote is to get it right, then release otherwise we create a lot of headaches and ramifications that may never get solved. So in the do-ocracy I’ll volunteer for some of the work below if we decide to go with separate repos.
1) I’ll volunteer to take a couple templates and put them in their own repos if we can get some created. 2) do we have a gallery page yet? we had a good proposal, is it implemented? 3) What templates are worth inclusion? Seems like some of the ALS recommender ones are supersets of others do we need them all? 4) do we need “release” of these except through Apache git and Github? Since they are all built by users I suspect the normal Apache release process can be avoided for them, the full push to maven, binary hosting, etc. 5) We need someone to file Infra Jira’s to get repos. If someone can coach me on this I will do it. This might only add a week or so to the release date. If we don’t do it I bet it will be much more time spent in support and fixing later. Who was converting package names in the templates? If that is done adding them to a repo will be easy On Aug 15, 2016, at 5:01 PM, Donald Szeto <don...@apache.org> wrote: I think the problem with templates in examples is that they are already modified and it will be hard to merge any upstream template changes back to these examples. We need a method to consistently test and support official templates. We can probably use separate repositories, but I am not sure if we have enough time if we are planning to release 0.10.0 this week. We probably want GitHub integration for all these official template repos as well. I do not have a strong opinion in mandating one engine one repo, but I think otherwise it would be hard for customized engines to merge upstream official changes. On Mon, Aug 15, 2016 at 3:05 PM, Pat Ferrel <p...@occamsmachete.com> wrote: > Possible but we have template in examples/ for testing and that’s ok imo. > If we think separate repos is the way to go why not ask? > > @mentors how hard would it be to get say 5 new repos tied to github > mirrors? > > > On Aug 15, 2016, at 1:47 PM, Chan Lee <chanlee...@gmail.com> wrote: > > Regarding 1), I agree with Pat's points for the need to maintain a > separate repository and release cycle for templates. But for the purpose of > making the donation process simple and ensuring consistency of templates > with each new release of PIO, I think having the code reside inside PIO > repo would be the cleanest solution. > > How about creating a separate "apache" branch on each official template > repo that is consistent with PIO release? So before every PIO release, we > will merge changes into this "apache" branch, merge them into the main PIO > repo, and run rigorous tests for each template to ensure that they're > compatible. > > That way people can continue to clone/fork templates as before, it will be > easy to tag/track template versions compatible with each PIO release, and > we can ensure that all official engines work with each new PIO release > which I feel is most important. Also, as Marcin mentioned, we can expand > tests to keep track of performance changes as well. > > Also, I think we should consider merging the current examples/ directory > with the templates/ directory. > > Thanks, > Chan > > > On Sun, Aug 14, 2016 at 8:30 AM, Pat Ferrel <p...@occamsmachete.com > <mailto:p...@occamsmachete.com>> wrote: > Important things unresolved for release: > > 1) What to do with Apache templates—my suggestion is separate repos for > the 3 reasons below. > 2) We need a Gallery page listing at least a couple converted tested > templates—couldn’t find this in the site but maybe I missed it. The UR is > now converted and tested as are the examples. > 3) AML fork healed, should be pushed this afternoon. > 4) the rest of the release magic > > PIO is rather moot without templates but if they are on a separate release > schedule like the doc site (which also needs work) we could release without > 1 & 2. I’d favor that rather than releasing with templates in a templates/ > directory. > > Anything else? How are people feeling about release? > > > On Aug 12, 2016, at 10:18 AM, Pat Ferrel <p...@occamsmachete.com <mailto: > p...@occamsmachete.com>> wrote: > > Independent of Apache I’d suggest each template have their own git repo as > they do now. Is this practically possible in ASF (another example of my > infrastructure rant but leave that for now). The semantics of this make > sense. The requirements for all apache or non-apache templates seem to be: > > 1) They need to be allowed to have separate release cycles. > 2) There should be only one way to install templates apache or non-apache > 3) This method should support tags and branches, and separate PRs > > Practically speaking for independent ones (the Universal Recommender will > remain independent until the issues above are resolved) the templates > *will* be a separate repo and `git pull` *will* be the method of download. > To me this implies the Apache ones should use the same pattern. > > The separate issue of rules for inclusion in Apache should include a test > requirement IMO. But inclusion in the Gallery seems a looser attachment to > the project—just one persons opinion > > > On Aug 11, 2016, at 3:42 PM, Chan Lee <chanlee...@gmail.com <mailto: > chanlee...@gmail.com>> wrote: > > I've begun working on migrating official PredictionIO templates into the > main apache repository, and there are a few points I'd like to bring to > attention. > > 1) Template code will be merged under templates/ directory, and all commit > history and authorship will be preserved. > > 2) Organization namespace will change in the template code from > "io.prediction" to "org.apache.predictionio". > > 3) In order to download an engine template, users could (1) git clone the > entire repo and copy the files in templates/<some-template>, or (2) use > something like svn to download only the necessary subdirectory. I'd like to > ask what everyone thinks should be the recommended approach. The > documentation should be updated accordingly (instead of `pio template get`). > > 4) I'd like to ask if there are any ActionML/outside templates that should > be included or updated. For instance, > template-scala-parallel-universal-recommendation > in PredictionIO repo seems outdate compared to one in actionml repo. > > > Thanks, > Chan > > > > On Mon, Aug 8, 2016 at 9:16 AM, Marcin Ziemiński <ziem...@gmail.com > <mailto:ziem...@gmail.com>> wrote: > I think that this is a very good idea. Obviously it would require > configuring more concurrent builds to finish in reasonable time, what > should not be a problem. Besides, it would make the official templates be > consistent with every next release of PredictionIO and what is very > important - it would provide many different tests for the repository. > Having working templates of different types could give us more material to > test not only the reliability of PredictionIO, but also its performance > with every change. It would require adding different types of tests and > tools to gather runtime statistics, but this is something I would like to > take a look at in the future. > > niedz., 7.08.2016 o 10:21 użytkownik Donald Szeto <don...@apache.org > <mailto:don...@apache.org>> > napisał: > >> We can also make these templates part of the integration test that Chan > and >> Marcin just submitted ( >> https://github.com/apache/incubator-predictionio/pull/267 < > https://github.com/apache/incubator-predictionio/pull/267>). That way we >> can >> make commitment to included templates be part of every committer's > effort. >> Just my 2c. >> >> On Sun, Aug 7, 2016 at 9:47 AM, Pat Ferrel <p...@occamsmachete.com > <mailto:p...@occamsmachete.com>> wrote: >> >>> If this sound ok, I propose the process be: >>> 1) inclusion in the gallery is a PR to the yaml gallery file. This is >>> reviewable by all but takes only one committer to approve and merge. >>> 2) submission for inclusion in the project can be a PR to the new >>> templates directory as Simon suggests. But this may require a more > formal >>> vote, and come with some commitment for support and possibly >> consideration >>> of the author for committer status. >>> >>> Someone who knows Apache rules better than me may know the type of vote >> to >>> have for #2, maybe this is only informal review and single committer >>> acceptance too as long as the committer is willing to support the >> template. >>> >>> As far as the Salesforce templates marked official, I’d hope that most >>> would be donated. According to the proposed rules all we need is a PR > for >>> each. And again a big thanks to Salesforce! >>> >>> >>> On Aug 4, 2016, at 2:55 PM, Pat Ferrel <p...@occamsmachete.com <mailto: > p...@occamsmachete.com>> wrote: >>> >>> Actually this is mostly a fine idea but I think the bigger question is >> how >>> do we treat templates in general. >>> >>> IMO the maintaining author can decide to contribute them or not and the >>> committers can decide to accept or not. For example in the case of the >> UR I >>> may decide to support and maintain it outside of Apache while some from >>> Salesforce might be submitted as donations. Donation comes at some >>> obligation. There are lots of examples in Mahout of algorithms whose >>> authors no longer support them. These are slowly being deprecated and >>> removed so I’d like to see a method that avoids this issue. >>> >>> If we allow maintainers to submit templates to the Gallery with a link > to >>> code as well as support then donating the template code to Apache is >>> independent of acceptance to the Gallery. Any template donated to > Apache >>> should come with some commitment by the author to support it there >>> indefinitely and perhaps even acceptance as a PIO committer—and if they >>> disappear or don’t support the template it is easy to deprecate/remove. >>> >>> That means if Salesforce wants to donate the templates it >> maintains—great. >>> Personally I'd would vote for acceptance of most of them. Then the >> support >>> link can be to the Apache user list or instructions for joining it. If > a >>> maintainer wants to stay out of the Apache repo and support separately >> then >>> this is possible also. With all templates treated equally—official or >>> not—and there is a route to becoming official for non-official ones. >>> >>> BTW it might be good to remove the “official” designation in favor of >>> “Apache supported” or the PredictionIO logo or something like that. We >>> really need to encourage template development even if they are not code >>> contributions. >>> >>> >>> On Aug 3, 2016, at 3:32 PM, Simon Chan <si...@salesforce.com <mailto: > si...@salesforce.com>> wrote: >>> >>> Hey guys, >>> >>> I just want to start the discussion about moving previously "official" >>> PredictionIO engine templates into Apache. >>> >>> Official templates are those marked "Official" here: >>> http://templates.prediction.io/ <http://templates.prediction.io/> >>> >>> In order to move them to ASF Git, would the best way be creating >>> sub-directory under PredictionIO repo? >>> >>> Simon
