Re: PIO-20 problems

[Pat Ferrel]

Alex, can you look at these unit test failures on the PR, they seem to be in 
JDBCPEvents 

https://travis-ci.org/apache/incubator-predictionio/builds/151905196


On Aug 12, 2016, at 1:18 PM, Pat Ferrel  wrote:

Can't install unittest with pip or pip3 even though the rest of the 
prerequisites work. Simply refuses to find it. Pip3 search unittest give this:

WebTestRunner (0.2)- Web-based interface for selectively 
executing client-side Python UnitTests
unittest (0.0) - 
nosetests-json-extended (0.1.0)- Create json logging output for 
pythonnosetests unittest framework

no description and 0.0 seems odd. Pip3 install unittest gives: 

Collecting unittest
 Could not find a version that satisfies the requirement unittest (from 
versions: )
No matching distribution found for unittest

Tried forcing the version to 0.0 but again no luck

Ideas? In the meantime waiting for Travis to do it—sigh


On Aug 11, 2016, at 2:31 PM, Pat Ferrel  wrote:

With the keystore my template-based integration test passes and I’ve put the 
keystore back in. The diffs on PRs on Github seem completely wonky right now. 
Git diff is trustworthy at least.

Will try the python tests now too.

Thanks guys, working smoothly now.


On Aug 11, 2016, at 11:53 AM, Donald Szeto  wrote:

Hi all,

I went ahead and pulled Pat's branch, performed a clean build, and repeated
the quick start guide of the Scala parallel recommendation template. I
could produce the same problem, and root caused it to a missing
conf/keystore.jks file.

I think with SSL now optional, we should not be distributing a KeyStore
file. We can either quick fix it now by putting this file back, or modify
SSLConfiguration to not look for this file when SSL support is off.

Regards,
Donald

On Thu, Aug 11, 2016 at 8:03 AM, Chan Lee  wrote:

> I want to clarify some points:
> 
> 2) Deploying templates do not require SSL. If you execute
> ./make-distribution.sh using the current develop branch (provided you
> change the namespace in the template from io.prediction to
> org.apache.predictionio), you can deploy without SSL on localhost. The
> travis tests also do not use SSL and pass as can be seen in
> https://travis-ci.org/Ziemin/incubator-predictionio.
> 
> 3) You can run the tests locally the same way it is run on travis
> ('python3 ${PIO_DIR}/tests/pio_tests/tests.py'). It is documented in the
> README in the same directory, but maybe this was not clear enough. If you
> have any additional questions or run into issues executing the tests,
> please let me know.
> 
> Thanks,
> Chan
> 
> On Thu, Aug 11, 2016 at 6:07 AM, Pat Ferrel  wrote:
> 
>> I’m always disinclined to commit code that has not been tested. However
>> many things in the current state of the develop branch seem broken or at
>> least too big to address in a single commit.
>> 
>> 1) Style test fail in Travis that are not runnable or at least have not
>> been documented for local build. I have fixed these.
>> 2) SSL seems to still be required for deploying an engine, PIO-1 may not
>> have addressed this. Has anyone tried to build and test a template yet
>> without SSL? Unless someone can state otherwise I’m inclined to ignore this
>> failure for now since it is out of scope for this commit.
>> 3) the test framework has been integrated into Travis but again no
>> documented way to run it locally and some of the travis errors seem
>> spurious. I am also not inclined to wait for this to be fixed unless
>> someone can point out real problems it is discovering and send instructions
>> for how to run it locally.
>> 
>> Before the extra travis tests I was able to build and train a template,
>> which fails on deploy with an SSL config error. Since the PR does not
>> strictly touch any templates I am going to have to ignore this serious
>> issue and address it in Jira.
>> 
>> Unless someone vetos this I plan to reluctantly push the PR with these
>> non-trivial issues remaining.
>> 
>> 
>> 
> 

Re: PIO-20 problems

[Pat Ferrel]

Can't install unittest with pip or pip3 even though the rest of the 
prerequisites work. Simply refuses to find it. Pip3 search unittest give this:

WebTestRunner (0.2)- Web-based interface for selectively 
executing client-side Python UnitTests
unittest (0.0) - 
nosetests-json-extended (0.1.0)- Create json logging output for 
pythonnosetests unittest framework

no description and 0.0 seems odd. Pip3 install unittest gives: 

Collecting unittest
  Could not find a version that satisfies the requirement unittest (from 
versions: )
No matching distribution found for unittest

Tried forcing the version to 0.0 but again no luck

Ideas? In the meantime waiting for Travis to do it—sigh


On Aug 11, 2016, at 2:31 PM, Pat Ferrel  wrote:

With the keystore my template-based integration test passes and I’ve put the 
keystore back in. The diffs on PRs on Github seem completely wonky right now. 
Git diff is trustworthy at least.

Will try the python tests now too.

Thanks guys, working smoothly now.


On Aug 11, 2016, at 11:53 AM, Donald Szeto  wrote:

Hi all,

I went ahead and pulled Pat's branch, performed a clean build, and repeated
the quick start guide of the Scala parallel recommendation template. I
could produce the same problem, and root caused it to a missing
conf/keystore.jks file.

I think with SSL now optional, we should not be distributing a KeyStore
file. We can either quick fix it now by putting this file back, or modify
SSLConfiguration to not look for this file when SSL support is off.

Regards,
Donald

On Thu, Aug 11, 2016 at 8:03 AM, Chan Lee  wrote:

> I want to clarify some points:
> 
> 2) Deploying templates do not require SSL. If you execute
> ./make-distribution.sh using the current develop branch (provided you
> change the namespace in the template from io.prediction to
> org.apache.predictionio), you can deploy without SSL on localhost. The
> travis tests also do not use SSL and pass as can be seen in
> https://travis-ci.org/Ziemin/incubator-predictionio.
> 
> 3) You can run the tests locally the same way it is run on travis
> ('python3 ${PIO_DIR}/tests/pio_tests/tests.py'). It is documented in the
> README in the same directory, but maybe this was not clear enough. If you
> have any additional questions or run into issues executing the tests,
> please let me know.
> 
> Thanks,
> Chan
> 
> On Thu, Aug 11, 2016 at 6:07 AM, Pat Ferrel  wrote:
> 
>> I’m always disinclined to commit code that has not been tested. However
>> many things in the current state of the develop branch seem broken or at
>> least too big to address in a single commit.
>> 
>> 1) Style test fail in Travis that are not runnable or at least have not
>> been documented for local build. I have fixed these.
>> 2) SSL seems to still be required for deploying an engine, PIO-1 may not
>> have addressed this. Has anyone tried to build and test a template yet
>> without SSL? Unless someone can state otherwise I’m inclined to ignore this
>> failure for now since it is out of scope for this commit.
>> 3) the test framework has been integrated into Travis but again no
>> documented way to run it locally and some of the travis errors seem
>> spurious. I am also not inclined to wait for this to be fixed unless
>> someone can point out real problems it is discovering and send instructions
>> for how to run it locally.
>> 
>> Before the extra travis tests I was able to build and train a template,
>> which fails on deploy with an SSL config error. Since the PR does not
>> strictly touch any templates I am going to have to ignore this serious
>> issue and address it in Jira.
>> 
>> Unless someone vetos this I plan to reluctantly push the PR with these
>> non-trivial issues remaining.
>> 
>> 
>> 
> 

Re: transition from "official" PredictionIO templates to Apache

[Pat Ferrel]

Independent of Apache I’d suggest each template have their own git repo as they 
do now. Is this practically possible in ASF (another example of my 
infrastructure rant but leave that for now). The semantics of this make sense. 
The requirements for all apache or non-apache templates seem to be:

1) They need to be allowed to have separate release cycles. 
2) There should be only one way to install templates apache or non-apache
3) This method should support tags and branches, and separate PRs

Practically speaking for independent ones (the Universal Recommender will 
remain independent until the issues above are resolved) the templates *will* be 
a separate repo and `git pull` *will* be the method of download. To me this 
implies the Apache ones should use the same pattern. 

The separate issue of rules for inclusion in Apache should include a test 
requirement IMO. But inclusion in the Gallery seems a looser attachment to the 
project—just one persons opinion


On Aug 11, 2016, at 3:42 PM, Chan Lee  wrote:

I've begun working on migrating official PredictionIO templates into the main 
apache repository, and there are a few points I'd like to bring to attention.

1) Template code will be merged under templates/ directory, and all commit 
history and authorship will be preserved.

2) Organization namespace will change in the template code from "io.prediction" 
to "org.apache.predictionio". 

3) In order to download an engine template, users could (1) git clone the 
entire repo and copy the files in templates/, or (2) use 
something like svn to download only the necessary subdirectory. I'd like to ask 
what everyone thinks should be the recommended approach. The documentation 
should be updated accordingly (instead of `pio template get`).

4) I'd like to ask if there are any ActionML/outside templates that should be 
included or updated. For instance, 
template-scala-parallel-universal-recommendation in PredictionIO repo seems 
outdate compared to one in actionml repo. 


Thanks,
Chan



On Mon, Aug 8, 2016 at 9:16 AM, Marcin Ziemiński > wrote:
I think that this is a very good idea. Obviously it would require
configuring more concurrent builds to finish in reasonable time, what
should not be a problem. Besides, it would make the official templates be
consistent with every next release of PredictionIO and what is very
important - it would provide many different tests for the repository.
Having working templates of different types could give us more material to
test not only the reliability of PredictionIO, but also its performance
with every change. It would require adding different types of tests and
tools to gather runtime statistics, but this is something I would like to
take a look at in the future.

niedz., 7.08.2016 o 10:21 użytkownik Donald Szeto >
napisał:

> We can also make these templates part of the integration test that Chan and
> Marcin just submitted (
> https://github.com/apache/incubator-predictionio/pull/267 
> ). That way we
> can
> make commitment to included templates be part of every committer's effort.
> Just my 2c.
>
> On Sun, Aug 7, 2016 at 9:47 AM, Pat Ferrel  > wrote:
>
> > If this sound ok, I propose the process be:
> > 1) inclusion in the gallery is a PR to the yaml gallery file. This is
> > reviewable by all but takes only one committer to approve and merge.
> > 2) submission for inclusion in the project can be a PR to the new
> > templates directory as Simon suggests. But this may require a more formal
> > vote, and come with some commitment for support and possibly
> consideration
> > of the author for committer status.
> >
> > Someone who knows Apache rules better than me may know the type of vote
> to
> > have for #2, maybe this is only informal review and single committer
> > acceptance too as long as the committer is willing to support the
> template.
> >
> > As far as the Salesforce templates marked official, I’d hope that most
> > would be donated. According to the proposed rules all we need is a PR for
> > each. And again a big thanks to Salesforce!
> >
> >
> > On Aug 4, 2016, at 2:55 PM, Pat Ferrel  > > wrote:
> >
> > Actually this is mostly a fine idea but I think the bigger question is
> how
> > do we treat templates in general.
> >
> > IMO the maintaining author can decide to contribute them or not and the
> > committers can decide to accept or not. For example in the case of the
> UR I
> > may decide to support and maintain it outside of Apache while some from
> > Salesforce might be submitted as donations.  Donation comes at some
> > obligation. There are lots of examples in Mahout of algorithms whose
> > authors no longer support them. These are slowly being deprecated and
> > 

[GitHub] incubator-predictionio pull request #234: Fix JSON

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-predictionio/pull/234


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-predictionio pull request #271: [MINOR] Miscellaneous typos and co...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-predictionio/pull/271


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: transition from "official" PredictionIO templates to Apache

[Chan Lee]

I've begun working on migrating official PredictionIO templates into the
main apache repository, and there are a few points I'd like to bring to
attention.

1) Template code will be merged under templates/ directory, and all commit
history and authorship will be preserved.

2) Organization namespace will change in the template code from
"io.prediction" to "org.apache.predictionio".

3) In order to download an engine template, users could (1) git clone the
entire repo and copy the files in templates/, or (2) use
something like svn to download only the necessary subdirectory. I'd like to
ask what everyone thinks should be the recommended approach. The
documentation should be updated accordingly (instead of `pio template get`).

4) I'd like to ask if there are any ActionML/outside templates that should
be included or updated. For instance,
template-scala-parallel-universal-recommendation
in PredictionIO repo seems outdate compared to one in actionml repo.


Thanks,
Chan



On Mon, Aug 8, 2016 at 9:16 AM, Marcin Ziemiński  wrote:

> I think that this is a very good idea. Obviously it would require
> configuring more concurrent builds to finish in reasonable time, what
> should not be a problem. Besides, it would make the official templates be
> consistent with every next release of PredictionIO and what is very
> important - it would provide many different tests for the repository.
> Having working templates of different types could give us more material to
> test not only the reliability of PredictionIO, but also its performance
> with every change. It would require adding different types of tests and
> tools to gather runtime statistics, but this is something I would like to
> take a look at in the future.
>
> niedz., 7.08.2016 o 10:21 użytkownik Donald Szeto 
> napisał:
>
> > We can also make these templates part of the integration test that Chan
> and
> > Marcin just submitted (
> > https://github.com/apache/incubator-predictionio/pull/267). That way we
> > can
> > make commitment to included templates be part of every committer's
> effort.
> > Just my 2c.
> >
> > On Sun, Aug 7, 2016 at 9:47 AM, Pat Ferrel 
> wrote:
> >
> > > If this sound ok, I propose the process be:
> > > 1) inclusion in the gallery is a PR to the yaml gallery file. This is
> > > reviewable by all but takes only one committer to approve and merge.
> > > 2) submission for inclusion in the project can be a PR to the new
> > > templates directory as Simon suggests. But this may require a more
> formal
> > > vote, and come with some commitment for support and possibly
> > consideration
> > > of the author for committer status.
> > >
> > > Someone who knows Apache rules better than me may know the type of vote
> > to
> > > have for #2, maybe this is only informal review and single committer
> > > acceptance too as long as the committer is willing to support the
> > template.
> > >
> > > As far as the Salesforce templates marked official, I’d hope that most
> > > would be donated. According to the proposed rules all we need is a PR
> for
> > > each. And again a big thanks to Salesforce!
> > >
> > >
> > > On Aug 4, 2016, at 2:55 PM, Pat Ferrel  wrote:
> > >
> > > Actually this is mostly a fine idea but I think the bigger question is
> > how
> > > do we treat templates in general.
> > >
> > > IMO the maintaining author can decide to contribute them or not and the
> > > committers can decide to accept or not. For example in the case of the
> > UR I
> > > may decide to support and maintain it outside of Apache while some from
> > > Salesforce might be submitted as donations.  Donation comes at some
> > > obligation. There are lots of examples in Mahout of algorithms whose
> > > authors no longer support them. These are slowly being deprecated and
> > > removed so I’d like to see a method that avoids this issue.
> > >
> > > If we allow maintainers to submit templates to the Gallery with a link
> to
> > > code as well as support then donating the template code to Apache is
> > > independent of acceptance to the Gallery. Any template donated to
> Apache
> > > should come with some commitment by the author to support it there
> > > indefinitely and perhaps even acceptance as a PIO committer—and if they
> > > disappear or don’t support the template it is easy to deprecate/remove.
> > >
> > > That means if Salesforce wants to donate the templates it
> > maintains—great.
> > > Personally I'd would vote for acceptance of most of them. Then the
> > support
> > > link can be to the Apache user list or instructions for joining it. If
> a
> > > maintainer wants to stay out of the Apache repo and support separately
> > then
> > > this is possible also. With all templates treated equally—official or
> > > not—and there is a route to becoming official for non-official ones.
> > >
> > > BTW it might be good to remove the “official” designation in favor of
> > > “Apache