Subscription request

2016-09-18 Thread Vijay Bhat 
Hi,

I'd like to join the Prediction IO dev mailing list.

Thanks,
Vijay

Re: [VOTE]: Apache PredictionIO (incubating) 0.10.0 Release

2016-09-18 Thread Donald Szeto 
Yes. Many of them are sources, docs, poms and signatures though.

On Sun, Sep 18, 2016 at 6:02 PM, Suneel Marthi 
wrote:

> So all of these artifacts need to be verified and validated ?
>
> https://repository.apache.org/content/repositories/
> orgapachepredictionio-1003/org/apache/predictionio/
>
>
>
> On Mon, Sep 19, 2016 at 2:59 AM, Donald Szeto  wrote:
>
> > I will start a new voting thread. The staging repo should contain just
> > PredictionIO artifacts.
> >
> > On Sun, Sep 18, 2016 at 5:57 PM, Suneel Marthi 
> wrote:
> >
> > > the MAven staging repo brings up whole load of artifacts, which ones r
> we
> > > voting on again ?
> > >
> > > On Mon, Sep 19, 2016 at 2:49 AM, Donald Szeto 
> wrote:
> > >
> > > > I have prepared a RC2. The artifacts can be downloaded here:
> > > > https://dist.apache.org/repos/dist/dev/incubator/
> predictionio/0.10.0-
> > > > incubating-rc2/
> > > > or from the Maven staging repo here:
> > > > https://repository.apache.org/content/repositories/
> > > > orgapachepredictionio-1003/
> > > > .
> > > >
> > > > Do we need to vote on the RC2 again?
> > > >
> > > > On Sat, Sep 17, 2016 at 2:10 PM, Pat Ferrel 
> > > wrote:
> > > >
> > > > > Hate to make more work for you but the new name would also maybe
> > avoid
> > > > > some confusion for older migrating users.
> > > > >
> > > > > On Sep 16, 2016, at 9:02 AM, Donald Szeto 
> wrote:
> > > > >
> > > > > If everyone agrees that the artifacts should have an "apache-"
> > prefix,
> > > I
> > > > > will roll an RC2 shortly. Since the namespace change is
> unavoidable,
> > > > might
> > > > > as well get all artifact name changes done in one shot for good.
> > > > >
> > > > > On Friday, September 16, 2016, Andrew Purtell <
> > > andrew.purt...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Let me double check.
> > > > > >
> > > > > >> On Sep 16, 2016, at 7:33 AM, Alex Merritt <
> > emergentor...@apache.org
> > > > > > > wrote:
> > > > > >>
> > > > > >> I believe it depends on which of the two votes you mean. For the
> > > > podling
> > > > > >> vote, PPMC votes are binding, for the incubator vote,  IPMC
> votes
> > > are,
> > > > > > no?
> > > > > >>
> > > > > >>> On Sep 15, 2016 9:42 PM, "Andrew Purtell" <
> > > andrew.purt...@gmail.com
> > > > > > > wrote:
> > > > > >>>
> > > > > >>> I believe 'binding' only applies to IPMC.
> > > > > >>>
> > > > > > On Sep 15, 2016, at 12:49 PM, Suneel Marthi <
> > > > suneel.mar...@gmail.com
> > > > > > >
> > > > >  wrote:
> > > > > 
> > > > >  Folks, When u vote please specify "+1 Binding" if u r a PMC
> > > member.
> > > > > Its
> > > > >  only the PMC votes that count for a release to pass.
> > > > > 
> > > > > 
> > > > > 
> > > > > > On Thu, Sep 15, 2016 at 2:11 PM, Robert Lu <
> > > robberp...@outlook.com
> > > > > > >
> > > > > >>> wrote:
> > > > > >
> > > > > > +1
> > > > > >
> > > > > >> On Sep 15, 2016, at 01:13, Matthew Tovbin <
> tovb...@apache.org
> > > > > > > wrote:
> > > > > >>
> > > > > >> +1
> > > > > >>
> > > > > >>> On Wed, Sep 14, 2016 at 10:12 AM, Pat Ferrel <
> > > > > p...@occamsmachete.com
> > > > > > >
> > > > > >> wrote:
> > > > > >>
> > > > > >>> +1
> > > > > >>>
> > > > > >>>
> > > > > >>> On Sep 13, 2016, at 11:55 AM, Donald Szeto <
> > > > donald.sz...@gmail.com
> > > > > > >
> > > > > > wrote:
> > > > > >>>
> > > > > >>> This is the vote for 0.10.0 of Apache PredictionIO
> > > (incubating).
> > > > > >>>
> > > > > >>> The vote will run for at least 72 hours and will close on
> > Sept
> > > > > 16th,
> > > > > > 2016.
> > > > > >>>
> > > > > >>> The artifacts can be downloaded here:
> > > > > >>> https://dist.apache.org/repos/
> dist/dev/incubator/predictioni
> > > > > >>> o/0.10.0-incubating-rc1/
> > > > > >>> or
> > > > > >>> from the Maven staging repo here:
> > > > > >>> https://repository.apache.org/
> content/repositories/orgapache
> > > > > >>> predictionio-1001/
> > > > > >>>
> > > > > >>> All JIRAs completed for this release are tagged with
> > > 'FixVersion
> > > > =
> > > > > > 0.10.0'.
> > > > > >>> You can view them here:
> > > > > >>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> proje
> > > > > >>> ctId=12320420=12337844
> > > > > >>>
> > > > > >>> The artifacts have been signed with Key : 8BF4ABEB
> > > > > >>>
> > > > > >>> Please vote accordingly:
> > > > > >>>
> > > > > >>> [ ] +1, accept RC as the official 0.10.0 release
> > > > > >>> [ ] -1, do not accept RC as the official 0.10.0 release
> > > > because...
> > > > > >>>
> > > > > >
> > > > >
> > > > >
> > > >
> > >
> >
>

Re: [VOTE]: Apache PredictionIO (incubating) 0.10.0 Release

2016-09-18 Thread Suneel Marthi 
So all of these artifacts need to be verified and validated ?

https://repository.apache.org/content/repositories/orgapachepredictionio-1003/org/apache/predictionio/



On Mon, Sep 19, 2016 at 2:59 AM, Donald Szeto  wrote:

> I will start a new voting thread. The staging repo should contain just
> PredictionIO artifacts.
>
> On Sun, Sep 18, 2016 at 5:57 PM, Suneel Marthi  wrote:
>
> > the MAven staging repo brings up whole load of artifacts, which ones r we
> > voting on again ?
> >
> > On Mon, Sep 19, 2016 at 2:49 AM, Donald Szeto  wrote:
> >
> > > I have prepared a RC2. The artifacts can be downloaded here:
> > > https://dist.apache.org/repos/dist/dev/incubator/predictionio/0.10.0-
> > > incubating-rc2/
> > > or from the Maven staging repo here:
> > > https://repository.apache.org/content/repositories/
> > > orgapachepredictionio-1003/
> > > .
> > >
> > > Do we need to vote on the RC2 again?
> > >
> > > On Sat, Sep 17, 2016 at 2:10 PM, Pat Ferrel 
> > wrote:
> > >
> > > > Hate to make more work for you but the new name would also maybe
> avoid
> > > > some confusion for older migrating users.
> > > >
> > > > On Sep 16, 2016, at 9:02 AM, Donald Szeto  wrote:
> > > >
> > > > If everyone agrees that the artifacts should have an "apache-"
> prefix,
> > I
> > > > will roll an RC2 shortly. Since the namespace change is unavoidable,
> > > might
> > > > as well get all artifact name changes done in one shot for good.
> > > >
> > > > On Friday, September 16, 2016, Andrew Purtell <
> > andrew.purt...@gmail.com>
> > > > wrote:
> > > >
> > > > > Let me double check.
> > > > >
> > > > >> On Sep 16, 2016, at 7:33 AM, Alex Merritt <
> emergentor...@apache.org
> > > > > > wrote:
> > > > >>
> > > > >> I believe it depends on which of the two votes you mean. For the
> > > podling
> > > > >> vote, PPMC votes are binding, for the incubator vote,  IPMC votes
> > are,
> > > > > no?
> > > > >>
> > > > >>> On Sep 15, 2016 9:42 PM, "Andrew Purtell" <
> > andrew.purt...@gmail.com
> > > > > > wrote:
> > > > >>>
> > > > >>> I believe 'binding' only applies to IPMC.
> > > > >>>
> > > > > On Sep 15, 2016, at 12:49 PM, Suneel Marthi <
> > > suneel.mar...@gmail.com
> > > > > >
> > > >  wrote:
> > > > 
> > > >  Folks, When u vote please specify "+1 Binding" if u r a PMC
> > member.
> > > > Its
> > > >  only the PMC votes that count for a release to pass.
> > > > 
> > > > 
> > > > 
> > > > > On Thu, Sep 15, 2016 at 2:11 PM, Robert Lu <
> > robberp...@outlook.com
> > > > > >
> > > > >>> wrote:
> > > > >
> > > > > +1
> > > > >
> > > > >> On Sep 15, 2016, at 01:13, Matthew Tovbin  > > > > > wrote:
> > > > >>
> > > > >> +1
> > > > >>
> > > > >>> On Wed, Sep 14, 2016 at 10:12 AM, Pat Ferrel <
> > > > p...@occamsmachete.com
> > > > > >
> > > > >> wrote:
> > > > >>
> > > > >>> +1
> > > > >>>
> > > > >>>
> > > > >>> On Sep 13, 2016, at 11:55 AM, Donald Szeto <
> > > donald.sz...@gmail.com
> > > > > >
> > > > > wrote:
> > > > >>>
> > > > >>> This is the vote for 0.10.0 of Apache PredictionIO
> > (incubating).
> > > > >>>
> > > > >>> The vote will run for at least 72 hours and will close on
> Sept
> > > > 16th,
> > > > > 2016.
> > > > >>>
> > > > >>> The artifacts can be downloaded here:
> > > > >>> https://dist.apache.org/repos/dist/dev/incubator/predictioni
> > > > >>> o/0.10.0-incubating-rc1/
> > > > >>> or
> > > > >>> from the Maven staging repo here:
> > > > >>> https://repository.apache.org/content/repositories/orgapache
> > > > >>> predictionio-1001/
> > > > >>>
> > > > >>> All JIRAs completed for this release are tagged with
> > 'FixVersion
> > > =
> > > > > 0.10.0'.
> > > > >>> You can view them here:
> > > > >>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?proje
> > > > >>> ctId=12320420=12337844
> > > > >>>
> > > > >>> The artifacts have been signed with Key : 8BF4ABEB
> > > > >>>
> > > > >>> Please vote accordingly:
> > > > >>>
> > > > >>> [ ] +1, accept RC as the official 0.10.0 release
> > > > >>> [ ] -1, do not accept RC as the official 0.10.0 release
> > > because...
> > > > >>>
> > > > >
> > > >
> > > >
> > >
> >
>

[VOTE]: Apache PredictionIO (incubating) 0.10.0 Release (RC2)

2016-09-18 Thread Donald Szeto 
This is the vote for 0.10.0 of Apache PredictionIO (incubating).

The only difference between RC1 and RC2 is adding the "apache-" prefix to
Maven artifacts.

The vote will run for at least 72 hours and will close on Sept 21st, 2016.

The artifacts can be downloaded here:
https://dist.apache.org/repos/dist/dev/incubator/predictionio/0.10.0-incubating-rc2/
or from the Maven staging repo here:
https://repository.apache.org/content/repositories/orgapachepredictionio-1003/

All JIRAs completed for this release are tagged with 'FixVersion = 0.10.0'.
You can view them here:
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12320420=12337844

The artifacts have been signed with Key : 8BF4ABEB

Please vote accordingly:

[ ] +1, accept RC as the official 0.10.0 release
[ ] -1, do not accept RC as the official 0.10.0 release because...

Re: [VOTE]: Apache PredictionIO (incubating) 0.10.0 Release

2016-09-18 Thread Suneel Marthi 
the MAven staging repo brings up whole load of artifacts, which ones r we
voting on again ?

On Mon, Sep 19, 2016 at 2:49 AM, Donald Szeto  wrote:

> I have prepared a RC2. The artifacts can be downloaded here:
> https://dist.apache.org/repos/dist/dev/incubator/predictionio/0.10.0-
> incubating-rc2/
> or from the Maven staging repo here:
> https://repository.apache.org/content/repositories/
> orgapachepredictionio-1003/
> .
>
> Do we need to vote on the RC2 again?
>
> On Sat, Sep 17, 2016 at 2:10 PM, Pat Ferrel  wrote:
>
> > Hate to make more work for you but the new name would also maybe avoid
> > some confusion for older migrating users.
> >
> > On Sep 16, 2016, at 9:02 AM, Donald Szeto  wrote:
> >
> > If everyone agrees that the artifacts should have an "apache-" prefix, I
> > will roll an RC2 shortly. Since the namespace change is unavoidable,
> might
> > as well get all artifact name changes done in one shot for good.
> >
> > On Friday, September 16, 2016, Andrew Purtell 
> > wrote:
> >
> > > Let me double check.
> > >
> > >> On Sep 16, 2016, at 7:33 AM, Alex Merritt  > > > wrote:
> > >>
> > >> I believe it depends on which of the two votes you mean. For the
> podling
> > >> vote, PPMC votes are binding, for the incubator vote,  IPMC votes are,
> > > no?
> > >>
> > >>> On Sep 15, 2016 9:42 PM, "Andrew Purtell"  > > > wrote:
> > >>>
> > >>> I believe 'binding' only applies to IPMC.
> > >>>
> > > On Sep 15, 2016, at 12:49 PM, Suneel Marthi <
> suneel.mar...@gmail.com
> > > >
> >  wrote:
> > 
> >  Folks, When u vote please specify "+1 Binding" if u r a PMC member.
> > Its
> >  only the PMC votes that count for a release to pass.
> > 
> > 
> > 
> > > On Thu, Sep 15, 2016 at 2:11 PM, Robert Lu  > > >
> > >>> wrote:
> > >
> > > +1
> > >
> > >> On Sep 15, 2016, at 01:13, Matthew Tovbin  > > > wrote:
> > >>
> > >> +1
> > >>
> > >>> On Wed, Sep 14, 2016 at 10:12 AM, Pat Ferrel <
> > p...@occamsmachete.com
> > > >
> > >> wrote:
> > >>
> > >>> +1
> > >>>
> > >>>
> > >>> On Sep 13, 2016, at 11:55 AM, Donald Szeto <
> donald.sz...@gmail.com
> > > >
> > > wrote:
> > >>>
> > >>> This is the vote for 0.10.0 of Apache PredictionIO (incubating).
> > >>>
> > >>> The vote will run for at least 72 hours and will close on Sept
> > 16th,
> > > 2016.
> > >>>
> > >>> The artifacts can be downloaded here:
> > >>> https://dist.apache.org/repos/dist/dev/incubator/predictioni
> > >>> o/0.10.0-incubating-rc1/
> > >>> or
> > >>> from the Maven staging repo here:
> > >>> https://repository.apache.org/content/repositories/orgapache
> > >>> predictionio-1001/
> > >>>
> > >>> All JIRAs completed for this release are tagged with 'FixVersion
> =
> > > 0.10.0'.
> > >>> You can view them here:
> > >>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?proje
> > >>> ctId=12320420=12337844
> > >>>
> > >>> The artifacts have been signed with Key : 8BF4ABEB
> > >>>
> > >>> Please vote accordingly:
> > >>>
> > >>> [ ] +1, accept RC as the official 0.10.0 release
> > >>> [ ] -1, do not accept RC as the official 0.10.0 release
> because...
> > >>>
> > >
> >
> >
>

Re: [VOTE]: Apache PredictionIO (incubating) 0.10.0 Release

2016-09-18 Thread Suneel Marthi 
Yes u do have to vote fresh each time u create a new RC.

On Mon, Sep 19, 2016 at 2:49 AM, Donald Szeto  wrote:

> I have prepared a RC2. The artifacts can be downloaded here:
> https://dist.apache.org/repos/dist/dev/incubator/predictionio/0.10.0-
> incubating-rc2/
> or from the Maven staging repo here:
> https://repository.apache.org/content/repositories/
> orgapachepredictionio-1003/
> .
>
> Do we need to vote on the RC2 again?
>
> On Sat, Sep 17, 2016 at 2:10 PM, Pat Ferrel  wrote:
>
> > Hate to make more work for you but the new name would also maybe avoid
> > some confusion for older migrating users.
> >
> > On Sep 16, 2016, at 9:02 AM, Donald Szeto  wrote:
> >
> > If everyone agrees that the artifacts should have an "apache-" prefix, I
> > will roll an RC2 shortly. Since the namespace change is unavoidable,
> might
> > as well get all artifact name changes done in one shot for good.
> >
> > On Friday, September 16, 2016, Andrew Purtell 
> > wrote:
> >
> > > Let me double check.
> > >
> > >> On Sep 16, 2016, at 7:33 AM, Alex Merritt  > > > wrote:
> > >>
> > >> I believe it depends on which of the two votes you mean. For the
> podling
> > >> vote, PPMC votes are binding, for the incubator vote,  IPMC votes are,
> > > no?
> > >>
> > >>> On Sep 15, 2016 9:42 PM, "Andrew Purtell"  > > > wrote:
> > >>>
> > >>> I believe 'binding' only applies to IPMC.
> > >>>
> > > On Sep 15, 2016, at 12:49 PM, Suneel Marthi <
> suneel.mar...@gmail.com
> > > >
> >  wrote:
> > 
> >  Folks, When u vote please specify "+1 Binding" if u r a PMC member.
> > Its
> >  only the PMC votes that count for a release to pass.
> > 
> > 
> > 
> > > On Thu, Sep 15, 2016 at 2:11 PM, Robert Lu  > > >
> > >>> wrote:
> > >
> > > +1
> > >
> > >> On Sep 15, 2016, at 01:13, Matthew Tovbin  > > > wrote:
> > >>
> > >> +1
> > >>
> > >>> On Wed, Sep 14, 2016 at 10:12 AM, Pat Ferrel <
> > p...@occamsmachete.com
> > > >
> > >> wrote:
> > >>
> > >>> +1
> > >>>
> > >>>
> > >>> On Sep 13, 2016, at 11:55 AM, Donald Szeto <
> donald.sz...@gmail.com
> > > >
> > > wrote:
> > >>>
> > >>> This is the vote for 0.10.0 of Apache PredictionIO (incubating).
> > >>>
> > >>> The vote will run for at least 72 hours and will close on Sept
> > 16th,
> > > 2016.
> > >>>
> > >>> The artifacts can be downloaded here:
> > >>> https://dist.apache.org/repos/dist/dev/incubator/predictioni
> > >>> o/0.10.0-incubating-rc1/
> > >>> or
> > >>> from the Maven staging repo here:
> > >>> https://repository.apache.org/content/repositories/orgapache
> > >>> predictionio-1001/
> > >>>
> > >>> All JIRAs completed for this release are tagged with 'FixVersion
> =
> > > 0.10.0'.
> > >>> You can view them here:
> > >>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?proje
> > >>> ctId=12320420=12337844
> > >>>
> > >>> The artifacts have been signed with Key : 8BF4ABEB
> > >>>
> > >>> Please vote accordingly:
> > >>>
> > >>> [ ] +1, accept RC as the official 0.10.0 release
> > >>> [ ] -1, do not accept RC as the official 0.10.0 release
> because...
> > >>>
> > >
> >
> >
>

Re: Remove engine registration

2016-09-18 Thread Pat Ferrel 
This sounds like a good case for Donald’s suggestion. 

What I was trying to add to the discussion is a way to make all commands rely 
on state in the megastore, rather than any file on any machine in a cluster or 
on ordering of execution or execution from a location in a directory structure. 
All commands would then be stateless.

This enables real use cases like provisioning PIO machines and running `pio 
deploy ` to get a new PredictionServer. Provisioning can be 
container and discovery based rather cleanly.


On Sep 17, 2016, at 5:26 PM, Mars Hall  wrote:

Hello folks,

Great to hear about this possibility. I've been working on running PredictionIO 
on Heroku https://www.heroku.com

Heroku's 12-factor architecture https://12factor.net prefers "stateless builds" 
to ensure that compiled artifacts result in processes which may be cheaply 
restarted, replaced, and scaled via process count & size. I imagine this 
stateless property would be valuable for others as well.

The fact that `pio build` inserts stateful metadata into a database causes 
ripples throughout the lifecycle of PIO engines on Heroku:

* An engine cannot be built for production without the production database 
available. When a production database contains PII (personally identifiable 
information) which has security compliance requirements, the build system may 
not be privileged to access that PII data. This also affects CI (continuous 
integration/testing), where engines would need to be rebuilt in production, 
defeating assurances CI is supposed to provide.

* The build artifacts cannot be reliably reused. "Slugs" at Heroku are intended 
to be stateless, so that you can rollback to a previous version during the 
lifetime of an app. With `pio build` causing database side-effects, there's a 
greater-than-zero probability of slug-to-metadata inconsistencies eventually 
surfacing in a long-running system.


From my user-perspective, a few changes to the CLI would fix it:

1. add a "skip registration" option, `pio build --without-engine-registration`
2. a new command `pio app register` that could be run separately in the built 
engine (before training)

Alas, I do not know PredictionIO internals, so I can only offer a suggestion 
for how this might be solved.


Donald, one specific note,

Regarding "No automatic version matching of PIO binary distribution and 
artifacts version used in the engine template":

The Heroku slug contains the PredictionIO binary distribution used to build the 
engine, so there's never a version matching issue. I guess some systems might 
deploy only the engine artifacts to production where a pre-existing PIO binary 
is available, but that seems like a risky practice for long-running systems.


Thanks for listening,

*Mars Hall
Customer Facing Architect
Salesforce App Cloud / Heroku
San Francisco, California

> On Sep 16, 2016, at 10:42, Donald Szeto  wrote:
> 
> Hi all,
> 
> I want to start the discussion of removing engine registration. How many 
> people actually take advantage of being able to run pio commands everywhere 
> outside of an engine template directory? This will be a nontrivial change on 
> the operational side so I want to gauge the potential impact to existing 
> users.
> 
> Pros:
> - Stateless build. This would work well with many PaaS.
> - Eliminate the "pio build" command once and for all.
> - Ability to use your own build system, i.e. Maven, Ant, Gradle, etc.
> - Potentially better experience with IDE since engine templates no longer 
> depends on an SBT plugin.
> 
> Cons:
> - Inability to run pio engine training and deployment commands outside of 
> engine template directory.
> - No automatic version matching of PIO binary distribution and artifacts 
> version used in the engine template.
> - A less unified user experience: from pio-build-train-deploy to build, then 
> pio-train-deploy.
> 
> Regards,
> Donald

[jira] [Created] (PIO-38) add Apache Parquet as a data source

2016-09-18 Thread Wojciech Indyk (JIRA) 
Wojciech Indyk created PIO-38:
-

 Summary: add Apache Parquet as a data source
 Key: PIO-38
 URL: https://issues.apache.org/jira/browse/PIO-38
 Project: PredictionIO
  Issue Type: New Feature
Reporter: Wojciech Indyk


Apache Parquet (https://parquet.apache.org/) is a columnar data store, native 
for Apache Spark and very well suited to storing batch data (as an input) for 
PredictionIO Engine.
Parquet is very popular to archive clickstream, so it would enable to use 
PredictionIO without additional import of data (and duplication) to HBase.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)