Re: Serving Spark ML models via a regular Python web app

Hi Chris, I was just checking out your project. I mentioned we use MLeap to serve predictions from a trained Spark ML RandomForest model. How would I do that with pipeline.io ? It isn't clear to me. Thanks! Michael > On Aug 11, 2016, at 9:42 AM, Chris Fregly

Re: Serving Spark ML models via a regular Python web app

Thanks for the additional reference Chris. Sounds like there are a few independent projects addressing this story. On Thu, Aug 11, 2016 at 12:42 PM Chris Fregly wrote: > And here's a recent slide deck on the pipeline.io that summarizes what > we're working on (all open

Re: Serving Spark ML models via a regular Python web app

And here's a recent slide deck on the pipeline.io that summarizes what we're working on (all open source): https://www.slideshare.net/mobile/cfregly/advanced-spark-and-tensorflow-meetup-08042016-one-click-spark-ml-pipeline-deploy-to-production mleap is heading the wrong direction and

Re: Serving Spark ML models via a regular Python web app

this is exactly what my http://pipeline.io project is addressing. check it out and send me feedback or create issues at that github location. > On Aug 11, 2016, at 7:42 AM, Nicholas Chammas > wrote: > > Thanks Michael for the reference, and thanks Nick for the

Re: Serving Spark ML models via a regular Python web app

Thanks Michael for the reference, and thanks Nick for the comprehensive overview of existing JIRA discussions about this. I've added myself as a watcher on the various tasks. On Thu, Aug 11, 2016 at 3:02 AM Nick Pentreath wrote: > Currently there is no direct way in

Who controls 'databricks-jenkins'?

Not a big deal but 'he' is commenting on a lot of ancient PRs for some reason, like https://github.com/apache/spark/pull/51 and it generates mails to the list. I assume this is a misconfiguration somewhere. - To unsubscribe

Re: Serving Spark ML models via a regular Python web app

Currently there is no direct way in Spark to serve models without bringing in all of Spark as a dependency. For Spark ML, there is actually no way to do it independently of DataFrames either (which for single-instance prediction makes things sub-optimal). That is covered here:

Sorting within partitions is not maintained in parquet?

Hi, It seems that something changed between Spark 1.6.2 and 2.0.0 that I wasn't expecting. If I have a DataFrame with records sorted within each partition, and I write it to parquet and read back from the parquet, previously the records would be iterated through in the same order they were