[
https://issues.apache.org/jira/browse/PIO-192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16682673#comment-16682673
]
ASF GitHub Bot commented on PIO-192:
------------------------------------
takezoe commented on a change in pull request #494: [PIO-192] Enhance PySpark
support
URL: https://github.com/apache/predictionio/pull/494#discussion_r232469362
##########
File path: python/pypio/pypio.py
##########
@@ -15,9 +15,35 @@
# limitations under the License.
#
+from __future__ import absolute_import
+
+import sys
+
from pypio.data import PEventStore
-from pypio.workflow import CleanupFunctions
+from pyspark.sql import SparkSession
+from pyspark.sql import utils
+
+
+def init():
+ global spark
+ spark = SparkSession.builder.getOrCreate()
+ global sc
+ sc = spark.sparkContext
+ global sqlContext
+ sqlContext = spark._wrapped
+ global p_event_store
+ p_event_store = PEventStore(spark._jsparkSession, sqlContext)
+ print("Initialized pypio")
+
+
+def find(app_name):
+ return p_event_store.find(app_name)
-p_event_store = PEventStore(spark._jsparkSession, sqlContext)
-cleanup_functions = CleanupFunctions(sqlContext)
+def save(model):
+ engine = sc._jvm.org.apache.predictionio.e2.engine.PythonEngine
+ engine.model().set(model._to_java())
+ main_args = utils.toJArray(sc._gateway, sc._gateway.jvm.String, sys.argv)
+ create_workflow = sc._jvm.org.apache.predictionio.workflow.CreateWorkflow
+ spark.stop()
+ create_workflow.main(main_args)
Review comment:
This functions is wrapper of `CreateWorkflow`. It seems to can run
`runEvaluation` not only `runTrain` by giving bootstrap arguments. Does it work?
https://github.com/apache/predictionio/blob/4342fcd9d0a7b549543b59467f5e1b008523fe4f/core/src/main/scala/org/apache/predictionio/workflow/CreateWorkflow.scala#L271
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Enhance PySpark support
> -----------------------
>
> Key: PIO-192
> URL: https://issues.apache.org/jira/browse/PIO-192
> Project: PredictionIO
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.13.0
> Reporter: Takako Shimamoto
> Assignee: Takako Shimamoto
> Priority: Major
>
> h3. Summary
> Enhance the pypio, which is the Python API for PIO.
> h3. Goals
> The limitations of the current Python support always force developers to have
> access to sbt. This enhancement will get rid of the build phase.
> h3. Description
> A Python engine template requires 3 files:
> * Python code to specify for the --main-py-file option
> * template.json
> {code:json}
> {"pio": {"version": { "min": "0.14.0-SNAPSHOT" }}}
> {code}
> * engine.json
> {code:json}
> {
> "id": "default",
> "description": "Default settings",
> "engineFactory": "org.apache.predictionio.e2.engine.PythonEngine",
> "algorithms": [
> {
> "name": "default",
> "params": {
> "name": "BHPApp"
> }
> }
> ],
> "serving": {
> "params": {
> "columns": ["prediction"]
> }
> }
> }
> {code}
> h4. pypio module
> Developers can use the pypio module with jupyter notebook and Python code.
> First, import the necessary modules.
> {code:python}
> from pypio import pypio
> {code}
> Once the module in imported, the first step is to initialize the pypio module.
> {code:python}
> pypio.init()
> {code}
> Next, find data from the event store.
> {code:python}
> event_df = pypio.find('BHPApp')
> {code}
> And then, save the model.
> {code:python}
> # model is a PipelineModel, which is produced after a Pipeline’s fit() method
> runs
> pipeline = Pipeline(...)
> model = pipeline.fit(train_df)
> pypio.save(model)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
