[
https://issues.apache.org/jira/browse/SYSTEMML-1471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15961117#comment-15961117
]
Deron Eriksson commented on SYSTEMML-1471:
------------------------------------------
Hi [~mboehm7]. I just wanted to mention that if compilation chains are being
consolidated, I would really prefer if the ScriptExecutor class continues to
maintain its current flexibility. ScriptExecutor decouples script execution
from scripts and a primary point of its architecture is to allow flexibility
for advanced users to specify their own compilation chains or to modify steps
in the compilation chain. As an example, because of this flexibility, it was
possible to create the above ScoringScriptExecutor class in under 5 minutes
with hardly any code. ScriptExecutor has a sensible default compilation chain
but is also extensible to easily allow modifications for use cases that we
haven't foreseen.
cc [~freiss] [~reinwald]
> Support PreparedScript for MLContext
> ------------------------------------
>
> Key: SYSTEMML-1471
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1471
> Project: SystemML
> Issue Type: Improvement
> Reporter: Niketan Pansare
>
> The intent of this JIRA is three-fold:
> 1. Allow MLContext to be used in prediction scenario.
> 2. Consolidate the code of JMLC and MLContext.
> 3. Explore what extensions are needed in SystemML to support Spark streaming.
> For prediction scenario, it is important to reduce the parsing/validation
> overhead as much as possible and reusing the JMLC infrastructure might be a
> good step in that direction. It is also important that MLContext continues to
> support dynamic recompilation and other optimization as the input size could
> be small (similar to JMLC), but could also be large (if window size is large,
> making MLContext ideal for this scenario).
> {code}
> val streamingContext = new StreamingContext(sc, SLIDE_INTERVAL)
> val windowDStream = .....window(WINDOW_LENGTH, SLIDE_INTERVAL)
> val preparedScript = ....prepareScript(....)
> windowDStream.foreachRDD(currentWindow => {
> if (currentWindow.count() > 0) {
> ml.execute(preparedScript.in("X", currentWindow.toDF()))
> ...
> }
> })
> {code}
> [~deron] [~mboehm7] [~reinwald] [~freiss] [~mwdus...@us.ibm.com] [~nakul02]
> Is this something that interest anyone of you ?
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
