[
https://issues.apache.org/jira/browse/SYSTEMML-1471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15959961#comment-15959961
]
Deron Eriksson edited comment on SYSTEMML-1471 at 4/6/17 11:31 PM:
-------------------------------------------------------------------
I think you might be able to do something like this with the existing API. You
could create a ScoringScriptExecutor class that extends ScriptExecutor. On this
class, create a prepare(Script) method that contains:
{code}
setup(script);
parseScript();
liveVariableAnalysis();
validateScript();
constructHops();
rewriteHops();
rewritePersistentReadsAndWrites();
constructLops();
generateRuntimeProgram();
showExplanation();
globalDataFlowOptimization();
countCompiledMRJobsAndSparkInstructions();
initializeCachingAndScratchSpace();
cleanupRuntimeProgram();
{code}
Then override ScriptExecutor's execute(Script) method and have it contain:
{code}
if(statistics) {
Statistics.startRunTimer();
}
createAndInitializeExecutionContext();
executeRuntimeProgram();
cleanupAfterExecution();
// add symbol table to MLResults
MLResults mlResults = new MLResults(script);
script.setResults(mlResults);
if (statistics) {
Statistics.stopRunTimer();
System.out.println(Statistics.display(statisticsMaxHeavyHitters));
}
return mlResults;
{code}
In the calling code, have something like:
{code}
ScoringScriptExecutor sse = new ScoringScriptExecutor();
sse.prepare(script); // create the dml program
while (....) {
...
MLResults results = ml.execute(script, sse); // execute the dml program
}
{code}
was (Author: deron):
I think you might be able to do something like this with the existing API. You
could create a ScoringScriptExecutor class that extends ScriptExecutor. On this
class, create a prepare(Script) method that contains:
{code}
setup(script);
parseScript();
liveVariableAnalysis();
validateScript();
constructHops();
rewriteHops();
rewritePersistentReadsAndWrites();
constructLops();
generateRuntimeProgram();
showExplanation();
globalDataFlowOptimization();
countCompiledMRJobsAndSparkInstructions();
initializeCachingAndScratchSpace();
cleanupRuntimeProgram();
{code}
Then override ScriptExecutor's execute(Script) method and have it contain:
{code}
script.clearAll();
if(statistics) {
Statistics.startRunTimer();
}
createAndInitializeExecutionContext();
executeRuntimeProgram();
cleanupAfterExecution();
// add symbol table to MLResults
MLResults mlResults = new MLResults(script);
script.setResults(mlResults);
if (statistics) {
Statistics.stopRunTimer();
System.out.println(Statistics.display(statisticsMaxHeavyHitters));
}
return mlResults;
{code}
In the calling code, have something like:
{code}
ScoringScriptExecutor sse = new ScoringScriptExecutor();
sse.prepare(script); // create the dml program
while (....) {
...
MLResults results = ml.execute(script, sse); // execute the dml program
}
{code}
> Support PreparedScript for MLContext
> ------------------------------------
>
> Key: SYSTEMML-1471
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1471
> Project: SystemML
> Issue Type: Improvement
> Reporter: Niketan Pansare
>
> The intent of this JIRA is three-fold:
> 1. Allow MLContext to be used in prediction scenario.
> 2. Consolidate the code of JMLC and MLContext.
> 3. Explore what extensions are needed in SystemML to support Spark streaming.
> For prediction scenario, it is important to reduce the parsing/validation
> overhead as much as possible and reusing the JMLC infrastructure might be a
> good step in that direction. It is also important that MLContext continues to
> support dynamic recompilation and other optimization as the input size could
> be small (similar to JMLC), but could also be large (if window size is large,
> making MLContext ideal for this scenario).
> {code}
> val streamingContext = new StreamingContext(sc, SLIDE_INTERVAL)
> val windowDStream = .....window(WINDOW_LENGTH, SLIDE_INTERVAL)
> val preparedScript = ....prepareScript(....)
> windowDStream.foreachRDD(currentWindow => {
> if (currentWindow.count() > 0) {
> ml.execute(preparedScript.in("X", currentWindow.toDF()))
> ...
> }
> })
> {code}
> [~deron] [~mboehm7] [~reinwald] [~freiss] [~mwdus...@us.ibm.com] [~nakul02]
> Is this something that interest anyone of you ?
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
