Repository: incubator-systemml Updated Branches: refs/heads/master a83e25cc1 -> 400a0d822
[SYSTEMML-640] Fix forced parfor program recompile (invalid selective) The parfor util script sample.dml failed with dimension mismatch in special cases, where the remote memory budget of map/reduce tasks was larger than the driver memory budget and the permutation matrix multiplication would be compiled to MR in local parfor but CP in remote parfor execution. If the parfor optimizer selects remote parfor in these cases, we trigger a forced recompile to CP which internally tries to reduce the overhead by recompiling only dags where the runtime plan contains MR instructions. This selective recompilation is invalid with permutation matrix multiplications that stretch two subsequent basic block dags and the first dag does not necessarily contain MR instructions. This patch removes this selective compilation because meanwhile we compile average dags in <1ms. Project: http://git-wip-us.apache.org/repos/asf/incubator-systemml/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-systemml/commit/95f17be5 Tree: http://git-wip-us.apache.org/repos/asf/incubator-systemml/tree/95f17be5 Diff: http://git-wip-us.apache.org/repos/asf/incubator-systemml/diff/95f17be5 Branch: refs/heads/master Commit: 95f17be5a38f554b30beae7380f69b456dace700 Parents: a83e25c Author: Matthias Boehm <[email protected]> Authored: Tue Apr 19 21:24:53 2016 -0700 Committer: Matthias Boehm <[email protected]> Committed: Tue Apr 19 21:24:53 2016 -0700 ---------------------------------------------------------------------- src/main/java/org/apache/sysml/hops/recompile/Recompiler.java | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-systemml/blob/95f17be5/src/main/java/org/apache/sysml/hops/recompile/Recompiler.java ---------------------------------------------------------------------- diff --git a/src/main/java/org/apache/sysml/hops/recompile/Recompiler.java b/src/main/java/org/apache/sysml/hops/recompile/Recompiler.java index 9452bc2..c21c3a7 100644 --- a/src/main/java/org/apache/sysml/hops/recompile/Recompiler.java +++ b/src/main/java/org/apache/sysml/hops/recompile/Recompiler.java @@ -1366,9 +1366,9 @@ public class Recompiler { StatementBlock sb = pb.getStatementBlock(); - //recompile hops dag to CP (opt: don't recompile if CP and no MR inst) - if( sb != null && !(et==ExecType.CP && !OptTreeConverter.containsMRJobInstruction(pb, true, true)) ) - { + //recompile hops dag to CP (note selective recompile 'if CP and no MR inst' + //would be invalid with permutation matrix mult across multiple dags) + if( sb != null ) { ArrayList<Instruction> tmp = pb.getInstructions(); tmp = Recompiler.recompileHopsDag2Forced(sb, sb.get_hops(), tid, et); pb.setInstructions( tmp );
