[
https://issues.apache.org/jira/browse/SYSTEMML-1071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15626597#comment-15626597
]
Glenn Weidner commented on SYSTEMML-1071:
-----------------------------------------
To workaround test data generation issue [SYSTEMML-1044], used MR to
successfully create 8GB data:
./genDescriptiveStatisticsData_1M.sh gwperftest MR &>> logs/genStatsData.out
Univar-Stats.dml completed successfully (no dimension mismatch) against the
complete data set:
./runAllStats_8g_univar.sh gwperftest SPARK
SystemML execution statistics:
Total elapsed time: 747.936 sec.
Total compilation time: 2.032 sec.
Total execution time: 745.905 sec.
Number of compiled Spark inst: 0.
Number of executed Spark inst: 0.
Cache hits (Mem, WB, FS, HDFS): 27904/0/0/2.
Cache writes (WB, FS, HDFS): 3138/0/1.
Cache times (ACQr/m, RLS, EXP): 9.478/0.027/0.463/0.365 sec.
HOP DAGs recompiled (PRED, SB): 0/100.
HOP DAGs recompile time: 0.413 sec.
Spark ctx create time (lazy): 0.000 sec.
Spark trans counts (par,bc,col):0/0/0.
Spark trans times (par,bc,col): 0.000/0.000/0.000 secs.
ParFor loops optimized: 1.
ParFor optimize time: 0.179 sec.
ParFor initialize time: 0.010 sec.
ParFor result merge time: 0.002 sec.
ParFor total update in-place: 0/0/13900
Total JIT compile time: 4.29 sec.
Total JVM GC count: 25.
Total JVM GC time: 1.365 sec.
Heavy hitter instructions (name, time, count):
-- 1) cm 463.780 sec 2700
-- 2) qsort 221.881 sec 900
-- 3) rangeReIndex 15.815 sec 2999
-- 4) uamean 12.531 sec 900
-- 5) qpick 12.011 sec 1800
-- 6) uacmax 10.099 sec 1
-- 7) uamax 2.754 sec 1101
-- 8) uamin 2.546 sec 1000
-- 9) ctable 2.444 sec 100
-- 10) leftIndex 0.432 sec 13900
> Perftest: Univar-Stats.dml fails with dimension mismatch for M (8GB) scenario
> -----------------------------------------------------------------------------
>
> Key: SYSTEMML-1071
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1071
> Project: SystemML
> Issue Type: Bug
> Reporter: Glenn Weidner
>
> Univar-Stats.dml fails for 8G medium scenario (runUnivar-Stats_A_1M) with
> below stack trace. Note smaller and larger (80G) scenarios completed without
> error.
> 16/10/27 10:20:20 WARN parser.Expression: WARNING:
> ../algorithms/Univar-Stats.dml -- line 64, column 21 -- ppred() has been
> deprecated. Please use the operator directly.
> 16/10/27 10:20:20 ERROR api.DMLScript: Failed to execute DML script.
> org.apache.sysml.parser.LanguageException: Invalid Parameters : ERROR:
> ../algorithms/Univar-Stats.dml -- line 64, column 21 -- Mismatch in
> dimensions for operation (PPRED(K,1,>) MULT maxs)
> at
> org.apache.sysml.parser.Expression.raiseValidateError(Expression.java:557)
> at
> org.apache.sysml.parser.BinaryExpression.checkAndSetDimensions(BinaryExpression.java:188)
> at
> org.apache.sysml.parser.BinaryExpression.validateExpression(BinaryExpression.java:141)
> at
> org.apache.sysml.parser.BuiltinFunctionExpression.validateExpression(BuiltinFunctionExpression.java:323)
> at
> org.apache.sysml.parser.StatementBlock.validate(StatementBlock.java:600)
> at
> org.apache.sysml.parser.DMLTranslator.validateParseTree(DMLTranslator.java:136)
> at org.apache.sysml.api.DMLScript.execute(DMLScript.java:595)
> at org.apache.sysml.api.DMLScript.executeScript(DMLScript.java:354)
> at org.apache.sysml.api.DMLScript.main(DMLScript.java:199)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
> at
> org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
> at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
> Exception in thread "main" org.apache.sysml.api.DMLException:
> org.apache.sysml.parser.LanguageException: Invalid Parameters : ERROR:
> ../algorithms/Univar-Stats.dml -- line 64, column 21 -- Mismatch in
> dimensions for operation (PPRED(K,1,>) MULT maxs)
> at org.apache.sysml.api.DMLScript.executeScript(DMLScript.java:368)
> at org.apache.sysml.api.DMLScript.main(DMLScript.java:199)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
> at
> org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
> at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
> Caused by: org.apache.sysml.parser.LanguageException: Invalid Parameters :
> ERROR: ../algorithms/Univar-Stats.dml -- line 64, column 21 -- Mismatch in
> dimensions for operation (PPRED(K,1,>) MULT maxs)
> at
> org.apache.sysml.parser.Expression.raiseValidateError(Expression.java:557)
> at
> org.apache.sysml.parser.BinaryExpression.checkAndSetDimensions(BinaryExpression.java:188)
> at
> org.apache.sysml.parser.BinaryExpression.validateExpression(BinaryExpression.java:141)
> at
> org.apache.sysml.parser.BuiltinFunctionExpression.validateExpression(BuiltinFunctionExpression.java:323)
> at
> org.apache.sysml.parser.StatementBlock.validate(StatementBlock.java:600)
> at
> org.apache.sysml.parser.DMLTranslator.validateParseTree(DMLTranslator.java:136)
> at org.apache.sysml.api.DMLScript.execute(DMLScript.java:595)
> at org.apache.sysml.api.DMLScript.executeScript(DMLScript.java:354)
> ... 10 more
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
