[ https://issues.apache.org/jira/browse/SYSTEMML-909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15489124#comment-15489124 ]
Mike Dusenberry commented on SYSTEMML-909: ------------------------------------------ cc [~acs_s] > `determineDataFrameDimensionsIfNeeded(...)` is a bottleneck. > ------------------------------------------------------------ > > Key: SYSTEMML-909 > URL: https://issues.apache.org/jira/browse/SYSTEMML-909 > Project: SystemML > Issue Type: Improvement > Reporter: Mike Dusenberry > > The {{[determineDataFrameDimensionsIfNeeded(...) | > https://github.com/apache/incubator-systemml/blob/master/src/main/java/org/apache/sysml/api/mlcontext/MLContextConversionUtil.java#L585]}} > function in {{MLContext}} is a major bottleneck, particularly due to the > `javaRDD` call. > The issue I'm seeing is that the javaRDD.count() function causes execution of > the lazy DataFrames I pass in, which are created from another DataFrame via > df.randomSplit([0.8, 0.2]), thus a shuffle occurs. I know that this is going > to happen anyways in the internal conversion, but it wastes a lot of time by > having to also do it in this step too. Assume that I have more data than I > can efficiently cache (~7TB with the potential for much more), so I need to > incur the shuffle step only once on the way into the engine. -- This message was sent by Atlassian JIRA (v6.3.4#6332)