[ https://issues.apache.org/jira/browse/SPARK-21476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16094182#comment-16094182 ]
Saurabh Agrawal edited comment on SPARK-21476 at 7/20/17 5:16 AM: ------------------------------------------------------------------ I'm saying that the trees in the model get serialized with each task which increases the task deserialization time if the forest is big. I see that there is a transformImpl in RandomForestClassificationModel which is broadcasting itself first and then calling predict on the broadcast value (https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/ml/classification/RandomForestClassifier.scala#L207-L213). But transformImpl is not getting invoked by the transform method in ProbabilisticClassificationModel. Instead ProbabilisticClassificationModel uses the concrete class definition of predictRaw. transorm is a distributed operation but the trees contained within the model do not get broadcast and are instead serialized with each task. Is this intended behavior? was (Author: sagraw): I'm saying that the trees in the model get serialized with each task which increases the task deserialization time if the forest is big. I see that there is a transformImpl in RandomForestClassificationModel which is broadcasting itself first and then calling predict on the broadcast value (https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/ml/classification/RandomForestClassifier.scala#L207-L213). But transformImpl is not getting invoked by the transform method in ProbabilisticClassificationModel. Instead ProbabilisticClassificationModel uses the concrete class definition of predictRaw. transorm is a distributed operation but the trees contained within the model do not get broadcast and instead are serialized with each task. Is this intended behavior? > RandomForest classification model not using broadcast in transform > ------------------------------------------------------------------ > > Key: SPARK-21476 > URL: https://issues.apache.org/jira/browse/SPARK-21476 > Project: Spark > Issue Type: Bug > Components: ML > Affects Versions: 2.2.0 > Reporter: Saurabh Agrawal > > I notice significant task deserialization latency while running prediction > with pipelines using RandomForestClassificationModel. While digging into the > source, found that the transform method in RandomForestClassificationModel > binds to its parent ProbabilisticClassificationModel and the only concrete > definition that RandomForestClassificationModel provides and which is > actually used in transform is that of predictRaw. Broadcasting is not being > used in predictRaw. -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org