[ https://issues.apache.org/jira/browse/SPARK-23333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16358048#comment-16358048 ]
Liang-Chi Hsieh commented on SPARK-23333: ----------------------------------------- Currently I think we don't have API in Dataset to just fetch an any row back. Is it reasonable to add a \{{def any(n: Int): Array[T]}} to Dataset? cc [~cloud_fan] > SparkML VectorAssembler.transform slow when needing to invoke .first() on > sorted DataFrame > ------------------------------------------------------------------------------------------ > > Key: SPARK-23333 > URL: https://issues.apache.org/jira/browse/SPARK-23333 > Project: Spark > Issue Type: Improvement > Components: ML, MLlib, SQL > Affects Versions: 2.2.1 > Reporter: V Luong > Priority: Major > > Under certain circumstances, newDF = vectorAssembler.transform(oldDF) invokes > oldDF.first() in order to establish some metadata/attributes: > [https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/ml/feature/VectorAssembler.scala#L88.] > When oldDF is sorted, the above triggering of oldDF.first() can be very slow. > For the purpose of establishing metadata, taking an arbitrary row from oldDF > will be just as good as taking oldDF.first(). Is there hence a way we can > speed up a great deal by somehow grabbing a random row, instead of relying on > oldDF.first()? -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org