Miles Crawford created SPARK-17669:
--------------------------------------
Summary: Strange UI behavior using Datasets
Key: SPARK-17669
URL: https://issues.apache.org/jira/browse/SPARK-17669
Project: Spark
Issue Type: Bug
Components: SQL, Web UI
Affects Versions: 2.0.0
Reporter: Miles Crawford
I recently migrated my application to Spark 2.0, and everything worked well,
except for one function that uses "toDS" and the ML libraries.
This stage used to complete in 15 minutes or so on 1.6.2, and now takes almost
two hours.
The UI shows very strange behavior - completed stages still being worked on,
concurrent work on tons of stages, including ones from downstream jobs:
https://dl.dropboxusercontent.com/u/231152/spark.png
The only source change I made was changing "toDF" to "toDS()" before handing my
RDDs to the ML libraries.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
