Hi Maciej, FYI, this fix is submitted at https://github.com/apache/spark/pull/16785.
Liang-Chi Hsieh wrote > Hi Maciej, > > After looking into the details of the time spent on preparing the executed > plan, the cause of the significant difference between 1.6 and current > codebase when running the example, is the optimization process to generate > constraints. > > There seems few operations in generating constraints which are not > optimized. Plus the fact the query plan grows continuously, the time spent > on generating constraints increases more and more. > > I am trying to reduce the time cost. Although not as low as 1.6 because we > can't remove the process of generating constraints, it is significantly > lower than current codebase (74294 ms -> 2573 ms). > > 385 ms > 107 ms > 46 ms > 58 ms > 64 ms > 105 ms > 86 ms > 122 ms > 115 ms > 114 ms > 100 ms > 109 ms > 169 ms > 196 ms > 174 ms > 212 ms > 290 ms > 254 ms > 318 ms > 405 ms > 347 ms > 443 ms > 432 ms > 500 ms > 544 ms > 619 ms > 697 ms > 683 ms > 807 ms > 802 ms > 960 ms > 1010 ms > 1155 ms > 1251 ms > 1298 ms > 1388 ms > 1503 ms > 1613 ms > 2279 ms > 2349 ms > 2573 ms > > Liang-Chi Hsieh wrote >> Hi Maciej, >> >> Thanks for the info you provided. >> >> I tried to run the same example with 1.6 and current branch and record >> the difference between the time cost on preparing the executed plan. >> >> Current branch: >> >> 292 ms >> >> 95 ms >> 57 ms >> 34 ms >> 128 ms >> 120 ms >> 63 ms >> 106 ms >> 179 ms >> 159 ms >> 235 ms >> 260 ms >> 334 ms >> 464 ms >> 547 ms >> 719 ms >> 942 ms >> 1130 ms >> 1928 ms >> 1751 ms >> 2159 ms >> 2767 ms >> 3333 ms >> 4175 ms >> 5106 ms >> 6269 ms >> 7683 ms >> 9210 ms >> 10931 ms >> 13237 ms >> 15651 ms >> 19222 ms >> 23841 ms >> 26135 ms >> 31299 ms >> 38437 ms >> 47392 ms >> 51420 ms >> 60285 ms >> 69840 ms >> 74294 ms >> >> 1.6: >> >> 3 ms >> 4 ms >> 10 ms >> 4 ms >> 17 ms >> 8 ms >> 12 ms >> 21 ms >> 15 ms >> 15 ms >> 19 ms >> 23 ms >> 28 ms >> 28 ms >> 58 ms >> 39 ms >> 43 ms >> 61 ms >> 56 ms >> 60 ms >> 81 ms >> 73 ms >> 100 ms >> 91 ms >> 96 ms >> 116 ms >> 111 ms >> 140 ms >> 127 ms >> 142 ms >> 148 ms >> 165 ms >> 171 ms >> 198 ms >> 200 ms >> 233 ms >> 237 ms >> 253 ms >> 256 ms >> 271 ms >> 292 ms >> 452 ms >> >> Although they both take more time after each iteration due to the grown >> query plan, it is obvious that current branch takes much more time than >> 1.6 branch. The optimizer and query planning in current branch is much >> more complicated than 1.6. >> zero323 wrote >>> Hi Liang-Chi, >>> >>> Thank you for your answer and PR but what I think I wasn't specific >>> enough. In hindsight I should have illustrate this better. What really >>> troubles me here is a pattern of growing delays. Difference between >>> 1.6.3 (roughly 20s runtime since the first job): >>> >>> >>> 1.6 timeline >>> >>> vs 2.1.0 (45 minutes or so in a bad case): >>> >>> 2.1.0 timeline >>> >>> The code is just an example and it is intentionally dumb. You easily >>> mask this with caching, or using significantly larger data sets. So I >>> guess the question I am really interested in is - what changed between >>> 1.6.3 and 2.x (this is more or less consistent across 2.0, 2.1 and >>> current master) to cause this and more important, is it a feature or is >>> it a bug? I admit, I choose a lazy path here, and didn't spend much time >>> (yet) trying to dig deeper. >>> >>> I can see a bit higher memory usage, a bit more intensive GC activity, >>> but nothing I would really blame for this behavior, and duration of >>> individual jobs is comparable with some favor of 2.x. Neither >>> StringIndexer nor OneHotEncoder changed much in 2.x. They used RDDs for >>> fitting in 1.6 and, as far as I can tell, they still do that in 2.x. And >>> the problem doesn't look that related to the data processing part in the >>> first place. >>> >>> >>> On 02/02/2017 07:22 AM, Liang-Chi Hsieh wrote: >>>> Hi Maciej, >>>> >>>> FYI, the PR is at https://github.com/apache/spark/pull/16775. >>>> >>>> >>>> Liang-Chi Hsieh wrote >>>>> Hi Maciej, >>>>> >>>>> Basically the fitting algorithm in Pipeline is an iterative operation. >>>>> Running iterative algorithm on Dataset would have RDD lineages and >>>>> query >>>>> plans that grow fast. Without cache and checkpoint, it gets slower >>>>> when >>>>> the iteration number increases. >>>>> >>>>> I think it is why when you run a Pipeline with long stages, it gets >>>>> much >>>>> longer time to finish. As I think it is not uncommon to have long >>>>> stages >>>>> in a Pipeline, we should improve this. I will submit a PR for this. >>>>> zero323 wrote >>>>>> Hi everyone, >>>>>> >>>>>> While experimenting with ML pipelines I experience a significant >>>>>> performance regression when switching from 1.6.x to 2.x. >>>>>> >>>>>> import org.apache.spark.ml.{Pipeline, PipelineStage} >>>>>> import org.apache.spark.ml.feature.{OneHotEncoder, StringIndexer, >>>>>> VectorAssembler} >>>>>> >>>>>> val df = (1 to 40).foldLeft(Seq((1, "foo"), (2, "bar"), (3, >>>>>> "baz")).toDF("id", "x0"))((df, i) => df.withColumn(s"x$i", $"x0")) >>>>>> val indexers = df.columns.tail.map(c => new StringIndexer() >>>>>> .setInputCol(c) >>>>>> .setOutputCol(s"${c}_indexed") >>>>>> .setHandleInvalid("skip")) >>>>>> >>>>>> val encoders = indexers.map(indexer => new OneHotEncoder() >>>>>> .setInputCol(indexer.getOutputCol) >>>>>> .setOutputCol(s"${indexer.getOutputCol}_encoded") >>>>>> .setDropLast(true)) >>>>>> >>>>>> val assembler = new >>>>>> VectorAssembler().setInputCols(encoders.map(_.getOutputCol)) >>>>>> val stages: Array[PipelineStage] = indexers ++ encoders :+ assembler >>>>>> >>>>>> new Pipeline().setStages(stages).fit(df).transform(df).show >>>>>> >>>>>> Task execution time is comparable and executors are most of the time >>>>>> idle so it looks like it is a problem with the optimizer. Is it a >>>>>> known >>>>>> issue? Are there any changes I've missed, that could lead to this >>>>>> behavior? >>>>>> >>>>>> -- >>>>>> Best, >>>>>> Maciej >>>>>> >>>>>> >>>>>> --------------------------------------------------------------------- >>>>>> To unsubscribe e-mail: >>>>>> dev-unsubscribe@.apache >>>> >>>> >>>> >>>> >>>> ----- >>>> Liang-Chi Hsieh | @viirya >>>> Spark Technology Center >>>> http://www.spark.tc/ >>>> -- >>>> View this message in context: >>>> http://apache-spark-developers-list.1001551.n3.nabble.com/SQL-ML-Pipeline-performance-regression-between-1-6-and-2-x-tp20803p20822.html >>>> Sent from the Apache Spark Developers List mailing list archive at >>>> Nabble.com. >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe e-mail: >>> dev-unsubscribe@.apache >>>> >>> >>> -- >>> Maciej Szymkiewicz >>> >>> >>> >>> nM15AWH.png (19K) >>> <http://apache-spark-developers-list.1001551.n3.nabble.com/attachment/20827/0/nM15AWH.png> >>> KHZa7hL.png (26K) >>> <http://apache-spark-developers-list.1001551.n3.nabble.com/attachment/20827/1/KHZa7hL.png> ----- Liang-Chi Hsieh | @viirya Spark Technology Center http://www.spark.tc/ -- View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/SQL-ML-Pipeline-performance-regression-between-1-6-and-2-x-tp20803p20837.html Sent from the Apache Spark Developers List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
