ahshahid commented on PR #58003: URL: https://github.com/apache/spark/pull/58003#issuecomment-5431560927
Thank you Xiduo You, for review. Will implement the feedback and update pr in a day.. Regards Asif A democracy of barbarians is worse than A dictatorship On Wed, Aug 26, 2026, 1:15 AM Xiduo You ***@***.***> wrote: > ***@***.**** commented on this pull request. > > thank you @ahshahid <https://github.com/ahshahid> , this fix looks > correct to me, some comments. > ------------------------------ > > In > sql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/TableCacheQueryStageExecSuite.scala > <https://github.com/apache/spark/pull/58003#discussion_r3860652338>: > > > + > +class TableCacheQueryStageExecSuite extends SharedSparkSession with AdaptiveSparkPlanHelper { > + import testImplicits._ > + > + test("SPARK-58767: re-use of exchange involving cached plans ") { > + withTempPaths(3) { paths => > + // to be safe use 3 data paths so that the 3 tables are not in any situation canonicalized > + // same. > + val data = Seq((1, 1), (2, 2), (3, 3), (4, 4), (6, 6), (5, 5)) > + paths.zipWithIndex.foreach{case(path, indx) => > + val tableNumber = indx + 1 > + spark.createDataFrame(data).toDF(s"c1_$tableNumber", s"c2_$tableNumber").write.mode( > + SaveMode.Overwrite).parquet(path.getAbsolutePath)} > + withTempView("v1") { > + withTable("t1", "t2", "t3") { > + spark.sql( > > please wrap with withCache() > ------------------------------ > > In > sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/QueryStageExec.scala > <https://github.com/apache/spark/pull/58003#discussion_r3860667820>: > > > @@ -307,6 +308,8 @@ case class TableCacheQueryStageExec( > } > } > > + override def doCanonicalize(): SparkPlan = _canonicalized > > can we move the following code from ExchangeQueryStageExec to > QueryStageExec ? > > val _canonicalized: SparkPlan > override def doCanonicalize(): SparkPlan = _canonicalized > > ------------------------------ > > In > sql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/TableCacheQueryStageExecSuite.scala > <https://github.com/apache/spark/pull/58003#discussion_r3860813039>: > > > + ) > + > + spark.table("t1").where($"c1_1" > 0).createTempView("v1") > + spark.catalog.cacheTable("v1") > + val al1 = spark.table("v1").as("al1") > + val al2 = spark.table("v1").as("al2") > + val t2 = spark.table("t2") > + val t3 = spark.table("t3") > + val j1 = t2.where($"c2_2" > 0).join(t3, t2.col("c1_2") === t3.col("c1_3")) > + assert(j1.collect().nonEmpty) > + val j2 = j1.join( > + broadcast(al1.where($"al1.c1_1" > 0)), $"al1.c1_1" === t2.col("c1_2")) > + assert(j2.collect().nonEmpty) > + val df = j2 > + .join(broadcast(al2.where($"al2.c1_1" > 0)), $"al2.c1_1" === t3.col("c1_3")) > + assert(df.collect().nonEmpty) > > can we use checkAnswer for more accurate result ? > > — > Reply to this email directly, view it on GitHub > <https://github.com/apache/spark/pull/58003?email_source=notifications&email_token=AC6XG2HEG62R4XTDESXQJ5D5L2MAJA5CNFSNUABKM5UWIORPF5TWS5BNNB2WEL2QOVWGYUTFOF2WK43UKJSXM2LFO4XTKMBSG44TENBWGQZKM4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJLDGN5XXIZLSL5RWY2LDNM#pullrequestreview-5027924642>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AC6XG2CBSG5XWO7VBBUNX6T5L2MAJAVCNFSNUABEKJSXA33TNF2G64TZHMYTOMJWGU3DKOB3JFZXG5LFHM2TCNBYGEZDSOBZHCQXMAQ> > . > Triage notifications, keep track of coding agent tasks and review pull > requests on the go with GitHub Mobile for iOS > <https://github.com/notifications/mobile/ios/AC6XG2BTKOGU5XEBKYLYTVT5L2MAJA5CNFSNUABKM5UWIORPF5TWS5BNNB2WEL2QOVWGYUTFOF2WK43UKJSXM2LFO4XTKMBSG44TENBWGQZKM4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJKTGN5XXIZLSL5UW64Y> > and Android > <https://github.com/notifications/mobile/android/AC6XG2FGPGHAB7MD76XNDKT5L2MAJA5CNFSNUABKM5UWIORPF5TWS5BNNB2WEL2QOVWGYUTFOF2WK43UKJSXM2LFO4XTKMBSG44TENBWGQZKM4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJLTGN5XXIZLSL5QW4ZDSN5UWI>. > Download it today! > You are receiving this because you were mentioned.Message ID: > ***@***.***> > -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
