Thanks for looking into it, Kazuaki! Pozdrawiam, Jacek Laskowski ---- https://about.me/JacekLaskowski Mastering Spark SQL https://bit.ly/mastering-spark-sql Spark Structured Streaming https://bit.ly/spark-structured-streaming Mastering Kafka Streams https://bit.ly/mastering-kafka-streams Follow me at https://twitter.com/jaceklaskowski
On Tue, Jan 2, 2018 at 4:27 AM, Kazuaki Ishizaki <ishiz...@jp.ibm.com> wrote: > Thank you for your correction :) > > I also made mistake in a report. What I reported at first never occurs > with the correct Java bean class. > Finally, I can reproduce a problem that Jacek reported even using the > master. In my environment, this problem occurs with or without whole-stage > codegen. I updated the JIRA ticket. > > I am still working for this. > > Kazuaki Ishizaki > > > > From: Herman van Hövell tot Westerflier <hvanhov...@databricks.com> > To: Kazuaki Ishizaki <ishiz...@jp.ibm.com> > Cc: Jacek Laskowski <ja...@japila.pl>, dev <dev@spark.apache.org> > Date: 2018/01/02 04:12 > > Subject: Re: Whole-stage codegen and SparkPlan.newPredicate > ------------------------------ > > > > Wrong ticket: *https://issues.apache.org/jira/browse/SPARK-22935* > <https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_SPARK-2D22935&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=b70dG_9wpCdZSkBJahHYQ4IwKMdp2hQM29f-ZCGj9Pg&m=VnpMA7cb_gBkvZ_GtGqLiS5Y6LRDg3d712mPbbbMbwI&s=xt733QXPlrceLDjgRGDVGiaFQafyzAGeE-eGTe-JxpU&e=> > > Thanks for working on this :) > > On Mon, Jan 1, 2018 at 2:22 PM, Kazuaki Ishizaki <*ishiz...@jp.ibm.com* > <ishiz...@jp.ibm.com>> wrote: > I ran the program in URL of stackoverflow with Spark 2.2.1 and master. I > cannot see the exception even when I disabled whole-stage codegen. Am I > wrong? > We would appreciate it if you could create a JIRA entry with simple > standalone repro. > > In addition to this report, I realized that this program produces > incorrect results. I created a JIRA entry > *https://issues.apache.org/jira/browse/SPARK-22934* > <https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_SPARK-2D22934&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=b70dG_9wpCdZSkBJahHYQ4IwKMdp2hQM29f-ZCGj9Pg&m=VnpMA7cb_gBkvZ_GtGqLiS5Y6LRDg3d712mPbbbMbwI&s=7cZZHNoJBbHY-tjrE1ut71cOjCyA82CwclHbWG2znno&e=> > . > > Best Regards, > Kazuaki Ishizaki > > > > From: Herman van Hövell tot Westerflier < > *hvanhov...@databricks.com* <hvanhov...@databricks.com>> > To: Jacek Laskowski <*ja...@japila.pl* <ja...@japila.pl>> > Cc: dev <*dev@spark.apache.org* <dev@spark.apache.org>> > Date: 2017/12/31 21:44 > Subject: Re: Whole-stage codegen and SparkPlan.newPredicate > ------------------------------ > > > > Hi Jacek, > > In this case whole stage code generation is turned off. However we still > use code generation for a lot of other things: projections, predicates, > orderings & encoders. You are currently seeing a compile time failure while > generating a predicate. There is currently no easy way to turn code > generation off entirely. > > The error itself is not great, but it still captures the problem in a > relatively timely fashion. We should have caught this during analysis > though. Can you file a ticket? > > - Herman > > On Sat, Dec 30, 2017 at 9:16 AM, Jacek Laskowski <*ja...@japila.pl* > <ja...@japila.pl>> wrote: > Hi, > > While working on an issue with Whole-stage codegen as reported @ > *https://stackoverflow.com/q/48026060/1305344* > <https://urldefense.proofpoint.com/v2/url?u=https-3A__stackoverflow.com_q_48026060_1305344&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=b70dG_9wpCdZSkBJahHYQ4IwKMdp2hQM29f-ZCGj9Pg&m=LNc_s5Vc87PNQAWCeE9iuJVzWzNEBkgolWvuze48L7k&s=RvT_q8DEXf0WcHO5PmpYKTonkiLk_CRCFDWIwHR7b1o&e=>I > found out that spark.sql.codegen.wholeStage=false does *not* turn > whole-stage codegen off completely. > > > It looks like SparkPlan.newPredicate [1] gets called regardless of the > value of spark.sql.codegen.wholeStage property. > > $ ./bin/spark-shell --conf spark.sql.codegen.wholeStage=false > ... > scala> spark.sessionState.conf.wholeStageEnabled > res7: Boolean = false > > That leads to an issue in the SO question with whole-stage codegen > regardless of the value: > > ... > at org.apache.spark.sql.execution.SparkPlan. > newPredicate(SparkPlan.scala:385) > at org.apache.spark.sql.execution.FilterExec$$anonfun$18.apply( > basicPhysicalOperators.scala:214) > at org.apache.spark.sql.execution.FilterExec$$anonfun$18.apply( > basicPhysicalOperators.scala:213) > at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsWithIndexInternal > $1$$anonfun$apply$24.apply(RDD.scala:816) > ... > > Is this a bug or does it work as intended? Why? > > [1] > *https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/SparkPlan.scala?utf8=%E2%9C%93#L386* > <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_spark_blob_master_sql_core_src_main_scala_org_apache_spark_sql_execution_SparkPlan.scala-3Futf8-3D-25E2-259C-2593-23L386&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=b70dG_9wpCdZSkBJahHYQ4IwKMdp2hQM29f-ZCGj9Pg&m=LNc_s5Vc87PNQAWCeE9iuJVzWzNEBkgolWvuze48L7k&s=vHxnoCNIUEN3ubKGZGGsWbkAxPDM5sbLewYKVLgwDY8&e=> > > Pozdrawiam, > Jacek Laskowski > ---- > *https://about.me/JacekLaskowski* > <https://urldefense.proofpoint.com/v2/url?u=https-3A__about.me_JacekLaskowski&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=b70dG_9wpCdZSkBJahHYQ4IwKMdp2hQM29f-ZCGj9Pg&m=LNc_s5Vc87PNQAWCeE9iuJVzWzNEBkgolWvuze48L7k&s=LH71LLLzVggGx5f1T9hE7BVortdTN6qh-Ji3OQGsfMY&e=> > Mastering Spark SQL *https://bit.ly/mastering-spark-sql* > <https://urldefense.proofpoint.com/v2/url?u=https-3A__bit.ly_mastering-2Dspark-2Dsql&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=b70dG_9wpCdZSkBJahHYQ4IwKMdp2hQM29f-ZCGj9Pg&m=LNc_s5Vc87PNQAWCeE9iuJVzWzNEBkgolWvuze48L7k&s=MxnWpkT9RJKvNrpPAFnfceiOl14n7CJ0SiRzWZc9nRA&e=> > Spark Structured Streaming *https://bit.ly/spark-structured-streaming* > <https://urldefense.proofpoint.com/v2/url?u=https-3A__bit.ly_spark-2Dstructured-2Dstreaming&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=b70dG_9wpCdZSkBJahHYQ4IwKMdp2hQM29f-ZCGj9Pg&m=LNc_s5Vc87PNQAWCeE9iuJVzWzNEBkgolWvuze48L7k&s=ssYJfznmDoI8I3uEOZW2r9sKaKw1fEJiD_DU2mzPg24&e=> > Mastering Apache Spark 2 *https://bit.ly/mastering-apache-spark* > <https://urldefense.proofpoint.com/v2/url?u=https-3A__bit.ly_mastering-2Dapache-2Dspark&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=b70dG_9wpCdZSkBJahHYQ4IwKMdp2hQM29f-ZCGj9Pg&m=LNc_s5Vc87PNQAWCeE9iuJVzWzNEBkgolWvuze48L7k&s=ZVqCRb9jygZwP8pTDorfMLRtQSPmal3P_HdgvPJ6_Qo&e=> > Follow me at *https://twitter.com/jaceklaskowski* > <https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_jaceklaskowski&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=b70dG_9wpCdZSkBJahHYQ4IwKMdp2hQM29f-ZCGj9Pg&m=LNc_s5Vc87PNQAWCeE9iuJVzWzNEBkgolWvuze48L7k&s=hf9Zczq71Qh0vcGJs8iYL5mG_M6rB-a6IEXeGtaIkfI&e=> > > > > > > >
