[
https://issues.apache.org/jira/browse/SPARK-15122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Davies Liu resolved SPARK-15122.
--------------------------------
Resolution: Fixed
Fix Version/s: 2.0.0
Issue resolved by pull request 12954
[https://github.com/apache/spark/pull/12954]
> TPC-DS Qury 41 fails with The correlated scalar subquery can only contain
> equality predicates
> ---------------------------------------------------------------------------------------------
>
> Key: SPARK-15122
> URL: https://issues.apache.org/jira/browse/SPARK-15122
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.0.0
> Reporter: JESSE CHEN
> Priority: Critical
> Fix For: 2.0.0
>
>
> The official TPC-DS query 41 fails with the following error:
> {noformat}
> Error in query: The correlated scalar subquery can only contain equality
> predicates: (((i_manufact#38 = i_manufact#16) && (((((i_category#36 = Women)
> && ((i_color#41 = powder) || (i_color#41 = khaki))) && (((i_units#42 = Ounce)
> || (i_units#42 = Oz)) && ((i_size#39 = medium) || (i_size#39 = extra
> large)))) || (((i_category#36 = Women) && ((i_color#41 = brown) ||
> (i_color#41 = honeydew))) && (((i_units#42 = Bunch) || (i_units#42 = Ton)) &&
> ((i_size#39 = N/A) || (i_size#39 = small))))) || ((((i_category#36 = Men) &&
> ((i_color#41 = floral) || (i_color#41 = deep))) && (((i_units#42 = N/A) ||
> (i_units#42 = Dozen)) && ((i_size#39 = petite) || (i_size#39 = large)))) ||
> (((i_category#36 = Men) && ((i_color#41 = light) || (i_color#41 =
> cornflower))) && (((i_units#42 = Box) || (i_units#42 = Pound)) && ((i_size#39
> = medium) || (i_size#39 = extra large))))))) || ((i_manufact#38 =
> i_manufact#16) && (((((i_category#36 = Women) && ((i_color#41 = midnight) ||
> (i_color#41 = snow))) && (((i_units#42 = Pallet) || (i_units#42 = Gross)) &&
> ((i_size#39 = medium) || (i_size#39 = extra large)))) || (((i_category#36 =
> Women) && ((i_color#41 = cyan) || (i_color#41 = papaya))) && (((i_units#42 =
> Cup) || (i_units#42 = Dram)) && ((i_size#39 = N/A) || (i_size#39 = small)))))
> || ((((i_category#36 = Men) && ((i_color#41 = orange) || (i_color#41 =
> frosted))) && (((i_units#42 = Each) || (i_units#42 = Tbl)) && ((i_size#39 =
> petite) || (i_size#39 = large)))) || (((i_category#36 = Men) && ((i_color#41
> = forest) || (i_color#41 = ghost))) && (((i_units#42 = Lb) || (i_units#42 =
> Bundle)) && ((i_size#39 = medium) || (i_size#39 = extra large))))))));
> {noformat}
> The output plans showed the following errors
> {noformat}
> == Parsed Logical Plan ==
> 'GlobalLimit 100
> +- 'LocalLimit 100
> +- 'Sort ['i_product_name ASC], true
> +- 'Distinct
> +- 'Project ['i_product_name]
> +- 'Filter ((('i_manufact_id >= 738) && ('i_manufact_id <= (738 +
> 40))) && (scalar-subquery#1 [] > 0))
> : +- 'SubqueryAlias scalar-subquery#1 []
> : +- 'Project ['count(1) AS item_cnt#0]
> : +- 'Filter ((('i_manufact = 'i1.i_manufact) &&
> ((((('i_category = Women) && (('i_color = powder) || ('i_color = khaki))) &&
> ((('i_units = Ounce) || ('i_units = Oz)) && (('i_size = medium) || ('i_size =
> extra large)))) || ((('i_category = Women) && (('i_color = brown) ||
> ('i_color = honeydew))) && ((('i_units = Bunch) || ('i_units = Ton)) &&
> (('i_size = N/A) || ('i_size = small))))) || (((('i_category = Men) &&
> (('i_color = floral) || ('i_color = deep))) && ((('i_units = N/A) ||
> ('i_units = Dozen)) && (('i_size = petite) || ('i_size = large)))) ||
> ((('i_category = Men) && (('i_color = light) || ('i_color = cornflower))) &&
> ((('i_units = Box) || ('i_units = Pound)) && (('i_size = medium) || ('i_size
> = extra large))))))) || (('i_manufact = 'i1.i_manufact) && ((((('i_category =
> Women) && (('i_color = midnight) || ('i_color = snow))) && ((('i_units =
> Pallet) || ('i_units = Gross)) && (('i_size = medium) || ('i_size = extra
> large)))) || ((('i_category = Women) && (('i_color = cyan) || ('i_color =
> papaya))) && ((('i_units = Cup) || ('i_units = Dram)) && (('i_size = N/A) ||
> ('i_size = small))))) || (((('i_category = Men) && (('i_color = orange) ||
> ('i_color = frosted))) && ((('i_units = Each) || ('i_units = Tbl)) &&
> (('i_size = petite) || ('i_size = large)))) || ((('i_category = Men) &&
> (('i_color = forest) || ('i_color = ghost))) && ((('i_units = Lb) ||
> ('i_units = Bundle)) && (('i_size = medium) || ('i_size = extra large))))))))
> : +- 'UnresolvedRelation `item`, None
> +- 'UnresolvedRelation `item`, Some(i1)
> == Analyzed Logical Plan ==
> i_product_name: string
> GlobalLimit 100
> +- LocalLimit 100
> +- Sort [i_product_name#24 ASC], true
> +- Distinct
> +- Project [i_product_name#24]
> +- Filter (((i_manufact_id#16L >= cast(738 as bigint)) &&
> (i_manufact_id#16L <= cast((738 + 40) as bigint))) && (scalar-subquery#1
> [(((i_manufact#39 = i_manufact#17) && (((((i_category#37 = Women) &&
> ((i_color#42 = powder) || (i_color#42 = khaki))) && (((i_units#43 = Ounce) ||
> (i_units#43 = Oz)) && ((i_size#40 = medium) || (i_size#40 = extra large))))
> || (((i_category#37 = Women) && ((i_color#42 = brown) || (i_color#42 =
> honeydew))) && (((i_units#43 = Bunch) || (i_units#43 = Ton)) && ((i_size#40 =
> N/A) || (i_size#40 = small))))) || ((((i_category#37 = Men) && ((i_color#42 =
> floral) || (i_color#42 = deep))) && (((i_units#43 = N/A) || (i_units#43 =
> Dozen)) && ((i_size#40 = petite) || (i_size#40 = large)))) ||
> (((i_category#37 = Men) && ((i_color#42 = light) || (i_color#42 =
> cornflower))) && (((i_units#43 = Box) || (i_units#43 = Pound)) && ((i_size#40
> = medium) || (i_size#40 = extra large))))))) || ((i_manufact#39 =
> i_manufact#17) && (((((i_category#37 = Women) && ((i_color#42 = midnight) ||
> (i_color#42 = snow))) && (((i_units#43 = Pallet) || (i_units#43 = Gross)) &&
> ((i_size#40 = medium) || (i_size#40 = extra large)))) || (((i_category#37 =
> Women) && ((i_color#42 = cyan) || (i_color#42 = papaya))) && (((i_units#43 =
> Cup) || (i_units#43 = Dram)) && ((i_size#40 = N/A) || (i_size#40 = small)))))
> || ((((i_category#37 = Men) && ((i_color#42 = orange) || (i_color#42 =
> frosted))) && (((i_units#43 = Each) || (i_units#43 = Tbl)) && ((i_size#40 =
> petite) || (i_size#40 = large)))) || (((i_category#37 = Men) && ((i_color#42
> = forest) || (i_color#42 = ghost))) && (((i_units#43 = Lb) || (i_units#43 =
> Bundle)) && ((i_size#40 = medium) || (i_size#40 = extra large))))))))] >
> cast(0 as bigint)))
> : +- SubqueryAlias scalar-subquery#1 [(((i_manufact#39 =
> i_manufact#17) && (((((i_category#37 = Women) && ((i_color#42 = powder) ||
> (i_color#42 = khaki))) && (((i_units#43 = Ounce) || (i_units#43 = Oz)) &&
> ((i_size#40 = medium) || (i_size#40 = extra large)))) || (((i_category#37 =
> Women) && ((i_color#42 = brown) || (i_color#42 = honeydew))) && (((i_units#43
> = Bunch) || (i_units#43 = Ton)) && ((i_size#40 = N/A) || (i_size#40 =
> small))))) || ((((i_category#37 = Men) && ((i_color#42 = floral) ||
> (i_color#42 = deep))) && (((i_units#43 = N/A) || (i_units#43 = Dozen)) &&
> ((i_size#40 = petite) || (i_size#40 = large)))) || (((i_category#37 = Men) &&
> ((i_color#42 = light) || (i_color#42 = cornflower))) && (((i_units#43 = Box)
> || (i_units#43 = Pound)) && ((i_size#40 = medium) || (i_size#40 = extra
> large))))))) || ((i_manufact#39 = i_manufact#17) && (((((i_category#37 =
> Women) && ((i_color#42 = midnight) || (i_color#42 = snow))) && (((i_units#43
> = Pallet) || (i_units#43 = Gross)) && ((i_size#40 = medium) || (i_size#40 =
> extra large)))) || (((i_category#37 = Women) && ((i_color#42 = cyan) ||
> (i_color#42 = papaya))) && (((i_units#43 = Cup) || (i_units#43 = Dram)) &&
> ((i_size#40 = N/A) || (i_size#40 = small))))) || ((((i_category#37 = Men) &&
> ((i_color#42 = orange) || (i_color#42 = frosted))) && (((i_units#43 = Each)
> || (i_units#43 = Tbl)) && ((i_size#40 = petite) || (i_size#40 = large)))) ||
> (((i_category#37 = Men) && ((i_color#42 = forest) || (i_color#42 = ghost)))
> && (((i_units#43 = Lb) || (i_units#43 = Bundle)) && ((i_size#40 = medium) ||
> (i_size#40 = extra large))))))))]
> : +- Aggregate
> [i_manufact#39,i_category#37,i_size#40,i_units#43,i_color#42],
> [(count(1),mode=Complete,isDistinct=false) AS
> item_cnt#0L,i_manufact#39,i_category#37,i_size#40,i_units#43,i_color#42]
> : +- MetastoreRelation hadoopds1g, item, None
> +- SubqueryAlias i1
> +-
> Relation[i_item_sk#3,i_item_id#4,i_rec_start_date#5,i_rec_end_date#6,i_item_desc#7,i_current_price#8,i_wholesale_cost#9,i_brand_id#10L,i_brand#11,i_class_id#12L,i_class#13,i_category_id#14L,i_category#15,i_manufact_id#16L,i_manufact#17,i_size#18,i_formulation#19,i_color#20,i_units#21,i_container#22,i_manager_id#23L,i_product_name#24]
> HadoopFiles
> {noformat}
> Note that the q41 in
> https://github.com/sameeragarwal/spark/blob/tpcds-more/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/tpcds/TPCDS_1_4_Benchmark.scala
> is NOT the official TPC-DS query nor does it have allowed "minor query
> modification". It works in the nightly build. But we cannot claim it is a
> TPC-DS query.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
