[GitHub] [spark] amanomer commented on a change in pull request #26811: [SPARK-29600][SQL] ArrayContains function may return incorrect result for DecimalType
amanomer commented on a change in pull request #26811: [SPARK-29600][SQL] ArrayContains function may return incorrect result for DecimalType URL: https://github.com/apache/spark/pull/26811#discussion_r358660693 ## File path: sql/core/src/test/scala/org/apache/spark/sql/DataFrameFunctionsSuite.scala ## @@ -850,7 +850,7 @@ class DataFrameFunctionsSuite extends QueryTest with SharedSparkSession { val errorMsg1 = s""" |Input to function array_contains should have been array followed by a - |value with same element type, but it's [array, decimal(29,29)]. + |value with same element type, but it's [array, decimal(38,29)]. Review comment: Above code is creating new expression by updating only right child. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] amanomer commented on a change in pull request #26811: [SPARK-29600][SQL] ArrayContains function may return incorrect result for DecimalType
amanomer commented on a change in pull request #26811: [SPARK-29600][SQL] ArrayContains function may return incorrect result for DecimalType URL: https://github.com/apache/spark/pull/26811#discussion_r358659882 ## File path: sql/core/src/test/scala/org/apache/spark/sql/DataFrameFunctionsSuite.scala ## @@ -850,7 +850,7 @@ class DataFrameFunctionsSuite extends QueryTest with SharedSparkSession { val errorMsg1 = s""" |Input to function array_contains should have been array followed by a - |value with same element type, but it's [array, decimal(29,29)]. + |value with same element type, but it's [array, decimal(38,29)]. Review comment: https://github.com/apache/spark/blob/1fc353d51a62cb554e6af23dbc9a613e214e3af1/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala#L864-L869 This code is to cast left and right expression one by one. Here, - `e.childern` is `Seq( array, decimal(29,29))`, and - `e.inputTypes` will return `Seq(array, decimal(38,29))` `impicitCast(array, array)` will return `None`, since `int` can't be casted to `decimal(38,29)`. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] amanomer commented on a change in pull request #26811: [SPARK-29600][SQL] ArrayContains function may return incorrect result for DecimalType
amanomer commented on a change in pull request #26811: [SPARK-29600][SQL] ArrayContains function may return incorrect result for DecimalType URL: https://github.com/apache/spark/pull/26811#discussion_r358618026 ## File path: sql/core/src/test/scala/org/apache/spark/sql/DataFrameFunctionsSuite.scala ## @@ -850,7 +850,7 @@ class DataFrameFunctionsSuite extends QueryTest with SharedSparkSession { val errorMsg1 = s""" |Input to function array_contains should have been array followed by a - |value with same element type, but it's [array, decimal(29,29)]. + |value with same element type, but it's [array, decimal(38,29)]. Review comment: cc @maropu @cloud-fan This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] amanomer commented on a change in pull request #26811: [SPARK-29600][SQL] ArrayContains function may return incorrect result for DecimalType
amanomer commented on a change in pull request #26811: [SPARK-29600][SQL] ArrayContains function may return incorrect result for DecimalType URL: https://github.com/apache/spark/pull/26811#discussion_r358076319 ## File path: sql/core/src/test/scala/org/apache/spark/sql/DataFrameFunctionsSuite.scala ## @@ -850,7 +850,7 @@ class DataFrameFunctionsSuite extends QueryTest with SharedSparkSession { val errorMsg1 = s""" |Input to function array_contains should have been array followed by a - |value with same element type, but it's [array, decimal(29,29)]. + |value with same element type, but it's [array, decimal(38,29)]. Review comment: cc @cloud-fan @maropu This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] amanomer commented on a change in pull request #26811: [SPARK-29600][SQL] ArrayContains function may return incorrect result for DecimalType
amanomer commented on a change in pull request #26811: [SPARK-29600][SQL] ArrayContains function may return incorrect result for DecimalType URL: https://github.com/apache/spark/pull/26811#discussion_r358617660 ## File path: sql/core/src/test/scala/org/apache/spark/sql/DataFrameFunctionsSuite.scala ## @@ -850,7 +850,7 @@ class DataFrameFunctionsSuite extends QueryTest with SharedSparkSession { val errorMsg1 = s""" |Input to function array_contains should have been array followed by a - |value with same element type, but it's [array, decimal(29,29)]. + |value with same element type, but it's [array, decimal(38,29)]. Review comment: Do you mean why in above test case query, `ArrayContains` is throwing `AnalysisException` instead of casting integer to Decimal? An integer cannot be casted to decimal with scale > 28. ``` decimalWith28Zeroes = 1. SELECT array_contains(array(1), decimalWith28Zeroes); Result =>> true ``` ``` decimalWith29Zeroes = 1.0 SELECT array_contains(array(1), decimalWith29Zeroes); Result =>> AnalysisException ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] amanomer commented on a change in pull request #26811: [SPARK-29600][SQL] ArrayContains function may return incorrect result for DecimalType
amanomer commented on a change in pull request #26811: [SPARK-29600][SQL] ArrayContains function may return incorrect result for DecimalType URL: https://github.com/apache/spark/pull/26811#discussion_r358076319 ## File path: sql/core/src/test/scala/org/apache/spark/sql/DataFrameFunctionsSuite.scala ## @@ -850,7 +850,7 @@ class DataFrameFunctionsSuite extends QueryTest with SharedSparkSession { val errorMsg1 = s""" |Input to function array_contains should have been array followed by a - |value with same element type, but it's [array, decimal(29,29)]. + |value with same element type, but it's [array, decimal(38,29)]. Review comment: cc @cloud-fan @maropu This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] amanomer commented on a change in pull request #26811: [SPARK-29600][SQL] ArrayContains function may return incorrect result for DecimalType
amanomer commented on a change in pull request #26811: [SPARK-29600][SQL] ArrayContains function may return incorrect result for DecimalType URL: https://github.com/apache/spark/pull/26811#discussion_r358076319 ## File path: sql/core/src/test/scala/org/apache/spark/sql/DataFrameFunctionsSuite.scala ## @@ -850,7 +850,7 @@ class DataFrameFunctionsSuite extends QueryTest with SharedSparkSession { val errorMsg1 = s""" |Input to function array_contains should have been array followed by a - |value with same element type, but it's [array, decimal(29,29)]. + |value with same element type, but it's [array, decimal(38,29)]. Review comment: cc @cloud-fan This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] amanomer commented on a change in pull request #26811: [SPARK-29600][SQL] ArrayContains function may return incorrect result for DecimalType
amanomer commented on a change in pull request #26811: [SPARK-29600][SQL] ArrayContains function may return incorrect result for DecimalType URL: https://github.com/apache/spark/pull/26811#discussion_r358076174 ## File path: sql/core/src/test/scala/org/apache/spark/sql/DataFrameFunctionsSuite.scala ## @@ -850,7 +850,7 @@ class DataFrameFunctionsSuite extends QueryTest with SharedSparkSession { val errorMsg1 = s""" |Input to function array_contains should have been array followed by a - |value with same element type, but it's [array, decimal(29,29)]. + |value with same element type, but it's [array, decimal(38,29)]. Review comment: https://github.com/apache/spark/blob/1fc353d51a62cb554e6af23dbc9a613e214e3af1/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala#L864-L869 For query `array_contains(array(1), .01234567890123456790123456780)`\ `e.inputTypes` will return `Seq(Array(Decimal(38,29)), Decimal(38,29))` and above code will cast `.01234567890123456790123456780` as `Decimal(38,29)`. Previously, when we were using `findWiderTypeForTwo`, decimal types were not getting upcasted but `findWiderTypeWithoutStringPromotionForTwo` will successfully upcast DecimalType This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org