[jira] [Updated] (SPARK-14580) HiveTypeCoercion.IfCoercion should preserve original predicates.
[ https://issues.apache.org/jira/browse/SPARK-14580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-14580: -- Description: Currently, `HiveTypeCoercion.IfCoercion` removes all predicates whose return-type is null. However, some UDFs need evaluations because they are designed to throw exceptions. *Hive* {code} hive> select if(assert_true(false),2,3); OK Failed with exception java.io.IOException:org.apache.hadoop.hive.ql.metadata.HiveException: ASSERT_TRUE(): assertion failed. {code} *Spark* {code} scala> sql("select if(assert_true(false),2,3)").head res2: org.apache.spark.sql.Row = [3] {code} `IfCoercion` works like the followings. {code} === Applying Rule org.apache.spark.sql.catalyst.analysis.HiveTypeCoercion$IfCoercion === !'Project [unresolvedalias(if (HiveGenericUDF#org.apache.hadoop.hive.ql.udf.generic.GenericUDFAssertTrue(false)) 2 else 3)] 'Project [unresolvedalias(if (nu ll) 2 else 3)] +- OneRowRelation$ +- OneRowRelation$ {code} This issue aims to fix this and to add `assert_true` as a Spark SQL function. was: Currently, `HiveTypeCoercion.IfCoercion` removes all predicates whose return-type is null. However, some UDFs need evaluations because they are designed to throw exceptions. *Hive* {code} hive> select if(assert_true(false),2,3); OK Failed with exception java.io.IOException:org.apache.hadoop.hive.ql.metadata.HiveException: ASSERT_TRUE(): assertion failed. {code} *Spark* {code} scala> sql("select if(assert_true(false),2,3)").head res2: org.apache.spark.sql.Row = [3] {code} `IfCoercion` works like the followings. {code} === Applying Rule org.apache.spark.sql.catalyst.analysis.HiveTypeCoercion$IfCoercion === !'Project [unresolvedalias(if (HiveGenericUDF#org.apache.hadoop.hive.ql.udf.generic.GenericUDFAssertTrue(false)) 2 else 3)] 'Project [unresolvedalias(if (nu ll) 2 else 3)] +- OneRowRelation$ +- OneRowRelation$ {code} This issue aims to fix this. > HiveTypeCoercion.IfCoercion should preserve original predicates. > > > Key: SPARK-14580 > URL: https://issues.apache.org/jira/browse/SPARK-14580 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 1.6.1, 2.0.0 >Reporter: Dongjoon Hyun > > Currently, `HiveTypeCoercion.IfCoercion` removes all predicates whose > return-type is null. However, some UDFs need evaluations because they are > designed to throw exceptions. > *Hive* > {code} > hive> select if(assert_true(false),2,3); > OK > Failed with exception > java.io.IOException:org.apache.hadoop.hive.ql.metadata.HiveException: > ASSERT_TRUE(): assertion failed. > {code} > > *Spark* > {code} > scala> sql("select if(assert_true(false),2,3)").head > res2: org.apache.spark.sql.Row = [3] > {code} > `IfCoercion` works like the followings. > {code} > === Applying Rule > org.apache.spark.sql.catalyst.analysis.HiveTypeCoercion$IfCoercion === > !'Project [unresolvedalias(if > (HiveGenericUDF#org.apache.hadoop.hive.ql.udf.generic.GenericUDFAssertTrue(false)) > 2 else 3)] 'Project [unresolvedalias(if (nu > ll) 2 else 3)] > +- OneRowRelation$ > +- OneRowRelation$ > {code} > This issue aims to fix this and to add `assert_true` as a Spark SQL function. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-14580) HiveTypeCoercion.IfCoercion should preserve original predicates.
[ https://issues.apache.org/jira/browse/SPARK-14580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-14580: -- Description: Currently, `HiveTypeCoercion.IfCoercion` removes all predicates whose return-type is null. However, some UDFs need evaluations because they are designed to throw exceptions. *Hive* {code} hive> select if(assert_true(false),2,3); OK Failed with exception java.io.IOException:org.apache.hadoop.hive.ql.metadata.HiveException: ASSERT_TRUE(): assertion failed. {code} *Spark* {code} scala> sql("select if(assert_true(false),2,3)").head res2: org.apache.spark.sql.Row = [3] {code} `IfCoercion` works like the followings. {code} === Applying Rule org.apache.spark.sql.catalyst.analysis.HiveTypeCoercion$IfCoercion === !'Project [unresolvedalias(if (HiveGenericUDF#org.apache.hadoop.hive.ql.udf.generic.GenericUDFAssertTrue(false)) 2 else 3)] 'Project [unresolvedalias(if (nu ll) 2 else 3)] +- OneRowRelation$ +- OneRowRelation$ {code} This issue aims to fix this. was: Currently, `HiveTypeCoercion.IfCoercion` removes all predicates whose return-type is null. However, some UDFs need evaluations because they are designed to throw exceptions. *Hive* {code} hive> select if(assert_true(false),2,3); OK Failed with exception java.io.IOException:org.apache.hadoop.hive.ql.metadata.HiveException: ASSERT_TRUE(): assertion failed. {code} *Spark* {code} scala> sql("select if(assert_true(false),2,3)").head res2: org.apache.spark.sql.Row = [3] {code} `IfCoercion` works like the followings. {code} === Applying Rule org.apache.spark.sql.catalyst.analysis.HiveTypeCoercion$IfCoercion === !'Project [unresolvedalias(if (HiveGenericUDF#org.apache.hadoop.hive.ql.udf.generic.GenericUDFAssertTrue(false)) 2 else 3)] 'Project [unresolvedalias(if (nu ll) 2 else 3)] +- OneRowRelation$ +- OneRowRelation$ {code} > HiveTypeCoercion.IfCoercion should preserve original predicates. > > > Key: SPARK-14580 > URL: https://issues.apache.org/jira/browse/SPARK-14580 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 1.6.1, 2.0.0 >Reporter: Dongjoon Hyun > > Currently, `HiveTypeCoercion.IfCoercion` removes all predicates whose > return-type is null. However, some UDFs need evaluations because they are > designed to throw exceptions. > *Hive* > {code} > hive> select if(assert_true(false),2,3); > OK > Failed with exception > java.io.IOException:org.apache.hadoop.hive.ql.metadata.HiveException: > ASSERT_TRUE(): assertion failed. > {code} > > *Spark* > {code} > scala> sql("select if(assert_true(false),2,3)").head > res2: org.apache.spark.sql.Row = [3] > {code} > `IfCoercion` works like the followings. > {code} > === Applying Rule > org.apache.spark.sql.catalyst.analysis.HiveTypeCoercion$IfCoercion === > !'Project [unresolvedalias(if > (HiveGenericUDF#org.apache.hadoop.hive.ql.udf.generic.GenericUDFAssertTrue(false)) > 2 else 3)] 'Project [unresolvedalias(if (nu > ll) 2 else 3)] > +- OneRowRelation$ > +- OneRowRelation$ > {code} > This issue aims to fix this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-14580) HiveTypeCoercion.IfCoercion should preserve original predicates.
[ https://issues.apache.org/jira/browse/SPARK-14580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-14580: -- Description: Currently, `HiveTypeCoercion.IfCoercion` removes all predicates whose return-type is null. However, some UDFs need evaluations because they are designed to throw exceptions. *Hive* {code} hive> select if(assert_true(false),2,3); OK Failed with exception java.io.IOException:org.apache.hadoop.hive.ql.metadata.HiveException: ASSERT_TRUE(): assertion failed. {code} *Spark* {code} scala> sql("select if(assert_true(false),2,3)").head res2: org.apache.spark.sql.Row = [3] {code} `IfCoercion` works like the followings. {code} === Applying Rule org.apache.spark.sql.catalyst.analysis.HiveTypeCoercion$IfCoercion === !'Project [unresolvedalias(if (HiveGenericUDF#org.apache.hadoop.hive.ql.udf.generic.GenericUDFAssertTrue(false)) 2 else 3)] 'Project [unresolvedalias(if (nu ll) 2 else 3)] +- OneRowRelation$ +- OneRowRelation$ {code} was: Currently, `HiveTypeCoercion.IfCoercion` removes all predicates whose return-type is null. However, some UDFs need evaluations because they are designed to throw exceptions. *Hive* {code} hive> select if(assert_true(false),2,3); OK Failed with exception java.io.IOException:org.apache.hadoop.hive.ql.metadata.HiveException: ASSERT_TRUE(): assertion failed. {code} *Before* {code} scala> sql("select if(assert_true(false),2,3)").head res2: org.apache.spark.sql.Row = [3] {code} `IfCoercion` works like the followings. {code} === Applying Rule org.apache.spark.sql.catalyst.analysis.HiveTypeCoercion$IfCoercion === !'Project [unresolvedalias(if (HiveGenericUDF#org.apache.hadoop.hive.ql.udf.generic.GenericUDFAssertTrue(false)) 2 else 3)] 'Project [unresolvedalias(if (nu ll) 2 else 3)] +- OneRowRelation$ +- OneRowRelation$ {code} > HiveTypeCoercion.IfCoercion should preserve original predicates. > > > Key: SPARK-14580 > URL: https://issues.apache.org/jira/browse/SPARK-14580 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 1.6.1, 2.0.0 >Reporter: Dongjoon Hyun > > Currently, `HiveTypeCoercion.IfCoercion` removes all predicates whose > return-type is null. However, some UDFs need evaluations because they are > designed to throw exceptions. > *Hive* > {code} > hive> select if(assert_true(false),2,3); > OK > Failed with exception > java.io.IOException:org.apache.hadoop.hive.ql.metadata.HiveException: > ASSERT_TRUE(): assertion failed. > {code} > > *Spark* > {code} > scala> sql("select if(assert_true(false),2,3)").head > res2: org.apache.spark.sql.Row = [3] > {code} > `IfCoercion` works like the followings. > {code} > === Applying Rule > org.apache.spark.sql.catalyst.analysis.HiveTypeCoercion$IfCoercion === > !'Project [unresolvedalias(if > (HiveGenericUDF#org.apache.hadoop.hive.ql.udf.generic.GenericUDFAssertTrue(false)) > 2 else 3)] 'Project [unresolvedalias(if (nu > ll) 2 else 3)] > +- OneRowRelation$ > +- OneRowRelation$ > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org