[jira] [Commented] (HIVE-9645) Constant folding case NULL equality
[
https://issues.apache.org/jira/browse/HIVE-9645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14321308#comment-14321308
]
Hive QA commented on HIVE-9645:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12698875/HIVE-9645.2.patch
{color:red}ERROR:{color} -1 due to 10 failed/errored test(s), 7550 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_lateral_view_outer
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_null_cast
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_null_check
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udaf_context_ngrams
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_array
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_percentile
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_remote_script
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_root_dir_external_table
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_scriptfile1
org.apache.hive.jdbc.TestSSL.testSSLFetchHttp
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2803/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2803/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2803/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 10 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12698875 - PreCommit-HIVE-TRUNK-Build
> Constant folding case NULL equality
> ---
>
> Key: HIVE-9645
> URL: https://issues.apache.org/jira/browse/HIVE-9645
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer
>Affects Versions: 1.2.0
>Reporter: Gopal V
>Assignee: Ashutosh Chauhan
> Attachments: HIVE-9645.1.patch, HIVE-9645.2.patch, HIVE-9645.patch
>
>
> Hive logical optimizer does not follow the Null scan codepath when
> encountering a NULL = 1;
> NULL = 1 is not evaluated as false in the constant propogation implementation.
> {code}
> hive> explain select count(1) from store_sales where null=1;
> ...
> TableScan
> alias: store_sales
> filterExpr: (null = 1) (type: boolean)
> Statistics: Num rows: 550076554 Data size: 49570324480
> Basic stats: COMPLETE Column stats: COMPLETE
> Filter Operator
> predicate: (null = 1) (type: boolean)
> Statistics: Num rows: 275038277 Data size: 0 Basic stats:
> PARTIAL Column stats: COMPLETE
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9645) Constant folding case NULL equality
[
https://issues.apache.org/jira/browse/HIVE-9645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14321167#comment-14321167
]
Gopal V commented on HIVE-9645:
---
This broke CTAS typing for me entirely, for cases with an IS_NULL filter.
NULL might be Void, but the Constant propogation optimizer does casts for the
all the other cases to the right type (i.e 1+1 -> 2.0 for floats).
> Constant folding case NULL equality
> ---
>
> Key: HIVE-9645
> URL: https://issues.apache.org/jira/browse/HIVE-9645
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer
>Affects Versions: 1.2.0
>Reporter: Gopal V
>Assignee: Ashutosh Chauhan
> Attachments: HIVE-9645.1.patch, HIVE-9645.2.patch, HIVE-9645.patch
>
>
> Hive logical optimizer does not follow the Null scan codepath when
> encountering a NULL = 1;
> NULL = 1 is not evaluated as false in the constant propogation implementation.
> {code}
> hive> explain select count(1) from store_sales where null=1;
> ...
> TableScan
> alias: store_sales
> filterExpr: (null = 1) (type: boolean)
> Statistics: Num rows: 550076554 Data size: 49570324480
> Basic stats: COMPLETE Column stats: COMPLETE
> Filter Operator
> predicate: (null = 1) (type: boolean)
> Statistics: Num rows: 275038277 Data size: 0 Basic stats:
> PARTIAL Column stats: COMPLETE
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9645) Constant folding case NULL equality
[
https://issues.apache.org/jira/browse/HIVE-9645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14321159#comment-14321159
]
Ashutosh Chauhan commented on HIVE-9645:
This patch doesnt change that semantics (of treating null as void type). It has
been there in Hive for a while.
> Constant folding case NULL equality
> ---
>
> Key: HIVE-9645
> URL: https://issues.apache.org/jira/browse/HIVE-9645
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer
>Affects Versions: 1.2.0
>Reporter: Gopal V
>Assignee: Ashutosh Chauhan
> Attachments: HIVE-9645.1.patch, HIVE-9645.2.patch, HIVE-9645.patch
>
>
> Hive logical optimizer does not follow the Null scan codepath when
> encountering a NULL = 1;
> NULL = 1 is not evaluated as false in the constant propogation implementation.
> {code}
> hive> explain select count(1) from store_sales where null=1;
> ...
> TableScan
> alias: store_sales
> filterExpr: (null = 1) (type: boolean)
> Statistics: Num rows: 550076554 Data size: 49570324480
> Basic stats: COMPLETE Column stats: COMPLETE
> Filter Operator
> predicate: (null = 1) (type: boolean)
> Statistics: Num rows: 275038277 Data size: 0 Basic stats:
> PARTIAL Column stats: COMPLETE
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9645) Constant folding case NULL equality
[
https://issues.apache.org/jira/browse/HIVE-9645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14321157#comment-14321157
]
Ashutosh Chauhan commented on HIVE-9645:
Constant null is of Void type.
> Constant folding case NULL equality
> ---
>
> Key: HIVE-9645
> URL: https://issues.apache.org/jira/browse/HIVE-9645
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer
>Affects Versions: 1.2.0
>Reporter: Gopal V
>Assignee: Ashutosh Chauhan
> Attachments: HIVE-9645.1.patch, HIVE-9645.2.patch, HIVE-9645.patch
>
>
> Hive logical optimizer does not follow the Null scan codepath when
> encountering a NULL = 1;
> NULL = 1 is not evaluated as false in the constant propogation implementation.
> {code}
> hive> explain select count(1) from store_sales where null=1;
> ...
> TableScan
> alias: store_sales
> filterExpr: (null = 1) (type: boolean)
> Statistics: Num rows: 550076554 Data size: 49570324480
> Basic stats: COMPLETE Column stats: COMPLETE
> Filter Operator
> predicate: (null = 1) (type: boolean)
> Statistics: Num rows: 275038277 Data size: 0 Basic stats:
> PARTIAL Column stats: COMPLETE
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9645) Constant folding case NULL equality
[
https://issues.apache.org/jira/browse/HIVE-9645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14321152#comment-14321152
]
Gopal V commented on HIVE-9645:
---
The type changes based on the WHERE clause, which is strange.
{code}
SELECT cfloat, cbigint, coalesce(cfloat, cbigint, 0)
FROM alltypesorc
{code}
is
vs
{code}
SELECT cfloat, cbigint, coalesce(cfloat, cbigint, 0)
FROM alltypesorc
WHERE (cfloat IS NULL AND cbigint IS NULL) LIMIT 10;
{code}
depending on the optimizer replacing NULL params.
> Constant folding case NULL equality
> ---
>
> Key: HIVE-9645
> URL: https://issues.apache.org/jira/browse/HIVE-9645
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer
>Affects Versions: 1.2.0
>Reporter: Gopal V
>Assignee: Ashutosh Chauhan
> Attachments: HIVE-9645.1.patch, HIVE-9645.2.patch, HIVE-9645.patch
>
>
> Hive logical optimizer does not follow the Null scan codepath when
> encountering a NULL = 1;
> NULL = 1 is not evaluated as false in the constant propogation implementation.
> {code}
> hive> explain select count(1) from store_sales where null=1;
> ...
> TableScan
> alias: store_sales
> filterExpr: (null = 1) (type: boolean)
> Statistics: Num rows: 550076554 Data size: 49570324480
> Basic stats: COMPLETE Column stats: COMPLETE
> Filter Operator
> predicate: (null = 1) (type: boolean)
> Statistics: Num rows: 275038277 Data size: 0 Basic stats:
> PARTIAL Column stats: COMPLETE
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9645) Constant folding case NULL equality
[
https://issues.apache.org/jira/browse/HIVE-9645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14321146#comment-14321146
]
Ashutosh Chauhan commented on HIVE-9645:
Not quite following what you mean. Do you mean for third expression, instead
of float it is now int? Coalesce is suppose to return first non-null value
which in this case is 0. Per Hive behavior a literal numeric is first parsed
into integer, which if it fails into float. So, to me new behavior seems to be
more consistent with Hive than earlier.
> Constant folding case NULL equality
> ---
>
> Key: HIVE-9645
> URL: https://issues.apache.org/jira/browse/HIVE-9645
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer
>Affects Versions: 1.2.0
>Reporter: Gopal V
>Assignee: Ashutosh Chauhan
> Attachments: HIVE-9645.1.patch, HIVE-9645.patch
>
>
> Hive logical optimizer does not follow the Null scan codepath when
> encountering a NULL = 1;
> NULL = 1 is not evaluated as false in the constant propogation implementation.
> {code}
> hive> explain select count(1) from store_sales where null=1;
> ...
> TableScan
> alias: store_sales
> filterExpr: (null = 1) (type: boolean)
> Statistics: Num rows: 550076554 Data size: 49570324480
> Basic stats: COMPLETE Column stats: COMPLETE
> Filter Operator
> predicate: (null = 1) (type: boolean)
> Statistics: Num rows: 275038277 Data size: 0 Basic stats:
> PARTIAL Column stats: COMPLETE
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9645) Constant folding case NULL equality
[
https://issues.apache.org/jira/browse/HIVE-9645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14321053#comment-14321053
]
Gopal V commented on HIVE-9645:
---
{code}
-expressions: null (type: void), null (type: void),
COALESCE(null,null,0) (type: float)
+expressions: null (type: void), null (type: void), 0 (type:
int)
{code}
The generation seems to be losing the type information between the
{{oldExprs.get(i).getTypeInfo()}} vs using Void.
> Constant folding case NULL equality
> ---
>
> Key: HIVE-9645
> URL: https://issues.apache.org/jira/browse/HIVE-9645
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer
>Affects Versions: 1.2.0
>Reporter: Gopal V
>Assignee: Ashutosh Chauhan
> Attachments: HIVE-9645.1.patch, HIVE-9645.patch
>
>
> Hive logical optimizer does not follow the Null scan codepath when
> encountering a NULL = 1;
> NULL = 1 is not evaluated as false in the constant propogation implementation.
> {code}
> hive> explain select count(1) from store_sales where null=1;
> ...
> TableScan
> alias: store_sales
> filterExpr: (null = 1) (type: boolean)
> Statistics: Num rows: 550076554 Data size: 49570324480
> Basic stats: COMPLETE Column stats: COMPLETE
> Filter Operator
> predicate: (null = 1) (type: boolean)
> Statistics: Num rows: 275038277 Data size: 0 Basic stats:
> PARTIAL Column stats: COMPLETE
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9645) Constant folding case NULL equality
[
https://issues.apache.org/jira/browse/HIVE-9645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14320484#comment-14320484
]
Hive QA commented on HIVE-9645:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12698690/HIVE-9645.1.patch
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 7542 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_windowing_navfn
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_orc_vectorization_ppd
org.apache.hadoop.hive.metastore.txn.TestCompactionTxnHandler.testRevokeTimedOutWorkers
org.apache.hadoop.hive.thrift.TestHadoop20SAuthBridge.testMetastoreProxyUser
org.apache.hadoop.hive.thrift.TestHadoop20SAuthBridge.testSaslWithHiveMetaStore
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2794/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2794/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2794/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12698690 - PreCommit-HIVE-TRUNK-Build
> Constant folding case NULL equality
> ---
>
> Key: HIVE-9645
> URL: https://issues.apache.org/jira/browse/HIVE-9645
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer
>Affects Versions: 1.2.0
>Reporter: Gopal V
>Assignee: Ashutosh Chauhan
> Attachments: HIVE-9645.1.patch, HIVE-9645.patch
>
>
> Hive logical optimizer does not follow the Null scan codepath when
> encountering a NULL = 1;
> NULL = 1 is not evaluated as false in the constant propogation implementation.
> {code}
> hive> explain select count(1) from store_sales where null=1;
> ...
> TableScan
> alias: store_sales
> filterExpr: (null = 1) (type: boolean)
> Statistics: Num rows: 550076554 Data size: 49570324480
> Basic stats: COMPLETE Column stats: COMPLETE
> Filter Operator
> predicate: (null = 1) (type: boolean)
> Statistics: Num rows: 275038277 Data size: 0 Basic stats:
> PARTIAL Column stats: COMPLETE
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9645) Constant folding case NULL equality
[
https://issues.apache.org/jira/browse/HIVE-9645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14319688#comment-14319688
]
Hive QA commented on HIVE-9645:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12698568/HIVE-9645.patch
{color:red}ERROR:{color} -1 due to 35 failed/errored test(s), 7542 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_decimal_udf
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input9
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_load_dyn_part14
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_num_op_type_conv
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_constant_expr
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf6
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_add_months
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_case
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_coalesce
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_decode
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_elt
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_greatest
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_if
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_instr
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_isnull_isnotnull
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_last_day
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_least
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_locate
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_next_day
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_nvl
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_size
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_when
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_coalesce
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_decimal_udf
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_windowing_navfn
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_optimize_nullscan
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_orc_vectorization_ppd
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_coalesce
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_decimal_udf
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join8
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join8
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_load_dyn_part14
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_optimize_nullscan
org.apache.hive.hcatalog.streaming.TestStreaming.testTransactionBatchEmptyCommit
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2787/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2787/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2787/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 35 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12698568 - PreCommit-HIVE-TRUNK-Build
> Constant folding case NULL equality
> ---
>
> Key: HIVE-9645
> URL: https://issues.apache.org/jira/browse/HIVE-9645
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer
>Affects Versions: 1.2.0
>Reporter: Gopal V
>Assignee: Ashutosh Chauhan
> Attachments: HIVE-9645.patch
>
>
> Hive logical optimizer does not follow the Null scan codepath when
> encountering a NULL = 1;
> NULL = 1 is not evaluated as false in the constant propogation implementation.
> {code}
> hive> explain select count(1) from store_sales where null=1;
> ...
> TableScan
> alias: store_sales
> filterExpr: (null = 1) (type: boolean)
> Statistics: Num rows: 550076554 Data size: 49570324480
> Basic stats: COMPLETE Column stats: COMPLETE
> Filter Operator
> predicate: (null = 1) (type: boolean)
> Statistics: Num rows: 275038277 Data size: 0 Basic stats:
> PARTIAL Column stats: COMPLETE
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
