[jira] [Commented] (HIVE-14324) ORC PPD for floats is broken

[Prasanth Jayachandran (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-14324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394374#comment-15394374
 ] 

Prasanth Jayachandran commented on HIVE-14324:
--

Test failures doesn't look related to me.

> ORC PPD for floats is broken
> 
>
> Key: HIVE-14324
> URL: https://issues.apache.org/jira/browse/HIVE-14324
> Project: Hive
>  Issue Type: Bug
>  Components: ORC
>Affects Versions: 1.3.0, 2.0.0, 2.1.0, 2.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Critical
> Attachments: HIVE-14324.1.patch, HIVE-14324.2.patch
>
>
> ORC stores min/max stats, bloom filters by passing floats as doubles using 
> java's widening conversion. So if we write a float value of 0.22 to ORC file, 
> the min/max stats and bloom filter will use 0.219988079071 double value.
> But when we do PPD, SARG creates literals by converting float to string and 
> then to double which compares 0.22 to 0.219988079071 and fails PPD 
> evaluation. 
> {code}
> hive> create table orc_float (f float) stored as orc;
> hive> insert into table orc_float values(0.22);
> hive> set hive.optimize.index.filter=true;
> hive> select * from orc_float where f=0.22;
> OK
> hive> set hive.optimize.index.filter=false;
> hive> select * from orc_float where f=0.22;
> OK
> 0.22
> {code}
> This is not a problem for doubles and decimals.
> This issue was introduced in HIVE-8460 but back then there was no strict type 
> check when SARGs are created and also PPD evaluation does not convert to 
> column type. But now predicate leaf creation in SARG enforces strict type 
> check for boxed literals and predicate type and PPD evaluation converts stats 
> and constants to column type (predicate).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-14324) ORC PPD for floats is broken

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-14324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15393419#comment-15393419
 ] 

Hive QA commented on HIVE-14324:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12820002/HIVE-14324.2.patch

{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 18 failed/errored test(s), 10321 tests 
executed
*Failed tests:*
{noformat}
TestColumn - did not produce a TEST-*.xml file
TestCookieSigner - did not produce a TEST-*.xml file
TestJdbcWithSQLAuthUDFBlacklist - did not produce a TEST-*.xml file
TestLdapAtnProviderWithMiniDS - did not produce a TEST-*.xml file
TestLdapAuthenticationProviderImpl - did not produce a TEST-*.xml file
TestMsgBusConnection - did not produce a TEST-*.xml file
TestPlainSaslHelper - did not produce a TEST-*.xml file
TestPluggableHiveSessionImpl - did not produce a TEST-*.xml file
TestSessionCleanup - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.llap.daemon.impl.TestLlapTokenChecker.testCheckPermissions
org.apache.hadoop.hive.llap.daemon.impl.TestLlapTokenChecker.testGetToken
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskSchedulerService.testDelayedLocalityNodeCommErrorImmediateAllocation
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.testConnections
{noformat}

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/644/testReport
Console output: 
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/644/console
Test logs: 
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-644/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 18 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12820002 - PreCommit-HIVE-MASTER-Build

> ORC PPD for floats is broken
> 
>
> Key: HIVE-14324
> URL: https://issues.apache.org/jira/browse/HIVE-14324
> Project: Hive
>  Issue Type: Bug
>  Components: ORC
>Affects Versions: 1.3.0, 2.0.0, 2.1.0, 2.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Critical
> Attachments: HIVE-14324.1.patch, HIVE-14324.2.patch
>
>
> ORC stores min/max stats, bloom filters by passing floats as doubles using 
> java's widening conversion. So if we write a float value of 0.22 to ORC file, 
> the min/max stats and bloom filter will use 0.219988079071 double value.
> But when we do PPD, SARG creates literals by converting float to string and 
> then to double which compares 0.22 to 0.219988079071 and fails PPD 
> evaluation. 
> {code}
> hive> create table orc_float (f float) stored as orc;
> hive> insert into table orc_float values(0.22);
> hive> set hive.optimize.index.filter=true;
> hive> select * from orc_float where f=0.22;
> OK
> hive> set hive.optimize.index.filter=false;
> hive> select * from orc_float where f=0.22;
> OK
> 0.22
> {code}
> This is not a problem for doubles and decimals.
> This issue was introduced in HIVE-8460 but back then there was no strict type 
> check when SARGs are created and also PPD evaluation does not convert to 
> column type. But now predicate leaf creation in SARG enforces strict type 
> check for boxed literals and predicate type and PPD evaluation converts stats 
> and constants to column type (predicate).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-14324) ORC PPD for floats is broken

[Gopal V (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-14324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15392781#comment-15392781
 ] 

Gopal V commented on HIVE-14324:


LGTM  - +1.

The narrowing to Float is necessary before widening it back to Double.

> ORC PPD for floats is broken
> 
>
> Key: HIVE-14324
> URL: https://issues.apache.org/jira/browse/HIVE-14324
> Project: Hive
>  Issue Type: Bug
>  Components: ORC
>Affects Versions: 1.3.0, 2.0.0, 2.1.0, 2.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Critical
> Attachments: HIVE-14324.1.patch, HIVE-14324.2.patch
>
>
> ORC stores min/max stats, bloom filters by passing floats as doubles using 
> java's widening conversion. So if we write a float value of 0.22 to ORC file, 
> the min/max stats and bloom filter will use 0.219988079071 double value.
> But when we do PPD, SARG creates literals by converting float to string and 
> then to double which compares 0.22 to 0.219988079071 and fails PPD 
> evaluation. 
> {code}
> hive> create table orc_float (f float) stored as orc;
> hive> insert into table orc_float values(0.22);
> hive> set hive.optimize.index.filter=true;
> hive> select * from orc_float where f=0.22;
> OK
> hive> set hive.optimize.index.filter=false;
> hive> select * from orc_float where f=0.22;
> OK
> 0.22
> {code}
> This is not a problem for doubles and decimals.
> This issue was introduced in HIVE-8460 but back then there was no strict type 
> check when SARGs are created and also PPD evaluation does not convert to 
> column type. But now predicate leaf creation in SARG enforces strict type 
> check for boxed literals and predicate type and PPD evaluation converts stats 
> and constants to column type (predicate).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-14324) ORC PPD for floats is broken

[Prasanth Jayachandran (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-14324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15392492#comment-15392492
 ] 

Prasanth Jayachandran commented on HIVE-14324:
--

.2 patch fixes test issue.

> ORC PPD for floats is broken
> 
>
> Key: HIVE-14324
> URL: https://issues.apache.org/jira/browse/HIVE-14324
> Project: Hive
>  Issue Type: Bug
>  Components: ORC
>Affects Versions: 1.3.0, 2.0.0, 2.1.0, 2.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Critical
> Attachments: HIVE-14324.1.patch, HIVE-14324.2.patch
>
>
> ORC stores min/max stats, bloom filters by passing floats as doubles using 
> java's widening conversion. So if we write a float value of 0.22 to ORC file, 
> the min/max stats and bloom filter will use 0.219988079071 double value.
> But when we do PPD, SARG creates literals by converting float to string and 
> then to double which compares 0.22 to 0.219988079071 and fails PPD 
> evaluation. 
> {code}
> hive> create table orc_float (f float) stored as orc;
> hive> insert into table orc_float values(0.22);
> hive> set hive.optimize.index.filter=true;
> hive> select * from orc_float where f=0.22;
> OK
> hive> set hive.optimize.index.filter=false;
> hive> select * from orc_float where f=0.22;
> OK
> 0.22
> {code}
> This is not a problem for doubles and decimals.
> This issue was introduced in HIVE-8460 but back then there was no strict type 
> check when SARGs are created and also PPD evaluation does not convert to 
> column type. But now predicate leaf creation in SARG enforces strict type 
> check for boxed literals and predicate type and PPD evaluation converts stats 
> and constants to column type (predicate).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-14324) ORC PPD for floats is broken

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-14324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15392445#comment-15392445
 ] 

Hive QA commented on HIVE-14324:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12819883/HIVE-14324.1.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 13 failed/errored test(s), 10339 tests 
executed
*Failed tests:*
{noformat}
TestMiniTezCliDriver-tez_joins_explain.q-vector_data_types.q-schema_evol_text_vecrow_mapwork_part_all_primitive.q-and-12-more
 - did not produce a TEST-*.xml file
TestMsgBusConnection - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_orc_ppd_basic
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.llap.daemon.impl.TestLlapTokenChecker.testCheckPermissions
org.apache.hadoop.hive.llap.daemon.impl.TestLlapTokenChecker.testGetToken
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.testConnections
org.apache.hadoop.hive.ql.io.sarg.TestConvertAstToSearchArg.TestFloatSarg
{noformat}

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/637/testReport
Console output: 
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/637/console
Test logs: 
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-637/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 13 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12819883 - PreCommit-HIVE-MASTER-Build

> ORC PPD for floats is broken
> 
>
> Key: HIVE-14324
> URL: https://issues.apache.org/jira/browse/HIVE-14324
> Project: Hive
>  Issue Type: Bug
>  Components: ORC
>Affects Versions: 1.3.0, 2.0.0, 2.1.0, 2.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Critical
> Attachments: HIVE-14324.1.patch
>
>
> ORC stores min/max stats, bloom filters by passing floats as doubles using 
> java's widening conversion. So if we write a float value of 0.22 to ORC file, 
> the min/max stats and bloom filter will use 0.219988079071 double value.
> But when we do PPD, SARG creates literals by converting float to string and 
> then to double which compares 0.22 to 0.219988079071 and fails PPD 
> evaluation. 
> {code}
> hive> create table orc_float (f float) stored as orc;
> hive> insert into table orc_float values(0.22);
> hive> set hive.optimize.index.filter=true;
> hive> select * from orc_float where f=0.22;
> OK
> hive> set hive.optimize.index.filter=false;
> hive> select * from orc_float where f=0.22;
> OK
> 0.22
> {code}
> This is not a problem for doubles and decimals.
> This issue was introduced in HIVE-8460 but back then there was no strict type 
> check when SARGs are created and also PPD evaluation does not convert to 
> column type. But now predicate leaf creation in SARG enforces strict type 
> check for boxed literals and predicate type and PPD evaluation converts stats 
> and constants to column type (predicate).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-14324) ORC PPD for floats is broken

[Prasanth Jayachandran (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-14324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15391598#comment-15391598
 ] 

Prasanth Jayachandran commented on HIVE-14324:
--

[~gopalv] Can you please review this patch?

> ORC PPD for floats is broken
> 
>
> Key: HIVE-14324
> URL: https://issues.apache.org/jira/browse/HIVE-14324
> Project: Hive
>  Issue Type: Bug
>  Components: ORC
>Affects Versions: 1.3.0, 2.0.0, 2.1.0, 2.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Critical
> Attachments: HIVE-14324.1.patch
>
>
> ORC stores min/max stats, bloom filters by passing floats as doubles using 
> java's widening conversion. So if we write a float value of 0.22 to ORC file, 
> the min/max stats and bloom filter will use 0.219988079071 double value.
> But when we do PPD, SARG creates literals by converting float to string and 
> then to double which compares 0.22 to 0.219988079071 and fails PPD 
> evaluation. 
> {code}
> hive> create table orc_float (f float) stored as orc;
> hive> insert into table orc_float values(0.22);
> hive> set hive.optimize.index.filter=true;
> hive> select * from orc_float where f=0.22;
> OK
> hive> set hive.optimize.index.filter=false;
> hive> select * from orc_float where f=0.22;
> OK
> 0.22
> {code}
> This is not a problem for doubles and decimals.
> This issue was introduced in HIVE-8460 but back then there was no strict type 
> check when SARGs are created and also PPD evaluation does not convert to 
> column type. But now predicate leaf creation in SARG enforces strict type 
> check for boxed literals and predicate type and PPD evaluation converts stats 
> and constants to column type (predicate).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)