Re: Hive 0.14.1 release
Oh, now I get it. The 1.0.0 *branch* of Hive. Okay. -- Lefty On Tue, Dec 30, 2014 at 11:43 PM, Lefty Leverenz leftylever...@gmail.com wrote: I thought x.x.# releases were just for fixups, x.#.x could include new features, and #.x.x were major releases that might have some backward-incompatible changes. But I guess we haven't agreed on that. As for documentation, we still have 84 jiras with TODOC14 labels https://issues.apache.org/jira/issues/?jql=project%20%3D%20HIVE%20AND%20labels%20%3D%20TODOC14. Not to mention 25 TODOC13 labels https://issues.apache.org/jira/issues/?jql=project%20%3D%20HIVE%20AND%20labels%20%3D%20TODOC13, eleven TODOC12 https://issues.apache.org/jira/issues/?jql=project%20%3D%20HIVE%20AND%20labels%20%3D%20TODOC12, seven TODOC11 https://issues.apache.org/jira/issues/?jql=project%20%3D%20HIVE%20AND%20labels%20%3D%20TODOC11, and seven TODOC10 https://issues.apache.org/jira/issues/?jql=project%20%3D%20HIVE%20AND%20labels%20%3D%20TODOC10 . That's 134 doc tasks to finish for a Hive 1.0.0 release -- preferably by the release date, not after. Because expectations are higher for 1.0.0 releases. -- Lefty On Tue, Dec 30, 2014 at 5:23 PM, Vikram Dixit K vikram.di...@gmail.com wrote: Hi Folks, Given that there have been a number of fixes that have gone into branch 0.14 in the past 8 weeks, I would like to make a release of 0.14.1 soon. I would like to fix some of the release issues as well this time around. I am thinking of some time around 15th January for getting a RC out. Please let me know if you have any concerns. Also, from a previous thread, I would like to make this release the 1.0 branch of hive. The process for getting jiras into this release is going to be the same as the previous one viz.: 1. Mark the jira with fix version 0.14.1 and update the status to blocker/critical. 2. If a committer +1s the patch for 0.14.1, it is good to go in. Please mention me in the jira in case you are not sure if the jira should make it for 0.14.1. Thanks Vikram.
[jira] [Commented] (HIVE-9205) Change default tez install directory to use /tmp instead of /user and create the directory if it does not exist
[ https://issues.apache.org/jira/browse/HIVE-9205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14262031#comment-14262031 ] Lefty Leverenz commented on HIVE-9205: -- Does this need any documentation? * [Hive on Tez -- Installation and Configuration | https://cwiki.apache.org/confluence/display/Hive/Hive+on+Tez#HiveonTez-InstallationandConfiguration] Change default tez install directory to use /tmp instead of /user and create the directory if it does not exist --- Key: HIVE-9205 URL: https://issues.apache.org/jira/browse/HIVE-9205 Project: Hive Issue Type: Bug Components: Tez Affects Versions: 0.14.0, 0.15.0 Reporter: Vikram Dixit K Assignee: Vikram Dixit K Fix For: 0.15.0, 0.14.1 Attachments: HIVE-9205.1.patch, HIVE-9205.2.patch The common deployment scenario is to install the packages and start services. Creating the /user/user directory is currently an extra step during manual installation. In case the user tries to bring up the hive shell with tez enabled, this would result in an exception. The solution is to change the default install directory to /tmp (so that we have the permissions to create the directory /tmp/user) and create the /tmp/user directory if it did not exist earlier. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-5530) null pointer exception when case returns null
[
https://issues.apache.org/jira/browse/HIVE-5530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14262033#comment-14262033
]
Hive QA commented on HIVE-5530:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12689616/HIVE-5530.2.patch.txt
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 6723 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udaf_covar_samp
org.apache.hive.hcatalog.streaming.TestStreaming.testEndpointConnection
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2229/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2229/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2229/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12689616 - PreCommit-HIVE-TRUNK-Build
null pointer exception when case returns null
-
Key: HIVE-5530
URL: https://issues.apache.org/jira/browse/HIVE-5530
Project: Hive
Issue Type: Bug
Components: SQL
Affects Versions: 0.11.0
Reporter: N Campbell
Assignee: Navis
Priority: Minor
Attachments: HIVE-5530.1.patch.txt, HIVE-5530.2.patch.txt
The following expression will cause an NPE
select case when 1 = 1 then null end from t
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9221) Remove deprecation warning for hive.metastore.local
[
https://issues.apache.org/jira/browse/HIVE-9221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14262037#comment-14262037
]
Lefty Leverenz commented on HIVE-9221:
--
No doc needed. The wiki already says that *hive.metastore.local* was removed
in 0.10.0 with HIVE-2585:
* [Hive Metastore Administration |
https://cwiki.apache.org/confluence/display/Hive/AdminManual+MetastoreAdmin]
** [Basic Configuration Parameters |
https://cwiki.apache.org/confluence/display/Hive/AdminManual+MetastoreAdmin#AdminManualMetastoreAdmin-BasicConfigurationParameters]
** [Local/Embedded Metastore Server |
https://cwiki.apache.org/confluence/display/Hive/AdminManual+MetastoreAdmin#AdminManualMetastoreAdmin-Local/EmbeddedMetastoreServer]
** [Remote Metastore Server -- Client Configuration Parameters |
https://cwiki.apache.org/confluence/display/Hive/AdminManual+MetastoreAdmin#AdminManualMetastoreAdmin-ClientConfigurationParameters]
* [Configuration Properties -- hive.metastore.local |
https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.metastore.local]
Remove deprecation warning for hive.metastore.local
---
Key: HIVE-9221
URL: https://issues.apache.org/jira/browse/HIVE-9221
Project: Hive
Issue Type: Bug
Affects Versions: 0.15.0
Reporter: Brock Noland
Assignee: Brock Noland
Priority: Minor
Fix For: 0.15.0
Attachments: HIVE-9221.patch
The property {{hive.metastore.local}} has been removed for years. We can
remove the warning.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9119) ZooKeeperHiveLockManager does not use zookeeper in the proper way
[ https://issues.apache.org/jira/browse/HIVE-9119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14262040#comment-14262040 ] Lefty Leverenz commented on HIVE-9119: -- +1 for the configuration parameter definitions. ZooKeeperHiveLockManager does not use zookeeper in the proper way - Key: HIVE-9119 URL: https://issues.apache.org/jira/browse/HIVE-9119 Project: Hive Issue Type: Improvement Components: Locking Affects Versions: 0.13.0, 0.14.0, 0.13.1 Reporter: Na Yang Assignee: Na Yang Attachments: HIVE-9119.1.patch, HIVE-9119.2.patch, HIVE-9119.3.patch ZooKeeperHiveLockManager does not use zookeeper in the proper way. Currently a new zookeeper client instance is created for each getlock/releaselock query which sometimes causes the number of open connections between HiveServer2 and ZooKeeper exceed the max connection number that zookeeper server allows. To use zookeeper as a distributed lock, there is no need to create a new zookeeper instance for every getlock try. A single zookeeper instance could be reused and shared by ZooKeeperHiveLockManagers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-7613) Research optimization of auto convert join to map join [Spark branch]
[ https://issues.apache.org/jira/browse/HIVE-7613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14262043#comment-14262043 ] Szehon Ho commented on HIVE-7613: - Thats a good idea, it would be useful. I'll look into that when I get back after New Years Research optimization of auto convert join to map join [Spark branch] - Key: HIVE-7613 URL: https://issues.apache.org/jira/browse/HIVE-7613 Project: Hive Issue Type: Sub-task Components: Spark Reporter: Chengxiang Li Assignee: Suhas Satish Priority: Minor Fix For: spark-branch Attachments: HIve on Spark Map join background.docx, Hive on Spark Join Master Design.pdf, small_table_broadcasting.pdf ConvertJoinMapJoin is an optimization the replaces a common join(aka shuffle join) with a map join(aka broadcast or fragment replicate join) when possible. we need to research how to make it workable with Hive on Spark. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9167) Enhance encryption testing framework to allow create keys zones inside .q files
[ https://issues.apache.org/jira/browse/HIVE-9167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14262055#comment-14262055 ] Lefty Leverenz commented on HIVE-9167: -- Does this need any documentation? (For example, CRYPTO in HiveCommand.java and an explanation of how to create keys zones in .q files.) Enhance encryption testing framework to allow create keys zones inside .q files - Key: HIVE-9167 URL: https://issues.apache.org/jira/browse/HIVE-9167 Project: Hive Issue Type: Sub-task Reporter: Sergio Peña Assignee: Sergio Peña Labels: Kanban Fix For: encryption-branch Attachments: HIVE-9167.4.patch The current implementation of the encryption testing framework on HIVE-8900 initializes a couple of encrypted databases to be used on .q test files. This is useful in order to make tests small, but it does not test all details found on the encryption implementation, such as: encrypted tables with different encryption strength in the same database. We need to allow this kind of encryption as it is how it will be used in the real world where a database will have a few encrypted tables (not all the DB). Also, we need to make this encryption framework flexible so that we can create/delete keys zones on demand when running the .q files. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-7613) Research optimization of auto convert join to map join [Spark branch]
[ https://issues.apache.org/jira/browse/HIVE-7613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lefty Leverenz updated HIVE-7613: - Labels: TODOC-SPARK (was: ) Research optimization of auto convert join to map join [Spark branch] - Key: HIVE-7613 URL: https://issues.apache.org/jira/browse/HIVE-7613 Project: Hive Issue Type: Sub-task Components: Spark Reporter: Chengxiang Li Assignee: Suhas Satish Priority: Minor Labels: TODOC-SPARK Fix For: spark-branch Attachments: HIve on Spark Map join background.docx, Hive on Spark Join Master Design.pdf, small_table_broadcasting.pdf ConvertJoinMapJoin is an optimization the replaces a common join(aka shuffle join) with a map join(aka broadcast or fragment replicate join) when possible. we need to research how to make it workable with Hive on Spark. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-7613) Research optimization of auto convert join to map join [Spark branch]
[ https://issues.apache.org/jira/browse/HIVE-7613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14262057#comment-14262057 ] Lefty Leverenz commented on HIVE-7613: -- Thanks [~szehon], and Happy New Year! I added a TODOC-SPARK label just to help us keep track of this. Research optimization of auto convert join to map join [Spark branch] - Key: HIVE-7613 URL: https://issues.apache.org/jira/browse/HIVE-7613 Project: Hive Issue Type: Sub-task Components: Spark Reporter: Chengxiang Li Assignee: Suhas Satish Priority: Minor Labels: TODOC-SPARK Fix For: spark-branch Attachments: HIve on Spark Map join background.docx, Hive on Spark Join Master Design.pdf, small_table_broadcasting.pdf ConvertJoinMapJoin is an optimization the replaces a common join(aka shuffle join) with a map join(aka broadcast or fragment replicate join) when possible. we need to research how to make it workable with Hive on Spark. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9217) UnionProcessor misses results for multi-insert when hive.optimize.union.remove=true
[
https://issues.apache.org/jira/browse/HIVE-9217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14262082#comment-14262082
]
Hive QA commented on HIVE-9217:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12689623/HIVE-9217.02.patch
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 6724 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_optimize_nullscan
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2230/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2230/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2230/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12689623 - PreCommit-HIVE-TRUNK-Build
UnionProcessor misses results for multi-insert when
hive.optimize.union.remove=true
---
Key: HIVE-9217
URL: https://issues.apache.org/jira/browse/HIVE-9217
Project: Hive
Issue Type: Bug
Reporter: Pengcheng Xiong
Assignee: Pengcheng Xiong
Attachments: HIVE-9217.01.patch, HIVE-9217.02.patch
To reproduce:
In union_remove_6.q, just change
FROM (
SELECT key, count(1) as values from inputTbl1 group by key
UNION ALL
SELECT key, count(1) as values from inputTbl1 group by key
) a
insert overwrite table outputTbl1 select *
insert overwrite table outputTbl2 select *;
to
FROM (
select * from(
SELECT key, count(1) as values from inputTbl1 group by key
UNION ALL
SELECT key, count(1) as values from inputTbl1 group by key
)subq
) a
insert overwrite table outputTbl1 select *
insert overwrite table outputTbl2 select *;
select * from outputtbl2 will output nothing
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Fwd: way to add custom udf jar in hadoop 2.x version
Forwarding Niels' question to hive mailing list. On Wed, Dec 31, 2014 at 1:24 AM, Niels Basjes ni...@basjes.nl wrote: Thanks for the pointer. This seems to work for functions. Is there something similar for CREATE EXTERNAL TABLE ?? Niels On Dec 31, 2014 8:13 AM, Ted Yu yuzhih...@gmail.com wrote: Have you seen this thread ? http://search-hadoop.com/m/8er9TcALc/Hive+udf+custom+jarsubj=Best+way+to+add+custom+UDF+jar+in+HiveServer2 On Dec 30, 2014, at 10:56 PM, reena upadhyay reena2...@gmail.com wrote: Hi, I am using hadoop 2.4.0 version. I have created custom udf jar. I am trying to execute a simple select udf query using java hive jdbc client program. When hive execute the query using map reduce job, then the query execution get fails because the mapper is not able to locate the udf class. So I wanted to add the udf jar in hadoop environment permanently. Please suggest me a way to add this external jar for single node and multi node hadoop cluster. PS: I am using hive 0.13.1 version and I already have this custom udf jar added in HIVE_HOME/lib directory. Thanks
[jira] [Commented] (HIVE-9167) Enhance encryption testing framework to allow create keys zones inside .q files
[ https://issues.apache.org/jira/browse/HIVE-9167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14262234#comment-14262234 ] Ferdinand Xu commented on HIVE-9167: Hi [~leftylev], I don't think crypto command needs documentations since it is visible for test only. Thank you! Enhance encryption testing framework to allow create keys zones inside .q files - Key: HIVE-9167 URL: https://issues.apache.org/jira/browse/HIVE-9167 Project: Hive Issue Type: Sub-task Reporter: Sergio Peña Assignee: Sergio Peña Labels: Kanban Fix For: encryption-branch Attachments: HIVE-9167.4.patch The current implementation of the encryption testing framework on HIVE-8900 initializes a couple of encrypted databases to be used on .q test files. This is useful in order to make tests small, but it does not test all details found on the encryption implementation, such as: encrypted tables with different encryption strength in the same database. We need to allow this kind of encryption as it is how it will be used in the real world where a database will have a few encrypted tables (not all the DB). Also, we need to make this encryption framework flexible so that we can create/delete keys zones on demand when running the .q files. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9205) Change default tez install directory to use /tmp instead of /user and create the directory if it does not exist
[ https://issues.apache.org/jira/browse/HIVE-9205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14262343#comment-14262343 ] Vikram Dixit K commented on HIVE-9205: -- [~leftylev] There is no change needed in the documentation. Only change is to try to create the directory if it did not exist. Change default tez install directory to use /tmp instead of /user and create the directory if it does not exist --- Key: HIVE-9205 URL: https://issues.apache.org/jira/browse/HIVE-9205 Project: Hive Issue Type: Bug Components: Tez Affects Versions: 0.14.0, 0.15.0 Reporter: Vikram Dixit K Assignee: Vikram Dixit K Fix For: 0.15.0, 0.14.1 Attachments: HIVE-9205.1.patch, HIVE-9205.2.patch The common deployment scenario is to install the packages and start services. Creating the /user/user directory is currently an extra step during manual installation. In case the user tries to bring up the hive shell with tez enabled, this would result in an exception. The solution is to change the default install directory to /tmp (so that we have the permissions to create the directory /tmp/user) and create the /tmp/user directory if it did not exist earlier. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-9239) Fix ordering differences due to Java 8 (Part 5)
Mohit Sabharwal created HIVE-9239:
-
Summary: Fix ordering differences due to Java 8 (Part 5)
Key: HIVE-9239
URL: https://issues.apache.org/jira/browse/HIVE-9239
Project: Hive
Issue Type: Sub-task
Components: Tests
Reporter: Mohit Sabharwal
Assignee: Mohit Sabharwal
This patch fixes the following tests:
(1) TestCliDriver.testCliDriver_column_access_stats
Converted all maps used to populate {{ColumnAccessInfo.tableToColumnAccessMap}}
to deterministic order maps. These include {{ParseContext.topToTable}} and
relevant maps in {{UpdateDeleteSemanticAnalyzer}}.
(2) TestCliDriver.testCliDriver_table_access_keys_stats
{{CheckTableAccessHook}} prints table access key information from
{{TableAccessInfo}} by operator. Made the relevant maps deterministic.
(3) TestCliDriver.testCliDriver_subquery_multiinsert
Tasks are printed in different order in {{ExplainTask}} in Java 8 vs Java 7.
TaskCompiler, specifically MapReduceCompiler should use deterministic map for
{{GenMRProcContext.opTaskMap}}
(4) TestCliDriver: outer_join_ppr.q and subquery_notin_having.q
{{FilterOperator}} has predicate ({{ExprNodeDesc}}) with children in different
order. This ordering looks to be coming from antlr. Generated version specific
output for these.
(5) TestCliDriver.testCliDriver_parquet_map_null
Avro API uses HashMap for deserialized map, which makes input to Parquet
non-deterministic. Generated version specific output.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9239) Fix ordering differences due to Java 8 (Part 5)
[
https://issues.apache.org/jira/browse/HIVE-9239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mohit Sabharwal updated HIVE-9239:
--
Description:
This patch fixes the following tests:
(1) TestCliDriver.testCliDriver_column_access_stats
Converted all maps used to populate {{ColumnAccessInfo.tableToColumnAccessMap}}
to deterministic order maps. These include {{ParseContext.topToTable}} and
relevant maps in {{UpdateDeleteSemanticAnalyzer}}.
(2) TestCliDriver.testCliDriver_table_access_keys_stats
{{CheckTableAccessHook}} prints table access key information from
{{TableAccessInfo}} by operator. Made the relevant maps deterministic.
(3) TestCliDriver.testCliDriver_subquery_multiinsert
Tasks are printed in different order in {{ExplainTask}} in Java 8 vs Java 7.
TaskCompiler, specifically MapReduceCompiler should use deterministic map for
{{GenMRProcContext.opTaskMap}}
(4) TestCliDriver: outer_join_ppr.q and subquery_notin_having.q
Predicate ({{ExprNodeDesc}}) has children in different order. This ordering
looks to be coming from antlr. Generated version specific output for these.
(5) TestCliDriver.testCliDriver_parquet_map_null
Avro API uses HashMap for deserialized map, which makes input to Parquet
non-deterministic. Generated version specific output.
was:
This patch fixes the following tests:
(1) TestCliDriver.testCliDriver_column_access_stats
Converted all maps used to populate {{ColumnAccessInfo.tableToColumnAccessMap}}
to deterministic order maps. These include {{ParseContext.topToTable}} and
relevant maps in {{UpdateDeleteSemanticAnalyzer}}.
(2) TestCliDriver.testCliDriver_table_access_keys_stats
{{CheckTableAccessHook}} prints table access key information from
{{TableAccessInfo}} by operator. Made the relevant maps deterministic.
(3) TestCliDriver.testCliDriver_subquery_multiinsert
Tasks are printed in different order in {{ExplainTask}} in Java 8 vs Java 7.
TaskCompiler, specifically MapReduceCompiler should use deterministic map for
{{GenMRProcContext.opTaskMap}}
(4) TestCliDriver: outer_join_ppr.q and subquery_notin_having.q
{{FilterOperator}} has predicate ({{ExprNodeDesc}}) with children in different
order. This ordering looks to be coming from antlr. Generated version specific
output for these.
(5) TestCliDriver.testCliDriver_parquet_map_null
Avro API uses HashMap for deserialized map, which makes input to Parquet
non-deterministic. Generated version specific output.
Fix ordering differences due to Java 8 (Part 5)
---
Key: HIVE-9239
URL: https://issues.apache.org/jira/browse/HIVE-9239
Project: Hive
Issue Type: Sub-task
Components: Tests
Reporter: Mohit Sabharwal
Assignee: Mohit Sabharwal
This patch fixes the following tests:
(1) TestCliDriver.testCliDriver_column_access_stats
Converted all maps used to populate
{{ColumnAccessInfo.tableToColumnAccessMap}} to deterministic order maps.
These include {{ParseContext.topToTable}} and relevant maps in
{{UpdateDeleteSemanticAnalyzer}}.
(2) TestCliDriver.testCliDriver_table_access_keys_stats
{{CheckTableAccessHook}} prints table access key information from
{{TableAccessInfo}} by operator. Made the relevant maps deterministic.
(3) TestCliDriver.testCliDriver_subquery_multiinsert
Tasks are printed in different order in {{ExplainTask}} in Java 8 vs Java 7.
TaskCompiler, specifically MapReduceCompiler should use deterministic map for
{{GenMRProcContext.opTaskMap}}
(4) TestCliDriver: outer_join_ppr.q and subquery_notin_having.q
Predicate ({{ExprNodeDesc}}) has children in different order. This ordering
looks to be coming from antlr. Generated version specific output for these.
(5) TestCliDriver.testCliDriver_parquet_map_null
Avro API uses HashMap for deserialized map, which makes input to Parquet
non-deterministic. Generated version specific output.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9239) Fix ordering differences due to Java 8 (Part 5)
[
https://issues.apache.org/jira/browse/HIVE-9239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mohit Sabharwal updated HIVE-9239:
--
Description:
This patch fixes the following tests:
(1) TestCliDriver.testCliDriver_column_access_stats
Converted all maps used to populate {{ColumnAccessInfo.tableToColumnAccessMap}}
to deterministic order maps. These include {{ParseContext.topToTable}} and
relevant maps in {{UpdateDeleteSemanticAnalyzer}}.
(2) TestCliDriver.testCliDriver_table_access_keys_stats
{{CheckTableAccessHook}} prints table access key information from
{{TableAccessInfo}} by operator. Made the relevant maps deterministic.
(3) TestCliDriver.testCliDriver_subquery_multiinsert
Tasks are printed in different order in {{ExplainTask}} in Java 8 vs Java 7.
TaskCompiler, specifically MapReduceCompiler should use deterministic map for
{{GenMRProcContext.opTaskMap}}
(4) TestCliDriver: outer_join_ppr.q and subquery_notin_having.q
Predicate {{ExprNodeDesc}} has children in different order. This ordering looks
to be coming from antlr. Generated version specific output for these.
(5) TestCliDriver.testCliDriver_parquet_map_null
Avro API uses HashMap for deserialized map, which makes input to Parquet
non-deterministic. Generated version specific output.
was:
This patch fixes the following tests:
(1) TestCliDriver.testCliDriver_column_access_stats
Converted all maps used to populate {{ColumnAccessInfo.tableToColumnAccessMap}}
to deterministic order maps. These include {{ParseContext.topToTable}} and
relevant maps in {{UpdateDeleteSemanticAnalyzer}}.
(2) TestCliDriver.testCliDriver_table_access_keys_stats
{{CheckTableAccessHook}} prints table access key information from
{{TableAccessInfo}} by operator. Made the relevant maps deterministic.
(3) TestCliDriver.testCliDriver_subquery_multiinsert
Tasks are printed in different order in {{ExplainTask}} in Java 8 vs Java 7.
TaskCompiler, specifically MapReduceCompiler should use deterministic map for
{{GenMRProcContext.opTaskMap}}
(4) TestCliDriver: outer_join_ppr.q and subquery_notin_having.q
Predicate ({{ExprNodeDesc}}) has children in different order. This ordering
looks to be coming from antlr. Generated version specific output for these.
(5) TestCliDriver.testCliDriver_parquet_map_null
Avro API uses HashMap for deserialized map, which makes input to Parquet
non-deterministic. Generated version specific output.
Fix ordering differences due to Java 8 (Part 5)
---
Key: HIVE-9239
URL: https://issues.apache.org/jira/browse/HIVE-9239
Project: Hive
Issue Type: Sub-task
Components: Tests
Reporter: Mohit Sabharwal
Assignee: Mohit Sabharwal
This patch fixes the following tests:
(1) TestCliDriver.testCliDriver_column_access_stats
Converted all maps used to populate
{{ColumnAccessInfo.tableToColumnAccessMap}} to deterministic order maps.
These include {{ParseContext.topToTable}} and relevant maps in
{{UpdateDeleteSemanticAnalyzer}}.
(2) TestCliDriver.testCliDriver_table_access_keys_stats
{{CheckTableAccessHook}} prints table access key information from
{{TableAccessInfo}} by operator. Made the relevant maps deterministic.
(3) TestCliDriver.testCliDriver_subquery_multiinsert
Tasks are printed in different order in {{ExplainTask}} in Java 8 vs Java 7.
TaskCompiler, specifically MapReduceCompiler should use deterministic map for
{{GenMRProcContext.opTaskMap}}
(4) TestCliDriver: outer_join_ppr.q and subquery_notin_having.q
Predicate {{ExprNodeDesc}} has children in different order. This ordering
looks to be coming from antlr. Generated version specific output for these.
(5) TestCliDriver.testCliDriver_parquet_map_null
Avro API uses HashMap for deserialized map, which makes input to Parquet
non-deterministic. Generated version specific output.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Review Request 29520: HIVE-9239 : Fix ordering differences due to Java 8 (Part 5)
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29520/ --- Review request for hive. Bugs: HIVE-9239 https://issues.apache.org/jira/browse/HIVE-9239 Repository: hive-git Description --- HIVE-9239 : Fix ordering differences due to Java 8 (Part 5) This patch fixes the following tests: (1) TestCliDriver.testCliDriver_column_access_stats Converted all maps used to populate ColumnAccessInfo.tableToColumnAccessMap to deterministic order maps. These include ParseContext.topToTable and relevant maps in UpdateDeleteSemanticAnalyzer. (2) TestCliDriver.testCliDriver_table_access_keys_stats CheckTableAccessHook prints table access key information from TableAccessInfo by operator. Made the relevant maps deterministic. (3) TestCliDriver.testCliDriver_subquery_multiinsert Tasks are printed in different order in ExplainTask in Java 8 vs Java 7. TaskCompiler, specifically MapReduceCompiler should use deterministic map for GenMRProcContext.opTaskMap (4) TestCliDriver: outer_join_ppr.q and subquery_notin_having.q Predicate ExprNodeDesc has children in different order. This ordering looks to be coming from antlr. Generated version specific output for these. (5) TestCliDriver.testCliDriver_parquet_map_null Avro API uses HashMap for deserialized map, which makes input to Parquet non-deterministic. Generated version specific output. Diffs - itests/util/src/main/java/org/apache/hadoop/hive/ql/hooks/CheckColumnAccessHook.java f6058e4b1e74a0d77d3d44d009fe52429e593ae8 itests/util/src/main/java/org/apache/hadoop/hive/ql/hooks/CheckTableAccessHook.java 8e19fad85092321e1fb1dc3494f0c83aa4445580 ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnAccessInfo.java 8416cff1001eb452a4d977163965b0f15e9115fe ql/src/java/org/apache/hadoop/hive/ql/parse/MapReduceCompiler.java 703c9d119685dabe7f2e6c4f73877172841b4a2f ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java d9da15e65f7cae18939082e4c7709bfb7b60bf60 ql/src/java/org/apache/hadoop/hive/ql/parse/TableAccessAnalyzer.java d847a16b4141aa876baeeab01ec4d4cff61e290d ql/src/java/org/apache/hadoop/hive/ql/parse/TableAccessInfo.java b69830dfb73075b96bc23f5760d327d4477e26b0 ql/src/java/org/apache/hadoop/hive/ql/parse/UpdateDeleteSemanticAnalyzer.java 65f5eaa6b7370455494b30111051339091180a88 ql/src/test/queries/clientpositive/outer_join_ppr.q c32e02305930b4e278cf74f29d794f1eb337f97b ql/src/test/queries/clientpositive/parquet_map_null.q d31cb9908a1f6aaea4eebc6dd1b69c8bb7ec9cda ql/src/test/queries/clientpositive/subquery_notin_having.q a181d593cc3006067ce84628af06d437c46a0a68 ql/src/test/results/clientpositive/column_access_stats.q.out 9330ff7341e106317d2ed880d8e557e7c3f474dd ql/src/test/results/clientpositive/outer_join_ppr.q.java1.7.out PRE-CREATION ql/src/test/results/clientpositive/outer_join_ppr.q.java1.8.out PRE-CREATION ql/src/test/results/clientpositive/outer_join_ppr.q.out 58369ea2b8c53f3695f0301fc09a34684434ddc3 ql/src/test/results/clientpositive/parquet_map_null.q.java1.7.out PRE-CREATION ql/src/test/results/clientpositive/parquet_map_null.q.java1.8.out PRE-CREATION ql/src/test/results/clientpositive/parquet_map_null.q.out 5772ef2811adf1e2b50972956db7be0a26ac3a97 ql/src/test/results/clientpositive/subquery_multiinsert.q.out 556e2891f197e8c4b0180f14597118543bb8d5dd ql/src/test/results/clientpositive/subquery_notin_having.q.java1.7.out PRE-CREATION ql/src/test/results/clientpositive/subquery_notin_having.q.java1.8.out PRE-CREATION ql/src/test/results/clientpositive/subquery_notin_having.q.out 2e84283f3936a11a0e7a76c8e2da909c242ce6a5 ql/src/test/results/clientpositive/table_access_keys_stats.q.out a0449aa39f8fc7960fe42680781a9f5ba01d42b8 Diff: https://reviews.apache.org/r/29520/diff/ Testing --- Tested with both Java 7 and Java 8 Thanks, Mohit Sabharwal
[jira] [Updated] (HIVE-9239) Fix ordering differences due to Java 8 (Part 5)
[
https://issues.apache.org/jira/browse/HIVE-9239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mohit Sabharwal updated HIVE-9239:
--
Attachment: HIVE-9239.patch
Fix ordering differences due to Java 8 (Part 5)
---
Key: HIVE-9239
URL: https://issues.apache.org/jira/browse/HIVE-9239
Project: Hive
Issue Type: Sub-task
Components: Tests
Reporter: Mohit Sabharwal
Assignee: Mohit Sabharwal
Attachments: HIVE-9239.patch
This patch fixes the following tests:
(1) TestCliDriver.testCliDriver_column_access_stats
Converted all maps used to populate
{{ColumnAccessInfo.tableToColumnAccessMap}} to deterministic order maps.
These include {{ParseContext.topToTable}} and relevant maps in
{{UpdateDeleteSemanticAnalyzer}}.
(2) TestCliDriver.testCliDriver_table_access_keys_stats
{{CheckTableAccessHook}} prints table access key information from
{{TableAccessInfo}} by operator. Made the relevant maps deterministic.
(3) TestCliDriver.testCliDriver_subquery_multiinsert
Tasks are printed in different order in {{ExplainTask}} in Java 8 vs Java 7.
TaskCompiler, specifically MapReduceCompiler should use deterministic map for
{{GenMRProcContext.opTaskMap}}
(4) TestCliDriver: outer_join_ppr.q and subquery_notin_having.q
Predicate {{ExprNodeDesc}} has children in different order. This ordering
looks to be coming from antlr. Generated version specific output for these.
(5) TestCliDriver.testCliDriver_parquet_map_null
Avro API uses HashMap for deserialized map, which makes input to Parquet
non-deterministic. Generated version specific output.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9239) Fix ordering differences due to Java 8 (Part 5)
[
https://issues.apache.org/jira/browse/HIVE-9239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mohit Sabharwal updated HIVE-9239:
--
Status: Patch Available (was: Open)
Fix ordering differences due to Java 8 (Part 5)
---
Key: HIVE-9239
URL: https://issues.apache.org/jira/browse/HIVE-9239
Project: Hive
Issue Type: Sub-task
Components: Tests
Reporter: Mohit Sabharwal
Assignee: Mohit Sabharwal
Attachments: HIVE-9239.patch
This patch fixes the following tests:
(1) TestCliDriver.testCliDriver_column_access_stats
Converted all maps used to populate
{{ColumnAccessInfo.tableToColumnAccessMap}} to deterministic order maps.
These include {{ParseContext.topToTable}} and relevant maps in
{{UpdateDeleteSemanticAnalyzer}}.
(2) TestCliDriver.testCliDriver_table_access_keys_stats
{{CheckTableAccessHook}} prints table access key information from
{{TableAccessInfo}} by operator. Made the relevant maps deterministic.
(3) TestCliDriver.testCliDriver_subquery_multiinsert
Tasks are printed in different order in {{ExplainTask}} in Java 8 vs Java 7.
TaskCompiler, specifically MapReduceCompiler should use deterministic map for
{{GenMRProcContext.opTaskMap}}
(4) TestCliDriver: outer_join_ppr.q and subquery_notin_having.q
Predicate {{ExprNodeDesc}} has children in different order. This ordering
looks to be coming from antlr. Generated version specific output for these.
(5) TestCliDriver.testCliDriver_parquet_map_null
Avro API uses HashMap for deserialized map, which makes input to Parquet
non-deterministic. Generated version specific output.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9205) Change default tez install directory to use /tmp instead of /user and create the directory if it does not exist
[ https://issues.apache.org/jira/browse/HIVE-9205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14262488#comment-14262488 ] Lefty Leverenz commented on HIVE-9205: -- Thanks [~vikram.dixit]. Happy New Year! Change default tez install directory to use /tmp instead of /user and create the directory if it does not exist --- Key: HIVE-9205 URL: https://issues.apache.org/jira/browse/HIVE-9205 Project: Hive Issue Type: Bug Components: Tez Affects Versions: 0.14.0, 0.15.0 Reporter: Vikram Dixit K Assignee: Vikram Dixit K Fix For: 0.15.0, 0.14.1 Attachments: HIVE-9205.1.patch, HIVE-9205.2.patch The common deployment scenario is to install the packages and start services. Creating the /user/user directory is currently an extra step during manual installation. In case the user tries to bring up the hive shell with tez enabled, this would result in an exception. The solution is to change the default install directory to /tmp (so that we have the permissions to create the directory /tmp/user) and create the /tmp/user directory if it did not exist earlier. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9167) Enhance encryption testing framework to allow create keys zones inside .q files
[ https://issues.apache.org/jira/browse/HIVE-9167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14262490#comment-14262490 ] Lefty Leverenz commented on HIVE-9167: -- Thanks [~xu], and Happy New Year! Enhance encryption testing framework to allow create keys zones inside .q files - Key: HIVE-9167 URL: https://issues.apache.org/jira/browse/HIVE-9167 Project: Hive Issue Type: Sub-task Reporter: Sergio Peña Assignee: Sergio Peña Labels: Kanban Fix For: encryption-branch Attachments: HIVE-9167.4.patch The current implementation of the encryption testing framework on HIVE-8900 initializes a couple of encrypted databases to be used on .q test files. This is useful in order to make tests small, but it does not test all details found on the encryption implementation, such as: encrypted tables with different encryption strength in the same database. We need to allow this kind of encryption as it is how it will be used in the real world where a database will have a few encrypted tables (not all the DB). Also, we need to make this encryption framework flexible so that we can create/delete keys zones on demand when running the .q files. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9239) Fix ordering differences due to Java 8 (Part 5)
[
https://issues.apache.org/jira/browse/HIVE-9239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14262506#comment-14262506
]
Brock Noland commented on HIVE-9239:
+1
Fix ordering differences due to Java 8 (Part 5)
---
Key: HIVE-9239
URL: https://issues.apache.org/jira/browse/HIVE-9239
Project: Hive
Issue Type: Sub-task
Components: Tests
Reporter: Mohit Sabharwal
Assignee: Mohit Sabharwal
Attachments: HIVE-9239.patch
This patch fixes the following tests:
(1) TestCliDriver.testCliDriver_column_access_stats
Converted all maps used to populate
{{ColumnAccessInfo.tableToColumnAccessMap}} to deterministic order maps.
These include {{ParseContext.topToTable}} and relevant maps in
{{UpdateDeleteSemanticAnalyzer}}.
(2) TestCliDriver.testCliDriver_table_access_keys_stats
{{CheckTableAccessHook}} prints table access key information from
{{TableAccessInfo}} by operator. Made the relevant maps deterministic.
(3) TestCliDriver.testCliDriver_subquery_multiinsert
Tasks are printed in different order in {{ExplainTask}} in Java 8 vs Java 7.
TaskCompiler, specifically MapReduceCompiler should use deterministic map for
{{GenMRProcContext.opTaskMap}}
(4) TestCliDriver: outer_join_ppr.q and subquery_notin_having.q
Predicate {{ExprNodeDesc}} has children in different order. This ordering
looks to be coming from antlr. Generated version specific output for these.
(5) TestCliDriver.testCliDriver_parquet_map_null
Avro API uses HashMap for deserialized map, which makes input to Parquet
non-deterministic. Generated version specific output.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9239) Fix ordering differences due to Java 8 (Part 5)
[
https://issues.apache.org/jira/browse/HIVE-9239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14262520#comment-14262520
]
Hive QA commented on HIVE-9239:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12689731/HIVE-9239.patch
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 6723 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2232/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2232/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2232/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12689731 - PreCommit-HIVE-TRUNK-Build
Fix ordering differences due to Java 8 (Part 5)
---
Key: HIVE-9239
URL: https://issues.apache.org/jira/browse/HIVE-9239
Project: Hive
Issue Type: Sub-task
Components: Tests
Reporter: Mohit Sabharwal
Assignee: Mohit Sabharwal
Attachments: HIVE-9239.patch
This patch fixes the following tests:
(1) TestCliDriver.testCliDriver_column_access_stats
Converted all maps used to populate
{{ColumnAccessInfo.tableToColumnAccessMap}} to deterministic order maps.
These include {{ParseContext.topToTable}} and relevant maps in
{{UpdateDeleteSemanticAnalyzer}}.
(2) TestCliDriver.testCliDriver_table_access_keys_stats
{{CheckTableAccessHook}} prints table access key information from
{{TableAccessInfo}} by operator. Made the relevant maps deterministic.
(3) TestCliDriver.testCliDriver_subquery_multiinsert
Tasks are printed in different order in {{ExplainTask}} in Java 8 vs Java 7.
TaskCompiler, specifically MapReduceCompiler should use deterministic map for
{{GenMRProcContext.opTaskMap}}
(4) TestCliDriver: outer_join_ppr.q and subquery_notin_having.q
Predicate {{ExprNodeDesc}} has children in different order. This ordering
looks to be coming from antlr. Generated version specific output for these.
(5) TestCliDriver.testCliDriver_parquet_map_null
Avro API uses HashMap for deserialized map, which makes input to Parquet
non-deterministic. Generated version specific output.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
