[jira] [Updated] (HIVE-17483) HS2 kill command to kill queries using query id
[ https://issues.apache.org/jira/browse/HIVE-17483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Teddy Choi updated HIVE-17483: -- Attachment: HIVE-17483.2.patch > HS2 kill command to kill queries using query id > --- > > Key: HIVE-17483 > URL: https://issues.apache.org/jira/browse/HIVE-17483 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Thejas M Nair >Assignee: Teddy Choi > Attachments: HIVE-17483.1.patch, HIVE-17483.2.patch, > HIVE-17483.2.patch > > > For administrators, it is important to be able to kill queries if required. > Currently, there is no clean way to do it. > It would help to have a "kill query " command that can be run using > odbc/jdbc against a HiveServer2 instance, to kill a query with that queryid > running in that instance. > Authorization will have to be done to ensure that the user that is invoking > the API is allowed to perform this action. > In case of SQL std authorization, this would require admin role. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17426) Execution framework in hive to run tasks in parallel
[ https://issues.apache.org/jira/browse/HIVE-17426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16169619#comment-16169619 ] Lefty Leverenz commented on HIVE-17426: --- Thanks for changing the description of *hive.repl.approx.max.load.tasks*, [~anishek]. > Execution framework in hive to run tasks in parallel > > > Key: HIVE-17426 > URL: https://issues.apache.org/jira/browse/HIVE-17426 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 3.0.0 >Reporter: anishek >Assignee: anishek > Fix For: 3.0.0 > > Attachments: HIVE-17426.0.patch, HIVE-17426.1.patch, > HIVE-17426.2.patch, HIVE-17426.3.patch, HIVE-17426.4.patch, HIVE-17426.5.patch > > > the execution framework currently only runs MR / Spark Tasks in parallel > when {{set hive.exec.parallel=true}}. > Allow other types of tasks to run in parallel as well to support replication > scenarios in hive. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17521) Improve defaults for few runtime configs
[ https://issues.apache.org/jira/browse/HIVE-17521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16169599#comment-16169599 ] Lefty Leverenz commented on HIVE-17521: --- Doc note: This changes default values for *hive.auto.convert.join.hashtable.max.entries*, *hive.hashtable.key.count.adjustment*, and *hive.tez.bloom.filter.factor* so the wiki needs to be updated (with version information). * [hive.auto.convert.join.hashtable.max.entries | https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.auto.convert.join.hashtable.max.entries] -- not documented yet, created by HIVE-12492 (this link will work after it is documented) * [hive.hashtable.key.count.adjustment | https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.hashtable.key.count.adjustment] * [hive.tez.bloom.filter.factor | https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.tez.bloom.filter.factor] Added a TODOC3.0 label. > Improve defaults for few runtime configs > > > Key: HIVE-17521 > URL: https://issues.apache.org/jira/browse/HIVE-17521 > Project: Hive > Issue Type: Task > Components: Configuration >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan >Priority: Minor > Labels: TODOC3.0 > Fix For: 3.0.0 > > Attachments: HIVE-17521.2.patch, HIVE-17521.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (HIVE-12492) MapJoin: 4 million unique integers seems to be a probe plateau
[ https://issues.apache.org/jira/browse/HIVE-12492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15887496#comment-15887496 ] Lefty Leverenz edited comment on HIVE-12492 at 9/18/17 5:19 AM: Doc note: This adds *hive.auto.convert.join.hashtable.max.entries* to HiveConf.java, so it needs to be documented in the wiki. * [Configuration Properties -- Query and DDL Execution | https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-QueryandDDLExecution] Added a TODOC2.2 label. Edit (7/Mar/17): HIVE-16137 changes the default value to 40,000,000 so that's what should be documented in the wiki. Typo alert: In the parameter description, "does not take affect" should be "does not take effect." This can be corrected in the wiki. Edit (18/Sep/17): HIVE-17276 fixes the typo in the parameter description and HIVE-17521 changes the default value to 21,000,000 (both in release 3.0.0). was (Author: le...@hortonworks.com): Doc note: This adds *hive.auto.convert.join.hashtable.max.entries* to HiveConf.java, so it needs to be documented in the wiki. * [Configuration Properties -- Query and DDL Execution | https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-QueryandDDLExecution] Added a TODOC2.2 label. Edit (7/Mar/17): HIVE-16137 changes the default value to 40,000,000 so that's what should be documented in the wiki. Typo alert: In the parameter description, "does not take affect" should be "does not take effect." This can be corrected in the wiki. > MapJoin: 4 million unique integers seems to be a probe plateau > -- > > Key: HIVE-12492 > URL: https://issues.apache.org/jira/browse/HIVE-12492 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Affects Versions: 1.3.0, 1.2.1, 2.0.0 >Reporter: Gopal V >Assignee: Jesus Camacho Rodriguez > Labels: TODOC2.2 > Fix For: 2.2.0 > > Attachments: HIVE-12492.01.patch, HIVE-12492.02.patch, > HIVE-12492.patch > > > After 4 million keys, the map-join implementation seems to suffer from a > performance degradation. > The hashtable build & probe time makes this very inefficient, even if the > data is very compact (i.e 2 ints). > Falling back onto the shuffle join or bucket map-join is useful after 2^22 > items. > (Note: this fixes a statsutil issue - due to the extra clone() in the column > stats path) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17521) Improve defaults for few runtime configs
[ https://issues.apache.org/jira/browse/HIVE-17521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lefty Leverenz updated HIVE-17521: -- Labels: TODOC3.0 (was: ) > Improve defaults for few runtime configs > > > Key: HIVE-17521 > URL: https://issues.apache.org/jira/browse/HIVE-17521 > Project: Hive > Issue Type: Task > Components: Configuration >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan >Priority: Minor > Labels: TODOC3.0 > Fix For: 3.0.0 > > Attachments: HIVE-17521.2.patch, HIVE-17521.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-13989) Extended ACLs are not handled according to specification
[ https://issues.apache.org/jira/browse/HIVE-13989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16169583#comment-16169583 ] Lefty Leverenz commented on HIVE-13989: --- [~vgumashta], this jira is marked as fixed in 2.3.0 and 2.2.1 but you committed it to branch-2 and branch-2.2 so I think it should say fixed in 2.4.0 and 2.2.1. > Extended ACLs are not handled according to specification > > > Key: HIVE-13989 > URL: https://issues.apache.org/jira/browse/HIVE-13989 > Project: Hive > Issue Type: Bug > Components: HCatalog >Affects Versions: 1.2.1, 2.0.0 >Reporter: Chris Drome >Assignee: Chris Drome > Fix For: 2.3.0, 2.2.1 > > Attachments: HIVE-13989.1-branch-1.patch, HIVE-13989.1.patch, > HIVE-13989.4-branch-2.2.patch, HIVE-13989.4-branch-2.patch, > HIVE-13989-branch-1.patch, HIVE-13989-branch-2.2.patch, > HIVE-13989-branch-2.2.patch, HIVE-13989-branch-2.2.patch > > > Hive takes two approaches to working with extended ACLs depending on whether > data is being produced via a Hive query or HCatalog APIs. A Hive query will > run an FsShell command to recursively set the extended ACLs for a directory > sub-tree. HCatalog APIs will attempt to build up the directory sub-tree > programmatically and runs some code to set the ACLs to match the parent > directory. > Some incorrect assumptions were made when implementing the extended ACLs > support. Refer to https://issues.apache.org/jira/browse/HDFS-4685 for the > design documents of extended ACLs in HDFS. These documents model the > implementation after the POSIX implementation on Linux, which can be found at > http://www.vanemery.com/Linux/ACL/POSIX_ACL_on_Linux.html. > The code for setting extended ACLs via HCatalog APIs is found in > HdfsUtils.java: > {code} > if (aclEnabled) { > aclStatus = sourceStatus.getAclStatus(); > if (aclStatus != null) { > LOG.trace(aclStatus.toString()); > aclEntries = aclStatus.getEntries(); > removeBaseAclEntries(aclEntries); > //the ACL api's also expect the tradition user/group/other permission > in the form of ACL > aclEntries.add(newAclEntry(AclEntryScope.ACCESS, AclEntryType.USER, > sourcePerm.getUserAction())); > aclEntries.add(newAclEntry(AclEntryScope.ACCESS, AclEntryType.GROUP, > sourcePerm.getGroupAction())); > aclEntries.add(newAclEntry(AclEntryScope.ACCESS, AclEntryType.OTHER, > sourcePerm.getOtherAction())); > } > } > {code} > We found that DEFAULT extended ACL rules were not being inherited properly by > the directory sub-tree, so the above code is incomplete because it > effectively drops the DEFAULT rules. The second problem is with the call to > {{sourcePerm.getGroupAction()}}, which is incorrect in the case of extended > ACLs. When extended ACLs are used the GROUP permission is replaced with the > extended ACL mask. So the above code will apply the wrong permissions to the > GROUP. Instead the correct GROUP permissions now need to be pulled from the > AclEntry as returned by {{getAclStatus().getEntries()}}. See the > implementation of the new method {{getDefaultAclEntries}} for details. > Similar issues exist with the HCatalog API. None of the API accounts for > setting extended ACLs on the directory sub-tree. The changes to the HCatalog > API allow the extended ACLs to be passed into the required methods similar to > how basic permissions are passed in. When building the directory sub-tree the > extended ACLs of the table directory are inherited by all sub-directories, > including the DEFAULT rules. > Replicating the problem: > Create a table to write data into (I will use acl_test as the destination and > words_text as the source) and set the ACLs as follows: > {noformat} > $ hdfs dfs -setfacl -m > default:user::rwx,default:group::r-x,default:mask::rwx,default:user:hdfs:rwx,group::r-x,user:hdfs:rwx > /user/cdrome/hive/acl_test > $ hdfs dfs -ls -d /user/cdrome/hive/acl_test > drwxrwx---+ - cdrome hdfs 0 2016-07-13 20:36 > /user/cdrome/hive/acl_test > $ hdfs dfs -getfacl -R /user/cdrome/hive/acl_test > # file: /user/cdrome/hive/acl_test > # owner: cdrome > # group: hdfs > user::rwx > user:hdfs:rwx > group::r-x > mask::rwx > other::--- > default:user::rwx > default:user:hdfs:rwx > default:group::r-x > default:mask::rwx > default:other::--- > {noformat} > Note that the basic GROUP permission is set to {{rwx}} after setting the > ACLs. The ACLs explicitly set the DEFAULT rules and a rule specifically for > the {{hdfs}} user. > Run the following query to populate the table: > {noformat} > insert into acl_test partition (dt='a', ds='b') select a, b from words_text > where dt = 'c'; > {noformat} > Note that words_text only has a single partition k
[jira] [Updated] (HIVE-17139) Conditional expressions optimization: skip the expression evaluation if the condition is not satisfied for vectorization engine.
[ https://issues.apache.org/jira/browse/HIVE-17139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ke Jia updated HIVE-17139: -- Attachment: HIVE-17139.10.patch > Conditional expressions optimization: skip the expression evaluation if the > condition is not satisfied for vectorization engine. > > > Key: HIVE-17139 > URL: https://issues.apache.org/jira/browse/HIVE-17139 > Project: Hive > Issue Type: Improvement >Reporter: Ke Jia >Assignee: Ke Jia > Attachments: HIVE-17139.10.patch, HIVE-17139.1.patch, > HIVE-17139.2.patch, HIVE-17139.3.patch, HIVE-17139.4.patch, > HIVE-17139.5.patch, HIVE-17139.6.patch, HIVE-17139.7.patch, > HIVE-17139.8.patch, HIVE-17139.9.patch > > > The case when and if statement execution for Hive vectorization is not > optimal, which all the conditional and else expressions are evaluated for > current implementation. The optimized approach is to update the selected > array of batch parameter after the conditional expression is executed. Then > the else expression will only do the selected rows instead of all. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17375) stddev_samp,var_samp standard compliance
[ https://issues.apache.org/jira/browse/HIVE-17375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16169580#comment-16169580 ] Lefty Leverenz commented on HIVE-17375: --- Thanks [~kgyrtkirk], I agree that this doesn't need to be documented. > stddev_samp,var_samp standard compliance > > > Key: HIVE-17375 > URL: https://issues.apache.org/jira/browse/HIVE-17375 > Project: Hive > Issue Type: Sub-task > Components: SQL >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Minor > Fix For: 3.0.0 > > Attachments: HIVE-17375.1.patch, HIVE-17375.2.patch > > > these two udaf-s are returning 0 in case of only one element - however the > stadard requires NULL to be returned -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17496) Bootstrap repl is not cleaning up staging dirs
[ https://issues.apache.org/jira/browse/HIVE-17496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17496: -- Status: Open (was: Patch Available) > Bootstrap repl is not cleaning up staging dirs > -- > > Key: HIVE-17496 > URL: https://issues.apache.org/jira/browse/HIVE-17496 > Project: Hive > Issue Type: Bug > Components: repl >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17496.1.patch, HIVE-17496.2.patch, > HIVE-17496.3.patch, HIVE-17496.4.patch, HIVE-17496.5.patch > > > This will put more pressure on the HDFS file limit. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HIVE-17548) ThriftCliService reports inaccurate the number of current sessions in the log message
[ https://issues.apache.org/jira/browse/HIVE-17548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang reassigned HIVE-17548: -- Assignee: Xuefu Zhang > ThriftCliService reports inaccurate the number of current sessions in the log > message > - > > Key: HIVE-17548 > URL: https://issues.apache.org/jira/browse/HIVE-17548 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 1.1.0 >Reporter: Xuefu Zhang >Assignee: Xuefu Zhang > > Currently ThriftCliService uses an atomic integer to keep track of the number > of currently open sessions. It reports it through the following two log > messages: > {code} > 2017-09-18 04:14:31,722 INFO [HiveServer2-Handler-Pool: Thread-729979]: > org.apache.hive.service.cli.thrift.ThriftCLIService: Opened a session: > SessionHandle [99ec30d7-5c44-4a45-a8d6-0f0e7ecf4879], current sessions: 345 > 2017-09-18 04:14:41,926 INFO [HiveServer2-Handler-Pool: Thread-717542]: > org.apache.hive.service.cli.thrift.ThriftCLIService: Closed session: > SessionHandle [f38f7890-cba4-459c-872e-4c261b897e00], current sessions: 344 > {code} > This assumes that all sessions are closed or opened thru Thrift API. This > assumption isn't correct because sessions may be closed by the server such as > in case of timeout. Therefore, such log messages tends to over-report the > number of open sessions. > In order to accurately report the number of outstanding sessions, session > manager should be consulted instead. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17496) Bootstrap repl is not cleaning up staging dirs
[ https://issues.apache.org/jira/browse/HIVE-17496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17496: -- Attachment: (was: HIVE-17496.5.patch) > Bootstrap repl is not cleaning up staging dirs > -- > > Key: HIVE-17496 > URL: https://issues.apache.org/jira/browse/HIVE-17496 > Project: Hive > Issue Type: Bug > Components: repl >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17496.1.patch, HIVE-17496.2.patch, > HIVE-17496.3.patch, HIVE-17496.4.patch, HIVE-17496.5.patch > > > This will put more pressure on the HDFS file limit. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17496) Bootstrap repl is not cleaning up staging dirs
[ https://issues.apache.org/jira/browse/HIVE-17496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17496: -- Attachment: HIVE-17496.5.patch > Bootstrap repl is not cleaning up staging dirs > -- > > Key: HIVE-17496 > URL: https://issues.apache.org/jira/browse/HIVE-17496 > Project: Hive > Issue Type: Bug > Components: repl >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17496.1.patch, HIVE-17496.2.patch, > HIVE-17496.3.patch, HIVE-17496.4.patch, HIVE-17496.5.patch > > > This will put more pressure on the HDFS file limit. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17496) Bootstrap repl is not cleaning up staging dirs
[ https://issues.apache.org/jira/browse/HIVE-17496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17496: -- Status: Patch Available (was: Open) > Bootstrap repl is not cleaning up staging dirs > -- > > Key: HIVE-17496 > URL: https://issues.apache.org/jira/browse/HIVE-17496 > Project: Hive > Issue Type: Bug > Components: repl >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17496.1.patch, HIVE-17496.2.patch, > HIVE-17496.3.patch, HIVE-17496.4.patch, HIVE-17496.5.patch > > > This will put more pressure on the HDFS file limit. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17542) Make HoS CombineEquivalentWorkResolver Configurable
[ https://issues.apache.org/jira/browse/HIVE-17542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16169564#comment-16169564 ] Hive QA commented on HIVE-17542: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12887577/HIVE-17542.2.patch {color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 12 failed/errored test(s), 11042 tests executed *Failed tests:* {noformat} TestAccumuloCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=230) TestDummy - did not produce a TEST-*.xml file (likely timed out) (batchId=230) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[create_view] (batchId=39) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udaf_context_ngrams] (batchId=70) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_mask_hash] (batchId=28) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats] (batchId=156) org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2] (batchId=100) org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[drop_table_failure2] (batchId=89) org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] (batchId=234) org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23] (batchId=234) org.apache.hadoop.hive.ql.TestAcidOnTez.testCtasTezUnion (batchId=215) org.apache.hadoop.hive.ql.TestAcidOnTez.testNonStandardConversion01 (batchId=215) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6858/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6858/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6858/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 12 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12887577 - PreCommit-HIVE-Build > Make HoS CombineEquivalentWorkResolver Configurable > --- > > Key: HIVE-17542 > URL: https://issues.apache.org/jira/browse/HIVE-17542 > Project: Hive > Issue Type: Improvement > Components: Physical Optimizer, Spark >Reporter: Sahil Takiar >Assignee: Sahil Takiar > Attachments: HIVE-17542.1.patch, HIVE-17542.2.patch > > > The {{CombineEquivalentWorkResolver}} is run by default. We should make it > configurable so that users can disable it in case there are any issues. We > can enable it by default to preserve backwards compatibility. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17496) Bootstrap repl is not cleaning up staging dirs
[ https://issues.apache.org/jira/browse/HIVE-17496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17496: -- Status: Patch Available (was: Open) > Bootstrap repl is not cleaning up staging dirs > -- > > Key: HIVE-17496 > URL: https://issues.apache.org/jira/browse/HIVE-17496 > Project: Hive > Issue Type: Bug > Components: repl >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17496.1.patch, HIVE-17496.2.patch, > HIVE-17496.3.patch, HIVE-17496.4.patch, HIVE-17496.5.patch > > > This will put more pressure on the HDFS file limit. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17496) Bootstrap repl is not cleaning up staging dirs
[ https://issues.apache.org/jira/browse/HIVE-17496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17496: -- Status: Open (was: Patch Available) > Bootstrap repl is not cleaning up staging dirs > -- > > Key: HIVE-17496 > URL: https://issues.apache.org/jira/browse/HIVE-17496 > Project: Hive > Issue Type: Bug > Components: repl >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17496.1.patch, HIVE-17496.2.patch, > HIVE-17496.3.patch, HIVE-17496.4.patch, HIVE-17496.5.patch > > > This will put more pressure on the HDFS file limit. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17496) Bootstrap repl is not cleaning up staging dirs
[ https://issues.apache.org/jira/browse/HIVE-17496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17496: -- Attachment: HIVE-17496.5.patch > Bootstrap repl is not cleaning up staging dirs > -- > > Key: HIVE-17496 > URL: https://issues.apache.org/jira/browse/HIVE-17496 > Project: Hive > Issue Type: Bug > Components: repl >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17496.1.patch, HIVE-17496.2.patch, > HIVE-17496.3.patch, HIVE-17496.4.patch, HIVE-17496.5.patch > > > This will put more pressure on the HDFS file limit. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17483) HS2 kill command to kill queries using query id
[ https://issues.apache.org/jira/browse/HIVE-17483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Teddy Choi updated HIVE-17483: -- Attachment: HIVE-17483.2.patch Removed unnecessary files > HS2 kill command to kill queries using query id > --- > > Key: HIVE-17483 > URL: https://issues.apache.org/jira/browse/HIVE-17483 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Thejas M Nair >Assignee: Teddy Choi > Attachments: HIVE-17483.1.patch, HIVE-17483.2.patch > > > For administrators, it is important to be able to kill queries if required. > Currently, there is no clean way to do it. > It would help to have a "kill query " command that can be run using > odbc/jdbc against a HiveServer2 instance, to kill a query with that queryid > running in that instance. > Authorization will have to be done to ensure that the user that is invoking > the API is allowed to perform this action. > In case of SQL std authorization, this would require admin role. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-15212) merge branch into master
[ https://issues.apache.org/jira/browse/HIVE-15212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16169531#comment-16169531 ] Hive QA commented on HIVE-15212: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12887575/HIVE-15212.14.patch {color:red}ERROR:{color} -1 due to build exiting with an error Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6857/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6857/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6857/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Tests exited with: NonZeroExitCodeException Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ date '+%Y-%m-%d %T.%3N' 2017-09-18 02:53:21.912 + [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]] + export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + export PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m ' + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m ' + export 'MAVEN_OPTS=-Xmx1g ' + MAVEN_OPTS='-Xmx1g ' + cd /data/hiveptest/working/ + tee /data/hiveptest/logs/PreCommit-HIVE-Build-6857/source-prep.txt + [[ false == \t\r\u\e ]] + mkdir -p maven ivy + [[ git = \s\v\n ]] + [[ git = \g\i\t ]] + [[ -z master ]] + [[ -d apache-github-source-source ]] + [[ ! -d apache-github-source-source/.git ]] + [[ ! -d apache-github-source-source ]] + date '+%Y-%m-%d %T.%3N' 2017-09-18 02:53:21.915 + cd apache-github-source-source + git fetch origin + git reset --hard HEAD HEAD is now at a51ae9c HIVE-17203: Add InterfaceAudience and InterfaceStability annotations for HCat APIs (Sahil Takiar, reviewed by Aihua Xu) + git clean -f -d Removing standalone-metastore/src/gen/org/ + git checkout master Already on 'master' Your branch is up-to-date with 'origin/master'. + git reset --hard origin/master HEAD is now at a51ae9c HIVE-17203: Add InterfaceAudience and InterfaceStability annotations for HCat APIs (Sahil Takiar, reviewed by Aihua Xu) + git merge --ff-only origin/master Already up-to-date. + date '+%Y-%m-%d %T.%3N' 2017-09-18 02:53:27.049 + patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh + patchFilePath=/data/hiveptest/working/scratch/build.patch + [[ -f /data/hiveptest/working/scratch/build.patch ]] + chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh + /data/hiveptest/working/scratch/smart-apply-patch.sh /data/hiveptest/working/scratch/build.patch error: metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreThread.java: No such file or directory The patch does not appear to apply with p0, p1, or p2 + exit 1 ' {noformat} This message is automatically generated. ATTACHMENT ID: 12887575 - PreCommit-HIVE-Build > merge branch into master > > > Key: HIVE-15212 > URL: https://issues.apache.org/jira/browse/HIVE-15212 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-15212.01.patch, HIVE-15212.02.patch, > HIVE-15212.03.patch, HIVE-15212.04.patch, HIVE-15212.05.patch, > HIVE-15212.06.patch, HIVE-15212.07.patch, HIVE-15212.08.patch, > HIVE-15212.09.patch, HIVE-15212.10.patch, HIVE-15212.11.patch, > HIVE-15212.12.patch, HIVE-15212.12.patch, HIVE-15212.13.patch, > HIVE-15212.13.patch, HIVE-15212.14.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17545) Make HoS RDD Cacheing Optimization Configurable
[ https://issues.apache.org/jira/browse/HIVE-17545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16169530#comment-16169530 ] Hive QA commented on HIVE-17545: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12887576/HIVE-17545.2.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 12 failed/errored test(s), 11041 tests executed *Failed tests:* {noformat} TestAccumuloCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=230) TestDummy - did not produce a TEST-*.xml file (likely timed out) (batchId=230) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[create_view] (batchId=39) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_mask_hash] (batchId=28) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[unionDistinct_1] (batchId=143) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats] (batchId=156) org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[drop_table_failure2] (batchId=89) org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] (batchId=234) org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23] (batchId=234) org.apache.hadoop.hive.ql.TestAcidOnTez.testCtasTezUnion (batchId=215) org.apache.hadoop.hive.ql.TestAcidOnTez.testNonStandardConversion01 (batchId=215) org.apache.hive.jdbc.TestJdbcDriver2.testSelectExecAsync2 (batchId=225) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6856/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6856/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6856/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 12 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12887576 - PreCommit-HIVE-Build > Make HoS RDD Cacheing Optimization Configurable > --- > > Key: HIVE-17545 > URL: https://issues.apache.org/jira/browse/HIVE-17545 > Project: Hive > Issue Type: Improvement > Components: Physical Optimizer, Spark >Reporter: Sahil Takiar >Assignee: Sahil Takiar > Attachments: HIVE-17545.1.patch, HIVE-17545.2.patch > > > The RDD cacheing optimization add in HIVE-10550 is enabled by default. We > should make it configurable in case users want to disable it. We can leave it > on by default to preserve backwards compatibility. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17344) LocalCache element memory usage is not calculated properly.
[ https://issues.apache.org/jira/browse/HIVE-17344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16169524#comment-16169524 ] Lefty Leverenz commented on HIVE-17344: --- [~kgyrtkirk], please add fix version 3.0.0 to this issue. Thanks. > LocalCache element memory usage is not calculated properly. > --- > > Key: HIVE-17344 > URL: https://issues.apache.org/jira/browse/HIVE-17344 > Project: Hive > Issue Type: Bug >Reporter: Janos Gub >Assignee: Janos Gub > Attachments: HIVE-17344.2.patch, HIVE-17344.patch > > > Orc footer cache has a calculation of memory usage: > {code:java} > public int getMemoryUsage() { > return bb.remaining() + 100; // 100 is for 2 longs, BB and java overheads > (semi-arbitrary). > } > {code} > ByteBuffer.remaining returns the remaining space in the bytebuffer, thus > allowing this cache have elements MAXWEIGHT/100 of arbitrary size. I think > the correct solution would be bb.capacity. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17465) Statistics: Drill-down filters don't reduce row-counts progressively
[ https://issues.apache.org/jira/browse/HIVE-17465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16169496#comment-16169496 ] Hive QA commented on HIVE-17465: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12887574/HIVE-17465.7.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 12 failed/errored test(s), 11041 tests executed *Failed tests:* {noformat} TestAccumuloCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=230) TestDummy - did not produce a TEST-*.xml file (likely timed out) (batchId=230) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[create_view] (batchId=39) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_mask_hash] (batchId=28) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats] (batchId=156) org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_explainuser_1] (batchId=170) org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[drop_table_failure2] (batchId=89) org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] (batchId=234) org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23] (batchId=234) org.apache.hadoop.hive.ql.TestAcidOnTez.testCtasTezUnion (batchId=215) org.apache.hadoop.hive.ql.TestAcidOnTez.testNonStandardConversion01 (batchId=215) org.apache.hive.jdbc.TestJdbcWithMiniHS2.testHttpRetryOnServerIdleTimeout (batchId=227) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6855/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6855/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6855/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 12 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12887574 - PreCommit-HIVE-Build > Statistics: Drill-down filters don't reduce row-counts progressively > > > Key: HIVE-17465 > URL: https://issues.apache.org/jira/browse/HIVE-17465 > Project: Hive > Issue Type: Bug > Components: Physical Optimizer, Statistics >Reporter: Gopal V >Assignee: Vineet Garg > Attachments: HIVE-17465.1.patch, HIVE-17465.2.patch, > HIVE-17465.3.patch, HIVE-17465.4.patch, HIVE-17465.5.patch, > HIVE-17465.6.patch, HIVE-17465.7.patch > > > {code} > explain select count(d_date_sk) from date_dim where d_year=2001 ; > explain select count(d_date_sk) from date_dim where d_year=2001 and d_moy = > 9; > explain select count(d_date_sk) from date_dim where d_year=2001 and d_moy = 9 > and d_dom = 21; > {code} > All 3 queries end up with the same row-count estimates after the filter. > {code} > Map Operator Tree: > TableScan > alias: date_dim > filterExpr: (d_year = 2001) (type: boolean) > Statistics: Num rows: 73049 Data size: 82034027 Basic > stats: COMPLETE Column stats: COMPLETE > Filter Operator > predicate: (d_year = 2001) (type: boolean) > Statistics: Num rows: 363 Data size: 4356 Basic stats: > COMPLETE Column stats: COMPLETE > > Map 1 > Map Operator Tree: > TableScan > alias: date_dim > filterExpr: ((d_year = 2001) and (d_moy = 9)) (type: > boolean) > Statistics: Num rows: 73049 Data size: 82034027 Basic > stats: COMPLETE Column stats: COMPLETE > Filter Operator > predicate: ((d_year = 2001) and (d_moy = 9)) (type: > boolean) > Statistics: Num rows: 363 Data size: 5808 Basic stats: > COMPLETE Column stats: COMPLETE > Map 1 > Map Operator Tree: > TableScan > alias: date_dim > filterExpr: ((d_year = 2001) and (d_moy = 9) and (d_dom = > 21)) (type: boolean) > Statistics: Num rows: 73049 Data size: 82034027 Basic > stats: COMPLETE Column stats: COMPLETE > Filter Operator > predicate: ((d_year = 2001) and (d_moy = 9) and (d_dom = > 21)) (type: boolean) > Statistics: Num rows: 363 Data size: 7260 Basic stats: > COMPLETE Column stats: COMPLETE > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-16886) HMS log notifications may have duplicated event IDs if multiple HMS are running concurrently
[ https://issues.apache.org/jira/browse/HIVE-16886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16169484#comment-16169484 ] Lefty Leverenz commented on HIVE-16886: --- Doc note: This adds *hive.notification.sequence.lock.max.retries* and *hive.notification.sequence.lock.retry.sleep.interval* to HiveConf.java, so they need to be documented in the wiki. Apparently they belong in the metastore section. * [Configuration Properties -- MetaStore | https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-MetaStore] Added a TODOC3.0 label. > HMS log notifications may have duplicated event IDs if multiple HMS are > running concurrently > > > Key: HIVE-16886 > URL: https://issues.apache.org/jira/browse/HIVE-16886 > Project: Hive > Issue Type: Bug > Components: Hive, Metastore >Reporter: Sergio Peña >Assignee: anishek > Labels: TODOC3.0 > Fix For: 3.0.0 > > Attachments: datastore-identity-holes.diff, HIVE-16886.1.patch, > HIVE-16886.2.patch, HIVE-16886.3.patch, HIVE-16886.4.patch, > HIVE-16886.5.patch, HIVE-16886.6.patch, HIVE-16886.7.patch, HIVE-16886.8.patch > > > When running multiple Hive Metastore servers and DB notifications are > enabled, I could see that notifications can be persisted with a duplicated > event ID. > This does not happen when running multiple threads in a single HMS node due > to the locking acquired on the DbNotificationsLog class, but multiple HMS > could cause conflicts. > The issue is in the ObjectStore#addNotificationEvent() method. The event ID > fetched from the datastore is used for the new notification, incremented in > the server itself, then persisted or updated back to the datastore. If 2 > servers read the same ID, then these 2 servers write a new notification with > the same ID. > The event ID is not unique nor a primary key. > Here's a test case using the TestObjectStore class that confirms this issue: > {noformat} > @Test > public void testConcurrentAddNotifications() throws ExecutionException, > InterruptedException { > final int NUM_THREADS = 2; > CountDownLatch countIn = new CountDownLatch(NUM_THREADS); > CountDownLatch countOut = new CountDownLatch(1); > HiveConf conf = new HiveConf(); > conf.setVar(HiveConf.ConfVars.METASTORE_EXPRESSION_PROXY_CLASS, > MockPartitionExpressionProxy.class.getName()); > ExecutorService executorService = > Executors.newFixedThreadPool(NUM_THREADS); > FutureTask tasks[] = new FutureTask[NUM_THREADS]; > for (int i=0; i final int n = i; > tasks[i] = new FutureTask(new Callable() { > @Override > public Void call() throws Exception { > ObjectStore store = new ObjectStore(); > store.setConf(conf); > NotificationEvent dbEvent = > new NotificationEvent(0, 0, > EventMessage.EventType.CREATE_DATABASE.toString(), "CREATE DATABASE DB" + n); > System.out.println("ADDING NOTIFICATION"); > countIn.countDown(); > countOut.await(); > store.addNotificationEvent(dbEvent); > System.out.println("FINISH NOTIFICATION"); > return null; > } > }); > executorService.execute(tasks[i]); > } > countIn.await(); > countOut.countDown(); > for (int i = 0; i < NUM_THREADS; ++i) { > tasks[i].get(); > } > NotificationEventResponse eventResponse = > objectStore.getNextNotification(new NotificationEventRequest()); > Assert.assertEquals(2, eventResponse.getEventsSize()); > Assert.assertEquals(1, eventResponse.getEvents().get(0).getEventId()); > // This fails because the next notification has an event ID = 1 > Assert.assertEquals(2, eventResponse.getEvents().get(1).getEventId()); > } > {noformat} > The last assertion fails expecting an event ID 1 instead of 2. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17203) Add InterfaceAudience and InterfaceStability annotations for HCat APIs
[ https://issues.apache.org/jira/browse/HIVE-17203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sahil Takiar updated HIVE-17203: Resolution: Fixed Status: Resolved (was: Patch Available) Committed to master. > Add InterfaceAudience and InterfaceStability annotations for HCat APIs > -- > > Key: HIVE-17203 > URL: https://issues.apache.org/jira/browse/HIVE-17203 > Project: Hive > Issue Type: Sub-task > Components: HCatalog >Affects Versions: 3.0.0 >Reporter: Sahil Takiar >Assignee: Sahil Takiar > Attachments: HIVE-17203.1.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17542) Make HoS CombineEquivalentWorkResolver Configurable
[ https://issues.apache.org/jira/browse/HIVE-17542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sahil Takiar updated HIVE-17542: Attachment: HIVE-17542.2.patch > Make HoS CombineEquivalentWorkResolver Configurable > --- > > Key: HIVE-17542 > URL: https://issues.apache.org/jira/browse/HIVE-17542 > Project: Hive > Issue Type: Improvement > Components: Physical Optimizer, Spark >Reporter: Sahil Takiar >Assignee: Sahil Takiar > Attachments: HIVE-17542.1.patch, HIVE-17542.2.patch > > > The {{CombineEquivalentWorkResolver}} is run by default. We should make it > configurable so that users can disable it in case there are any issues. We > can enable it by default to preserve backwards compatibility. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17545) Make HoS RDD Cacheing Optimization Configurable
[ https://issues.apache.org/jira/browse/HIVE-17545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sahil Takiar updated HIVE-17545: Attachment: HIVE-17545.2.patch > Make HoS RDD Cacheing Optimization Configurable > --- > > Key: HIVE-17545 > URL: https://issues.apache.org/jira/browse/HIVE-17545 > Project: Hive > Issue Type: Improvement > Components: Physical Optimizer, Spark >Reporter: Sahil Takiar >Assignee: Sahil Takiar > Attachments: HIVE-17545.1.patch, HIVE-17545.2.patch > > > The RDD cacheing optimization add in HIVE-10550 is enabled by default. We > should make it configurable in case users want to disable it. We can leave it > on by default to preserve backwards compatibility. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-15212) merge branch into master
[ https://issues.apache.org/jira/browse/HIVE-15212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-15212: Status: Patch Available (was: Open) > merge branch into master > > > Key: HIVE-15212 > URL: https://issues.apache.org/jira/browse/HIVE-15212 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-15212.01.patch, HIVE-15212.02.patch, > HIVE-15212.03.patch, HIVE-15212.04.patch, HIVE-15212.05.patch, > HIVE-15212.06.patch, HIVE-15212.07.patch, HIVE-15212.08.patch, > HIVE-15212.09.patch, HIVE-15212.10.patch, HIVE-15212.11.patch, > HIVE-15212.12.patch, HIVE-15212.12.patch, HIVE-15212.13.patch, > HIVE-15212.13.patch, HIVE-15212.14.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-15212) merge branch into master
[ https://issues.apache.org/jira/browse/HIVE-15212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-15212: Attachment: HIVE-15212.14.patch HiveQA runs just keep disappearing... again > merge branch into master > > > Key: HIVE-15212 > URL: https://issues.apache.org/jira/browse/HIVE-15212 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-15212.01.patch, HIVE-15212.02.patch, > HIVE-15212.03.patch, HIVE-15212.04.patch, HIVE-15212.05.patch, > HIVE-15212.06.patch, HIVE-15212.07.patch, HIVE-15212.08.patch, > HIVE-15212.09.patch, HIVE-15212.10.patch, HIVE-15212.11.patch, > HIVE-15212.12.patch, HIVE-15212.12.patch, HIVE-15212.13.patch, > HIVE-15212.13.patch, HIVE-15212.14.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-15212) merge branch into master
[ https://issues.apache.org/jira/browse/HIVE-15212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-15212: Status: Open (was: Patch Available) > merge branch into master > > > Key: HIVE-15212 > URL: https://issues.apache.org/jira/browse/HIVE-15212 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-15212.01.patch, HIVE-15212.02.patch, > HIVE-15212.03.patch, HIVE-15212.04.patch, HIVE-15212.05.patch, > HIVE-15212.06.patch, HIVE-15212.07.patch, HIVE-15212.08.patch, > HIVE-15212.09.patch, HIVE-15212.10.patch, HIVE-15212.11.patch, > HIVE-15212.12.patch, HIVE-15212.12.patch, HIVE-15212.13.patch, > HIVE-15212.13.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-16886) HMS log notifications may have duplicated event IDs if multiple HMS are running concurrently
[ https://issues.apache.org/jira/browse/HIVE-16886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lefty Leverenz updated HIVE-16886: -- Labels: TODOC3.0 (was: ) > HMS log notifications may have duplicated event IDs if multiple HMS are > running concurrently > > > Key: HIVE-16886 > URL: https://issues.apache.org/jira/browse/HIVE-16886 > Project: Hive > Issue Type: Bug > Components: Hive, Metastore >Reporter: Sergio Peña >Assignee: anishek > Labels: TODOC3.0 > Fix For: 3.0.0 > > Attachments: datastore-identity-holes.diff, HIVE-16886.1.patch, > HIVE-16886.2.patch, HIVE-16886.3.patch, HIVE-16886.4.patch, > HIVE-16886.5.patch, HIVE-16886.6.patch, HIVE-16886.7.patch, HIVE-16886.8.patch > > > When running multiple Hive Metastore servers and DB notifications are > enabled, I could see that notifications can be persisted with a duplicated > event ID. > This does not happen when running multiple threads in a single HMS node due > to the locking acquired on the DbNotificationsLog class, but multiple HMS > could cause conflicts. > The issue is in the ObjectStore#addNotificationEvent() method. The event ID > fetched from the datastore is used for the new notification, incremented in > the server itself, then persisted or updated back to the datastore. If 2 > servers read the same ID, then these 2 servers write a new notification with > the same ID. > The event ID is not unique nor a primary key. > Here's a test case using the TestObjectStore class that confirms this issue: > {noformat} > @Test > public void testConcurrentAddNotifications() throws ExecutionException, > InterruptedException { > final int NUM_THREADS = 2; > CountDownLatch countIn = new CountDownLatch(NUM_THREADS); > CountDownLatch countOut = new CountDownLatch(1); > HiveConf conf = new HiveConf(); > conf.setVar(HiveConf.ConfVars.METASTORE_EXPRESSION_PROXY_CLASS, > MockPartitionExpressionProxy.class.getName()); > ExecutorService executorService = > Executors.newFixedThreadPool(NUM_THREADS); > FutureTask tasks[] = new FutureTask[NUM_THREADS]; > for (int i=0; i final int n = i; > tasks[i] = new FutureTask(new Callable() { > @Override > public Void call() throws Exception { > ObjectStore store = new ObjectStore(); > store.setConf(conf); > NotificationEvent dbEvent = > new NotificationEvent(0, 0, > EventMessage.EventType.CREATE_DATABASE.toString(), "CREATE DATABASE DB" + n); > System.out.println("ADDING NOTIFICATION"); > countIn.countDown(); > countOut.await(); > store.addNotificationEvent(dbEvent); > System.out.println("FINISH NOTIFICATION"); > return null; > } > }); > executorService.execute(tasks[i]); > } > countIn.await(); > countOut.countDown(); > for (int i = 0; i < NUM_THREADS; ++i) { > tasks[i].get(); > } > NotificationEventResponse eventResponse = > objectStore.getNextNotification(new NotificationEventRequest()); > Assert.assertEquals(2, eventResponse.getEventsSize()); > Assert.assertEquals(1, eventResponse.getEvents().get(0).getEventId()); > // This fails because the next notification has an event ID = 1 > Assert.assertEquals(2, eventResponse.getEvents().get(1).getEventId()); > } > {noformat} > The last assertion fails expecting an event ID 1 instead of 2. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (HIVE-14309) Fix naming of classes in orc module to not conflict with standalone orc
[ https://issues.apache.org/jira/browse/HIVE-14309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16047439#comment-16047439 ] Lefty Leverenz edited comment on HIVE-14309 at 9/18/17 12:08 AM: - [~owen.omalley], you committed this to branch-2.2 on May 31, so please update the status and fix version. See commit fd1188a6a79c6da868d463c1e5db50c017c3b1a2. Edit 17/Sep/17: [~owen.omalley], nudge. was (Author: le...@hortonworks.com): [~owen.omalley], you committed this to branch-2.2 on May 31, so please update the status and fix version. See commit fd1188a6a79c6da868d463c1e5db50c017c3b1a2. > Fix naming of classes in orc module to not conflict with standalone orc > --- > > Key: HIVE-14309 > URL: https://issues.apache.org/jira/browse/HIVE-14309 > Project: Hive > Issue Type: Bug >Reporter: Owen O'Malley >Assignee: Owen O'Malley > > The current Hive 2.0 and 2.1 releases have classes in the org.apache.orc > namespace that clash with the ORC project's classes. From Hive 2.2 onward, > the classes will only be on ORC, but we'll reduce the problems of classpath > issues if we rename the classes to org.apache.hive.orc. > I've looked at a set of projects (pig, spark, oozie, flume, & storm) and > can't find any uses of Hive's versions of the org.apache.orc classes, so I > believe this is a safe change that will reduce the integration problems down > stream. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17451) Cannot read decimal from avro file created with HIVE
[ https://issues.apache.org/jira/browse/HIVE-17451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16169474#comment-16169474 ] Matt McCline commented on HIVE-17451: - Seems like avro-tools.jar tojson isn't converting the binary (physical type) to decimal (logical type). > Cannot read decimal from avro file created with HIVE > > > Key: HIVE-17451 > URL: https://issues.apache.org/jira/browse/HIVE-17451 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 1.1.0 >Reporter: liviu >Assignee: Ganesh Tripathi >Priority: Blocker > > Hi, > When we export decimal data from a hive managed table to a hive avro external > table (as bytes with decimal logicalType) the value from avro file cannot be > read with any other tools (ex: avro-tools, spark, datastage..) > _+Scenario:+_ > *create hive managed table an insert a decimal record:* > {code:java} > create table test_decimal (col1 decimal(20,2)); > insert into table test_decimal values (3.12); > {code} > *create avro schema /tmp/test_decimal.avsc with below content:* > {code:java} > { > "type" : "record", > "name" : "decimal_test_avro", > "fields" : [ { > "name" : "col1", > "type" : [ "null", { > "type" : "bytes", > "logicalType" : "decimal", > "precision" : 20, > "scale" : 2 > } ], > "default" : null, > "columnName" : "col1", > "sqlType" : "2" > }], > "tableName" : "decimal_test_avro" > } > {code} > *create an hive external table stored as avro:* > {code:java} > create external table test_decimal_avro > STORED AS AVRO > LOCATION '/tmp/test_decimal' > TBLPROPERTIES ( > 'avro.schema.url'='/tmp/test_decimal.avsc', > 'orc.compress'='SNAPPY'); > {code} > *insert data in avro external table from hive managed table:* > {code:java} > set hive.exec.compress.output=true; > set hive.exec.compress.intermediate=true; > set avro.output.codec=snappy; > insert overwrite table test_decimal_avro select * from test_decimal; > {code} > *successfully reading data from hive avro table through hive cli:* > {code:java} > select * from test_decimal_avro; > OK > 3.12 > {code} > *avro schema from avro created file is ok:* > {code:java} > hadoop jar /avro-tools.jar getschema /tmp/test_decimal/00_0 > { > "type" : "record", > "name" : "decimal_test_avro", > "fields" : [ { > "name" : "col1", > "type" : [ "null", { > "type" : "bytes", > "logicalType" : "decimal", > "precision" : 20, > "scale" : 2 > } ], > "default" : null, > "columnName" : "col1", > "sqlType" : "2" > } ], > "tableName" : "decimal_test_avro" > } > {code} > *read data from avro file with avro-tools {color:#d04437}error{color}, got > {color:#d04437}"\u00018"{color} value instead of the correct one:* > {code:java} > hadoop jar avro-tools.jar tojson /tmp/test_decimal/00_0 > {"col1":{"bytes":"\u00018"}} > {code} > *Read data in a spark dataframe error, got {color:#d04437}[01 38]{color} > and{color:#d04437} 8{color} when converted to string instead of correct > "3.12" value :* > {code:java} > val df = sql.read.avro("/tmp/test_decimal") > df: org.apache.spark.sql.DataFrame = [col1: binary] > scala> df.show() > +---+ > | col1| > +---+ > |[01 38]| > +---+ > scala> df.withColumn("col2", 'col1.cast("String")).select("col2").show() > ++ > |col2| > ++ > | 8| > ++ > {code} > Is this a Hive bug or there is anything else I can do in order to get correct > values in the avro file created by Hive? > Thanks, -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17387) implement Tez AM registry in Hive
[ https://issues.apache.org/jira/browse/HIVE-17387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16169473#comment-16169473 ] Lefty Leverenz commented on HIVE-17387: --- Doc note: This adds three configuration parameters to HiveConf.java (*hive.llap.task.scheduler.am.registry*, *hive.llap.task.scheduler.am.registry.principal*, and *hive.llap.task.scheduler.am.registry.keytab.file*) so they need to be documented in the wiki. * [Configuration Properties -- LLAP | https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-LLAP] Added a TODOC3.0 label. > implement Tez AM registry in Hive > - > > Key: HIVE-17387 > URL: https://issues.apache.org/jira/browse/HIVE-17387 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Labels: TODOC3.0 > Fix For: 3.0.0 > > Attachments: HIVE-17387.01.patch, HIVE-17387.02.patch, > HIVE-17387.patch > > > Necessary for HS2 HA, to transfer AMs between HS2s, etc. > Helpful for workload management. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17387) implement Tez AM registry in Hive
[ https://issues.apache.org/jira/browse/HIVE-17387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lefty Leverenz updated HIVE-17387: -- Labels: TODOC3.0 (was: ) > implement Tez AM registry in Hive > - > > Key: HIVE-17387 > URL: https://issues.apache.org/jira/browse/HIVE-17387 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Labels: TODOC3.0 > Fix For: 3.0.0 > > Attachments: HIVE-17387.01.patch, HIVE-17387.02.patch, > HIVE-17387.patch > > > Necessary for HS2 HA, to transfer AMs between HS2s, etc. > Helpful for workload management. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17451) Cannot read decimal from avro file created with HIVE
[ https://issues.apache.org/jira/browse/HIVE-17451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16169472#comment-16169472 ] Matt McCline commented on HIVE-17451: - I believe Avro stores the decimal as the serialization of a Java BigInteger. That is, we call BigInteger's toByteArray function that returns a byte array with two's compliment reprentation. Why do we use BigInteger? Because decimal's max precion of 38 digits cannot be stored in a Java 64 bit signed long. 128 bits are needed. I believe Spark represents decimal with precision > 18 as Java BigDecimal. BigDecimal.unscaledValue() returns a BigInteger. So logicalType decimal and stored as bytes makes sense. Does specifying [col1:decimal(20,2)] in the data frame detail not work? > Cannot read decimal from avro file created with HIVE > > > Key: HIVE-17451 > URL: https://issues.apache.org/jira/browse/HIVE-17451 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 1.1.0 >Reporter: liviu >Assignee: Ganesh Tripathi >Priority: Blocker > > Hi, > When we export decimal data from a hive managed table to a hive avro external > table (as bytes with decimal logicalType) the value from avro file cannot be > read with any other tools (ex: avro-tools, spark, datastage..) > _+Scenario:+_ > *create hive managed table an insert a decimal record:* > {code:java} > create table test_decimal (col1 decimal(20,2)); > insert into table test_decimal values (3.12); > {code} > *create avro schema /tmp/test_decimal.avsc with below content:* > {code:java} > { > "type" : "record", > "name" : "decimal_test_avro", > "fields" : [ { > "name" : "col1", > "type" : [ "null", { > "type" : "bytes", > "logicalType" : "decimal", > "precision" : 20, > "scale" : 2 > } ], > "default" : null, > "columnName" : "col1", > "sqlType" : "2" > }], > "tableName" : "decimal_test_avro" > } > {code} > *create an hive external table stored as avro:* > {code:java} > create external table test_decimal_avro > STORED AS AVRO > LOCATION '/tmp/test_decimal' > TBLPROPERTIES ( > 'avro.schema.url'='/tmp/test_decimal.avsc', > 'orc.compress'='SNAPPY'); > {code} > *insert data in avro external table from hive managed table:* > {code:java} > set hive.exec.compress.output=true; > set hive.exec.compress.intermediate=true; > set avro.output.codec=snappy; > insert overwrite table test_decimal_avro select * from test_decimal; > {code} > *successfully reading data from hive avro table through hive cli:* > {code:java} > select * from test_decimal_avro; > OK > 3.12 > {code} > *avro schema from avro created file is ok:* > {code:java} > hadoop jar /avro-tools.jar getschema /tmp/test_decimal/00_0 > { > "type" : "record", > "name" : "decimal_test_avro", > "fields" : [ { > "name" : "col1", > "type" : [ "null", { > "type" : "bytes", > "logicalType" : "decimal", > "precision" : 20, > "scale" : 2 > } ], > "default" : null, > "columnName" : "col1", > "sqlType" : "2" > } ], > "tableName" : "decimal_test_avro" > } > {code} > *read data from avro file with avro-tools {color:#d04437}error{color}, got > {color:#d04437}"\u00018"{color} value instead of the correct one:* > {code:java} > hadoop jar avro-tools.jar tojson /tmp/test_decimal/00_0 > {"col1":{"bytes":"\u00018"}} > {code} > *Read data in a spark dataframe error, got {color:#d04437}[01 38]{color} > and{color:#d04437} 8{color} when converted to string instead of correct > "3.12" value :* > {code:java} > val df = sql.read.avro("/tmp/test_decimal") > df: org.apache.spark.sql.DataFrame = [col1: binary] > scala> df.show() > +---+ > | col1| > +---+ > |[01 38]| > +---+ > scala> df.withColumn("col2", 'col1.cast("String")).select("col2").show() > ++ > |col2| > ++ > | 8| > ++ > {code} > Is this a Hive bug or there is anything else I can do in order to get correct > values in the avro file created by Hive? > Thanks, -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17465) Statistics: Drill-down filters don't reduce row-counts progressively
[ https://issues.apache.org/jira/browse/HIVE-17465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16169469#comment-16169469 ] Vineet Garg commented on HIVE-17465: [~ashutoshc] can you take a look at updated review? > Statistics: Drill-down filters don't reduce row-counts progressively > > > Key: HIVE-17465 > URL: https://issues.apache.org/jira/browse/HIVE-17465 > Project: Hive > Issue Type: Bug > Components: Physical Optimizer, Statistics >Reporter: Gopal V >Assignee: Vineet Garg > Attachments: HIVE-17465.1.patch, HIVE-17465.2.patch, > HIVE-17465.3.patch, HIVE-17465.4.patch, HIVE-17465.5.patch, > HIVE-17465.6.patch, HIVE-17465.7.patch > > > {code} > explain select count(d_date_sk) from date_dim where d_year=2001 ; > explain select count(d_date_sk) from date_dim where d_year=2001 and d_moy = > 9; > explain select count(d_date_sk) from date_dim where d_year=2001 and d_moy = 9 > and d_dom = 21; > {code} > All 3 queries end up with the same row-count estimates after the filter. > {code} > Map Operator Tree: > TableScan > alias: date_dim > filterExpr: (d_year = 2001) (type: boolean) > Statistics: Num rows: 73049 Data size: 82034027 Basic > stats: COMPLETE Column stats: COMPLETE > Filter Operator > predicate: (d_year = 2001) (type: boolean) > Statistics: Num rows: 363 Data size: 4356 Basic stats: > COMPLETE Column stats: COMPLETE > > Map 1 > Map Operator Tree: > TableScan > alias: date_dim > filterExpr: ((d_year = 2001) and (d_moy = 9)) (type: > boolean) > Statistics: Num rows: 73049 Data size: 82034027 Basic > stats: COMPLETE Column stats: COMPLETE > Filter Operator > predicate: ((d_year = 2001) and (d_moy = 9)) (type: > boolean) > Statistics: Num rows: 363 Data size: 5808 Basic stats: > COMPLETE Column stats: COMPLETE > Map 1 > Map Operator Tree: > TableScan > alias: date_dim > filterExpr: ((d_year = 2001) and (d_moy = 9) and (d_dom = > 21)) (type: boolean) > Statistics: Num rows: 73049 Data size: 82034027 Basic > stats: COMPLETE Column stats: COMPLETE > Filter Operator > predicate: ((d_year = 2001) and (d_moy = 9) and (d_dom = > 21)) (type: boolean) > Statistics: Num rows: 363 Data size: 7260 Basic stats: > COMPLETE Column stats: COMPLETE > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17465) Statistics: Drill-down filters don't reduce row-counts progressively
[ https://issues.apache.org/jira/browse/HIVE-17465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-17465: --- Attachment: HIVE-17465.7.patch > Statistics: Drill-down filters don't reduce row-counts progressively > > > Key: HIVE-17465 > URL: https://issues.apache.org/jira/browse/HIVE-17465 > Project: Hive > Issue Type: Bug > Components: Physical Optimizer, Statistics >Reporter: Gopal V >Assignee: Vineet Garg > Attachments: HIVE-17465.1.patch, HIVE-17465.2.patch, > HIVE-17465.3.patch, HIVE-17465.4.patch, HIVE-17465.5.patch, > HIVE-17465.6.patch, HIVE-17465.7.patch > > > {code} > explain select count(d_date_sk) from date_dim where d_year=2001 ; > explain select count(d_date_sk) from date_dim where d_year=2001 and d_moy = > 9; > explain select count(d_date_sk) from date_dim where d_year=2001 and d_moy = 9 > and d_dom = 21; > {code} > All 3 queries end up with the same row-count estimates after the filter. > {code} > Map Operator Tree: > TableScan > alias: date_dim > filterExpr: (d_year = 2001) (type: boolean) > Statistics: Num rows: 73049 Data size: 82034027 Basic > stats: COMPLETE Column stats: COMPLETE > Filter Operator > predicate: (d_year = 2001) (type: boolean) > Statistics: Num rows: 363 Data size: 4356 Basic stats: > COMPLETE Column stats: COMPLETE > > Map 1 > Map Operator Tree: > TableScan > alias: date_dim > filterExpr: ((d_year = 2001) and (d_moy = 9)) (type: > boolean) > Statistics: Num rows: 73049 Data size: 82034027 Basic > stats: COMPLETE Column stats: COMPLETE > Filter Operator > predicate: ((d_year = 2001) and (d_moy = 9)) (type: > boolean) > Statistics: Num rows: 363 Data size: 5808 Basic stats: > COMPLETE Column stats: COMPLETE > Map 1 > Map Operator Tree: > TableScan > alias: date_dim > filterExpr: ((d_year = 2001) and (d_moy = 9) and (d_dom = > 21)) (type: boolean) > Statistics: Num rows: 73049 Data size: 82034027 Basic > stats: COMPLETE Column stats: COMPLETE > Filter Operator > predicate: ((d_year = 2001) and (d_moy = 9) and (d_dom = > 21)) (type: boolean) > Statistics: Num rows: 363 Data size: 7260 Basic stats: > COMPLETE Column stats: COMPLETE > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17465) Statistics: Drill-down filters don't reduce row-counts progressively
[ https://issues.apache.org/jira/browse/HIVE-17465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-17465: --- Status: Patch Available (was: Open) > Statistics: Drill-down filters don't reduce row-counts progressively > > > Key: HIVE-17465 > URL: https://issues.apache.org/jira/browse/HIVE-17465 > Project: Hive > Issue Type: Bug > Components: Physical Optimizer, Statistics >Reporter: Gopal V >Assignee: Vineet Garg > Attachments: HIVE-17465.1.patch, HIVE-17465.2.patch, > HIVE-17465.3.patch, HIVE-17465.4.patch, HIVE-17465.5.patch, > HIVE-17465.6.patch, HIVE-17465.7.patch > > > {code} > explain select count(d_date_sk) from date_dim where d_year=2001 ; > explain select count(d_date_sk) from date_dim where d_year=2001 and d_moy = > 9; > explain select count(d_date_sk) from date_dim where d_year=2001 and d_moy = 9 > and d_dom = 21; > {code} > All 3 queries end up with the same row-count estimates after the filter. > {code} > Map Operator Tree: > TableScan > alias: date_dim > filterExpr: (d_year = 2001) (type: boolean) > Statistics: Num rows: 73049 Data size: 82034027 Basic > stats: COMPLETE Column stats: COMPLETE > Filter Operator > predicate: (d_year = 2001) (type: boolean) > Statistics: Num rows: 363 Data size: 4356 Basic stats: > COMPLETE Column stats: COMPLETE > > Map 1 > Map Operator Tree: > TableScan > alias: date_dim > filterExpr: ((d_year = 2001) and (d_moy = 9)) (type: > boolean) > Statistics: Num rows: 73049 Data size: 82034027 Basic > stats: COMPLETE Column stats: COMPLETE > Filter Operator > predicate: ((d_year = 2001) and (d_moy = 9)) (type: > boolean) > Statistics: Num rows: 363 Data size: 5808 Basic stats: > COMPLETE Column stats: COMPLETE > Map 1 > Map Operator Tree: > TableScan > alias: date_dim > filterExpr: ((d_year = 2001) and (d_moy = 9) and (d_dom = > 21)) (type: boolean) > Statistics: Num rows: 73049 Data size: 82034027 Basic > stats: COMPLETE Column stats: COMPLETE > Filter Operator > predicate: ((d_year = 2001) and (d_moy = 9) and (d_dom = > 21)) (type: boolean) > Statistics: Num rows: 363 Data size: 7260 Basic stats: > COMPLETE Column stats: COMPLETE > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17465) Statistics: Drill-down filters don't reduce row-counts progressively
[ https://issues.apache.org/jira/browse/HIVE-17465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-17465: --- Status: Open (was: Patch Available) > Statistics: Drill-down filters don't reduce row-counts progressively > > > Key: HIVE-17465 > URL: https://issues.apache.org/jira/browse/HIVE-17465 > Project: Hive > Issue Type: Bug > Components: Physical Optimizer, Statistics >Reporter: Gopal V >Assignee: Vineet Garg > Attachments: HIVE-17465.1.patch, HIVE-17465.2.patch, > HIVE-17465.3.patch, HIVE-17465.4.patch, HIVE-17465.5.patch, > HIVE-17465.6.patch, HIVE-17465.7.patch > > > {code} > explain select count(d_date_sk) from date_dim where d_year=2001 ; > explain select count(d_date_sk) from date_dim where d_year=2001 and d_moy = > 9; > explain select count(d_date_sk) from date_dim where d_year=2001 and d_moy = 9 > and d_dom = 21; > {code} > All 3 queries end up with the same row-count estimates after the filter. > {code} > Map Operator Tree: > TableScan > alias: date_dim > filterExpr: (d_year = 2001) (type: boolean) > Statistics: Num rows: 73049 Data size: 82034027 Basic > stats: COMPLETE Column stats: COMPLETE > Filter Operator > predicate: (d_year = 2001) (type: boolean) > Statistics: Num rows: 363 Data size: 4356 Basic stats: > COMPLETE Column stats: COMPLETE > > Map 1 > Map Operator Tree: > TableScan > alias: date_dim > filterExpr: ((d_year = 2001) and (d_moy = 9)) (type: > boolean) > Statistics: Num rows: 73049 Data size: 82034027 Basic > stats: COMPLETE Column stats: COMPLETE > Filter Operator > predicate: ((d_year = 2001) and (d_moy = 9)) (type: > boolean) > Statistics: Num rows: 363 Data size: 5808 Basic stats: > COMPLETE Column stats: COMPLETE > Map 1 > Map Operator Tree: > TableScan > alias: date_dim > filterExpr: ((d_year = 2001) and (d_moy = 9) and (d_dom = > 21)) (type: boolean) > Statistics: Num rows: 73049 Data size: 82034027 Basic > stats: COMPLETE Column stats: COMPLETE > Filter Operator > predicate: ((d_year = 2001) and (d_moy = 9) and (d_dom = > 21)) (type: boolean) > Statistics: Num rows: 363 Data size: 7260 Basic stats: > COMPLETE Column stats: COMPLETE > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (HIVE-17451) Cannot read decimal from avro file created with HIVE
[ https://issues.apache.org/jira/browse/HIVE-17451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16169433#comment-16169433 ] Matt McCline edited comment on HIVE-17451 at 9/17/17 11:38 PM: --- Hexidecimal 0x138 = Decimal 312 (unscaled) and, hexidecimal bytes 01 and 38 are a non-visible control character \001 and ASCII "8" was (Author: mmccline): Hexidecimal 0x138 = Decimal 312 > Cannot read decimal from avro file created with HIVE > > > Key: HIVE-17451 > URL: https://issues.apache.org/jira/browse/HIVE-17451 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 1.1.0 >Reporter: liviu >Assignee: Ganesh Tripathi >Priority: Blocker > > Hi, > When we export decimal data from a hive managed table to a hive avro external > table (as bytes with decimal logicalType) the value from avro file cannot be > read with any other tools (ex: avro-tools, spark, datastage..) > _+Scenario:+_ > *create hive managed table an insert a decimal record:* > {code:java} > create table test_decimal (col1 decimal(20,2)); > insert into table test_decimal values (3.12); > {code} > *create avro schema /tmp/test_decimal.avsc with below content:* > {code:java} > { > "type" : "record", > "name" : "decimal_test_avro", > "fields" : [ { > "name" : "col1", > "type" : [ "null", { > "type" : "bytes", > "logicalType" : "decimal", > "precision" : 20, > "scale" : 2 > } ], > "default" : null, > "columnName" : "col1", > "sqlType" : "2" > }], > "tableName" : "decimal_test_avro" > } > {code} > *create an hive external table stored as avro:* > {code:java} > create external table test_decimal_avro > STORED AS AVRO > LOCATION '/tmp/test_decimal' > TBLPROPERTIES ( > 'avro.schema.url'='/tmp/test_decimal.avsc', > 'orc.compress'='SNAPPY'); > {code} > *insert data in avro external table from hive managed table:* > {code:java} > set hive.exec.compress.output=true; > set hive.exec.compress.intermediate=true; > set avro.output.codec=snappy; > insert overwrite table test_decimal_avro select * from test_decimal; > {code} > *successfully reading data from hive avro table through hive cli:* > {code:java} > select * from test_decimal_avro; > OK > 3.12 > {code} > *avro schema from avro created file is ok:* > {code:java} > hadoop jar /avro-tools.jar getschema /tmp/test_decimal/00_0 > { > "type" : "record", > "name" : "decimal_test_avro", > "fields" : [ { > "name" : "col1", > "type" : [ "null", { > "type" : "bytes", > "logicalType" : "decimal", > "precision" : 20, > "scale" : 2 > } ], > "default" : null, > "columnName" : "col1", > "sqlType" : "2" > } ], > "tableName" : "decimal_test_avro" > } > {code} > *read data from avro file with avro-tools {color:#d04437}error{color}, got > {color:#d04437}"\u00018"{color} value instead of the correct one:* > {code:java} > hadoop jar avro-tools.jar tojson /tmp/test_decimal/00_0 > {"col1":{"bytes":"\u00018"}} > {code} > *Read data in a spark dataframe error, got {color:#d04437}[01 38]{color} > and{color:#d04437} 8{color} when converted to string instead of correct > "3.12" value :* > {code:java} > val df = sql.read.avro("/tmp/test_decimal") > df: org.apache.spark.sql.DataFrame = [col1: binary] > scala> df.show() > +---+ > | col1| > +---+ > |[01 38]| > +---+ > scala> df.withColumn("col2", 'col1.cast("String")).select("col2").show() > ++ > |col2| > ++ > | 8| > ++ > {code} > Is this a Hive bug or there is anything else I can do in order to get correct > values in the avro file created by Hive? > Thanks, -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17422) Skip non-native/temporary tables for all major table/partition related scenarios
[ https://issues.apache.org/jira/browse/HIVE-17422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16169445#comment-16169445 ] Tao Li commented on HIVE-17422: --- Test resut looks good. > Skip non-native/temporary tables for all major table/partition related > scenarios > > > Key: HIVE-17422 > URL: https://issues.apache.org/jira/browse/HIVE-17422 > Project: Hive > Issue Type: Improvement > Components: repl >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17422.1.patch, HIVE-17422.2.patch, > HIVE-17422.3.patch, HIVE-17422.4.patch > > > Currently during incremental dump, the non-native/temporary table info is > partially dumped in metadata file and will be ignored later by the repl load. > We can optimize it by moving the check (whether the table should be exported > or not) earlier so that we don't save any info to dump file for such types of > tables. CreateTableHandler already has this optimization, so we just need to > apply similar logic to other scenarios. > The change is to apply the EximUtil.shouldExportTable check to all scenarios > (e.g. alter table) that calls into the common dump method. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17275) Auto-merge fails on writes of UNION ALL output to ORC file with dynamic partitioning
[ https://issues.apache.org/jira/browse/HIVE-17275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16169442#comment-16169442 ] Lefty Leverenz commented on HIVE-17275: --- This issue needs fix versions. (Will it also be committed to branch-2.3?) > Auto-merge fails on writes of UNION ALL output to ORC file with dynamic > partitioning > > > Key: HIVE-17275 > URL: https://issues.apache.org/jira/browse/HIVE-17275 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 2.2.0 >Reporter: Chris Drome >Assignee: Chris Drome > Attachments: HIVE-17275.2-branch-2.2.patch, > HIVE-17275.2-branch-2.patch, HIVE-17275.2.patch, HIVE-17275-branch-2.2.patch, > HIVE-17275-branch-2.patch, HIVE-17275.patch > > > If dynamic partitioning is used to write the output of UNION or UNION ALL > queries into ORC files with hive.merge.tezfiles=true, the merge step fails as > follows: > {noformat} > 2017-08-08T11:27:19,958 ERROR [e7b1f06d-d632-408a-9dff-f7ae042cd25a main] > SessionState: Vertex failed, vertexName=File Merge, > vertexId=vertex_1502216690354_0001_33_00, diagnostics=[Task failed, > taskId=task_1502216690354_0001_33_00_00, diagnostics=[TaskAttempt 0 > failed, info=[Error: Error while running task ( failure ) : > attempt_1502216690354_0001_33_00_00_0:java.lang.RuntimeException: > java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: > java.io.IOException: Multiple partitions for one merge mapper: > hdfs://localhost:39943/build/ql/test/data/warehouse/partunion1/.hive-staging_hive_2017-08-08_11-27-09_105_286405133968521828-1/-ext-10002/part1=2014/1 > NOT EQUAL TO > hdfs://localhost:39943/build/ql/test/data/warehouse/partunion1/.hive-staging_hive_2017-08-08_11-27-09_105_286405133968521828-1/-ext-10002/part1=2014/2 > at > org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:211) > at > org.apache.hadoop.hive.ql.exec.tez.MergeFileTezProcessor.run(MergeFileTezProcessor.java:42) > at > org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:370) > at > org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73) > at > org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1807) > at > org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61) > at > org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37) > at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.lang.RuntimeException: > org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException: > Multiple partitions for one merge mapper: > hdfs://localhost:39943/build/ql/test/data/warehouse/partunion1/.hive-staging_hive_2017-08-08_11-27-09_105_286405133968521828-1/-ext-10002/part1=2014/1 > NOT EQUAL TO > hdfs://localhost:39943/build/ql/test/data/warehouse/partunion1/.hive-staging_hive_2017-08-08_11-27-09_105_286405133968521828-1/-ext-10002/part1=2014/2 > at > org.apache.hadoop.hive.ql.exec.tez.MergeFileRecordProcessor.processRow(MergeFileRecordProcessor.java:225) > at > org.apache.hadoop.hive.ql.exec.tez.MergeFileRecordProcessor.run(MergeFileRecordProcessor.java:154) > at > org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:185) > ... 14 more > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: > java.io.IOException: Multiple partitions for one merge mapper: > hdfs://localhost:39943/build/ql/test/data/warehouse/partunion1/.hive-staging_hive_2017-08-08_11-27-09_105_286405133968521828-1/-ext-10002/part1=2014/1 > NOT EQUAL TO > hdfs://localhost:39943/build/ql/test/data/warehouse/partunion1/.hive-staging_hive_2017-08-08_11-27-09_105_286405133968521828-1/-ext-10002/part1=2014/2 > at > org.apache.hadoop.hive.ql.exec.OrcFileMergeOperator.processKeyValuePairs(OrcFileMergeOperator.java:169) > at > org.apache.hadoop.hive.ql.exec.OrcFileMergeOperator.process(OrcFileMergeOperator.java:72) > at > org.apache.hadoop.hive.ql.exec.tez.MergeFileRecordProcessor.processRow(MergeFileRecordProcessor.java:216) > ... 16 more > Caused by: java.io.
[jira] [Commented] (HIVE-17422) Skip non-native/temporary tables for all major table/partition related scenarios
[ https://issues.apache.org/jira/browse/HIVE-17422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16169438#comment-16169438 ] Hive QA commented on HIVE-17422: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12887565/HIVE-17422.4.patch {color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 12 failed/errored test(s), 11041 tests executed *Failed tests:* {noformat} TestAccumuloCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=230) TestDummy - did not produce a TEST-*.xml file (likely timed out) (batchId=230) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[create_view] (batchId=39) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_mask_hash] (batchId=28) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[unionDistinct_1] (batchId=143) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats] (batchId=156) org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[drop_table_failure2] (batchId=89) org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] (batchId=234) org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23] (batchId=234) org.apache.hadoop.hive.ql.TestAcidOnTez.testCtasTezUnion (batchId=215) org.apache.hadoop.hive.ql.TestAcidOnTez.testNonStandardConversion01 (batchId=215) org.apache.hive.jdbc.TestJdbcWithMiniHS2.testHttpRetryOnServerIdleTimeout (batchId=227) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6854/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6854/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6854/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 12 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12887565 - PreCommit-HIVE-Build > Skip non-native/temporary tables for all major table/partition related > scenarios > > > Key: HIVE-17422 > URL: https://issues.apache.org/jira/browse/HIVE-17422 > Project: Hive > Issue Type: Improvement > Components: repl >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17422.1.patch, HIVE-17422.2.patch, > HIVE-17422.3.patch, HIVE-17422.4.patch > > > Currently during incremental dump, the non-native/temporary table info is > partially dumped in metadata file and will be ignored later by the repl load. > We can optimize it by moving the check (whether the table should be exported > or not) earlier so that we don't save any info to dump file for such types of > tables. CreateTableHandler already has this optimization, so we just need to > apply similar logic to other scenarios. > The change is to apply the EximUtil.shouldExportTable check to all scenarios > (e.g. alter table) that calls into the common dump method. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17522) cleanup old 'repl dump' dirs
[ https://issues.apache.org/jira/browse/HIVE-17522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16169435#comment-16169435 ] Lefty Leverenz commented on HIVE-17522: --- Doc note: This adds *hive.repl.dumpdir.clean.freq* and *hive.repl.dumpdir.ttl* to HiveConf.java, so they need to be documented in the wiki for release 3.0.0. * [Configuration Properties -- Replication | https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-Replication] Added a TODOC3.0 label. Question: What does TTL mean? > cleanup old 'repl dump' dirs > > > Key: HIVE-17522 > URL: https://issues.apache.org/jira/browse/HIVE-17522 > Project: Hive > Issue Type: Bug > Components: repl >Reporter: Tao Li >Assignee: Tao Li > Labels: TODOC3.0 > Fix For: 3.0.0 > > Attachments: HIVE-17522.1.patch > > > We want to clean up the old dump dirs to save space and reduce scan time when > needed. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17451) Cannot read decimal from avro file created with HIVE
[ https://issues.apache.org/jira/browse/HIVE-17451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16169433#comment-16169433 ] Matt McCline commented on HIVE-17451: - Hexidecimal 0x138 = Decimal 312 > Cannot read decimal from avro file created with HIVE > > > Key: HIVE-17451 > URL: https://issues.apache.org/jira/browse/HIVE-17451 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 1.1.0 >Reporter: liviu >Assignee: Ganesh Tripathi >Priority: Blocker > > Hi, > When we export decimal data from a hive managed table to a hive avro external > table (as bytes with decimal logicalType) the value from avro file cannot be > read with any other tools (ex: avro-tools, spark, datastage..) > _+Scenario:+_ > *create hive managed table an insert a decimal record:* > {code:java} > create table test_decimal (col1 decimal(20,2)); > insert into table test_decimal values (3.12); > {code} > *create avro schema /tmp/test_decimal.avsc with below content:* > {code:java} > { > "type" : "record", > "name" : "decimal_test_avro", > "fields" : [ { > "name" : "col1", > "type" : [ "null", { > "type" : "bytes", > "logicalType" : "decimal", > "precision" : 20, > "scale" : 2 > } ], > "default" : null, > "columnName" : "col1", > "sqlType" : "2" > }], > "tableName" : "decimal_test_avro" > } > {code} > *create an hive external table stored as avro:* > {code:java} > create external table test_decimal_avro > STORED AS AVRO > LOCATION '/tmp/test_decimal' > TBLPROPERTIES ( > 'avro.schema.url'='/tmp/test_decimal.avsc', > 'orc.compress'='SNAPPY'); > {code} > *insert data in avro external table from hive managed table:* > {code:java} > set hive.exec.compress.output=true; > set hive.exec.compress.intermediate=true; > set avro.output.codec=snappy; > insert overwrite table test_decimal_avro select * from test_decimal; > {code} > *successfully reading data from hive avro table through hive cli:* > {code:java} > select * from test_decimal_avro; > OK > 3.12 > {code} > *avro schema from avro created file is ok:* > {code:java} > hadoop jar /avro-tools.jar getschema /tmp/test_decimal/00_0 > { > "type" : "record", > "name" : "decimal_test_avro", > "fields" : [ { > "name" : "col1", > "type" : [ "null", { > "type" : "bytes", > "logicalType" : "decimal", > "precision" : 20, > "scale" : 2 > } ], > "default" : null, > "columnName" : "col1", > "sqlType" : "2" > } ], > "tableName" : "decimal_test_avro" > } > {code} > *read data from avro file with avro-tools {color:#d04437}error{color}, got > {color:#d04437}"\u00018"{color} value instead of the correct one:* > {code:java} > hadoop jar avro-tools.jar tojson /tmp/test_decimal/00_0 > {"col1":{"bytes":"\u00018"}} > {code} > *Read data in a spark dataframe error, got {color:#d04437}[01 38]{color} > and{color:#d04437} 8{color} when converted to string instead of correct > "3.12" value :* > {code:java} > val df = sql.read.avro("/tmp/test_decimal") > df: org.apache.spark.sql.DataFrame = [col1: binary] > scala> df.show() > +---+ > | col1| > +---+ > |[01 38]| > +---+ > scala> df.withColumn("col2", 'col1.cast("String")).select("col2").show() > ++ > |col2| > ++ > | 8| > ++ > {code} > Is this a Hive bug or there is anything else I can do in order to get correct > values in the avro file created by Hive? > Thanks, -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17522) cleanup old 'repl dump' dirs
[ https://issues.apache.org/jira/browse/HIVE-17522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lefty Leverenz updated HIVE-17522: -- Labels: TODOC3.0 (was: ) > cleanup old 'repl dump' dirs > > > Key: HIVE-17522 > URL: https://issues.apache.org/jira/browse/HIVE-17522 > Project: Hive > Issue Type: Bug > Components: repl >Reporter: Tao Li >Assignee: Tao Li > Labels: TODOC3.0 > Fix For: 3.0.0 > > Attachments: HIVE-17522.1.patch > > > We want to clean up the old dump dirs to save space and reduce scan time when > needed. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17422) Skip non-native/temporary tables for all major table/partition related scenarios
[ https://issues.apache.org/jira/browse/HIVE-17422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16169407#comment-16169407 ] Hive QA commented on HIVE-17422: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12887565/HIVE-17422.4.patch {color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 12 failed/errored test(s), 11041 tests executed *Failed tests:* {noformat} TestAccumuloCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=230) TestDummy - did not produce a TEST-*.xml file (likely timed out) (batchId=230) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[create_view] (batchId=39) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_mask_hash] (batchId=28) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[unionDistinct_1] (batchId=143) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats] (batchId=156) org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2] (batchId=100) org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[drop_table_failure2] (batchId=89) org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] (batchId=234) org.apache.hadoop.hive.ql.TestAcidOnTez.testCtasTezUnion (batchId=215) org.apache.hadoop.hive.ql.TestAcidOnTez.testNonStandardConversion01 (batchId=215) org.apache.hive.jdbc.TestJdbcWithMiniHS2.testHttpRetryOnServerIdleTimeout (batchId=227) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6853/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6853/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6853/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 12 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12887565 - PreCommit-HIVE-Build > Skip non-native/temporary tables for all major table/partition related > scenarios > > > Key: HIVE-17422 > URL: https://issues.apache.org/jira/browse/HIVE-17422 > Project: Hive > Issue Type: Improvement > Components: repl >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17422.1.patch, HIVE-17422.2.patch, > HIVE-17422.3.patch, HIVE-17422.4.patch > > > Currently during incremental dump, the non-native/temporary table info is > partially dumped in metadata file and will be ignored later by the repl load. > We can optimize it by moving the check (whether the table should be exported > or not) earlier so that we don't save any info to dump file for such types of > tables. CreateTableHandler already has this optimization, so we just need to > apply similar logic to other scenarios. > The change is to apply the EximUtil.shouldExportTable check to all scenarios > (e.g. alter table) that calls into the common dump method. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17493) Improve PKFK cardinality estimation in Physical planning
[ https://issues.apache.org/jira/browse/HIVE-17493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-17493: --- Resolution: Fixed Status: Resolved (was: Patch Available) Pushed to master > Improve PKFK cardinality estimation in Physical planning > > > Key: HIVE-17493 > URL: https://issues.apache.org/jira/browse/HIVE-17493 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Reporter: Vineet Garg >Assignee: Vineet Garg > Attachments: HIVE-17493.1.patch, HIVE-17493.2.patch, > HIVE-17493.3.patch, HIVE-17493.4.patch > > > Cardinality estimation of a join, after PK-FK relation has been ascertained, > could be improved if parent of the join operator is LEFT outer or RIGHT outer > join. > Currently estimation is done by estimating reduction of rows occurred on PK > side, then multiplying the reduction to FK side row count. This estimation of > reduction currently doesn't distinguish b/w INNER or OUTER joins. This could > be improved to handle outer joins better. > TPC-DS query45 is impacted by this. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17422) Skip non-native/temporary tables for all major table/partition related scenarios
[ https://issues.apache.org/jira/browse/HIVE-17422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17422: -- Attachment: HIVE-17422.4.patch > Skip non-native/temporary tables for all major table/partition related > scenarios > > > Key: HIVE-17422 > URL: https://issues.apache.org/jira/browse/HIVE-17422 > Project: Hive > Issue Type: Improvement > Components: repl >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17422.1.patch, HIVE-17422.2.patch, > HIVE-17422.3.patch, HIVE-17422.4.patch > > > Currently during incremental dump, the non-native/temporary table info is > partially dumped in metadata file and will be ignored later by the repl load. > We can optimize it by moving the check (whether the table should be exported > or not) earlier so that we don't save any info to dump file for such types of > tables. CreateTableHandler already has this optimization, so we just need to > apply similar logic to other scenarios. > The change is to apply the EximUtil.shouldExportTable check to all scenarios > (e.g. alter table) that calls into the common dump method. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17422) Skip non-native/temporary tables for all major table/partition related scenarios
[ https://issues.apache.org/jira/browse/HIVE-17422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17422: -- Status: Patch Available (was: Open) > Skip non-native/temporary tables for all major table/partition related > scenarios > > > Key: HIVE-17422 > URL: https://issues.apache.org/jira/browse/HIVE-17422 > Project: Hive > Issue Type: Improvement > Components: repl >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17422.1.patch, HIVE-17422.2.patch, > HIVE-17422.3.patch, HIVE-17422.4.patch > > > Currently during incremental dump, the non-native/temporary table info is > partially dumped in metadata file and will be ignored later by the repl load. > We can optimize it by moving the check (whether the table should be exported > or not) earlier so that we don't save any info to dump file for such types of > tables. CreateTableHandler already has this optimization, so we just need to > apply similar logic to other scenarios. > The change is to apply the EximUtil.shouldExportTable check to all scenarios > (e.g. alter table) that calls into the common dump method. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17422) Skip non-native/temporary tables for all major table/partition related scenarios
[ https://issues.apache.org/jira/browse/HIVE-17422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17422: -- Attachment: (was: HIVE-17422.4.patch) > Skip non-native/temporary tables for all major table/partition related > scenarios > > > Key: HIVE-17422 > URL: https://issues.apache.org/jira/browse/HIVE-17422 > Project: Hive > Issue Type: Improvement > Components: repl >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17422.1.patch, HIVE-17422.2.patch, > HIVE-17422.3.patch > > > Currently during incremental dump, the non-native/temporary table info is > partially dumped in metadata file and will be ignored later by the repl load. > We can optimize it by moving the check (whether the table should be exported > or not) earlier so that we don't save any info to dump file for such types of > tables. CreateTableHandler already has this optimization, so we just need to > apply similar logic to other scenarios. > The change is to apply the EximUtil.shouldExportTable check to all scenarios > (e.g. alter table) that calls into the common dump method. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17422) Skip non-native/temporary tables for all major table/partition related scenarios
[ https://issues.apache.org/jira/browse/HIVE-17422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17422: -- Status: Open (was: Patch Available) > Skip non-native/temporary tables for all major table/partition related > scenarios > > > Key: HIVE-17422 > URL: https://issues.apache.org/jira/browse/HIVE-17422 > Project: Hive > Issue Type: Improvement > Components: repl >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17422.1.patch, HIVE-17422.2.patch, > HIVE-17422.3.patch > > > Currently during incremental dump, the non-native/temporary table info is > partially dumped in metadata file and will be ignored later by the repl load. > We can optimize it by moving the check (whether the table should be exported > or not) earlier so that we don't save any info to dump file for such types of > tables. CreateTableHandler already has this optimization, so we just need to > apply similar logic to other scenarios. > The change is to apply the EximUtil.shouldExportTable check to all scenarios > (e.g. alter table) that calls into the common dump method. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17422) Skip non-native/temporary tables for all major table/partition related scenarios
[ https://issues.apache.org/jira/browse/HIVE-17422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17422: -- Status: Patch Available (was: Open) > Skip non-native/temporary tables for all major table/partition related > scenarios > > > Key: HIVE-17422 > URL: https://issues.apache.org/jira/browse/HIVE-17422 > Project: Hive > Issue Type: Improvement > Components: repl >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17422.1.patch, HIVE-17422.2.patch, > HIVE-17422.3.patch, HIVE-17422.4.patch > > > Currently during incremental dump, the non-native/temporary table info is > partially dumped in metadata file and will be ignored later by the repl load. > We can optimize it by moving the check (whether the table should be exported > or not) earlier so that we don't save any info to dump file for such types of > tables. CreateTableHandler already has this optimization, so we just need to > apply similar logic to other scenarios. > The change is to apply the EximUtil.shouldExportTable check to all scenarios > (e.g. alter table) that calls into the common dump method. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17422) Skip non-native/temporary tables for all major table/partition related scenarios
[ https://issues.apache.org/jira/browse/HIVE-17422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17422: -- Attachment: HIVE-17422.4.patch > Skip non-native/temporary tables for all major table/partition related > scenarios > > > Key: HIVE-17422 > URL: https://issues.apache.org/jira/browse/HIVE-17422 > Project: Hive > Issue Type: Improvement > Components: repl >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17422.1.patch, HIVE-17422.2.patch, > HIVE-17422.3.patch, HIVE-17422.4.patch > > > Currently during incremental dump, the non-native/temporary table info is > partially dumped in metadata file and will be ignored later by the repl load. > We can optimize it by moving the check (whether the table should be exported > or not) earlier so that we don't save any info to dump file for such types of > tables. CreateTableHandler already has this optimization, so we just need to > apply similar logic to other scenarios. > The change is to apply the EximUtil.shouldExportTable check to all scenarios > (e.g. alter table) that calls into the common dump method. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17422) Skip non-native/temporary tables for all major table/partition related scenarios
[ https://issues.apache.org/jira/browse/HIVE-17422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17422: -- Status: Open (was: Patch Available) > Skip non-native/temporary tables for all major table/partition related > scenarios > > > Key: HIVE-17422 > URL: https://issues.apache.org/jira/browse/HIVE-17422 > Project: Hive > Issue Type: Improvement > Components: repl >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17422.1.patch, HIVE-17422.2.patch, > HIVE-17422.3.patch, HIVE-17422.4.patch > > > Currently during incremental dump, the non-native/temporary table info is > partially dumped in metadata file and will be ignored later by the repl load. > We can optimize it by moving the check (whether the table should be exported > or not) earlier so that we don't save any info to dump file for such types of > tables. CreateTableHandler already has this optimization, so we just need to > apply similar logic to other scenarios. > The change is to apply the EximUtil.shouldExportTable check to all scenarios > (e.g. alter table) that calls into the common dump method. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-6590) Hive does not work properly with boolean partition columns (wrong results and inserts to incorrect HDFS path)
[ https://issues.apache.org/jira/browse/HIVE-6590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16169319#comment-16169319 ] Hive QA commented on HIVE-6590: --- Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12885121/HIVE-6590.5.patch {color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 10 failed/errored test(s), 11037 tests executed *Failed tests:* {noformat} TestAccumuloCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=230) TestDummy - did not produce a TEST-*.xml file (likely timed out) (batchId=230) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_rp_gby2_map_multi_distinct] (batchId=79) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[create_view] (batchId=39) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_mask_hash] (batchId=28) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats] (batchId=156) org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[drop_table_failure2] (batchId=89) org.apache.hadoop.hive.cli.TestSparkNegativeCliDriver.org.apache.hadoop.hive.cli.TestSparkNegativeCliDriver (batchId=241) org.apache.hadoop.hive.ql.TestAcidOnTez.testCtasTezUnion (batchId=215) org.apache.hadoop.hive.ql.TestAcidOnTez.testNonStandardConversion01 (batchId=215) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6851/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6851/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6851/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 10 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12885121 - PreCommit-HIVE-Build > Hive does not work properly with boolean partition columns (wrong results and > inserts to incorrect HDFS path) > - > > Key: HIVE-6590 > URL: https://issues.apache.org/jira/browse/HIVE-6590 > Project: Hive > Issue Type: Bug > Components: Database/Schema, Metastore >Affects Versions: 0.10.0 >Reporter: Lenni Kuff >Assignee: Zoltan Haindrich > Attachments: HIVE-6590.1.patch, HIVE-6590.2.patch, HIVE-6590.3.patch, > HIVE-6590.4.patch, HIVE-6590.5.patch, HIVE-6590.5.patch > > > Hive does not work properly with boolean partition columns. Queries return > wrong results and also insert to incorrect HDFS paths. > {code} > create table bool_part(int_col int) partitioned by(bool_col boolean); > # This works, creating 3 unique partitions! > ALTER TABLE bool_table ADD PARTITION (bool_col=FALSE); > ALTER TABLE bool_table ADD PARTITION (bool_col=false); > ALTER TABLE bool_table ADD PARTITION (bool_col=False); > {code} > The first problem is that Hive cannot filter on a bool partition key column. > "select * from bool_part" returns the correct results, but if you apply a > filter on the bool partition key column hive won't return any results. > The second problem is that Hive seems to just call "toString()" on the > boolean literal value. This means you can end up with multiple partitions > (FALSE, false, FaLSE, etc) mapping to the literal value 'FALSE'. For example, > if you can add three partition in have for the same logic value "false" doing: > ALTER TABLE bool_table ADD PARTITION (bool_col=FALSE) -> > /test-warehouse/bool_table/bool_col=FALSE/ > ALTER TABLE bool_table ADD PARTITION (bool_col=false) -> > /test-warehouse/bool_table/bool_col=false/ > ALTER TABLE bool_table ADD PARTITION (bool_col=False) -> > /test-warehouse/bool_table/bool_col=False/ -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-16355) Service: embedded mode should only be available if service is loaded onto the classpath
[ https://issues.apache.org/jira/browse/HIVE-16355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16169295#comment-16169295 ] Hive QA commented on HIVE-16355: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12885123/HIVE-16355.5.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 11 failed/errored test(s), 11035 tests executed *Failed tests:* {noformat} TestAccumuloCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=230) TestDummy - did not produce a TEST-*.xml file (likely timed out) (batchId=230) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[create_view] (batchId=39) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_mask_hash] (batchId=28) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats] (batchId=156) org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[drop_table_failure2] (batchId=89) org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23] (batchId=234) org.apache.hadoop.hive.cli.TestSparkNegativeCliDriver.org.apache.hadoop.hive.cli.TestSparkNegativeCliDriver (batchId=241) org.apache.hadoop.hive.ql.TestAcidOnTez.testCtasTezUnion (batchId=215) org.apache.hadoop.hive.ql.TestAcidOnTez.testNonStandardConversion01 (batchId=215) org.apache.hive.jdbc.TestJdbcWithMiniHS2.testHttpRetryOnServerIdleTimeout (batchId=227) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6850/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6850/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6850/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 11 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12885123 - PreCommit-HIVE-Build > Service: embedded mode should only be available if service is loaded onto the > classpath > --- > > Key: HIVE-16355 > URL: https://issues.apache.org/jira/browse/HIVE-16355 > Project: Hive > Issue Type: Sub-task > Components: Metastore, Server Infrastructure >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich > Attachments: HIVE-16355.1.patch, HIVE-16355.2.patch, > HIVE-16355.2.patch, HIVE-16355.3.patch, HIVE-16355.4.patch, > HIVE-16355.4.patch, HIVE-16355.5.patch > > > I would like to relax the hard reference to > {{EmbeddedThriftBinaryCLIService}} to be only used in case {{service}} module > is loaded onto the classpath. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17519) Transpose column stats display
[ https://issues.apache.org/jira/browse/HIVE-17519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16169270#comment-16169270 ] Hive QA commented on HIVE-17519: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12887328/HIVE-17519.01.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 392 failed/errored test(s), 11041 tests executed *Failed tests:* {noformat} TestAccumuloCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=230) TestDummy - did not produce a TEST-*.xml file (likely timed out) (batchId=230) org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[escape_comments] (batchId=239) org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_12] (batchId=239) org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_13] (batchId=239) org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_addpartition_blobstore_to_blobstore] (batchId=242) org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_addpartition_blobstore_to_local] (batchId=242) org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_addpartition_blobstore_to_warehouse] (batchId=242) org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_addpartition_local_to_blobstore] (batchId=242) org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_blobstore_to_blobstore] (batchId=242) org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_blobstore_to_local] (batchId=242) org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_blobstore_to_warehouse] (batchId=242) org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_local_to_blobstore] (batchId=242) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_table_stats] (batchId=51) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter2] (batchId=9) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter3] (batchId=20) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter5] (batchId=40) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alterColumnStatsPart] (batchId=82) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alterColumnStats] (batchId=53) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_file_format] (batchId=56) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_merge_2] (batchId=39) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_merge_2_orc] (batchId=72) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_merge_stats] (batchId=58) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_numbuckets_partitioned_table2_h23] (batchId=14) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_numbuckets_partitioned_table_h23] (batchId=66) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_partition_change_col] (batchId=24) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_partition_clusterby_sortby] (batchId=38) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_partition_coltype] (batchId=25) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_partition_format_loc] (batchId=51) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_partition_onto_nocurrent_db] (batchId=84) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_partition_update_status] (batchId=86) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_skewed_table] (batchId=35) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_table_add_partition] (batchId=17) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_table_cascade] (batchId=84) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_table_column_stats] (batchId=62) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_table_not_sorted] (batchId=61) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_table_serde2] (batchId=25) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_table_serde] (batchId=7) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_table_stats_status] (batchId=52) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_table_update_status] (batchId=76) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_table_update_status_disable_bitvector] (batchId=76) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_view_as_select] (batchId=9) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[analyze_table_null_partition] (batchId=78) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[analyze_tbl_date] (batchId=33) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[analyze_tbl_part] (batchId=47) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[authorization_index] (batchId=13) org.apache.hadoop.hive.cli.T
[jira] [Commented] (HIVE-17541) Move testing related methods from MetaStoreUtils to some testing related utility
[ https://issues.apache.org/jira/browse/HIVE-17541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16169262#comment-16169262 ] Hive QA commented on HIVE-17541: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12887348/HIVE-17541.01.patch {color:red}ERROR:{color} -1 due to build exiting with an error Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6848/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6848/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6848/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Tests exited with: NonZeroExitCodeException Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ date '+%Y-%m-%d %T.%3N' 2017-09-17 10:40:23.402 + [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]] + export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + export PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m ' + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m ' + export 'MAVEN_OPTS=-Xmx1g ' + MAVEN_OPTS='-Xmx1g ' + cd /data/hiveptest/working/ + tee /data/hiveptest/logs/PreCommit-HIVE-Build-6848/source-prep.txt + [[ false == \t\r\u\e ]] + mkdir -p maven ivy + [[ git = \s\v\n ]] + [[ git = \g\i\t ]] + [[ -z master ]] + [[ -d apache-github-source-source ]] + [[ ! -d apache-github-source-source/.git ]] + [[ ! -d apache-github-source-source ]] + date '+%Y-%m-%d %T.%3N' 2017-09-17 10:40:23.404 + cd apache-github-source-source + git fetch origin + git reset --hard HEAD HEAD is now at f772ba3 HIVE-17013: Delete request with a subquery based on select over a view (Eugene Koifman, reviewed by Ashutosh Chauhan) + git clean -f -d Removing ql/src/java/org/apache/hadoop/hive/ql/Context.java.orig Removing ql/src/java/org/apache/hadoop/hive/ql/optimizer/QueryPlanPostProcessor.java Removing standalone-metastore/src/gen/org/ + git checkout master Already on 'master' Your branch is up-to-date with 'origin/master'. + git reset --hard origin/master HEAD is now at f772ba3 HIVE-17013: Delete request with a subquery based on select over a view (Eugene Koifman, reviewed by Ashutosh Chauhan) + git merge --ff-only origin/master Already up-to-date. + date '+%Y-%m-%d %T.%3N' 2017-09-17 10:40:28.744 + patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh + patchFilePath=/data/hiveptest/working/scratch/build.patch + [[ -f /data/hiveptest/working/scratch/build.patch ]] + chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh + /data/hiveptest/working/scratch/smart-apply-patch.sh /data/hiveptest/working/scratch/build.patch Going to apply patch with: patch -p0 patching file hcatalog/core/src/test/java/org/apache/hive/hcatalog/cli/TestPermsGrp.java patching file hcatalog/core/src/test/java/org/apache/hive/hcatalog/mapreduce/TestHCatMultiOutputFormat.java patching file hcatalog/core/src/test/java/org/apache/hive/hcatalog/mapreduce/TestHCatPartitionPublish.java patching file hcatalog/webhcat/java-client/pom.xml patching file hcatalog/webhcat/java-client/src/test/java/org/apache/hive/hcatalog/api/TestHCatClient.java patching file hcatalog/webhcat/svr/src/test/java/org/apache/hive/hcatalog/templeton/TestWebHCatE2e.java patching file itests/hive-unit-hadoop2/src/test/java/org/apache/hadoop/hive/metastore/security/TestHadoopAuthBridge23.java patching file itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/AbstractTestAuthorizationApiAuthorizer.java patching file itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/TestFilterHooks.java patching file itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/TestHiveMetaStoreWithEnvironmentContext.java patching file itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/TestMarkPartitionRemote.java patching file itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/TestMetaStoreAuthorization.java patching file itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/TestMetaStoreEndFunctionListener.java patching file itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/TestMetaStoreEventListener.java patching file itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/TestMetaStoreEventListenerOnlyOnCommit.java patching file itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/TestMetaStoreInitListener.java patching file itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/TestMetaStoreListenersError.java patching file itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/TestMetaStoreMetri
[jira] [Commented] (HIVE-15899) check CTAS over acid table
[ https://issues.apache.org/jira/browse/HIVE-15899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16169243#comment-16169243 ] Hive QA commented on HIVE-15899: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12887533/HIVE-15899.10.patch {color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 11 failed/errored test(s), 11048 tests executed *Failed tests:* {noformat} TestAccumuloCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=230) TestDummy - did not produce a TEST-*.xml file (likely timed out) (batchId=230) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[create_view] (batchId=39) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_mask_hash] (batchId=28) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[unionDistinct_1] (batchId=143) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats] (batchId=156) org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2] (batchId=100) org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[drop_table_failure2] (batchId=89) org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] (batchId=234) org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23] (batchId=234) org.apache.hadoop.hive.ql.TestAcidOnTez.testNonStandardConversion02 (batchId=215) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6847/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6847/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6847/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 11 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12887533 - PreCommit-HIVE-Build > check CTAS over acid table > --- > > Key: HIVE-15899 > URL: https://issues.apache.org/jira/browse/HIVE-15899 > Project: Hive > Issue Type: Sub-task >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Attachments: HIVE-15899.01.patch, HIVE-15899.02.patch, > HIVE-15899.03.patch, HIVE-15899.04.patch, HIVE-15899.05.patch, > HIVE-15899.07.patch, HIVE-15899.08.patch, HIVE-15899.09.patch, > HIVE-15899.10.patch > > > need to add a test to check if create table as works correctly with acid > tables -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17496) Bootstrap repl is not cleaning up staging dirs
[ https://issues.apache.org/jira/browse/HIVE-17496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16169223#comment-16169223 ] Hive QA commented on HIVE-17496: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12887403/HIVE-17496.4.patch {color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 11 failed/errored test(s), 11042 tests executed *Failed tests:* {noformat} TestAccumuloCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=230) TestDummy - did not produce a TEST-*.xml file (likely timed out) (batchId=230) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[create_view] (batchId=39) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join_literals] (batchId=84) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_mask_hash] (batchId=28) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats] (batchId=156) org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2] (batchId=100) org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[drop_table_failure2] (batchId=89) org.apache.hadoop.hive.ql.TestAcidOnTez.testCtasTezUnion (batchId=215) org.apache.hadoop.hive.ql.TestAcidOnTez.testNonStandardConversion01 (batchId=215) org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testDeleteStagingDir (batchId=218) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6846/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6846/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6846/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 11 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12887403 - PreCommit-HIVE-Build > Bootstrap repl is not cleaning up staging dirs > -- > > Key: HIVE-17496 > URL: https://issues.apache.org/jira/browse/HIVE-17496 > Project: Hive > Issue Type: Bug > Components: repl >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17496.1.patch, HIVE-17496.2.patch, > HIVE-17496.3.patch, HIVE-17496.4.patch > > > This will put more pressure on the HDFS file limit. -- This message was sent by Atlassian JIRA (v6.4.14#64029)