[jira] [Updated] (HIVE-21022) Fix remote metastore tests which use ZooKeeper
[ https://issues.apache.org/jira/browse/HIVE-21022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Bapat updated HIVE-21022: -- Status: In Progress (was: Patch Available) > Fix remote metastore tests which use ZooKeeper > -- > > Key: HIVE-21022 > URL: https://issues.apache.org/jira/browse/HIVE-21022 > Project: Hive > Issue Type: Bug > Components: Tests >Affects Versions: 4.0.0 >Reporter: Ashutosh Bapat >Assignee: Ashutosh Bapat >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21022.01, HIVE-21022.01, HIVE-21022.01, > HIVE-21022.02.patch > > > Per [~vgarg]'s comment on HIVE-20794 at > https://issues.apache.org/jira/browse/HIVE-20794?focusedCommentId=16714093=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16714093, > the remote metatstore tests using ZooKeeper are flaky. They are failing with > error "Got exception: org.apache.zookeeper.KeeperException$NoNodeException > KeeperErrorCode = NoNode for /hs2mszktest". > Both of these tests are using the same root namespace and hence the reason > for this failure could be that the root namespace becomes unavailable to one > test when the other drops it. The drop seems to be happening automatically > through TestingServer code. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21030) Add credential store env properties redaction in JobConf
[ https://issues.apache.org/jira/browse/HIVE-21030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko updated HIVE-21030: -- Attachment: HIVE-21030.2.patch > Add credential store env properties redaction in JobConf > > > Key: HIVE-21030 > URL: https://issues.apache.org/jira/browse/HIVE-21030 > Project: Hive > Issue Type: Bug >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > Attachments: HIVE-21030.1.patch, HIVE-21030.2.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21022) Fix remote metastore tests which use ZooKeeper
[
https://issues.apache.org/jira/browse/HIVE-21022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16717615#comment-16717615
]
Hive QA commented on HIVE-21022:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
30s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
22s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
54s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
15s{color} | {color:blue} standalone-metastore/metastore-common in master has
29 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
5s{color} | {color:blue} standalone-metastore/metastore-server in master has
188 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
7s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 21m 11s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15261/dev-support/hive-personality.sh
|
| git revision | master / 9925eb1 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: standalone-metastore/metastore-common
standalone-metastore/metastore-server U: standalone-metastore |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15261/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Fix remote metastore tests which use ZooKeeper
> --
>
> Key: HIVE-21022
> URL: https://issues.apache.org/jira/browse/HIVE-21022
> Project: Hive
> Issue Type: Bug
> Components: Tests
>Affects Versions: 4.0.0
>Reporter: Ashutosh Bapat
>Assignee: Ashutosh Bapat
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-21022.01, HIVE-21022.01, HIVE-21022.01,
> HIVE-21022.02, HIVE-21022.02.patch
>
>
> Per [~vgarg]'s comment on HIVE-20794 at
> https://issues.apache.org/jira/browse/HIVE-20794?focusedCommentId=16714093=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16714093,
> the remote metatstore tests using ZooKeeper are flaky. They are failing with
> error "Got exception: org.apache.zookeeper.KeeperException$NoNodeException
> KeeperErrorCode = NoNode for /hs2mszktest".
> Both of these tests are using the same root namespace and hence the reason
> for this failure could be that the root namespace becomes
[jira] [Commented] (HIVE-21030) Add credential store env properties redaction in JobConf
[
https://issues.apache.org/jira/browse/HIVE-21030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16717362#comment-16717362
]
Hive QA commented on HIVE-21030:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12951361/HIVE-21030.1.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15259/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15259/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15259/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL
https://issues.apache.org/jira/secure/attachment/12951361/HIVE-21030.1.patch
was found in seen patch url's cache and a test was probably run already on it.
Aborting...
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12951361 - PreCommit-HIVE-Build
> Add credential store env properties redaction in JobConf
>
>
> Key: HIVE-21030
> URL: https://issues.apache.org/jira/browse/HIVE-21030
> Project: Hive
> Issue Type: Bug
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-21030.1.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20733) GenericUDFOPEqualNS may not use = in plan descriptions
[
https://issues.apache.org/jira/browse/HIVE-20733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16717423#comment-16717423
]
Hive QA commented on HIVE-20733:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
42s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
43s{color} | {color:blue} ql in master has 2311 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} ql: The patch generated 0 new + 1 unchanged - 3
fixed = 1 total (was 4) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 22m 18s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15260/dev-support/hive-personality.sh
|
| git revision | master / 9925eb1 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15260/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> GenericUDFOPEqualNS may not use = in plan descriptions
> --
>
> Key: HIVE-20733
> URL: https://issues.apache.org/jira/browse/HIVE-20733
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: David Lavati
>Priority: Major
> Attachments: HIVE-20733.2.patch, HIVE-20733.3.patch,
> HIVE-20733.4.patch, HIVE-20733.5.patch, HIVE-20733.patch
>
>
> right now GenericUDFOPEqualNS is displayed a "=" in explains; however it
> should be "<=>"
> this may cause some confusion...
> related qtest: is_distinct_from.q
> same: GenericUDFOPNotEqualNS
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21022) Fix remote metastore tests which use ZooKeeper
[ https://issues.apache.org/jira/browse/HIVE-21022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Bapat updated HIVE-21022: -- Attachment: HIVE-21022.02 Status: Patch Available (was: In Progress) Patch with solution described above. > Fix remote metastore tests which use ZooKeeper > -- > > Key: HIVE-21022 > URL: https://issues.apache.org/jira/browse/HIVE-21022 > Project: Hive > Issue Type: Bug > Components: Tests >Affects Versions: 4.0.0 >Reporter: Ashutosh Bapat >Assignee: Ashutosh Bapat >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21022.01, HIVE-21022.01, HIVE-21022.01, > HIVE-21022.02, HIVE-21022.02.patch > > > Per [~vgarg]'s comment on HIVE-20794 at > https://issues.apache.org/jira/browse/HIVE-20794?focusedCommentId=16714093=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16714093, > the remote metatstore tests using ZooKeeper are flaky. They are failing with > error "Got exception: org.apache.zookeeper.KeeperException$NoNodeException > KeeperErrorCode = NoNode for /hs2mszktest". > Both of these tests are using the same root namespace and hence the reason > for this failure could be that the root namespace becomes unavailable to one > test when the other drops it. The drop seems to be happening automatically > through TestingServer code. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20733) GenericUDFOPEqualNS may not use = in plan descriptions
[
https://issues.apache.org/jira/browse/HIVE-20733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16717503#comment-16717503
]
Hive QA commented on HIVE-20733:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12951360/HIVE-20733.5.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15660 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cbo_limit]
(batchId=182)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15260/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15260/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15260/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12951360 - PreCommit-HIVE-Build
> GenericUDFOPEqualNS may not use = in plan descriptions
> --
>
> Key: HIVE-20733
> URL: https://issues.apache.org/jira/browse/HIVE-20733
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: David Lavati
>Priority: Major
> Attachments: HIVE-20733.2.patch, HIVE-20733.3.patch,
> HIVE-20733.4.patch, HIVE-20733.5.patch, HIVE-20733.patch
>
>
> right now GenericUDFOPEqualNS is displayed a "=" in explains; however it
> should be "<=>"
> this may cause some confusion...
> related qtest: is_distinct_from.q
> same: GenericUDFOPNotEqualNS
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20914) MRScratchDir permission denied when "hive.server2.enable.doAs", "hive.exec.submitviachild" are set to "true" and impersonated/proxy user is used
[
https://issues.apache.org/jira/browse/HIVE-20914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko updated HIVE-20914:
--
Attachment: HIVE-20914.8.patch
> MRScratchDir permission denied when "hive.server2.enable.doAs",
> "hive.exec.submitviachild" are set to "true" and impersonated/proxy user is
> used
>
>
> Key: HIVE-20914
> URL: https://issues.apache.org/jira/browse/HIVE-20914
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-20914.1.patch, HIVE-20914.2.patch,
> HIVE-20914.3.patch, HIVE-20914.4.patch, HIVE-20914.5.patch,
> HIVE-20914.6.patch, HIVE-20914.7.patch, HIVE-20914.8.patch
>
>
> The above issue could be reproduced in none Kerberos cluster using the below
> steps:
> 1. Set "hive.exec.submitviachild" value to "true".
> 2. Run a count query not using "hive" user.
> {code}beeline -u 'jdbc:hive2://localhost:1' -n hdfs{code}
> There is no issue when we try to execute the same query using the "hive" user.
> {code:java}
> Exception in thread "main" java.lang.RuntimeException:
> org.apache.hadoop.security.AccessControlException: Permission denied:
> user=hive, access=EXECUTE, inode="/tmp/hive/hdfs":hdfs:supergroup:drwx--
> at
> org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkFsPermission(DefaultAuthorizationProvider.java:279)
> at
> org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.check(DefaultAuthorizationProvider.java:260)
> at
> org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkTraverse(DefaultAuthorizationProvider.java:201)
> at
> org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkPermission(DefaultAuthorizationProvider.java:154)
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:152)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:3877)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:3860)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkTraverse(FSDirectory.java:3847)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkTraverse(FSNamesystem.java:6822)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInternal(FSNamesystem.java:4551)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInt(FSNamesystem.java:4529)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:4502)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:884)
> at
> org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.mkdirs(AuthorizationProviderProxyClientProtocol.java:328)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:641)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073) at
> org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2281) at
> org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2277) at
> java.security.AccessController.doPrivileged(Native Method) at
> javax.security.auth.Subject.doAs(Subject.java:422) at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2275) at
> org.apache.hadoop.hive.ql.Context.getScratchDir(Context.java:285) at
> org.apache.hadoop.hive.ql.Context.getMRScratchDir(Context.java:328) at
> org.apache.hadoop.hive.ql.Context.getMRTmpPath(Context.java:444) at
> org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:243) at
> org.apache.hadoop.hive.ql.exec.mr.ExecDriver.main(ExecDriver.java:771) at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.apache.hadoop.util.RunJar.run(RunJar.java:221) at
> org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21030) Add credential store env properties redaction in JobConf
[
https://issues.apache.org/jira/browse/HIVE-21030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16717361#comment-16717361
]
Hive QA commented on HIVE-21030:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12951361/HIVE-21030.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15651 tests
executed
*Failed tests:*
{noformat}
org.apache.hive.beeline.TestBeelinePasswordOption.org.apache.hive.beeline.TestBeelinePasswordOption
(batchId=252)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15258/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15258/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15258/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12951361 - PreCommit-HIVE-Build
> Add credential store env properties redaction in JobConf
>
>
> Key: HIVE-21030
> URL: https://issues.apache.org/jira/browse/HIVE-21030
> Project: Hive
> Issue Type: Bug
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-21030.1.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20936) Allow the Worker thread in the metastore to run outside of it
[ https://issues.apache.org/jira/browse/HIVE-20936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16718215#comment-16718215 ] Jaume M commented on HIVE-20936: I've updated the patch [~ekoifman], also in reviewboard. A bunch of test are failing but they are failing as well in other runs in jenkins. > Allow the Worker thread in the metastore to run outside of it > - > > Key: HIVE-20936 > URL: https://issues.apache.org/jira/browse/HIVE-20936 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Jaume M >Assignee: Jaume M >Priority: Major > Attachments: HIVE-20936.1.patch, HIVE-20936.2.patch, > HIVE-20936.3.patch, HIVE-20936.4.patch, HIVE-20936.5.patch, > HIVE-20936.6.patch, HIVE-20936.7.patch, HIVE-20936.8.patch, HIVE-20936.8.patch > > > Currently the Worker thread in the metastore in bounded to the metastore, > mainly because of the TxnHandler that it has. This thread runs some map > reduce jobs which may not only be an option wherever the metastore is > running. A solution for this can be to run this thread in HS2 depending on a > flag. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-17935) Turn on hive.optimize.sort.dynamic.partition by default
[ https://issues.apache.org/jira/browse/HIVE-17935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16718295#comment-16718295 ] Vineet Garg commented on HIVE-17935: [~asherman] Since now this optimization is turned on by default (HIVE-20703 & HIVE-20915) I don't believe we need this JIRA anymore. Is it ok to close it? > Turn on hive.optimize.sort.dynamic.partition by default > --- > > Key: HIVE-17935 > URL: https://issues.apache.org/jira/browse/HIVE-17935 > Project: Hive > Issue Type: Bug >Reporter: Andrew Sherman >Priority: Major > Attachments: HIVE-17935.1.patch, HIVE-17935.2.patch, > HIVE-17935.3.patch, HIVE-17935.4.patch, HIVE-17935.5.patch, > HIVE-17935.6.patch, HIVE-17935.7.patch, HIVE-17935.8.patch > > > The config option hive.optimize.sort.dynamic.partition is an optimization for > Hive’s dynamic partitioning feature. It was originally implemented in > [HIVE-6455|https://issues.apache.org/jira/browse/HIVE-6455]. With this > optimization, the dynamic partition columns and bucketing columns (in case of > bucketed tables) are sorted before being fed to the reducers. Since the > partitioning and bucketing columns are sorted, each reducer can keep only one > record writer open at any time thereby reducing the memory pressure on the > reducers. There were some early problems with this optimization and it was > disabled by default in HiveConf in > [HIVE-8151|https://issues.apache.org/jira/browse/HIVE-8151]. Since then > setting hive.optimize.sort.dynamic.partition=true has been used to solve > problems where dynamic partitioning produces with (1) too many small files on > HDFS, which is bad for the cluster and can increase overhead for future Hive > queries over those partitions, and (2) OOM issues in the map tasks because it > trying to simultaneously write to 100 different files. > It now seems that the feature is probably mature enough that it can be > enabled by default. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21031) Array with one empty string is inserted as an empty array
[
https://issues.apache.org/jira/browse/HIVE-21031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16718320#comment-16718320
]
Eric Wohlstadter commented on HIVE-21031:
-
[~pbyrnes]
This may be related to HIVE-20827
/cc [~teddy.choi]
> Array with one empty string is inserted as an empty array
> -
>
> Key: HIVE-21031
> URL: https://issues.apache.org/jira/browse/HIVE-21031
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.3.2
>Reporter: Patrick Byrnes
>Priority: Major
>
> In beeline the output of
> {code:java}
> select array("");{code}
> is:
> {code:java}
> [""]
> {code}
> However, the output of
> {code:java}
> insert into table a select array("");select * from a;{code}
> is one row of:
> {code:java}
> []{code}
>
>
> Similarly, the output of
> {code:java}
> select array(array()){code}
> is:
> {code:java}
> [[]]{code}
> However, the output of
> {code:java}
> insert into table b select array(array());select a,size(a) from b;{code}
> is one row of:
> {code:java}
> []{code}
>
> Is there a way to insert an array whose only element is an empty string or an
> array whose only element is an empty array into a table?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20936) Allow the Worker thread in the metastore to run outside of it
[
https://issues.apache.org/jira/browse/HIVE-20936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16718144#comment-16718144
]
Hive QA commented on HIVE-20936:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
32s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
43s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
50s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
17s{color} | {color:blue} standalone-metastore/metastore-common in master has
29 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
2s{color} | {color:blue} standalone-metastore/metastore-server in master has
188 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
39s{color} | {color:blue} ql in master has 2311 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
36s{color} | {color:blue} service in master has 48 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
27s{color} | {color:blue} hcatalog/streaming in master has 11 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
24s{color} | {color:blue} streaming in master has 2 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
38s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
2s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
23s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m
37s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
43s{color} | {color:red} ql: The patch generated 3 new + 641 unchanged - 6
fixed = 644 total (was 647) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
16s{color} | {color:red} itests/hive-unit: The patch generated 1 new + 173
unchanged - 0 fixed = 174 total (was 173) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 105 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
0s{color} | {color:red} ql generated 3 new + 2310 unchanged - 1 fixed = 2313
total (was 2311) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
54s{color} | {color:red} standalone-metastore_metastore-common generated 1 new
+ 16 unchanged - 0 fixed = 17 total (was 16) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 51m 6s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | Field MetaStoreCompactorThread.threadId masks field in superclass
org.apache.hadoop.hive.ql.txn.compactor.CompactorThread In
MetaStoreCompactorThread.java:superclass
org.apache.hadoop.hive.ql.txn.compactor.CompactorThread In
MetaStoreCompactorThread.java |
| | Field MetaStoreCompactorThread.rs masks field in superclass
org.apache.hadoop.hive.ql.txn.compactor.CompactorThread In
MetaStoreCompactorThread.java:superclass
org.apache.hadoop.hive.ql.txn.compactor.CompactorThread In
MetaStoreCompactorThread.java |
| |
[jira] [Commented] (HIVE-16100) Dynamic Sorted Partition optimizer loses sibling operators
[
https://issues.apache.org/jira/browse/HIVE-16100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16718270#comment-16718270
]
Vineet Garg commented on HIVE-16100:
Pushed to master
> Dynamic Sorted Partition optimizer loses sibling operators
> --
>
> Key: HIVE-16100
> URL: https://issues.apache.org/jira/browse/HIVE-16100
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Affects Versions: 1.2.1, 2.1.1, 2.2.0
>Reporter: Gopal V
>Assignee: Vineet Garg
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-16100.1.patch, HIVE-16100.10.patch,
> HIVE-16100.11.patch, HIVE-16100.12.patch, HIVE-16100.2.patch,
> HIVE-16100.2.patch, HIVE-16100.3.patch, HIVE-16100.4.patch,
> HIVE-16100.5.patch, HIVE-16100.6.patch, HIVE-16100.7.patch,
> HIVE-16100.8.patch, HIVE-16100.9.patch
>
>
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/SortedDynPartitionOptimizer.java#L173
> {code}
> // unlink connection between FS and its parent
> fsParent = fsOp.getParentOperators().get(0);
> fsParent.getChildOperators().clear();
> {code}
> The optimizer discards any cases where the fsParent has another SEL child
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-16100) Dynamic Sorted Partition optimizer loses sibling operators
[
https://issues.apache.org/jira/browse/HIVE-16100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-16100:
---
Resolution: Fixed
Fix Version/s: 4.0.0
Status: Resolved (was: Patch Available)
> Dynamic Sorted Partition optimizer loses sibling operators
> --
>
> Key: HIVE-16100
> URL: https://issues.apache.org/jira/browse/HIVE-16100
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Affects Versions: 1.2.1, 2.1.1, 2.2.0
>Reporter: Gopal V
>Assignee: Vineet Garg
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-16100.1.patch, HIVE-16100.10.patch,
> HIVE-16100.11.patch, HIVE-16100.12.patch, HIVE-16100.2.patch,
> HIVE-16100.2.patch, HIVE-16100.3.patch, HIVE-16100.4.patch,
> HIVE-16100.5.patch, HIVE-16100.6.patch, HIVE-16100.7.patch,
> HIVE-16100.8.patch, HIVE-16100.9.patch
>
>
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/SortedDynPartitionOptimizer.java#L173
> {code}
> // unlink connection between FS and its parent
> fsParent = fsOp.getParentOperators().get(0);
> fsParent.getChildOperators().clear();
> {code}
> The optimizer discards any cases where the fsParent has another SEL child
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-16957) Support CTAS for auto gather column stats
[
https://issues.apache.org/jira/browse/HIVE-16957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-16957:
---
Description:
The idea is to rely as much as possible on the logic in
ColumnStatsSemanticAnalyzer as other operations do. In particular, they create
a 'analyze table t compute statistics for columns', use
ColumnStatsSemanticAnalyzer to parse it, and connect resulting plan to existing
INSERT/INSERT OVERWRITE statement. The challenge for CTAS or CREATE
MATERIALIZED VIEW is that the table object does not exist yet, hence we cannot
rely fully on ColumnStatsSemanticAnalyzer.
Thus, we use same process, but ColumnStatsSemanticAnalyzer produces a statement
for column stats collection that uses a table values clause instead of the
original table reference:
{code}
select compute_stats(col1), compute_stats(col2), compute_stats(col3)
from table(values(cast(null as int), cast(null as int), cast(null as string)))
as t(col1, col2, col3);
{code}
> Support CTAS for auto gather column stats
> -
>
> Key: HIVE-16957
> URL: https://issues.apache.org/jira/browse/HIVE-16957
> Project: Hive
> Issue Type: Sub-task
>Reporter: Pengcheng Xiong
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
>
> The idea is to rely as much as possible on the logic in
> ColumnStatsSemanticAnalyzer as other operations do. In particular, they
> create a 'analyze table t compute statistics for columns', use
> ColumnStatsSemanticAnalyzer to parse it, and connect resulting plan to
> existing INSERT/INSERT OVERWRITE statement. The challenge for CTAS or CREATE
> MATERIALIZED VIEW is that the table object does not exist yet, hence we
> cannot rely fully on ColumnStatsSemanticAnalyzer.
> Thus, we use same process, but ColumnStatsSemanticAnalyzer produces a
> statement for column stats collection that uses a table values clause instead
> of the original table reference:
> {code}
> select compute_stats(col1), compute_stats(col2), compute_stats(col3)
> from table(values(cast(null as int), cast(null as int), cast(null as
> string))) as t(col1, col2, col3);
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-16957) Support CTAS for auto gather column stats
[
https://issues.apache.org/jira/browse/HIVE-16957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-16957:
---
Status: Patch Available (was: In Progress)
> Support CTAS for auto gather column stats
> -
>
> Key: HIVE-16957
> URL: https://issues.apache.org/jira/browse/HIVE-16957
> Project: Hive
> Issue Type: Sub-task
>Reporter: Pengcheng Xiong
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
>
> The idea is to rely as much as possible on the logic in
> ColumnStatsSemanticAnalyzer as other operations do. In particular, they
> create a 'analyze table t compute statistics for columns', use
> ColumnStatsSemanticAnalyzer to parse it, and connect resulting plan to
> existing INSERT/INSERT OVERWRITE statement. The challenge for CTAS or CREATE
> MATERIALIZED VIEW is that the table object does not exist yet, hence we
> cannot rely fully on ColumnStatsSemanticAnalyzer.
> Thus, we use same process, but ColumnStatsSemanticAnalyzer produces a
> statement for column stats collection that uses a table values clause instead
> of the original table reference:
> {code}
> select compute_stats(col1), compute_stats(col2), compute_stats(col3)
> from table(values(cast(null as int), cast(null as int), cast(null as
> string))) as t(col1, col2, col3);
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-17020) Aggressive RS dedup can incorrectly remove OP tree branch
[
https://issues.apache.org/jira/browse/HIVE-17020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16718072#comment-16718072
]
Hive QA commented on HIVE-17020:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12951395/HIVE-17020.3.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15660 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15264/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15264/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15264/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12951395 - PreCommit-HIVE-Build
> Aggressive RS dedup can incorrectly remove OP tree branch
> -
>
> Key: HIVE-17020
> URL: https://issues.apache.org/jira/browse/HIVE-17020
> Project: Hive
> Issue Type: Bug
>Reporter: Rui Li
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-17020.1.patch, HIVE-17020.2.patch,
> HIVE-17020.3.patch
>
>
> Suppose we have an OP tree like this:
> {noformat}
> ...
> |
> RS[1]
> |
> SEL[2]
> /\
> SEL[3] SEL[4]
> | |
> RS[5] FS[6]
> |
> ...
> {noformat}
> When doing aggressive RS dedup, we'll remove all the operators between RS5
> and RS1, and thus the branch containing FS6 is lost.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20936) Allow the Worker thread in the metastore to run outside of it
[
https://issues.apache.org/jira/browse/HIVE-20936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16718164#comment-16718164
]
Hive QA commented on HIVE-20936:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12951398/HIVE-20936.8.patch
{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 47 failed/errored test(s), 15645 tests
executed
*Failed tests:*
{noformat}
TestMiniLlapCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=155)
[orc_ppd_schema_evol_3a.q,import_exported_table.q,tez_union_dynamic_partition.q,except_distinct.q,orc_merge10.q,mapreduce1.q,mm_all.q,orc_merge4.q,tez_union_dynamic_partition_2.q,dynamic_semijoin_user_level.q,temp_table_external.q,llap_udf.q,schemeAuthority.q,external_table_with_space_in_location_path.q,remote_script.q]
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cbo_limit]
(batchId=182)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testAlterPartition
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testAlterTable
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testAlterTableCascade
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testAlterViewParititon
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testColumnStatistics
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testComplexTable
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testComplexTypeApi
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testConcurrentMetastores
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testCreateAndGetTableWithDriver
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testCreateTableSettingId
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testDBLocationChange
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testDBOwner
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testDBOwnerChange
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testDatabase
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testDatabaseLocation
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testDatabaseLocationWithPermissionProblems
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testDropDatabaseCascadeMVMultiDB
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testDropTable
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testFilterLastPartition
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testFilterSinglePartition
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testFunctionWithResources
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testGetConfigValue
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testGetMetastoreUuid
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testGetPartitionsWithSpec
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testGetSchemaWithNoClassDefFoundError
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testGetTableObjects
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testGetUUIDInParallel
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testJDOPersistanceManagerCleanup
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testListPartitionNames
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testListPartitions
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testListPartitionsWihtLimitEnabled
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testNameMethods
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testPartition
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testPartitionFilter
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testRenamePartition
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testRetriableClientWithConnLifetime
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testSimpleFunction
(batchId=229)
[jira] [Commented] (HIVE-16100) Dynamic Sorted Partition optimizer loses sibling operators
[
https://issues.apache.org/jira/browse/HIVE-16100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16718266#comment-16718266
]
Hive QA commented on HIVE-16100:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12951403/HIVE-16100.12.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15660 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15266/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15266/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15266/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12951403 - PreCommit-HIVE-Build
> Dynamic Sorted Partition optimizer loses sibling operators
> --
>
> Key: HIVE-16100
> URL: https://issues.apache.org/jira/browse/HIVE-16100
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Affects Versions: 1.2.1, 2.1.1, 2.2.0
>Reporter: Gopal V
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-16100.1.patch, HIVE-16100.10.patch,
> HIVE-16100.11.patch, HIVE-16100.12.patch, HIVE-16100.2.patch,
> HIVE-16100.2.patch, HIVE-16100.3.patch, HIVE-16100.4.patch,
> HIVE-16100.5.patch, HIVE-16100.6.patch, HIVE-16100.7.patch,
> HIVE-16100.8.patch, HIVE-16100.9.patch
>
>
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/SortedDynPartitionOptimizer.java#L173
> {code}
> // unlink connection between FS and its parent
> fsParent = fsOp.getParentOperators().get(0);
> fsParent.getChildOperators().clear();
> {code}
> The optimizer discards any cases where the fsParent has another SEL child
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-17020) Aggressive RS dedup can incorrectly remove OP tree branch
[
https://issues.apache.org/jira/browse/HIVE-17020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16717680#comment-16717680
]
Vineet Garg commented on HIVE-17020:
[~lirui] Good catch! Uploaded a new patch.
> Aggressive RS dedup can incorrectly remove OP tree branch
> -
>
> Key: HIVE-17020
> URL: https://issues.apache.org/jira/browse/HIVE-17020
> Project: Hive
> Issue Type: Bug
>Reporter: Rui Li
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-17020.1.patch, HIVE-17020.2.patch,
> HIVE-17020.3.patch
>
>
> Suppose we have an OP tree like this:
> {noformat}
> ...
> |
> RS[1]
> |
> SEL[2]
> /\
> SEL[3] SEL[4]
> | |
> RS[5] FS[6]
> |
> ...
> {noformat}
> When doing aggressive RS dedup, we'll remove all the operators between RS5
> and RS1, and thus the branch containing FS6 is lost.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-17020) Aggressive RS dedup can incorrectly remove OP tree branch
[
https://issues.apache.org/jira/browse/HIVE-17020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-17020:
---
Attachment: HIVE-17020.3.patch
> Aggressive RS dedup can incorrectly remove OP tree branch
> -
>
> Key: HIVE-17020
> URL: https://issues.apache.org/jira/browse/HIVE-17020
> Project: Hive
> Issue Type: Bug
>Reporter: Rui Li
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-17020.1.patch, HIVE-17020.2.patch,
> HIVE-17020.3.patch
>
>
> Suppose we have an OP tree like this:
> {noformat}
> ...
> |
> RS[1]
> |
> SEL[2]
> /\
> SEL[3] SEL[4]
> | |
> RS[5] FS[6]
> |
> ...
> {noformat}
> When doing aggressive RS dedup, we'll remove all the operators between RS5
> and RS1, and thus the branch containing FS6 is lost.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-17020) Aggressive RS dedup can incorrectly remove OP tree branch
[
https://issues.apache.org/jira/browse/HIVE-17020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-17020:
---
Status: Open (was: Patch Available)
> Aggressive RS dedup can incorrectly remove OP tree branch
> -
>
> Key: HIVE-17020
> URL: https://issues.apache.org/jira/browse/HIVE-17020
> Project: Hive
> Issue Type: Bug
>Reporter: Rui Li
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-17020.1.patch, HIVE-17020.2.patch,
> HIVE-17020.3.patch
>
>
> Suppose we have an OP tree like this:
> {noformat}
> ...
> |
> RS[1]
> |
> SEL[2]
> /\
> SEL[3] SEL[4]
> | |
> RS[5] FS[6]
> |
> ...
> {noformat}
> When doing aggressive RS dedup, we'll remove all the operators between RS5
> and RS1, and thus the branch containing FS6 is lost.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-17020) Aggressive RS dedup can incorrectly remove OP tree branch
[
https://issues.apache.org/jira/browse/HIVE-17020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-17020:
---
Status: Patch Available (was: Open)
> Aggressive RS dedup can incorrectly remove OP tree branch
> -
>
> Key: HIVE-17020
> URL: https://issues.apache.org/jira/browse/HIVE-17020
> Project: Hive
> Issue Type: Bug
>Reporter: Rui Li
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-17020.1.patch, HIVE-17020.2.patch,
> HIVE-17020.3.patch
>
>
> Suppose we have an OP tree like this:
> {noformat}
> ...
> |
> RS[1]
> |
> SEL[2]
> /\
> SEL[3] SEL[4]
> | |
> RS[5] FS[6]
> |
> ...
> {noformat}
> When doing aggressive RS dedup, we'll remove all the operators between RS5
> and RS1, and thus the branch containing FS6 is lost.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21022) Fix remote metastore tests which use ZooKeeper
[
https://issues.apache.org/jira/browse/HIVE-21022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16717698#comment-16717698
]
Hive QA commented on HIVE-21022:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12951386/HIVE-21022.02
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 28 failed/errored test(s), 15227 tests
executed
*Failed tests:*
{noformat}
TestAdminUser - did not produce a TEST-*.xml file (likely timed out)
(batchId=227)
TestCachedStore - did not produce a TEST-*.xml file (likely timed out)
(batchId=227)
TestCatalogCaching - did not produce a TEST-*.xml file (likely timed out)
(batchId=227)
TestCatalogNonDefaultSvr - did not produce a TEST-*.xml file (likely timed out)
(batchId=227)
TestDataSourceProviderFactory - did not produce a TEST-*.xml file (likely timed
out) (batchId=229)
TestDeadline - did not produce a TEST-*.xml file (likely timed out)
(batchId=227)
TestHLLNoBias - did not produce a TEST-*.xml file (likely timed out)
(batchId=229)
TestHLLSerialization - did not produce a TEST-*.xml file (likely timed out)
(batchId=229)
TestHdfsUtils - did not produce a TEST-*.xml file (likely timed out)
(batchId=227)
TestHiveMetaStoreSchemaMethods - did not produce a TEST-*.xml file (likely
timed out) (batchId=227)
TestHyperLogLogDense - did not produce a TEST-*.xml file (likely timed out)
(batchId=229)
TestHyperLogLogMerge - did not produce a TEST-*.xml file (likely timed out)
(batchId=229)
TestHyperLogLogSparse - did not produce a TEST-*.xml file (likely timed out)
(batchId=229)
TestJSONMessageDeserializer - did not produce a TEST-*.xml file (likely timed
out) (batchId=227)
TestMetaStoreEventListenerOnlyOnCommit - did not produce a TEST-*.xml file
(likely timed out) (batchId=227)
TestMetaStoreListenersError - did not produce a TEST-*.xml file (likely timed
out) (batchId=227)
TestMetaStoreSchemaFactory - did not produce a TEST-*.xml file (likely timed
out) (batchId=227)
TestMetaStoreSchemaInfo - did not produce a TEST-*.xml file (likely timed out)
(batchId=227)
TestMetaStoreServerUtils - did not produce a TEST-*.xml file (likely timed out)
(batchId=227)
TestObjectStore - did not produce a TEST-*.xml file (likely timed out)
(batchId=229)
TestObjectStoreInitRetry - did not produce a TEST-*.xml file (likely timed out)
(batchId=227)
TestOldSchema - did not produce a TEST-*.xml file (likely timed out)
(batchId=227)
TestPartitionProjectionEvaluator - did not produce a TEST-*.xml file (likely
timed out) (batchId=229)
TestRemoteHiveMetaStoreZK - did not produce a TEST-*.xml file (likely timed
out) (batchId=227)
TestRemoteHiveMetaStoreZKBindHost - did not produce a TEST-*.xml file (likely
timed out) (batchId=229)
TestRetriesInRetryingHMSHandler - did not produce a TEST-*.xml file (likely
timed out) (batchId=229)
TestRetryingHMSHandler - did not produce a TEST-*.xml file (likely timed out)
(batchId=227)
TestSparseEncodeHash - did not produce a TEST-*.xml file (likely timed out)
(batchId=229)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15261/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15261/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15261/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 28 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12951386 - PreCommit-HIVE-Build
> Fix remote metastore tests which use ZooKeeper
> --
>
> Key: HIVE-21022
> URL: https://issues.apache.org/jira/browse/HIVE-21022
> Project: Hive
> Issue Type: Bug
> Components: Tests
>Affects Versions: 4.0.0
>Reporter: Ashutosh Bapat
>Assignee: Ashutosh Bapat
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-21022.01, HIVE-21022.01, HIVE-21022.01,
> HIVE-21022.02, HIVE-21022.02.patch
>
>
> Per [~vgarg]'s comment on HIVE-20794 at
> https://issues.apache.org/jira/browse/HIVE-20794?focusedCommentId=16714093=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16714093,
> the remote metatstore tests using ZooKeeper are flaky. They are failing with
> error "Got exception: org.apache.zookeeper.KeeperException$NoNodeException
> KeeperErrorCode = NoNode for /hs2mszktest".
> Both of these tests are using the same root namespace and hence the reason
> for this failure could be that the root namespace becomes unavailable to one
> test
[jira] [Updated] (HIVE-20734) Beeline: When beeline-site.xml is and hive CLI redirects to beeline, it should use the system username/dummy password instead of prompting for one
[ https://issues.apache.org/jira/browse/HIVE-20734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-20734: - Resolution: Fixed Fix Version/s: 4.0.0 Status: Resolved (was: Patch Available) Patch committed to master. Thanks [~maheshk114]! > Beeline: When beeline-site.xml is and hive CLI redirects to beeline, it > should use the system username/dummy password instead of prompting for one > -- > > Key: HIVE-20734 > URL: https://issues.apache.org/jira/browse/HIVE-20734 > Project: Hive > Issue Type: Bug > Components: Beeline >Affects Versions: 3.1.0 >Reporter: Vaibhav Gumashta >Assignee: mahesh kumar behera >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-20734.1.patch, HIVE-20734.2.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20914) MRScratchDir permission denied when "hive.server2.enable.doAs", "hive.exec.submitviachild" are set to "true" and impersonated/proxy user is used
[
https://issues.apache.org/jira/browse/HIVE-20914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16717854#comment-16717854
]
Hive QA commented on HIVE-20914:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12951390/HIVE-20914.8.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 45 failed/errored test(s), 15661 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testAlterPartition
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testAlterTable
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testAlterTableCascade
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testAlterViewParititon
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testColumnStatistics
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testComplexTable
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testComplexTypeApi
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testConcurrentMetastores
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testCreateAndGetTableWithDriver
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testCreateTableSettingId
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testDBLocationChange
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testDBOwner
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testDBOwnerChange
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testDatabase
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testDatabaseLocation
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testDatabaseLocationWithPermissionProblems
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testDropDatabaseCascadeMVMultiDB
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testDropTable
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testFilterLastPartition
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testFilterSinglePartition
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testFunctionWithResources
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testGetConfigValue
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testGetMetastoreUuid
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testGetPartitionsWithSpec
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testGetSchemaWithNoClassDefFoundError
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testGetTableObjects
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testGetUUIDInParallel
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testJDOPersistanceManagerCleanup
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testListPartitionNames
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testListPartitions
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testListPartitionsWihtLimitEnabled
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testNameMethods
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testPartition
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testPartitionFilter
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testRenamePartition
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testRetriableClientWithConnLifetime
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testSimpleFunction
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testSimpleTable
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testSimpleTypeApi
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testStatsFastTrivial
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testSynchronized
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testTableDatabase
(batchId=229)
[jira] [Commented] (HIVE-21030) Add credential store env properties redaction in JobConf
[
https://issues.apache.org/jira/browse/HIVE-21030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16717974#comment-16717974
]
Hive QA commented on HIVE-21030:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12951392/HIVE-21030.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 11 failed/errored test(s), 15660 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.metastore.TestObjectStore.catalogs (batchId=229)
org.apache.hadoop.hive.metastore.TestObjectStore.testDatabaseOps (batchId=229)
org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSQLDropParitionsCleanup
(batchId=229)
org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSQLDropPartitionsCacheCrossSession
(batchId=229)
org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSqlErrorMetrics
(batchId=229)
org.apache.hadoop.hive.metastore.TestObjectStore.testMasterKeyOps (batchId=229)
org.apache.hadoop.hive.metastore.TestObjectStore.testMaxEventResponse
(batchId=229)
org.apache.hadoop.hive.metastore.TestObjectStore.testPartitionOps (batchId=229)
org.apache.hadoop.hive.metastore.TestObjectStore.testQueryCloseOnError
(batchId=229)
org.apache.hadoop.hive.metastore.TestObjectStore.testRoleOps (batchId=229)
org.apache.hadoop.hive.metastore.TestObjectStore.testTableOps (batchId=229)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15263/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15263/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15263/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 11 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12951392 - PreCommit-HIVE-Build
> Add credential store env properties redaction in JobConf
>
>
> Key: HIVE-21030
> URL: https://issues.apache.org/jira/browse/HIVE-21030
> Project: Hive
> Issue Type: Bug
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-21030.1.patch, HIVE-21030.2.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-17020) Aggressive RS dedup can incorrectly remove OP tree branch
[
https://issues.apache.org/jira/browse/HIVE-17020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16718007#comment-16718007
]
Hive QA commented on HIVE-17020:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
33s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
39s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
37s{color} | {color:blue} ql in master has 2311 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 23m 25s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15264/dev-support/hive-personality.sh
|
| git revision | master / 896f10d |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql itests U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15264/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Aggressive RS dedup can incorrectly remove OP tree branch
> -
>
> Key: HIVE-17020
> URL: https://issues.apache.org/jira/browse/HIVE-17020
> Project: Hive
> Issue Type: Bug
>Reporter: Rui Li
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-17020.1.patch, HIVE-17020.2.patch,
> HIVE-17020.3.patch
>
>
> Suppose we have an OP tree like this:
> {noformat}
> ...
> |
> RS[1]
> |
> SEL[2]
> /\
> SEL[3] SEL[4]
> | |
> RS[5] FS[6]
> |
> ...
> {noformat}
> When doing aggressive RS dedup, we'll remove all the operators between RS5
> and RS1, and thus the branch containing FS6 is lost.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-18358) from_unixtime returns wrong year for Dec 31 timestamps with format 'YYYY'
[
https://issues.apache.org/jira/browse/HIVE-18358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Sherman reassigned HIVE-18358:
-
Assignee: (was: Andrew Sherman)
> from_unixtime returns wrong year for Dec 31 timestamps with format ''
> -
>
> Key: HIVE-18358
> URL: https://issues.apache.org/jira/browse/HIVE-18358
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.0
> Environment: AWS EMR with Hive 2.1.0-amzn-0
>Reporter: Nick Orka
>Priority: Major
> Labels: timezone
>
> If you use capital Ys as a year format in from_unixtime() it returns next
> year for Dec 31 only. All other days work as intended.
> Here is reproduction code:
> {code:sql}
> hive> select from_unixtime(1514754599, '-MM-dd HH-mm-ss'),
> from_unixtime(1514754599, '-MM-dd HH-mm-ss');
> OK
> 2018-12-31 21-09-59 2017-12-31 21-09-59
> Time taken: 0.025 seconds, Fetched: 1 row(s)
> hive>
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-16957) Support CTAS for auto gather column stats
[
https://issues.apache.org/jira/browse/HIVE-16957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16718332#comment-16718332
]
Jesus Camacho Rodriguez commented on HIVE-16957:
Uploaded initial patch to trigger ptests. I realized that ALTER MV... REBUILD
is still not collecting the column stats automatically yet, I will work on that.
Cc [~ashutoshc]
> Support CTAS for auto gather column stats
> -
>
> Key: HIVE-16957
> URL: https://issues.apache.org/jira/browse/HIVE-16957
> Project: Hive
> Issue Type: Sub-task
>Reporter: Pengcheng Xiong
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-16957.patch
>
>
> The idea is to rely as much as possible on the logic in
> ColumnStatsSemanticAnalyzer as other operations do. In particular, they
> create a 'analyze table t compute statistics for columns', use
> ColumnStatsSemanticAnalyzer to parse it, and connect resulting plan to
> existing INSERT/INSERT OVERWRITE statement. The challenge for CTAS or CREATE
> MATERIALIZED VIEW is that the table object does not exist yet, hence we
> cannot rely fully on ColumnStatsSemanticAnalyzer.
> Thus, we use same process, but ColumnStatsSemanticAnalyzer produces a
> statement for column stats collection that uses a table values clause instead
> of the original table reference:
> {code}
> select compute_stats(col1), compute_stats(col2), compute_stats(col3)
> from table(values(cast(null as int), cast(null as int), cast(null as
> string))) as t(col1, col2, col3);
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20998) HiveStrictManagedMigration utility should update DB/Table location as last migration steps
[ https://issues.apache.org/jira/browse/HIVE-20998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16718102#comment-16718102 ] Ashutosh Chauhan commented on HIVE-20998: - +1 > HiveStrictManagedMigration utility should update DB/Table location as last > migration steps > -- > > Key: HIVE-20998 > URL: https://issues.apache.org/jira/browse/HIVE-20998 > Project: Hive > Issue Type: Sub-task >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Attachments: HIVE-20998.1.patch > > > When processing a database or table, the HiveStrictManagedMigration utility > currently changes the database/table locations as the first step in > processing that database/table. Unfortunately if an error occurs while > processing this database or table, then there may still be migration work > that needs to continue for that db/table by running the migration again. > However the migration tool only processes dbs/tables that have the old > warehouse location, then the tool will skip over the db/table when the > migration is run again. > One fix here is to set the new location as the last step after all of the > migration work is done: > - The new table location will not be set until all of its partitions have > been successfully migrated. > - The new database location will not be set until all of its tables have > been successfully migrated. > For existing migrations that failed with an error, the following workaround > can be done so that the db/tables can be re-processed by the migration tool: > 1) Use the migration tool logs to find which databases/tables failed during > processing. > 2) For each db/table, change location of of the database and table back to > old location: > ALTER DATABASE tpcds_bin_partitioned_orc_10 SET LOCATION > 'hdfs://ns1/apps/hive/warehouse/tpcds_bin_partitioned_orc_10.db'; > ALTER TABLE tpcds_bin_partitioned_orc_10.store_sales SET LOCATION > 'hdfs://ns1/apps/hive/warehouse/tpcds_bin_partitioned_orc_10.db/store_sales'; > 2) Rerun the migration tool -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-16100) Dynamic Sorted Partition optimizer loses sibling operators
[
https://issues.apache.org/jira/browse/HIVE-16100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16718212#comment-16718212
]
Hive QA commented on HIVE-16100:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
46s{color} | {color:blue} ql in master has 2311 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 22m 53s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15266/dev-support/hive-personality.sh
|
| git revision | master / 896f10d |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15266/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Dynamic Sorted Partition optimizer loses sibling operators
> --
>
> Key: HIVE-16100
> URL: https://issues.apache.org/jira/browse/HIVE-16100
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Affects Versions: 1.2.1, 2.1.1, 2.2.0
>Reporter: Gopal V
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-16100.1.patch, HIVE-16100.10.patch,
> HIVE-16100.11.patch, HIVE-16100.12.patch, HIVE-16100.2.patch,
> HIVE-16100.2.patch, HIVE-16100.3.patch, HIVE-16100.4.patch,
> HIVE-16100.5.patch, HIVE-16100.6.patch, HIVE-16100.7.patch,
> HIVE-16100.8.patch, HIVE-16100.9.patch
>
>
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/SortedDynPartitionOptimizer.java#L173
> {code}
> // unlink connection between FS and its parent
> fsParent = fsOp.getParentOperators().get(0);
> fsParent.getChildOperators().clear();
> {code}
> The optimizer discards any cases where the fsParent has another SEL child
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20914) MRScratchDir permission denied when "hive.server2.enable.doAs", "hive.exec.submitviachild" are set to "true" and impersonated/proxy user is used
[
https://issues.apache.org/jira/browse/HIVE-20914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16718304#comment-16718304
]
Hive QA commented on HIVE-20914:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
40s{color} | {color:blue} ql in master has 2311 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 22m 39s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15267/dev-support/hive-personality.sh
|
| git revision | master / b650083 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15267/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> MRScratchDir permission denied when "hive.server2.enable.doAs",
> "hive.exec.submitviachild" are set to "true" and impersonated/proxy user is
> used
>
>
> Key: HIVE-20914
> URL: https://issues.apache.org/jira/browse/HIVE-20914
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-20914.1.patch, HIVE-20914.2.patch,
> HIVE-20914.3.patch, HIVE-20914.4.patch, HIVE-20914.5.patch,
> HIVE-20914.6.patch, HIVE-20914.7.patch, HIVE-20914.8.patch, HIVE-20914.9.patch
>
>
> The above issue could be reproduced in none Kerberos cluster using the below
> steps:
> 1. Set "hive.exec.submitviachild" value to "true".
> 2. Run a count query not using "hive" user.
> {code}beeline -u 'jdbc:hive2://localhost:1' -n hdfs{code}
> There is no issue when we try to execute the same query using the "hive" user.
> {code:java}
> Exception in thread "main" java.lang.RuntimeException:
> org.apache.hadoop.security.AccessControlException: Permission denied:
> user=hive, access=EXECUTE, inode="/tmp/hive/hdfs":hdfs:supergroup:drwx--
> at
> org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkFsPermission(DefaultAuthorizationProvider.java:279)
> at
> org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.check(DefaultAuthorizationProvider.java:260)
> at
>
[jira] [Assigned] (HIVE-16957) Support CTAS for auto gather column stats
[ https://issues.apache.org/jira/browse/HIVE-16957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez reassigned HIVE-16957: -- Assignee: Jesus Camacho Rodriguez (was: Pengcheng Xiong) > Support CTAS for auto gather column stats > - > > Key: HIVE-16957 > URL: https://issues.apache.org/jira/browse/HIVE-16957 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Jesus Camacho Rodriguez >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21023) Add test for replication to a target with hive.strict.managed.tables enabled
[ https://issues.apache.org/jira/browse/HIVE-21023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16717885#comment-16717885 ] Vihang Karajgaonkar commented on HIVE-21023: Can we split these tests into smaller ones? 40 min for a test is probably cutting it too close and the batch even with a single test may timeout. > Add test for replication to a target with hive.strict.managed.tables enabled > > > Key: HIVE-21023 > URL: https://issues.apache.org/jira/browse/HIVE-21023 > Project: Hive > Issue Type: Bug >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Attachments: HIVE-21023.01.patch > > > Tests added are timing out in ptest run. Need to skip these test cases from > batching and run them separately. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20914) MRScratchDir permission denied when "hive.server2.enable.doAs", "hive.exec.submitviachild" are set to "true" and impersonated/proxy user is used
[
https://issues.apache.org/jira/browse/HIVE-20914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16717746#comment-16717746
]
Hive QA commented on HIVE-20914:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
58s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
43s{color} | {color:blue} ql in master has 2311 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
37s{color} | {color:red} ql: The patch generated 3 new + 11 unchanged - 0 fixed
= 14 total (was 11) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 22m 42s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15262/dev-support/hive-personality.sh
|
| git revision | master / 9925eb1 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15262/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15262/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> MRScratchDir permission denied when "hive.server2.enable.doAs",
> "hive.exec.submitviachild" are set to "true" and impersonated/proxy user is
> used
>
>
> Key: HIVE-20914
> URL: https://issues.apache.org/jira/browse/HIVE-20914
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-20914.1.patch, HIVE-20914.2.patch,
> HIVE-20914.3.patch, HIVE-20914.4.patch, HIVE-20914.5.patch,
> HIVE-20914.6.patch, HIVE-20914.7.patch, HIVE-20914.8.patch
>
>
> The above issue could be reproduced in none Kerberos cluster using the below
> steps:
> 1. Set "hive.exec.submitviachild" value to "true".
> 2. Run a count query not using "hive" user.
> {code}beeline -u 'jdbc:hive2://localhost:1' -n hdfs{code}
> There is no issue when we try to execute the same query using the "hive" user.
> {code:java}
> Exception in thread "main" java.lang.RuntimeException:
> org.apache.hadoop.security.AccessControlException: Permission denied:
> user=hive, access=EXECUTE, inode="/tmp/hive/hdfs":hdfs:supergroup:drwx--
> at
> org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkFsPermission(DefaultAuthorizationProvider.java:279)
> at
>
[jira] [Commented] (HIVE-19081) Add partition should prevent loading acid files
[
https://issues.apache.org/jira/browse/HIVE-19081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16717749#comment-16717749
]
Igor Kryvenko commented on HIVE-19081:
--
[~ekoifman] Yeah, sure.
> Add partition should prevent loading acid files
> ---
>
> Key: HIVE-19081
> URL: https://issues.apache.org/jira/browse/HIVE-19081
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Vaibhav Gumashta
>Priority: Major
>
> similar to HIVE-19029
> {{Alter Table T add Partition ...} T is acid should check to make sure input
> files were not copied from another Acid table, i.e. make sure the files don't
> have Acid metadata columns.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21030) Add credential store env properties redaction in JobConf
[ https://issues.apache.org/jira/browse/HIVE-21030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16717881#comment-16717881 ] Vihang Karajgaonkar commented on HIVE-21030: +1 (pending tests) > Add credential store env properties redaction in JobConf > > > Key: HIVE-21030 > URL: https://issues.apache.org/jira/browse/HIVE-21030 > Project: Hive > Issue Type: Bug >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > Attachments: HIVE-21030.1.patch, HIVE-21030.2.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-19081) Add partition should prevent loading acid files
[
https://issues.apache.org/jira/browse/HIVE-19081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Igor Kryvenko reassigned HIVE-19081:
Assignee: Igor Kryvenko (was: Vaibhav Gumashta)
> Add partition should prevent loading acid files
> ---
>
> Key: HIVE-19081
> URL: https://issues.apache.org/jira/browse/HIVE-19081
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Igor Kryvenko
>Priority: Major
>
> similar to HIVE-19029
> {{Alter Table T add Partition ...} T is acid should check to make sure input
> files were not copied from another Acid table, i.e. make sure the files don't
> have Acid metadata columns.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20934) Query based compactor for minor compaction of full CRUD Acid tables
[ https://issues.apache.org/jira/browse/HIVE-20934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-20934: Summary: Query based compactor for minor compaction of full CRUD Acid tables (was: Query based compactor for full CRUD Acid tables) > Query based compactor for minor compaction of full CRUD Acid tables > --- > > Key: HIVE-20934 > URL: https://issues.apache.org/jira/browse/HIVE-20934 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.1.1 >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta >Priority: Major > > Follow up of HIVE-20699. This is to enable running minor compactions as a > HiveQL query -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21005) LLAP: Reading more stripes per-split leaks ZlibCodecs
[
https://issues.apache.org/jira/browse/HIVE-21005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V updated HIVE-21005:
---
Resolution: Fixed
Fix Version/s: 4.0.0
Status: Resolved (was: Patch Available)
Pushed to master, thanks [~ndembla]
> LLAP: Reading more stripes per-split leaks ZlibCodecs
> -
>
> Key: HIVE-21005
> URL: https://issues.apache.org/jira/browse/HIVE-21005
> Project: Hive
> Issue Type: Bug
> Components: llap
>Reporter: Gopal V
>Assignee: Nita Dembla
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-21005.patch
>
>
> OrcEncodedDataReader - calls ensureDataReader in a loop, overwriting itself
> {code}
> for (int stripeIxMod = 0; stripeIxMod < stripeRgs.length; ++stripeIxMod) {
>
> // 6.2. Ensure we have stripe metadata. We might have read it before
> for RG filtering.
> if (stripeMetadatas != null) {
> stripeMetadata = stripeMetadatas.get(stripeIxMod);
> } else {
> ...
> ensureDataReader();
> ...
> }
> {code}
> {code}
> private void ensureDataReader() throws IOException {
> ...
> stripeReader = orcReader.encodedReader(
> fileKey, dw, dw, useObjectPools ? POOL_FACTORY : null, trace,
> useCodecPool, cacheTag);
> {code}
> creates new encodedReader without closing previous stripe's encoded reader.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20936) Allow the Worker thread in the metastore to run outside of it
[ https://issues.apache.org/jira/browse/HIVE-20936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jaume M updated HIVE-20936: --- Attachment: HIVE-20936.8.patch Status: Patch Available (was: Open) > Allow the Worker thread in the metastore to run outside of it > - > > Key: HIVE-20936 > URL: https://issues.apache.org/jira/browse/HIVE-20936 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Jaume M >Assignee: Jaume M >Priority: Major > Attachments: HIVE-20936.1.patch, HIVE-20936.2.patch, > HIVE-20936.3.patch, HIVE-20936.4.patch, HIVE-20936.5.patch, > HIVE-20936.6.patch, HIVE-20936.7.patch, HIVE-20936.8.patch, HIVE-20936.8.patch > > > Currently the Worker thread in the metastore in bounded to the metastore, > mainly because of the TxnHandler that it has. This thread runs some map > reduce jobs which may not only be an option wherever the metastore is > running. A solution for this can be to run this thread in HS2 depending on a > flag. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20914) MRScratchDir permission denied when "hive.server2.enable.doAs", "hive.exec.submitviachild" are set to "true" and impersonated/proxy user is used
[
https://issues.apache.org/jira/browse/HIVE-20914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko updated HIVE-20914:
--
Attachment: HIVE-20914.9.patch
> MRScratchDir permission denied when "hive.server2.enable.doAs",
> "hive.exec.submitviachild" are set to "true" and impersonated/proxy user is
> used
>
>
> Key: HIVE-20914
> URL: https://issues.apache.org/jira/browse/HIVE-20914
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-20914.1.patch, HIVE-20914.2.patch,
> HIVE-20914.3.patch, HIVE-20914.4.patch, HIVE-20914.5.patch,
> HIVE-20914.6.patch, HIVE-20914.7.patch, HIVE-20914.8.patch, HIVE-20914.9.patch
>
>
> The above issue could be reproduced in none Kerberos cluster using the below
> steps:
> 1. Set "hive.exec.submitviachild" value to "true".
> 2. Run a count query not using "hive" user.
> {code}beeline -u 'jdbc:hive2://localhost:1' -n hdfs{code}
> There is no issue when we try to execute the same query using the "hive" user.
> {code:java}
> Exception in thread "main" java.lang.RuntimeException:
> org.apache.hadoop.security.AccessControlException: Permission denied:
> user=hive, access=EXECUTE, inode="/tmp/hive/hdfs":hdfs:supergroup:drwx--
> at
> org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkFsPermission(DefaultAuthorizationProvider.java:279)
> at
> org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.check(DefaultAuthorizationProvider.java:260)
> at
> org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkTraverse(DefaultAuthorizationProvider.java:201)
> at
> org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkPermission(DefaultAuthorizationProvider.java:154)
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:152)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:3877)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:3860)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkTraverse(FSDirectory.java:3847)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkTraverse(FSNamesystem.java:6822)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInternal(FSNamesystem.java:4551)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInt(FSNamesystem.java:4529)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:4502)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:884)
> at
> org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.mkdirs(AuthorizationProviderProxyClientProtocol.java:328)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:641)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073) at
> org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2281) at
> org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2277) at
> java.security.AccessController.doPrivileged(Native Method) at
> javax.security.auth.Subject.doAs(Subject.java:422) at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2275) at
> org.apache.hadoop.hive.ql.Context.getScratchDir(Context.java:285) at
> org.apache.hadoop.hive.ql.Context.getMRScratchDir(Context.java:328) at
> org.apache.hadoop.hive.ql.Context.getMRTmpPath(Context.java:444) at
> org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:243) at
> org.apache.hadoop.hive.ql.exec.mr.ExecDriver.main(ExecDriver.java:771) at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.apache.hadoop.util.RunJar.run(RunJar.java:221) at
> org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-17572) Warnings from SparkCrossProductCheck for MapJoins are confusing
[
https://issues.apache.org/jira/browse/HIVE-17572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Sherman reassigned HIVE-17572:
-
Assignee: (was: Andrew Sherman)
> Warnings from SparkCrossProductCheck for MapJoins are confusing
> ---
>
> Key: HIVE-17572
> URL: https://issues.apache.org/jira/browse/HIVE-17572
> Project: Hive
> Issue Type: Improvement
> Components: Spark
>Reporter: Sahil Takiar
>Priority: Major
>
> When the {{SparkCrossProductCheck}} detects a cross-product in a map-join, it
> prints out a confusing warning - e.g. {{Map Join MAPJOIN\[9\]\[bigTable=?\]
> in task 'Stage-1:MAPRED' is a cross product}}
> I see a few ways this can be imrpoved:
> * {{bigTable}} should actually specify the big table
> * I'm not sure why the stage id is printed instead of the work id, when a
> cross product is detected in a shuffle join the work id is shown (e.g.
> {{Warning: Shuffle Join JOIN\[13\]\[tables = \[$hdt$_1, $hdt$_2, $hdt$_0\]\]
> in Work 'Reducer 3' is a cross product}})
> * It shouldn't say {{MAPRED}} that can be confusing to users
> * The {{MAPJOIN}} id doesn't need to be printed, it doesn't have any meaning
> to the user and the value just keeps on going up and up the longer a session
> lives
> On a somewhat related note, could we just stick this warning in the explain
> plan? Otherwise users may not even notice it
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-17727) HoS Queries Print "Starting task [Stage-x:MAPRED] in serial mode"
[
https://issues.apache.org/jira/browse/HIVE-17727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Sherman reassigned HIVE-17727:
-
Assignee: (was: Andrew Sherman)
> HoS Queries Print "Starting task [Stage-x:MAPRED] in serial mode"
> -
>
> Key: HIVE-17727
> URL: https://issues.apache.org/jira/browse/HIVE-17727
> Project: Hive
> Issue Type: Improvement
> Components: Spark
>Reporter: Sahil Takiar
>Priority: Major
>
> Whenever a HoS query is run something like "Starting task [Stage-3:MAPRED] in
> serial mode" in printed out for each {{SparkTask}}, which is confusing
> because this isn't a MAPRED job. We should change {{StageType}} to include a
> {{SPARK}} type.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-17677) Investigate using hive statistics information to optimize HoS parallel order by
[ https://issues.apache.org/jira/browse/HIVE-17677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Sherman reassigned HIVE-17677: - Assignee: (was: Andrew Sherman) > Investigate using hive statistics information to optimize HoS parallel order > by > --- > > Key: HIVE-17677 > URL: https://issues.apache.org/jira/browse/HIVE-17677 > Project: Hive > Issue Type: Improvement >Affects Versions: 3.0.0 >Reporter: Andrew Sherman >Priority: Major > > I think Spark's native parallel order by works in a similar way to what we do > for Hive-on-MR. That is, it scans the RDD once and sample the data to > determine what ranges the data should be partitioned into, and then scans the > RDD again to do the actual order by (with multiple reducers). > One optimization suggested by [~stakiar] is that if we have column stats > about the col we are ordering by, then the first scan on the RDD is not > necessary. If we have histogram data about the RDD, we already know what the > ranges of the order by should be. This should work when running parallel > order by on simple tables, will be harder when we run it on derived datasets > (although not impossible). > To do his we would have to understand more about the internals of > JavaPairRDD. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-17760) Create a unit test which validates HIVE-9423 does not regress
[ https://issues.apache.org/jira/browse/HIVE-17760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Sherman updated HIVE-17760: -- Resolution: Won't Fix Status: Resolved (was: Patch Available) > Create a unit test which validates HIVE-9423 does not regress > -- > > Key: HIVE-17760 > URL: https://issues.apache.org/jira/browse/HIVE-17760 > Project: Hive > Issue Type: Bug >Reporter: Andrew Sherman >Assignee: Andrew Sherman >Priority: Major > Attachments: HIVE-17760.1.patch, HIVE-17760.2.patch, > HIVE-17760.3.patch, HIVE-17760.4.patch > > > During [HIVE-9423] we verified that when the Thrift server pool is exhausted, > then Beeline connection times out, and provide a meaningful error message. > Create a unit test which verifies this, and helps to keep this feature working -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21030) Add credential store env properties redaction in JobConf
[
https://issues.apache.org/jira/browse/HIVE-21030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16717880#comment-16717880
]
Hive QA commented on HIVE-21030:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
30s{color} | {color:blue} common in master has 65 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
11s{color} | {color:red} common: The patch generated 2 new + 6 unchanged - 0
fixed = 8 total (was 6) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 11m 21s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15263/dev-support/hive-personality.sh
|
| git revision | master / 896f10d |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15263/yetus/diff-checkstyle-common.txt
|
| modules | C: common U: common |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15263/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Add credential store env properties redaction in JobConf
>
>
> Key: HIVE-21030
> URL: https://issues.apache.org/jira/browse/HIVE-21030
> Project: Hive
> Issue Type: Bug
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-21030.1.patch, HIVE-21030.2.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20914) MRScratchDir permission denied when "hive.server2.enable.doAs", "hive.exec.submitviachild" are set to "true" and impersonated/proxy user is used
[
https://issues.apache.org/jira/browse/HIVE-20914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko updated HIVE-20914:
--
Attachment: HIVE-20914.9.patch
> MRScratchDir permission denied when "hive.server2.enable.doAs",
> "hive.exec.submitviachild" are set to "true" and impersonated/proxy user is
> used
>
>
> Key: HIVE-20914
> URL: https://issues.apache.org/jira/browse/HIVE-20914
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-20914.1.patch, HIVE-20914.2.patch,
> HIVE-20914.3.patch, HIVE-20914.4.patch, HIVE-20914.5.patch,
> HIVE-20914.6.patch, HIVE-20914.7.patch, HIVE-20914.8.patch, HIVE-20914.9.patch
>
>
> The above issue could be reproduced in none Kerberos cluster using the below
> steps:
> 1. Set "hive.exec.submitviachild" value to "true".
> 2. Run a count query not using "hive" user.
> {code}beeline -u 'jdbc:hive2://localhost:1' -n hdfs{code}
> There is no issue when we try to execute the same query using the "hive" user.
> {code:java}
> Exception in thread "main" java.lang.RuntimeException:
> org.apache.hadoop.security.AccessControlException: Permission denied:
> user=hive, access=EXECUTE, inode="/tmp/hive/hdfs":hdfs:supergroup:drwx--
> at
> org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkFsPermission(DefaultAuthorizationProvider.java:279)
> at
> org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.check(DefaultAuthorizationProvider.java:260)
> at
> org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkTraverse(DefaultAuthorizationProvider.java:201)
> at
> org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkPermission(DefaultAuthorizationProvider.java:154)
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:152)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:3877)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:3860)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkTraverse(FSDirectory.java:3847)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkTraverse(FSNamesystem.java:6822)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInternal(FSNamesystem.java:4551)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInt(FSNamesystem.java:4529)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:4502)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:884)
> at
> org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.mkdirs(AuthorizationProviderProxyClientProtocol.java:328)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:641)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073) at
> org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2281) at
> org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2277) at
> java.security.AccessController.doPrivileged(Native Method) at
> javax.security.auth.Subject.doAs(Subject.java:422) at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2275) at
> org.apache.hadoop.hive.ql.Context.getScratchDir(Context.java:285) at
> org.apache.hadoop.hive.ql.Context.getMRScratchDir(Context.java:328) at
> org.apache.hadoop.hive.ql.Context.getMRTmpPath(Context.java:444) at
> org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:243) at
> org.apache.hadoop.hive.ql.exec.mr.ExecDriver.main(ExecDriver.java:771) at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.apache.hadoop.util.RunJar.run(RunJar.java:221) at
> org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20914) MRScratchDir permission denied when "hive.server2.enable.doAs", "hive.exec.submitviachild" are set to "true" and impersonated/proxy user is used
[
https://issues.apache.org/jira/browse/HIVE-20914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko updated HIVE-20914:
--
Attachment: (was: HIVE-20914.9.patch)
> MRScratchDir permission denied when "hive.server2.enable.doAs",
> "hive.exec.submitviachild" are set to "true" and impersonated/proxy user is
> used
>
>
> Key: HIVE-20914
> URL: https://issues.apache.org/jira/browse/HIVE-20914
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-20914.1.patch, HIVE-20914.2.patch,
> HIVE-20914.3.patch, HIVE-20914.4.patch, HIVE-20914.5.patch,
> HIVE-20914.6.patch, HIVE-20914.7.patch, HIVE-20914.8.patch, HIVE-20914.9.patch
>
>
> The above issue could be reproduced in none Kerberos cluster using the below
> steps:
> 1. Set "hive.exec.submitviachild" value to "true".
> 2. Run a count query not using "hive" user.
> {code}beeline -u 'jdbc:hive2://localhost:1' -n hdfs{code}
> There is no issue when we try to execute the same query using the "hive" user.
> {code:java}
> Exception in thread "main" java.lang.RuntimeException:
> org.apache.hadoop.security.AccessControlException: Permission denied:
> user=hive, access=EXECUTE, inode="/tmp/hive/hdfs":hdfs:supergroup:drwx--
> at
> org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkFsPermission(DefaultAuthorizationProvider.java:279)
> at
> org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.check(DefaultAuthorizationProvider.java:260)
> at
> org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkTraverse(DefaultAuthorizationProvider.java:201)
> at
> org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkPermission(DefaultAuthorizationProvider.java:154)
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:152)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:3877)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:3860)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkTraverse(FSDirectory.java:3847)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkTraverse(FSNamesystem.java:6822)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInternal(FSNamesystem.java:4551)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInt(FSNamesystem.java:4529)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:4502)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:884)
> at
> org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.mkdirs(AuthorizationProviderProxyClientProtocol.java:328)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:641)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073) at
> org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2281) at
> org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2277) at
> java.security.AccessController.doPrivileged(Native Method) at
> javax.security.auth.Subject.doAs(Subject.java:422) at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2275) at
> org.apache.hadoop.hive.ql.Context.getScratchDir(Context.java:285) at
> org.apache.hadoop.hive.ql.Context.getMRScratchDir(Context.java:328) at
> org.apache.hadoop.hive.ql.Context.getMRTmpPath(Context.java:444) at
> org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:243) at
> org.apache.hadoop.hive.ql.exec.mr.ExecDriver.main(ExecDriver.java:771) at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.apache.hadoop.util.RunJar.run(RunJar.java:221) at
> org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-18884) Simplify Logging in Hive Metastore Client
[
https://issues.apache.org/jira/browse/HIVE-18884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Sherman reassigned HIVE-18884:
-
Assignee: (was: Andrew Sherman)
> Simplify Logging in Hive Metastore Client
> -
>
> Key: HIVE-18884
> URL: https://issues.apache.org/jira/browse/HIVE-18884
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Affects Versions: 3.0.0
>Reporter: BELUGA BEHR
>Priority: Minor
> Labels: noob
>
> https://github.com/apache/hive/blob/4047befe48c8f762c58d8854e058385c1df151c6/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
> The current logging is:
> {code}
> 2018-02-26 07:02:44,883 INFO hive.metastore: [HiveServer2-Handler-Pool:
> Thread-65]: Trying to connect to metastore with URI
> thrift://host.company.com:9083
> 2018-02-26 07:02:44,892 INFO hive.metastore: [HiveServer2-Handler-Pool:
> Thread-65]: Connected to metastore.
> 2018-02-26 07:02:44,892 INFO hive.metastore: [HiveServer2-Handler-Pool:
> Thread-65]: Opened a connection to metastore, current connections: 2
> {code}
> Please simplify to something like:
> {code}
> 2018-02-26 07:02:44,892 INFO hive.metastore: [HiveServer2-Handler-Pool:
> Thread-65]: Opened a connection to the Metastore Server (URI
> thrift://host.company.com:9083), current connections: 2
> ... or ...
> 2018-02-26 07:02:44,892 ERROR hive.metastore: [HiveServer2-Handler-Pool:
> Thread-65]: Failed to connect to the Metastore Server (URI
> thrift://host.company.com:9083)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-14615) Temp table leaves behind insert command
[
https://issues.apache.org/jira/browse/HIVE-14615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Sherman reassigned HIVE-14615:
-
Assignee: (was: Andrew Sherman)
> Temp table leaves behind insert command
> ---
>
> Key: HIVE-14615
> URL: https://issues.apache.org/jira/browse/HIVE-14615
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Reporter: Chaoyu Tang
>Priority: Major
> Attachments: HIVE-14615.1.patch, HIVE-14615.2.patch,
> HIVE-14615.3.patch, HIVE-14615.4.patch
>
>
> {code}
> create table test (key int, value string);
> insert into test values (1, 'val1');
> show tables;
> test
> values__tmp__table__1
> {code}
> the temp table values__tmp__table__1 was resulted from insert into ...values
> and exists until logout the session.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-17935) Turn on hive.optimize.sort.dynamic.partition by default
[ https://issues.apache.org/jira/browse/HIVE-17935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Sherman reassigned HIVE-17935: - Assignee: (was: Andrew Sherman) > Turn on hive.optimize.sort.dynamic.partition by default > --- > > Key: HIVE-17935 > URL: https://issues.apache.org/jira/browse/HIVE-17935 > Project: Hive > Issue Type: Bug >Reporter: Andrew Sherman >Priority: Major > Attachments: HIVE-17935.1.patch, HIVE-17935.2.patch, > HIVE-17935.3.patch, HIVE-17935.4.patch, HIVE-17935.5.patch, > HIVE-17935.6.patch, HIVE-17935.7.patch, HIVE-17935.8.patch > > > The config option hive.optimize.sort.dynamic.partition is an optimization for > Hive’s dynamic partitioning feature. It was originally implemented in > [HIVE-6455|https://issues.apache.org/jira/browse/HIVE-6455]. With this > optimization, the dynamic partition columns and bucketing columns (in case of > bucketed tables) are sorted before being fed to the reducers. Since the > partitioning and bucketing columns are sorted, each reducer can keep only one > record writer open at any time thereby reducing the memory pressure on the > reducers. There were some early problems with this optimization and it was > disabled by default in HiveConf in > [HIVE-8151|https://issues.apache.org/jira/browse/HIVE-8151]. Since then > setting hive.optimize.sort.dynamic.partition=true has been used to solve > problems where dynamic partitioning produces with (1) too many small files on > HDFS, which is bad for the cluster and can increase overhead for future Hive > queries over those partitions, and (2) OOM issues in the map tasks because it > trying to simultaneously write to 100 different files. > It now seems that the feature is probably mature enough that it can be > enabled by default. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-19841) Upgrade commons-collections to commons-collections4
[
https://issues.apache.org/jira/browse/HIVE-19841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Sherman reassigned HIVE-19841:
-
Assignee: (was: Andrew Sherman)
> Upgrade commons-collections to commons-collections4
> ---
>
> Key: HIVE-19841
> URL: https://issues.apache.org/jira/browse/HIVE-19841
> Project: Hive
> Issue Type: Task
>Affects Versions: 3.0.0, 4.0.0
>Reporter: BELUGA BEHR
>Priority: Minor
>
> Perhaps time to drink the Apache champagne (eat the Apache dog food) and
> upgrade the commons-collections library from 3.x to 4.x.
> {code}
> 3.2.2
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20030) Fix Java compile errors that show up in IntelliJ from ConvertJoinMapJoin.java and AnnotateRunTimeStatsOptimizer.java
[
https://issues.apache.org/jira/browse/HIVE-20030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Sherman updated HIVE-20030:
--
Resolution: Won't Fix
Status: Resolved (was: Patch Available)
> Fix Java compile errors that show up in IntelliJ from ConvertJoinMapJoin.java
> and AnnotateRunTimeStatsOptimizer.java
>
>
> Key: HIVE-20030
> URL: https://issues.apache.org/jira/browse/HIVE-20030
> Project: Hive
> Issue Type: Task
>Reporter: Andrew Sherman
>Assignee: Andrew Sherman
>Priority: Major
> Attachments: HIVE-20030.1.patch
>
>
> For some reason the Java compiler in IntellJ is more strict that the Oracle
> jdk compiler. Maybe this is something that can be configured away, but as it
> is simple I propose to make the code more type correct.
> {code}
> /Users/asherman/git/asf/hive2/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConvertJoinMapJoin.java
> Error:(613, 24) java: no suitable method found for
> findOperatorsUpstream(java.util.List extends
> org.apache.hadoop.hive.ql.plan.OperatorDesc>>,java.lang.Class)
> method
> org.apache.hadoop.hive.ql.exec.OperatorUtils.findOperatorsUpstream(org.apache.hadoop.hive.ql.exec.Operator,java.lang.Class)
> is not applicable
> (cannot infer type-variable(s) T
> (argument mismatch;
> java.util.List org.apache.hadoop.hive.ql.plan.OperatorDesc>> cannot be converted to
> org.apache.hadoop.hive.ql.exec.Operator))
> method
> org.apache.hadoop.hive.ql.exec.OperatorUtils.findOperatorsUpstream(java.util.Collection>,java.lang.Class)
> is not applicable
> (cannot infer type-variable(s) T
> (argument mismatch;
> java.util.List org.apache.hadoop.hive.ql.plan.OperatorDesc>> cannot be converted to
> java.util.Collection>))
> method
> org.apache.hadoop.hive.ql.exec.OperatorUtils.findOperatorsUpstream(org.apache.hadoop.hive.ql.exec.Operator,java.lang.Class,java.util.Set)
> is not applicable
> (cannot infer type-variable(s) T
> (actual and formal argument lists differ in length))
> {code}
> and
> {code}
> /Users/asherman/git/asf/hive2/ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/AnnotateRunTimeStatsOptimizer.java
> Error:(76, 12) java: no suitable method found for
> addAll(java.util.List>)
> method java.util.Collection.addAll(java.util.Collection org.apache.hadoop.hive.ql.exec.Operator org.apache.hadoop.hive.ql.plan.OperatorDesc>>) is not applicable
> (argument mismatch;
> java.util.List> cannot be
> converted to java.util.Collection org.apache.hadoop.hive.ql.exec.Operator org.apache.hadoop.hive.ql.plan.OperatorDesc>>)
> method java.util.Set.addAll(java.util.Collection org.apache.hadoop.hive.ql.exec.Operator org.apache.hadoop.hive.ql.plan.OperatorDesc>>) is not applicable
> (argument mismatch;
> java.util.List> cannot be
> converted to java.util.Collection org.apache.hadoop.hive.ql.exec.Operator org.apache.hadoop.hive.ql.plan.OperatorDesc>>)
> Error:(80, 14) java: no suitable method found for
> addAll(java.util.Set>)
> method java.util.Collection.addAll(java.util.Collection org.apache.hadoop.hive.ql.exec.Operator org.apache.hadoop.hive.ql.plan.OperatorDesc>>) is not applicable
> (argument mismatch;
> java.util.Set> cannot be converted
> to java.util.Collection extends org.apache.hadoop.hive.ql.plan.OperatorDesc>>)
> method java.util.Set.addAll(java.util.Collection org.apache.hadoop.hive.ql.exec.Operator org.apache.hadoop.hive.ql.plan.OperatorDesc>>) is not applicable
> (argument mismatch;
> java.util.Set> cannot be converted
> to java.util.Collection extends org.apache.hadoop.hive.ql.plan.OperatorDesc>>)
> Error:(85, 14) java: no suitable method found for
> addAll(java.util.Set>)
> method java.util.Collection.addAll(java.util.Collection org.apache.hadoop.hive.ql.exec.Operator org.apache.hadoop.hive.ql.plan.OperatorDesc>>) is not applicable
> (argument mismatch;
> java.util.Set> cannot be converted
> to java.util.Collection extends org.apache.hadoop.hive.ql.plan.OperatorDesc>>)
> method java.util.Set.addAll(java.util.Collection org.apache.hadoop.hive.ql.exec.Operator org.apache.hadoop.hive.ql.plan.OperatorDesc>>) is not applicable
> (argument mismatch;
> java.util.Set> cannot be converted
> to java.util.Collection extends org.apache.hadoop.hive.ql.plan.OperatorDesc>>)
> /Users/asherman/git/asf/hive2/ql/target/generated-sources/java/org/apache/hadoop/hive/ql/exec/vector/expressions/gen/IntervalYearMonthScalarAddTimestampColumn.java
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (HIVE-18116) Hive + HDFS EC Supportability and Testing Improvements
[ https://issues.apache.org/jira/browse/HIVE-18116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Sherman resolved HIVE-18116. --- Resolution: Fixed > Hive + HDFS EC Supportability and Testing Improvements > -- > > Key: HIVE-18116 > URL: https://issues.apache.org/jira/browse/HIVE-18116 > Project: Hive > Issue Type: Improvement >Reporter: Sahil Takiar >Assignee: Andrew Sherman >Priority: Major > > Now that we are on Hadoop 3.x, we can start integrating with HDFS Erasure > Coding (see > https://hadoop.apache.org/docs/r3.0.0-alpha2/hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html > for details). > First step is to add some tests using a custom CliDriver - we can do > something similar to what we did for encryption. > Next step will be some supportability improvements - like printing out in the > explain plan when a query is reading a EC file. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21023) Add test for replication to a target with hive.strict.managed.tables enabled
[ https://issues.apache.org/jira/browse/HIVE-21023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16717891#comment-16717891 ] Vihang Karajgaonkar commented on HIVE-21023: excluded TestReplicationScenariosMigration and TestReplicationWithTableMigration from batching for now. Can you send me your public keys to my email so that I can give you ssh access to the ptest server. > Add test for replication to a target with hive.strict.managed.tables enabled > > > Key: HIVE-21023 > URL: https://issues.apache.org/jira/browse/HIVE-21023 > Project: Hive > Issue Type: Bug >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Attachments: HIVE-21023.01.patch > > > Tests added are timing out in ptest run. Need to skip these test cases from > batching and run them separately. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21021) Scalar subquery with only aggregate in subquery (no group by) has unnecessary sq_count_check branch
[
https://issues.apache.org/jira/browse/HIVE-21021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-21021:
---
Status: Open (was: Patch Available)
> Scalar subquery with only aggregate in subquery (no group by) has unnecessary
> sq_count_check branch
> ---
>
> Key: HIVE-21021
> URL: https://issues.apache.org/jira/browse/HIVE-21021
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.0.0
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-21021.1.patch, HIVE-21021.2.patch,
> HIVE-21021.3.patch, HIVE-21021.4.patch
>
>
> {code:sql}
> CREATE TABLE `store_sales`(
> `ss_sold_date_sk` int,
> `ss_quantity` int,
> `ss_list_price` decimal(7,2));
> CREATE TABLE `date_dim`(
> `d_date_sk` int,
> `d_year` int);
> explain cbo with avg_sales as
> (select avg(quantity*list_price) average_sales
> from (select ss_quantity quantity
> ,ss_list_price list_price
>from store_sales
>,date_dim
>where ss_sold_date_sk = d_date_sk
> and d_year between 1999 and 2001 ) x)
> select * from store_sales where ss_list_price > (select average_sales from
> avg_sales);
> {code}
> {noformat}
> CBO PLAN:
> HiveProject(ss_sold_date_sk=[$0], ss_quantity=[$1], ss_list_price=[$2])
> HiveJoin(condition=[true], joinType=[inner], algorithm=[none], cost=[{2.0
> rows, 0.0 cpu, 0.0 io}])
> HiveJoin(condition=[>($2, $3)], joinType=[inner], algorithm=[none],
> cost=[{2.0 rows, 0.0 cpu, 0.0 io}])
> HiveProject(ss_sold_date_sk=[$0], ss_quantity=[$1], ss_list_price=[$2])
> HiveTableScan(table=[[sub, store_sales]], table:alias=[store_sales])
> HiveProject($f0=[/($0, $1)])
> HiveAggregate(group=[{}], agg#0=[sum($0)], agg#1=[count($0)])
> HiveProject($f0=[*(CAST($1):DECIMAL(10, 0), $2)])
> HiveJoin(condition=[=($0, $3)], joinType=[inner],
> algorithm=[none], cost=[{2.0 rows, 0.0 cpu, 0.0 io}])
> HiveProject(ss_sold_date_sk=[$0], ss_quantity=[$1],
> ss_list_price=[$2])
> HiveFilter(condition=[IS NOT NULL($0)])
> HiveTableScan(table=[[sub, store_sales]],
> table:alias=[store_sales])
> HiveProject(d_date_sk=[$0])
> HiveFilter(condition=[AND(BETWEEN(false, $1, 1999, 2001), IS
> NOT NULL($0))])
> HiveTableScan(table=[[sub, date_dim]],
> table:alias=[date_dim])
> HiveProject(cnt=[$0])
> HiveFilter(condition=[<=(sq_count_check($0), 1)])
> HiveProject(cnt=[$0])
> HiveAggregate(group=[{}], cnt=[COUNT()])
> HiveProject
> HiveProject($f0=[$0])
> HiveAggregate(group=[{}], agg#0=[count($0)])
> HiveJoin(condition=[=($0, $3)], joinType=[inner],
> algorithm=[none], cost=[{2.0 rows, 0.0 cpu, 0.0 io}])
> HiveProject(ss_sold_date_sk=[$0], ss_quantity=[$1],
> ss_list_price=[$2])
> HiveFilter(condition=[IS NOT NULL($0)])
> HiveTableScan(table=[[sub, store_sales]],
> table:alias=[store_sales])
> HiveProject(d_date_sk=[$0])
> HiveFilter(condition=[AND(BETWEEN(false, $1, 1999,
> 2001), IS NOT NULL($0))])
> HiveTableScan(table=[[sub, date_dim]],
> table:alias=[date_dim])
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21021) Scalar subquery with only aggregate in subquery (no group by) has unnecessary sq_count_check branch
[
https://issues.apache.org/jira/browse/HIVE-21021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-21021:
---
Status: Patch Available (was: Open)
> Scalar subquery with only aggregate in subquery (no group by) has unnecessary
> sq_count_check branch
> ---
>
> Key: HIVE-21021
> URL: https://issues.apache.org/jira/browse/HIVE-21021
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.0.0
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-21021.1.patch, HIVE-21021.2.patch,
> HIVE-21021.3.patch, HIVE-21021.4.patch
>
>
> {code:sql}
> CREATE TABLE `store_sales`(
> `ss_sold_date_sk` int,
> `ss_quantity` int,
> `ss_list_price` decimal(7,2));
> CREATE TABLE `date_dim`(
> `d_date_sk` int,
> `d_year` int);
> explain cbo with avg_sales as
> (select avg(quantity*list_price) average_sales
> from (select ss_quantity quantity
> ,ss_list_price list_price
>from store_sales
>,date_dim
>where ss_sold_date_sk = d_date_sk
> and d_year between 1999 and 2001 ) x)
> select * from store_sales where ss_list_price > (select average_sales from
> avg_sales);
> {code}
> {noformat}
> CBO PLAN:
> HiveProject(ss_sold_date_sk=[$0], ss_quantity=[$1], ss_list_price=[$2])
> HiveJoin(condition=[true], joinType=[inner], algorithm=[none], cost=[{2.0
> rows, 0.0 cpu, 0.0 io}])
> HiveJoin(condition=[>($2, $3)], joinType=[inner], algorithm=[none],
> cost=[{2.0 rows, 0.0 cpu, 0.0 io}])
> HiveProject(ss_sold_date_sk=[$0], ss_quantity=[$1], ss_list_price=[$2])
> HiveTableScan(table=[[sub, store_sales]], table:alias=[store_sales])
> HiveProject($f0=[/($0, $1)])
> HiveAggregate(group=[{}], agg#0=[sum($0)], agg#1=[count($0)])
> HiveProject($f0=[*(CAST($1):DECIMAL(10, 0), $2)])
> HiveJoin(condition=[=($0, $3)], joinType=[inner],
> algorithm=[none], cost=[{2.0 rows, 0.0 cpu, 0.0 io}])
> HiveProject(ss_sold_date_sk=[$0], ss_quantity=[$1],
> ss_list_price=[$2])
> HiveFilter(condition=[IS NOT NULL($0)])
> HiveTableScan(table=[[sub, store_sales]],
> table:alias=[store_sales])
> HiveProject(d_date_sk=[$0])
> HiveFilter(condition=[AND(BETWEEN(false, $1, 1999, 2001), IS
> NOT NULL($0))])
> HiveTableScan(table=[[sub, date_dim]],
> table:alias=[date_dim])
> HiveProject(cnt=[$0])
> HiveFilter(condition=[<=(sq_count_check($0), 1)])
> HiveProject(cnt=[$0])
> HiveAggregate(group=[{}], cnt=[COUNT()])
> HiveProject
> HiveProject($f0=[$0])
> HiveAggregate(group=[{}], agg#0=[count($0)])
> HiveJoin(condition=[=($0, $3)], joinType=[inner],
> algorithm=[none], cost=[{2.0 rows, 0.0 cpu, 0.0 io}])
> HiveProject(ss_sold_date_sk=[$0], ss_quantity=[$1],
> ss_list_price=[$2])
> HiveFilter(condition=[IS NOT NULL($0)])
> HiveTableScan(table=[[sub, store_sales]],
> table:alias=[store_sales])
> HiveProject(d_date_sk=[$0])
> HiveFilter(condition=[AND(BETWEEN(false, $1, 1999,
> 2001), IS NOT NULL($0))])
> HiveTableScan(table=[[sub, date_dim]],
> table:alias=[date_dim])
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21021) Scalar subquery with only aggregate in subquery (no group by) has unnecessary sq_count_check branch
[
https://issues.apache.org/jira/browse/HIVE-21021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-21021:
---
Attachment: HIVE-21021.4.patch
> Scalar subquery with only aggregate in subquery (no group by) has unnecessary
> sq_count_check branch
> ---
>
> Key: HIVE-21021
> URL: https://issues.apache.org/jira/browse/HIVE-21021
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.0.0
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-21021.1.patch, HIVE-21021.2.patch,
> HIVE-21021.3.patch, HIVE-21021.4.patch
>
>
> {code:sql}
> CREATE TABLE `store_sales`(
> `ss_sold_date_sk` int,
> `ss_quantity` int,
> `ss_list_price` decimal(7,2));
> CREATE TABLE `date_dim`(
> `d_date_sk` int,
> `d_year` int);
> explain cbo with avg_sales as
> (select avg(quantity*list_price) average_sales
> from (select ss_quantity quantity
> ,ss_list_price list_price
>from store_sales
>,date_dim
>where ss_sold_date_sk = d_date_sk
> and d_year between 1999 and 2001 ) x)
> select * from store_sales where ss_list_price > (select average_sales from
> avg_sales);
> {code}
> {noformat}
> CBO PLAN:
> HiveProject(ss_sold_date_sk=[$0], ss_quantity=[$1], ss_list_price=[$2])
> HiveJoin(condition=[true], joinType=[inner], algorithm=[none], cost=[{2.0
> rows, 0.0 cpu, 0.0 io}])
> HiveJoin(condition=[>($2, $3)], joinType=[inner], algorithm=[none],
> cost=[{2.0 rows, 0.0 cpu, 0.0 io}])
> HiveProject(ss_sold_date_sk=[$0], ss_quantity=[$1], ss_list_price=[$2])
> HiveTableScan(table=[[sub, store_sales]], table:alias=[store_sales])
> HiveProject($f0=[/($0, $1)])
> HiveAggregate(group=[{}], agg#0=[sum($0)], agg#1=[count($0)])
> HiveProject($f0=[*(CAST($1):DECIMAL(10, 0), $2)])
> HiveJoin(condition=[=($0, $3)], joinType=[inner],
> algorithm=[none], cost=[{2.0 rows, 0.0 cpu, 0.0 io}])
> HiveProject(ss_sold_date_sk=[$0], ss_quantity=[$1],
> ss_list_price=[$2])
> HiveFilter(condition=[IS NOT NULL($0)])
> HiveTableScan(table=[[sub, store_sales]],
> table:alias=[store_sales])
> HiveProject(d_date_sk=[$0])
> HiveFilter(condition=[AND(BETWEEN(false, $1, 1999, 2001), IS
> NOT NULL($0))])
> HiveTableScan(table=[[sub, date_dim]],
> table:alias=[date_dim])
> HiveProject(cnt=[$0])
> HiveFilter(condition=[<=(sq_count_check($0), 1)])
> HiveProject(cnt=[$0])
> HiveAggregate(group=[{}], cnt=[COUNT()])
> HiveProject
> HiveProject($f0=[$0])
> HiveAggregate(group=[{}], agg#0=[count($0)])
> HiveJoin(condition=[=($0, $3)], joinType=[inner],
> algorithm=[none], cost=[{2.0 rows, 0.0 cpu, 0.0 io}])
> HiveProject(ss_sold_date_sk=[$0], ss_quantity=[$1],
> ss_list_price=[$2])
> HiveFilter(condition=[IS NOT NULL($0)])
> HiveTableScan(table=[[sub, store_sales]],
> table:alias=[store_sales])
> HiveProject(d_date_sk=[$0])
> HiveFilter(condition=[AND(BETWEEN(false, $1, 1999,
> 2001), IS NOT NULL($0))])
> HiveTableScan(table=[[sub, date_dim]],
> table:alias=[date_dim])
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-16957) Support CTAS for auto gather column stats
[
https://issues.apache.org/jira/browse/HIVE-16957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-16957:
---
Attachment: HIVE-16957.patch
> Support CTAS for auto gather column stats
> -
>
> Key: HIVE-16957
> URL: https://issues.apache.org/jira/browse/HIVE-16957
> Project: Hive
> Issue Type: Sub-task
>Reporter: Pengcheng Xiong
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-16957.patch
>
>
> The idea is to rely as much as possible on the logic in
> ColumnStatsSemanticAnalyzer as other operations do. In particular, they
> create a 'analyze table t compute statistics for columns', use
> ColumnStatsSemanticAnalyzer to parse it, and connect resulting plan to
> existing INSERT/INSERT OVERWRITE statement. The challenge for CTAS or CREATE
> MATERIALIZED VIEW is that the table object does not exist yet, hence we
> cannot rely fully on ColumnStatsSemanticAnalyzer.
> Thus, we use same process, but ColumnStatsSemanticAnalyzer produces a
> statement for column stats collection that uses a table values clause instead
> of the original table reference:
> {code}
> select compute_stats(col1), compute_stats(col2), compute_stats(col3)
> from table(values(cast(null as int), cast(null as int), cast(null as
> string))) as t(col1, col2, col3);
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work started] (HIVE-16957) Support CTAS for auto gather column stats
[
https://issues.apache.org/jira/browse/HIVE-16957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-16957 started by Jesus Camacho Rodriguez.
--
> Support CTAS for auto gather column stats
> -
>
> Key: HIVE-16957
> URL: https://issues.apache.org/jira/browse/HIVE-16957
> Project: Hive
> Issue Type: Sub-task
>Reporter: Pengcheng Xiong
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
>
> The idea is to rely as much as possible on the logic in
> ColumnStatsSemanticAnalyzer as other operations do. In particular, they
> create a 'analyze table t compute statistics for columns', use
> ColumnStatsSemanticAnalyzer to parse it, and connect resulting plan to
> existing INSERT/INSERT OVERWRITE statement. The challenge for CTAS or CREATE
> MATERIALIZED VIEW is that the table object does not exist yet, hence we
> cannot rely fully on ColumnStatsSemanticAnalyzer.
> Thus, we use same process, but ColumnStatsSemanticAnalyzer produces a
> statement for column stats collection that uses a table values clause instead
> of the original table reference:
> {code}
> select compute_stats(col1), compute_stats(col2), compute_stats(col3)
> from table(values(cast(null as int), cast(null as int), cast(null as
> string))) as t(col1, col2, col3);
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-16957) Support CTAS for auto gather column stats
[
https://issues.apache.org/jira/browse/HIVE-16957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16718424#comment-16718424
]
Hive QA commented on HIVE-16957:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
55s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
45s{color} | {color:blue} ql in master has 2311 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
41s{color} | {color:red} ql: The patch generated 3 new + 562 unchanged - 5
fixed = 565 total (was 567) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m
57s{color} | {color:red} ql generated 1 new + 2309 unchanged - 2 fixed = 2310
total (was 2311) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
56s{color} | {color:red} ql generated 2 new + 98 unchanged - 2 fixed = 100
total (was 100) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 23m 10s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| |
org.apache.hadoop.hive.ql.parse.ColumnStatsSemanticAnalyzer.genPartitionClause(Table,
Map) makes inefficient use of keySet iterator instead of entrySet iterator At
ColumnStatsSemanticAnalyzer.java:of keySet iterator instead of entrySet
iterator At ColumnStatsSemanticAnalyzer.java:[line 160] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15271/dev-support/hive-personality.sh
|
| git revision | master / b650083 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15271/yetus/diff-checkstyle-ql.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15271/yetus/new-findbugs-ql.html
|
| javadoc |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15271/yetus/diff-javadoc-javadoc-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15271/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Support CTAS for auto gather column stats
> -
>
> Key: HIVE-16957
> URL: https://issues.apache.org/jira/browse/HIVE-16957
> Project: Hive
> Issue Type: Sub-task
>Reporter: Pengcheng Xiong
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-16957.patch
>
>
> The idea is to rely as much as possible on the logic in
> ColumnStatsSemanticAnalyzer as other operations do. In particular, they
> create a 'analyze table t compute statistics for columns', use
> ColumnStatsSemanticAnalyzer to parse it, and connect resulting plan to
> existing INSERT/INSERT OVERWRITE statement. The challenge for CTAS or CREATE
> MATERIALIZED VIEW is that the table object does not exist yet, hence we
> cannot rely fully on ColumnStatsSemanticAnalyzer.
> Thus, we use same
[jira] [Commented] (HIVE-16957) Support CTAS for auto gather column stats
[
https://issues.apache.org/jira/browse/HIVE-16957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16718441#comment-16718441
]
Hive QA commented on HIVE-16957:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12951441/HIVE-16957.patch
{color:green}SUCCESS:{color} +1 due to 27 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 136 failed/errored test(s), 15660 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[materialized_view_create_rewrite]
(batchId=270)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_table_update_status]
(batchId=86)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_table_update_status_disable_bitvector]
(batchId=85)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_rp_cross_product_check_2]
(batchId=22)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ctas] (batchId=7)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ctas_colname]
(batchId=63)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ctas_uses_database_location]
(batchId=37)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cte_3] (batchId=37)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cte_4] (batchId=91)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cte_mat_3] (batchId=26)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cte_mat_4] (batchId=6)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cte_mat_5] (batchId=3)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[decimal_serde]
(batchId=90)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[explain_ddl] (batchId=51)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby_duplicate_key]
(batchId=7)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join42] (batchId=26)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join_filters_overlap]
(batchId=37)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[masking_mv] (batchId=88)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[merge3] (batchId=63)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mm_default] (batchId=90)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[named_column_join]
(batchId=83)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[nonmr_fetch] (batchId=22)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[nullformatCTAS]
(batchId=42)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_createas1]
(batchId=95)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parallel_orderby]
(batchId=58)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[rcfile_default_format]
(batchId=49)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample_islocalmode_hook_use_metadata]
(batchId=67)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[show_materialized_views]
(batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoin_noskew]
(batchId=18)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoin_onesideskew]
(batchId=77)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[smb_mapjoin9]
(batchId=44)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats5] (batchId=53)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[union24] (batchId=65)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[union25] (batchId=75)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_decimal_6]
(batchId=15)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_mv]
(batchId=194)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[cte_4]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_nullscan]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_udf]
(batchId=155)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llapdecider]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_ppd_basic]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[rcfile_createas1]
(batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[unionDistinct_1]
(batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[convert_decimal64_to_decimal]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cross_prod_1]
(batchId=177)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cross_prod_3]
(batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cross_prod_4]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cross_product_check_1]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cross_product_check_2]
(batchId=182)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[ctas]
(batchId=159)
[jira] [Updated] (HIVE-20911) External Table Replication for Hive
[
https://issues.apache.org/jira/browse/HIVE-20911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HIVE-20911:
--
Labels: pull-request-available (was: )
> External Table Replication for Hive
> ---
>
> Key: HIVE-20911
> URL: https://issues.apache.org/jira/browse/HIVE-20911
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 4.0.0
>Reporter: anishek
>Assignee: anishek
>Priority: Critical
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-20911.01.patch
>
>
> External tables are not replicated currently as part of hive replication. As
> part of this jira we want to enable that.
> Approach:
> * Target cluster will have a top level base directory config that will be
> used to copy all data relevant to external tables. This will be provided via
> the *with* clause in the *repl load* command. This base path will be prefixed
> to the path of the same external table on source cluster.
> * Since changes to directories on the external table can happen without hive
> knowing it, hence we cant capture the relevant events when ever new data is
> added or removed, we will have to copy the data from the source path to
> target path for external tables every time we run incremental replication.
> ** this will require incremental *repl dump* to now create an additional
> file *\_external\_tables\_info* with data in the following form
> {code}
> tableName,base64Encoded(tableDataLocation)
> {code}
> In case there are different partitions in the table pointing to different
> locations there will be multiple entries in the file for the same table name
> with location pointing to different partition locations. For partitions
> created in a table without specifying the _set location_ command will be
> within the same table Data location and hence there will not be different
> entries in the file above
> ** *repl load* will read the *\_external\_tables\_info* to identify what
> locations are to be copied from source to target and create corresponding
> tasks for them.
> * New External tables will be created with metadata only with no data copied
> as part of regular tasks while incremental load/bootstrap load.
> * Bootstrap dump will also create *\_external\_tables\_info* which will be
> used to copy data from source to target as part of boostrap load.
> * Bootstrap load will create a DAG, that can use parallelism in the execution
> phase, the hdfs copy related tasks are created, once the bootstrap phase is
> complete.
> * Since incremental load results in a DAG with only sequential execution (
> events applied in sequence ) to effectively use the parallelism capability in
> execution mode, we create tasks for hdfs copy along with the incremental DAG.
> This requires a few basic calculations to approximately meet the configured
> value in "hive.repl.approx.max.load.tasks"
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-16957) Support CTAS for auto gather column stats
[
https://issues.apache.org/jira/browse/HIVE-16957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16718351#comment-16718351
]
Jesus Camacho Rodriguez commented on HIVE-16957:
ALTER MV... REBUILD is working correctly. When incremental rebuild translates
into a MERGE operation, i.e. MV contains a Group By statement, column stats are
not present because the MERGE contains in turn an UPDATE operation, which
currently invalidates column stats. When incremental rebuild translates into
INSERT operation, i.e., MV does not contain a Group By statement, column stats
for the MV are updated correctly.
> Support CTAS for auto gather column stats
> -
>
> Key: HIVE-16957
> URL: https://issues.apache.org/jira/browse/HIVE-16957
> Project: Hive
> Issue Type: Sub-task
>Reporter: Pengcheng Xiong
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-16957.patch
>
>
> The idea is to rely as much as possible on the logic in
> ColumnStatsSemanticAnalyzer as other operations do. In particular, they
> create a 'analyze table t compute statistics for columns', use
> ColumnStatsSemanticAnalyzer to parse it, and connect resulting plan to
> existing INSERT/INSERT OVERWRITE statement. The challenge for CTAS or CREATE
> MATERIALIZED VIEW is that the table object does not exist yet, hence we
> cannot rely fully on ColumnStatsSemanticAnalyzer.
> Thus, we use same process, but ColumnStatsSemanticAnalyzer produces a
> statement for column stats collection that uses a table values clause instead
> of the original table reference:
> {code}
> select compute_stats(col1), compute_stats(col2), compute_stats(col3)
> from table(values(cast(null as int), cast(null as int), cast(null as
> string))) as t(col1, col2, col3);
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21021) Scalar subquery with only aggregate in subquery (no group by) has unnecessary sq_count_check branch
[
https://issues.apache.org/jira/browse/HIVE-21021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16718376#comment-16718376
]
Hive QA commented on HIVE-21021:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
57s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
47s{color} | {color:blue} ql in master has 2311 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 22m 57s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15268/dev-support/hive-personality.sh
|
| git revision | master / b650083 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15268/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Scalar subquery with only aggregate in subquery (no group by) has unnecessary
> sq_count_check branch
> ---
>
> Key: HIVE-21021
> URL: https://issues.apache.org/jira/browse/HIVE-21021
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.0.0
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-21021.1.patch, HIVE-21021.2.patch,
> HIVE-21021.3.patch, HIVE-21021.4.patch
>
>
> {code:sql}
> CREATE TABLE `store_sales`(
> `ss_sold_date_sk` int,
> `ss_quantity` int,
> `ss_list_price` decimal(7,2));
> CREATE TABLE `date_dim`(
> `d_date_sk` int,
> `d_year` int);
> explain cbo with avg_sales as
> (select avg(quantity*list_price) average_sales
> from (select ss_quantity quantity
> ,ss_list_price list_price
>from store_sales
>,date_dim
>where ss_sold_date_sk = d_date_sk
> and d_year between 1999 and 2001 ) x)
> select * from store_sales where ss_list_price > (select average_sales from
> avg_sales);
> {code}
> {noformat}
> CBO PLAN:
> HiveProject(ss_sold_date_sk=[$0], ss_quantity=[$1], ss_list_price=[$2])
> HiveJoin(condition=[true], joinType=[inner], algorithm=[none], cost=[{2.0
> rows, 0.0 cpu, 0.0 io}])
> HiveJoin(condition=[>($2, $3)], joinType=[inner], algorithm=[none],
> cost=[{2.0 rows, 0.0 cpu, 0.0 io}])
> HiveProject(ss_sold_date_sk=[$0], ss_quantity=[$1], ss_list_price=[$2])
> HiveTableScan(table=[[sub, store_sales]], table:alias=[store_sales])
> HiveProject($f0=[/($0,
[jira] [Updated] (HIVE-21022) Fix remote metastore tests which use ZooKeeper
[ https://issues.apache.org/jira/browse/HIVE-21022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Bapat updated HIVE-21022: -- Attachment: HIVE-21022.03 Status: Patch Available (was: In Progress) Because of a bug in previous patch the loop kept connecting to the ZooKeeper even after success. Hence the tests failed with timeout. Attaching here a patch which fixes that bug. > Fix remote metastore tests which use ZooKeeper > -- > > Key: HIVE-21022 > URL: https://issues.apache.org/jira/browse/HIVE-21022 > Project: Hive > Issue Type: Bug > Components: Tests >Affects Versions: 4.0.0 >Reporter: Ashutosh Bapat >Assignee: Ashutosh Bapat >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21022.01, HIVE-21022.01, HIVE-21022.01, > HIVE-21022.02, HIVE-21022.02.patch, HIVE-21022.03 > > > Per [~vgarg]'s comment on HIVE-20794 at > https://issues.apache.org/jira/browse/HIVE-20794?focusedCommentId=16714093=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16714093, > the remote metatstore tests using ZooKeeper are flaky. They are failing with > error "Got exception: org.apache.zookeeper.KeeperException$NoNodeException > KeeperErrorCode = NoNode for /hs2mszktest". > Both of these tests are using the same root namespace and hence the reason > for this failure could be that the root namespace becomes unavailable to one > test when the other drops it. The drop seems to be happening automatically > through TestingServer code. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21022) Fix remote metastore tests which use ZooKeeper
[ https://issues.apache.org/jira/browse/HIVE-21022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Bapat updated HIVE-21022: -- Status: In Progress (was: Patch Available) > Fix remote metastore tests which use ZooKeeper > -- > > Key: HIVE-21022 > URL: https://issues.apache.org/jira/browse/HIVE-21022 > Project: Hive > Issue Type: Bug > Components: Tests >Affects Versions: 4.0.0 >Reporter: Ashutosh Bapat >Assignee: Ashutosh Bapat >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21022.01, HIVE-21022.01, HIVE-21022.01, > HIVE-21022.02, HIVE-21022.02.patch > > > Per [~vgarg]'s comment on HIVE-20794 at > https://issues.apache.org/jira/browse/HIVE-20794?focusedCommentId=16714093=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16714093, > the remote metatstore tests using ZooKeeper are flaky. They are failing with > error "Got exception: org.apache.zookeeper.KeeperException$NoNodeException > KeeperErrorCode = NoNode for /hs2mszktest". > Both of these tests are using the same root namespace and hence the reason > for this failure could be that the root namespace becomes unavailable to one > test when the other drops it. The drop seems to be happening automatically > through TestingServer code. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21022) Fix remote metastore tests which use ZooKeeper
[
https://issues.apache.org/jira/browse/HIVE-21022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16718470#comment-16718470
]
Hive QA commented on HIVE-21022:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
45s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
48s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
51s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
13s{color} | {color:blue} standalone-metastore/metastore-common in master has
29 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
3s{color} | {color:blue} standalone-metastore/metastore-server in master has
188 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 21m 34s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15272/dev-support/hive-personality.sh
|
| git revision | master / b650083 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: standalone-metastore/metastore-common
standalone-metastore/metastore-server U: standalone-metastore |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15272/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Fix remote metastore tests which use ZooKeeper
> --
>
> Key: HIVE-21022
> URL: https://issues.apache.org/jira/browse/HIVE-21022
> Project: Hive
> Issue Type: Bug
> Components: Tests
>Affects Versions: 4.0.0
>Reporter: Ashutosh Bapat
>Assignee: Ashutosh Bapat
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-21022.01, HIVE-21022.01, HIVE-21022.01,
> HIVE-21022.02, HIVE-21022.02.patch, HIVE-21022.03
>
>
> Per [~vgarg]'s comment on HIVE-20794 at
> https://issues.apache.org/jira/browse/HIVE-20794?focusedCommentId=16714093=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16714093,
> the remote metatstore tests using ZooKeeper are flaky. They are failing with
> error "Got exception: org.apache.zookeeper.KeeperException$NoNodeException
> KeeperErrorCode = NoNode for /hs2mszktest".
> Both of these tests are using the same root namespace and hence the reason
> for this failure could be that the root
[jira] [Commented] (HIVE-20911) External Table Replication for Hive
[
https://issues.apache.org/jira/browse/HIVE-20911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16718483#comment-16718483
]
anishek commented on HIVE-20911:
submitting initial patch for tests,
[~maheshk114]/[~sankarh]/[~ashutosh.bapat] please review!
> External Table Replication for Hive
> ---
>
> Key: HIVE-20911
> URL: https://issues.apache.org/jira/browse/HIVE-20911
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 4.0.0
>Reporter: anishek
>Assignee: anishek
>Priority: Critical
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-20911.01.patch
>
>
> External tables are not replicated currently as part of hive replication. As
> part of this jira we want to enable that.
> Approach:
> * Target cluster will have a top level base directory config that will be
> used to copy all data relevant to external tables. This will be provided via
> the *with* clause in the *repl load* command. This base path will be prefixed
> to the path of the same external table on source cluster.
> * Since changes to directories on the external table can happen without hive
> knowing it, hence we cant capture the relevant events when ever new data is
> added or removed, we will have to copy the data from the source path to
> target path for external tables every time we run incremental replication.
> ** this will require incremental *repl dump* to now create an additional
> file *\_external\_tables\_info* with data in the following form
> {code}
> tableName,base64Encoded(tableDataLocation)
> {code}
> In case there are different partitions in the table pointing to different
> locations there will be multiple entries in the file for the same table name
> with location pointing to different partition locations. For partitions
> created in a table without specifying the _set location_ command will be
> within the same table Data location and hence there will not be different
> entries in the file above
> ** *repl load* will read the *\_external\_tables\_info* to identify what
> locations are to be copied from source to target and create corresponding
> tasks for them.
> * New External tables will be created with metadata only with no data copied
> as part of regular tasks while incremental load/bootstrap load.
> * Bootstrap dump will also create *\_external\_tables\_info* which will be
> used to copy data from source to target as part of boostrap load.
> * Bootstrap load will create a DAG, that can use parallelism in the execution
> phase, the hdfs copy related tasks are created, once the bootstrap phase is
> complete.
> * Since incremental load results in a DAG with only sequential execution (
> events applied in sequence ) to effectively use the parallelism capability in
> execution mode, we create tasks for hdfs copy along with the incremental DAG.
> This requires a few basic calculations to approximately meet the configured
> value in "hive.repl.approx.max.load.tasks"
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21022) Fix remote metastore tests which use ZooKeeper
[ https://issues.apache.org/jira/browse/HIVE-21022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Bapat updated HIVE-21022: -- Status: In Progress (was: Patch Available) > Fix remote metastore tests which use ZooKeeper > -- > > Key: HIVE-21022 > URL: https://issues.apache.org/jira/browse/HIVE-21022 > Project: Hive > Issue Type: Bug > Components: Tests >Affects Versions: 4.0.0 >Reporter: Ashutosh Bapat >Assignee: Ashutosh Bapat >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21022.01, HIVE-21022.01, HIVE-21022.01, > HIVE-21022.02, HIVE-21022.02.patch, HIVE-21022.03 > > > Per [~vgarg]'s comment on HIVE-20794 at > https://issues.apache.org/jira/browse/HIVE-20794?focusedCommentId=16714093=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16714093, > the remote metatstore tests using ZooKeeper are flaky. They are failing with > error "Got exception: org.apache.zookeeper.KeeperException$NoNodeException > KeeperErrorCode = NoNode for /hs2mszktest". > Both of these tests are using the same root namespace and hence the reason > for this failure could be that the root namespace becomes unavailable to one > test when the other drops it. The drop seems to be happening automatically > through TestingServer code. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HIVE-16957) Support CTAS for auto gather column stats
[
https://issues.apache.org/jira/browse/HIVE-16957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16718332#comment-16718332
]
Jesus Camacho Rodriguez edited comment on HIVE-16957 at 12/12/18 1:33 AM:
--
Uploaded initial patch to trigger ptests. I realized that ALTER MV... REBUILD
is still not collecting the column stats automatically yet for incremental
rebuild, I will work on that.
Cc [~ashutoshc]
was (Author: jcamachorodriguez):
Uploaded initial patch to trigger ptests. I realized that ALTER MV... REBUILD
is still not collecting the column stats automatically yet, I will work on that.
Cc [~ashutoshc]
> Support CTAS for auto gather column stats
> -
>
> Key: HIVE-16957
> URL: https://issues.apache.org/jira/browse/HIVE-16957
> Project: Hive
> Issue Type: Sub-task
>Reporter: Pengcheng Xiong
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-16957.patch
>
>
> The idea is to rely as much as possible on the logic in
> ColumnStatsSemanticAnalyzer as other operations do. In particular, they
> create a 'analyze table t compute statistics for columns', use
> ColumnStatsSemanticAnalyzer to parse it, and connect resulting plan to
> existing INSERT/INSERT OVERWRITE statement. The challenge for CTAS or CREATE
> MATERIALIZED VIEW is that the table object does not exist yet, hence we
> cannot rely fully on ColumnStatsSemanticAnalyzer.
> Thus, we use same process, but ColumnStatsSemanticAnalyzer produces a
> statement for column stats collection that uses a table values clause instead
> of the original table reference:
> {code}
> select compute_stats(col1), compute_stats(col2), compute_stats(col3)
> from table(values(cast(null as int), cast(null as int), cast(null as
> string))) as t(col1, col2, col3);
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-17935) Turn on hive.optimize.sort.dynamic.partition by default
[
https://issues.apache.org/jira/browse/HIVE-17935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16718413#comment-16718413
]
Hive QA commented on HIVE-17935:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12908851/HIVE-17935.8.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15270/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15270/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15270/
Messages:
{noformat}
This message was trimmed, see log for full details
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-15270/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2018-12-12 02:54:29.157
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at b650083 HIVE-16100: Dynamic Sorted Partition optimizer loses
sibling operators (Vineet Garg, Gopal V reviewed by Ashutosh Chauhan)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at b650083 HIVE-16100: Dynamic Sorted Partition optimizer loses
sibling operators (Vineet Garg, Gopal V reviewed by Ashutosh Chauhan)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2018-12-12 02:54:29.717
+ rm -rf ../yetus_PreCommit-HIVE-Build-15270
+ mkdir ../yetus_PreCommit-HIVE-Build-15270
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-15270
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-15270/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error: patch failed:
ql/src/test/results/clientpositive/spark/load_dyn_part1.q.out:61
Falling back to three-way merge...
Applied patch to
'ql/src/test/results/clientpositive/spark/load_dyn_part1.q.out' with conflicts.
error: patch failed:
ql/src/test/results/clientpositive/spark/load_dyn_part10.q.out:49
Falling back to three-way merge...
Applied patch to
'ql/src/test/results/clientpositive/spark/load_dyn_part10.q.out' with conflicts.
error: patch failed:
ql/src/test/results/clientpositive/spark/load_dyn_part14.q.out:79
Falling back to three-way merge...
Applied patch to
'ql/src/test/results/clientpositive/spark/load_dyn_part14.q.out' with conflicts.
error: patch failed:
ql/src/test/results/clientpositive/spark/load_dyn_part3.q.out:47
Falling back to three-way merge...
Applied patch to
'ql/src/test/results/clientpositive/spark/load_dyn_part3.q.out' with conflicts.
error: patch failed:
ql/src/test/results/clientpositive/spark/load_dyn_part4.q.out:57
Falling back to three-way merge...
Applied patch to
'ql/src/test/results/clientpositive/spark/load_dyn_part4.q.out' with conflicts.
error: patch failed:
ql/src/test/results/clientpositive/spark/load_dyn_part5.q.out:34
Falling back to three-way merge...
Applied patch to
'ql/src/test/results/clientpositive/spark/load_dyn_part5.q.out' with conflicts.
error: patch failed:
ql/src/test/results/clientpositive/spark/load_dyn_part8.q.out:53
Falling back to three-way merge...
Applied patch to
'ql/src/test/results/clientpositive/spark/load_dyn_part8.q.out' with conflicts.
error: patch failed:
ql/src/test/results/clientpositive/spark/load_dyn_part9.q.out:49
Falling back to three-way merge...
Applied patch to
'ql/src/test/results/clientpositive/spark/load_dyn_part9.q.out' with conflicts.
error: patch failed:
ql/src/test/results/clientpositive/spark/orc_merge2.q.out:37
Falling back to three-way merge...
Applied patch to 'ql/src/test/results/clientpositive/spark/orc_merge2.q.out'
with conflicts.
error: patch failed: ql/src/test/results/clientpositive/spark/stats2.q.out:19
Falling back to three-way merge...
Applied patch to 'ql/src/test/results/clientpositive/spark/stats2.q.out' with
conflicts.
error: patch failed: ql/src/test/results/clientpositive/spark/union14.q.out:122
Falling back to three-way merge...
[jira] [Commented] (HIVE-20911) External Table Replication for Hive
[
https://issues.apache.org/jira/browse/HIVE-20911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16718482#comment-16718482
]
ASF GitHub Bot commented on HIVE-20911:
---
GitHub user anishek opened a pull request:
https://github.com/apache/hive/pull/506
HIVE-20911: External Table Replication for Hive
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/anishek/hive HIVE-20911
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/hive/pull/506.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #506
commit 57fa9f0c2c3e00e92b8ba472b05668ff3767c5f4
Author: Anishek Agarwal
Date: 2018-10-30T08:24:41Z
HIVE-20911: External Table Replication for Hive
> External Table Replication for Hive
> ---
>
> Key: HIVE-20911
> URL: https://issues.apache.org/jira/browse/HIVE-20911
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 4.0.0
>Reporter: anishek
>Assignee: anishek
>Priority: Critical
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-20911.01.patch
>
>
> External tables are not replicated currently as part of hive replication. As
> part of this jira we want to enable that.
> Approach:
> * Target cluster will have a top level base directory config that will be
> used to copy all data relevant to external tables. This will be provided via
> the *with* clause in the *repl load* command. This base path will be prefixed
> to the path of the same external table on source cluster.
> * Since changes to directories on the external table can happen without hive
> knowing it, hence we cant capture the relevant events when ever new data is
> added or removed, we will have to copy the data from the source path to
> target path for external tables every time we run incremental replication.
> ** this will require incremental *repl dump* to now create an additional
> file *\_external\_tables\_info* with data in the following form
> {code}
> tableName,base64Encoded(tableDataLocation)
> {code}
> In case there are different partitions in the table pointing to different
> locations there will be multiple entries in the file for the same table name
> with location pointing to different partition locations. For partitions
> created in a table without specifying the _set location_ command will be
> within the same table Data location and hence there will not be different
> entries in the file above
> ** *repl load* will read the *\_external\_tables\_info* to identify what
> locations are to be copied from source to target and create corresponding
> tasks for them.
> * New External tables will be created with metadata only with no data copied
> as part of regular tasks while incremental load/bootstrap load.
> * Bootstrap dump will also create *\_external\_tables\_info* which will be
> used to copy data from source to target as part of boostrap load.
> * Bootstrap load will create a DAG, that can use parallelism in the execution
> phase, the hdfs copy related tasks are created, once the bootstrap phase is
> complete.
> * Since incremental load results in a DAG with only sequential execution (
> events applied in sequence ) to effectively use the parallelism capability in
> execution mode, we create tasks for hdfs copy along with the incremental DAG.
> This requires a few basic calculations to approximately meet the configured
> value in "hive.repl.approx.max.load.tasks"
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21022) Fix remote metastore tests which use ZooKeeper
[ https://issues.apache.org/jira/browse/HIVE-21022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Bapat updated HIVE-21022: -- Attachment: HIVE-21022.03 Status: Patch Available (was: In Progress) Since the test is not failing always, re-triggering ptest by attaching the same patch again to verify that the problem is really fixed. > Fix remote metastore tests which use ZooKeeper > -- > > Key: HIVE-21022 > URL: https://issues.apache.org/jira/browse/HIVE-21022 > Project: Hive > Issue Type: Bug > Components: Tests >Affects Versions: 4.0.0 >Reporter: Ashutosh Bapat >Assignee: Ashutosh Bapat >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21022.01, HIVE-21022.01, HIVE-21022.01, > HIVE-21022.02, HIVE-21022.02.patch, HIVE-21022.03, HIVE-21022.03 > > > Per [~vgarg]'s comment on HIVE-20794 at > https://issues.apache.org/jira/browse/HIVE-20794?focusedCommentId=16714093=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16714093, > the remote metatstore tests using ZooKeeper are flaky. They are failing with > error "Got exception: org.apache.zookeeper.KeeperException$NoNodeException > KeeperErrorCode = NoNode for /hs2mszktest". > Both of these tests are using the same root namespace and hence the reason > for this failure could be that the root namespace becomes unavailable to one > test when the other drops it. The drop seems to be happening automatically > through TestingServer code. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20911) External Table Replication for Hive
[
https://issues.apache.org/jira/browse/HIVE-20911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16718507#comment-16718507
]
Hive QA commented on HIVE-20911:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
36s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
54s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
15s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
32s{color} | {color:blue} common in master has 65 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
37s{color} | {color:blue} ql in master has 2311 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
38s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
26s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
23s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
55s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
16s{color} | {color:red} common: The patch generated 1 new + 454 unchanged - 0
fixed = 455 total (was 454) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
39s{color} | {color:red} ql: The patch generated 17 new + 329 unchanged - 12
fixed = 346 total (was 341) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
21s{color} | {color:red} itests/hive-unit: The patch generated 23 new + 708
unchanged - 8 fixed = 731 total (was 716) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
0s{color} | {color:red} ql generated 4 new + 2310 unchanged - 1 fixed = 2314
total (was 2311) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 31m 17s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | Found reliance on default encoding in
org.apache.hadoop.hive.ql.exec.repl.ReplExternalTables$Reader.reader(FileSystem,
Path):in
org.apache.hadoop.hive.ql.exec.repl.ReplExternalTables$Reader.reader(FileSystem,
Path): new java.io.InputStreamReader(InputStream) At
ReplExternalTables.java:[line 214] |
| | The field
org.apache.hadoop.hive.ql.exec.repl.ReplLoadWork.pathsToCopyIterator is
transient but isn't set by deserialization In ReplLoadWork.java:but isn't set
by deserialization In ReplLoadWork.java |
| | Write to static field
org.apache.hadoop.hive.ql.exec.repl.incremental.IncrementalLoadTasksBuilder.numIteration
from instance method
org.apache.hadoop.hive.ql.exec.repl.incremental.IncrementalLoadTasksBuilder.build(DriverContext,
Hive, Logger, ReplLoadWork, TaskTracker) At
IncrementalLoadTasksBuilder.java:from instance method
org.apache.hadoop.hive.ql.exec.repl.incremental.IncrementalLoadTasksBuilder.build(DriverContext,
Hive, Logger, ReplLoadWork, TaskTracker) At
IncrementalLoadTasksBuilder.java:[line 100] |
| | Exception is caught when Exception is not thrown in
org.apache.hadoop.hive.ql.parse.repl.load.message.TableHandler.handle(MessageHandler$Context)
At
[jira] [Commented] (HIVE-20914) MRScratchDir permission denied when "hive.server2.enable.doAs", "hive.exec.submitviachild" are set to "true" and impersonated/proxy user is used
[
https://issues.apache.org/jira/browse/HIVE-20914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16718342#comment-16718342
]
Hive QA commented on HIVE-20914:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12951422/HIVE-20914.9.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 46 failed/errored test(s), 15661 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testAlterPartition
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testAlterTable
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testAlterTableCascade
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testAlterViewParititon
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testColumnStatistics
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testComplexTable
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testComplexTypeApi
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testConcurrentMetastores
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testCreateAndGetTableWithDriver
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testCreateTableSettingId
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testDBLocationChange
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testDBOwner
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testDBOwnerChange
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testDatabase
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testDatabaseLocation
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testDatabaseLocationWithPermissionProblems
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testDropDatabaseCascadeMVMultiDB
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testDropTable
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testFilterLastPartition
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testFilterSinglePartition
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testFunctionWithResources
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testGetConfigValue
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testGetMetastoreUuid
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testGetPartitionsWithSpec
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testGetSchemaWithNoClassDefFoundError
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testGetTableObjects
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testGetUUIDInParallel
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testJDOPersistanceManagerCleanup
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testListPartitionNames
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testListPartitions
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testListPartitionsWihtLimitEnabled
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testNameMethods
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testPartition
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testPartitionFilter
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testRenamePartition
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testRetriableClientWithConnLifetime
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testSimpleFunction
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testSimpleTable
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testSimpleTypeApi
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testStatsFastTrivial
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testSynchronized
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testTableDatabase
(batchId=229)
[jira] [Commented] (HIVE-17020) Aggressive RS dedup can incorrectly remove OP tree branch
[
https://issues.apache.org/jira/browse/HIVE-17020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16718383#comment-16718383
]
Rui Li commented on HIVE-17020:
---
+1
> Aggressive RS dedup can incorrectly remove OP tree branch
> -
>
> Key: HIVE-17020
> URL: https://issues.apache.org/jira/browse/HIVE-17020
> Project: Hive
> Issue Type: Bug
>Reporter: Rui Li
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-17020.1.patch, HIVE-17020.2.patch,
> HIVE-17020.3.patch
>
>
> Suppose we have an OP tree like this:
> {noformat}
> ...
> |
> RS[1]
> |
> SEL[2]
> /\
> SEL[3] SEL[4]
> | |
> RS[5] FS[6]
> |
> ...
> {noformat}
> When doing aggressive RS dedup, we'll remove all the operators between RS5
> and RS1, and thus the branch containing FS6 is lost.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20911) External Table Replication for Hive
[
https://issues.apache.org/jira/browse/HIVE-20911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
anishek updated HIVE-20911:
---
Status: Patch Available (was: In Progress)
> External Table Replication for Hive
> ---
>
> Key: HIVE-20911
> URL: https://issues.apache.org/jira/browse/HIVE-20911
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 4.0.0
>Reporter: anishek
>Assignee: anishek
>Priority: Critical
> Fix For: 4.0.0
>
> Attachments: HIVE-20911.01.patch
>
>
> External tables are not replicated currently as part of hive replication. As
> part of this jira we want to enable that.
> Approach:
> * Target cluster will have a top level base directory config that will be
> used to copy all data relevant to external tables. This will be provided via
> the *with* clause in the *repl load* command. This base path will be prefixed
> to the path of the same external table on source cluster.
> * Since changes to directories on the external table can happen without hive
> knowing it, hence we cant capture the relevant events when ever new data is
> added or removed, we will have to copy the data from the source path to
> target path for external tables every time we run incremental replication.
> ** this will require incremental *repl dump* to now create an additional
> file *\_external\_tables\_info* with data in the following form
> {code}
> tableName,base64Encoded(tableDataLocation)
> {code}
> In case there are different partitions in the table pointing to different
> locations there will be multiple entries in the file for the same table name
> with location pointing to different partition locations. For partitions
> created in a table without specifying the _set location_ command will be
> within the same table Data location and hence there will not be different
> entries in the file above
> ** *repl load* will read the *\_external\_tables\_info* to identify what
> locations are to be copied from source to target and create corresponding
> tasks for them.
> * New External tables will be created with metadata only with no data copied
> as part of regular tasks while incremental load/bootstrap load.
> * Bootstrap dump will also create *\_external\_tables\_info* which will be
> used to copy data from source to target as part of boostrap load.
> * Bootstrap load will create a DAG, that can use parallelism in the execution
> phase, the hdfs copy related tasks are created, once the bootstrap phase is
> complete.
> * Since incremental load results in a DAG with only sequential execution (
> events applied in sequence ) to effectively use the parallelism capability in
> execution mode, we create tasks for hdfs copy along with the incremental DAG.
> This requires a few basic calculations to approximately meet the configured
> value in "hive.repl.approx.max.load.tasks"
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20911) External Table Replication for Hive
[
https://issues.apache.org/jira/browse/HIVE-20911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
anishek updated HIVE-20911:
---
Attachment: HIVE-20911.01.patch
> External Table Replication for Hive
> ---
>
> Key: HIVE-20911
> URL: https://issues.apache.org/jira/browse/HIVE-20911
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 4.0.0
>Reporter: anishek
>Assignee: anishek
>Priority: Critical
> Fix For: 4.0.0
>
> Attachments: HIVE-20911.01.patch
>
>
> External tables are not replicated currently as part of hive replication. As
> part of this jira we want to enable that.
> Approach:
> * Target cluster will have a top level base directory config that will be
> used to copy all data relevant to external tables. This will be provided via
> the *with* clause in the *repl load* command. This base path will be prefixed
> to the path of the same external table on source cluster.
> * Since changes to directories on the external table can happen without hive
> knowing it, hence we cant capture the relevant events when ever new data is
> added or removed, we will have to copy the data from the source path to
> target path for external tables every time we run incremental replication.
> ** this will require incremental *repl dump* to now create an additional
> file *\_external\_tables\_info* with data in the following form
> {code}
> tableName,base64Encoded(tableDataLocation)
> {code}
> In case there are different partitions in the table pointing to different
> locations there will be multiple entries in the file for the same table name
> with location pointing to different partition locations. For partitions
> created in a table without specifying the _set location_ command will be
> within the same table Data location and hence there will not be different
> entries in the file above
> ** *repl load* will read the *\_external\_tables\_info* to identify what
> locations are to be copied from source to target and create corresponding
> tasks for them.
> * New External tables will be created with metadata only with no data copied
> as part of regular tasks while incremental load/bootstrap load.
> * Bootstrap dump will also create *\_external\_tables\_info* which will be
> used to copy data from source to target as part of boostrap load.
> * Bootstrap load will create a DAG, that can use parallelism in the execution
> phase, the hdfs copy related tasks are created, once the bootstrap phase is
> complete.
> * Since incremental load results in a DAG with only sequential execution (
> events applied in sequence ) to effectively use the parallelism capability in
> execution mode, we create tasks for hdfs copy along with the incremental DAG.
> This requires a few basic calculations to approximately meet the configured
> value in "hive.repl.approx.max.load.tasks"
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work started] (HIVE-20911) External Table Replication for Hive
[
https://issues.apache.org/jira/browse/HIVE-20911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-20911 started by anishek.
--
> External Table Replication for Hive
> ---
>
> Key: HIVE-20911
> URL: https://issues.apache.org/jira/browse/HIVE-20911
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 4.0.0
>Reporter: anishek
>Assignee: anishek
>Priority: Critical
> Fix For: 4.0.0
>
> Attachments: HIVE-20911.01.patch
>
>
> External tables are not replicated currently as part of hive replication. As
> part of this jira we want to enable that.
> Approach:
> * Target cluster will have a top level base directory config that will be
> used to copy all data relevant to external tables. This will be provided via
> the *with* clause in the *repl load* command. This base path will be prefixed
> to the path of the same external table on source cluster.
> * Since changes to directories on the external table can happen without hive
> knowing it, hence we cant capture the relevant events when ever new data is
> added or removed, we will have to copy the data from the source path to
> target path for external tables every time we run incremental replication.
> ** this will require incremental *repl dump* to now create an additional
> file *\_external\_tables\_info* with data in the following form
> {code}
> tableName,base64Encoded(tableDataLocation)
> {code}
> In case there are different partitions in the table pointing to different
> locations there will be multiple entries in the file for the same table name
> with location pointing to different partition locations. For partitions
> created in a table without specifying the _set location_ command will be
> within the same table Data location and hence there will not be different
> entries in the file above
> ** *repl load* will read the *\_external\_tables\_info* to identify what
> locations are to be copied from source to target and create corresponding
> tasks for them.
> * New External tables will be created with metadata only with no data copied
> as part of regular tasks while incremental load/bootstrap load.
> * Bootstrap dump will also create *\_external\_tables\_info* which will be
> used to copy data from source to target as part of boostrap load.
> * Bootstrap load will create a DAG, that can use parallelism in the execution
> phase, the hdfs copy related tasks are created, once the bootstrap phase is
> complete.
> * Since incremental load results in a DAG with only sequential execution (
> events applied in sequence ) to effectively use the parallelism capability in
> execution mode, we create tasks for hdfs copy along with the incremental DAG.
> This requires a few basic calculations to approximately meet the configured
> value in "hive.repl.approx.max.load.tasks"
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21022) Fix remote metastore tests which use ZooKeeper
[
https://issues.apache.org/jira/browse/HIVE-21022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16718489#comment-16718489
]
Hive QA commented on HIVE-21022:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12951458/HIVE-21022.03
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15660 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15272/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15272/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15272/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12951458 - PreCommit-HIVE-Build
> Fix remote metastore tests which use ZooKeeper
> --
>
> Key: HIVE-21022
> URL: https://issues.apache.org/jira/browse/HIVE-21022
> Project: Hive
> Issue Type: Bug
> Components: Tests
>Affects Versions: 4.0.0
>Reporter: Ashutosh Bapat
>Assignee: Ashutosh Bapat
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-21022.01, HIVE-21022.01, HIVE-21022.01,
> HIVE-21022.02, HIVE-21022.02.patch, HIVE-21022.03
>
>
> Per [~vgarg]'s comment on HIVE-20794 at
> https://issues.apache.org/jira/browse/HIVE-20794?focusedCommentId=16714093=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16714093,
> the remote metatstore tests using ZooKeeper are flaky. They are failing with
> error "Got exception: org.apache.zookeeper.KeeperException$NoNodeException
> KeeperErrorCode = NoNode for /hs2mszktest".
> Both of these tests are using the same root namespace and hence the reason
> for this failure could be that the root namespace becomes unavailable to one
> test when the other drops it. The drop seems to be happening automatically
> through TestingServer code.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-14615) Temp table leaves behind insert command
[
https://issues.apache.org/jira/browse/HIVE-14615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16718408#comment-16718408
]
Hive QA commented on HIVE-14615:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12904833/HIVE-14615.4.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15269/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15269/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15269/
Messages:
{noformat}
This message was trimmed, see log for full details
error: test/results/clientpositive/llap/orc_ppd_basic.q.out: does not exist in
index
error: test/results/clientpositive/llap/order_null.q.out: does not exist in
index
error: test/results/clientpositive/llap/partition_pruning.q.out: does not exist
in index
error: test/results/clientpositive/llap/reduce_deduplicate_distinct.q.out: does
not exist in index
error: test/results/clientpositive/llap/resourceplan.q.out: does not exist in
index
error: test/results/clientpositive/llap/schema_evol_orc_acid_part.q.out: does
not exist in index
error: test/results/clientpositive/llap/schema_evol_orc_acid_table.q.out: does
not exist in index
error: test/results/clientpositive/llap/schema_evol_orc_acidvec_part.q.out:
does not exist in index
error: test/results/clientpositive/llap/schema_evol_orc_acidvec_table.q.out:
does not exist in index
error: test/results/clientpositive/llap/schema_evol_orc_nonvec_part.q.out: does
not exist in index
error:
test/results/clientpositive/llap/schema_evol_orc_nonvec_part_all_complex.q.out:
does not exist in index
error: test/results/clientpositive/llap/schema_evol_orc_nonvec_table.q.out:
does not exist in index
error: test/results/clientpositive/llap/schema_evol_orc_vec_part.q.out: does
not exist in index
error:
test/results/clientpositive/llap/schema_evol_orc_vec_part_all_complex.q.out:
does not exist in index
error: test/results/clientpositive/llap/schema_evol_orc_vec_table.q.out: does
not exist in index
error: test/results/clientpositive/llap/schema_evol_stats.q.out: does not exist
in index
error: test/results/clientpositive/llap/schema_evol_text_nonvec_part.q.out:
does not exist in index
error:
test/results/clientpositive/llap/schema_evol_text_nonvec_part_all_complex.q.out:
does not exist in index
error: test/results/clientpositive/llap/schema_evol_text_nonvec_table.q.out:
does not exist in index
error: test/results/clientpositive/llap/schema_evol_text_vec_part.q.out: does
not exist in index
error:
test/results/clientpositive/llap/schema_evol_text_vec_part_all_complex.q.out:
does not exist in index
error: test/results/clientpositive/llap/schema_evol_text_vec_table.q.out: does
not exist in index
error: test/results/clientpositive/llap/schema_evol_text_vecrow_part.q.out:
does not exist in index
error:
test/results/clientpositive/llap/schema_evol_text_vecrow_part_all_complex.q.out:
does not exist in index
error: test/results/clientpositive/llap/schema_evol_text_vecrow_table.q.out:
does not exist in index
error: test/results/clientpositive/llap/semijoin6.q.out: does not exist in index
error: test/results/clientpositive/llap/semijoin7.q.out: does not exist in index
error: test/results/clientpositive/llap/skiphf_aggr.q.out: does not exist in
index
error: test/results/clientpositive/llap/special_character_in_tabnames_1.q.out:
does not exist in index
error: test/results/clientpositive/llap/subquery_exists.q.out: does not exist
in index
error: test/results/clientpositive/llap/subquery_in.q.out: does not exist in
index
error: test/results/clientpositive/llap/subquery_in_having.q.out: does not
exist in index
error: test/results/clientpositive/llap/subquery_multi.q.out: does not exist in
index
error: test/results/clientpositive/llap/subquery_notin.q.out: does not exist in
index
error: test/results/clientpositive/llap/subquery_scalar.q.out: does not exist
in index
error: test/results/clientpositive/llap/subquery_select.q.out: does not exist
in index
error: test/results/clientpositive/llap/sysdb.q.out: does not exist in index
error: test/results/clientpositive/llap/tez_nway_join.q.out: does not exist in
index
error: test/results/clientpositive/llap/tez_self_join.q.out: does not exist in
index
error: test/results/clientpositive/llap/tez_union_dynamic_partition.q.out: does
not exist in index
error: test/results/clientpositive/llap/tez_union_dynamic_partition_2.q.out:
does not exist in index
error: test/results/clientpositive/llap/update_after_multiple_inserts.q.out:
does not exist in index
error: test/results/clientpositive/llap/vector_acid3.q.out: does not exist in
index
error: test/results/clientpositive/llap/vector_adaptor_usage_mode.q.out: does
not exist in index
error:
[jira] [Commented] (HIVE-21021) Scalar subquery with only aggregate in subquery (no group by) has unnecessary sq_count_check branch
[
https://issues.apache.org/jira/browse/HIVE-21021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16718407#comment-16718407
]
Hive QA commented on HIVE-21021:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12951424/HIVE-21021.4.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 15660 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_memcheck]
(batchId=45)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_scalar]
(batchId=169)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_scalar]
(batchId=128)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15268/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15268/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15268/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12951424 - PreCommit-HIVE-Build
> Scalar subquery with only aggregate in subquery (no group by) has unnecessary
> sq_count_check branch
> ---
>
> Key: HIVE-21021
> URL: https://issues.apache.org/jira/browse/HIVE-21021
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.0.0
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-21021.1.patch, HIVE-21021.2.patch,
> HIVE-21021.3.patch, HIVE-21021.4.patch
>
>
> {code:sql}
> CREATE TABLE `store_sales`(
> `ss_sold_date_sk` int,
> `ss_quantity` int,
> `ss_list_price` decimal(7,2));
> CREATE TABLE `date_dim`(
> `d_date_sk` int,
> `d_year` int);
> explain cbo with avg_sales as
> (select avg(quantity*list_price) average_sales
> from (select ss_quantity quantity
> ,ss_list_price list_price
>from store_sales
>,date_dim
>where ss_sold_date_sk = d_date_sk
> and d_year between 1999 and 2001 ) x)
> select * from store_sales where ss_list_price > (select average_sales from
> avg_sales);
> {code}
> {noformat}
> CBO PLAN:
> HiveProject(ss_sold_date_sk=[$0], ss_quantity=[$1], ss_list_price=[$2])
> HiveJoin(condition=[true], joinType=[inner], algorithm=[none], cost=[{2.0
> rows, 0.0 cpu, 0.0 io}])
> HiveJoin(condition=[>($2, $3)], joinType=[inner], algorithm=[none],
> cost=[{2.0 rows, 0.0 cpu, 0.0 io}])
> HiveProject(ss_sold_date_sk=[$0], ss_quantity=[$1], ss_list_price=[$2])
> HiveTableScan(table=[[sub, store_sales]], table:alias=[store_sales])
> HiveProject($f0=[/($0, $1)])
> HiveAggregate(group=[{}], agg#0=[sum($0)], agg#1=[count($0)])
> HiveProject($f0=[*(CAST($1):DECIMAL(10, 0), $2)])
> HiveJoin(condition=[=($0, $3)], joinType=[inner],
> algorithm=[none], cost=[{2.0 rows, 0.0 cpu, 0.0 io}])
> HiveProject(ss_sold_date_sk=[$0], ss_quantity=[$1],
> ss_list_price=[$2])
> HiveFilter(condition=[IS NOT NULL($0)])
> HiveTableScan(table=[[sub, store_sales]],
> table:alias=[store_sales])
> HiveProject(d_date_sk=[$0])
> HiveFilter(condition=[AND(BETWEEN(false, $1, 1999, 2001), IS
> NOT NULL($0))])
> HiveTableScan(table=[[sub, date_dim]],
> table:alias=[date_dim])
> HiveProject(cnt=[$0])
> HiveFilter(condition=[<=(sq_count_check($0), 1)])
> HiveProject(cnt=[$0])
> HiveAggregate(group=[{}], cnt=[COUNT()])
> HiveProject
> HiveProject($f0=[$0])
> HiveAggregate(group=[{}], agg#0=[count($0)])
> HiveJoin(condition=[=($0, $3)], joinType=[inner],
> algorithm=[none], cost=[{2.0 rows, 0.0 cpu, 0.0 io}])
> HiveProject(ss_sold_date_sk=[$0], ss_quantity=[$1],
> ss_list_price=[$2])
> HiveFilter(condition=[IS NOT NULL($0)])
> HiveTableScan(table=[[sub, store_sales]],
> table:alias=[store_sales])
> HiveProject(d_date_sk=[$0])
> HiveFilter(condition=[AND(BETWEEN(false, $1, 1999,
> 2001), IS NOT NULL($0))])
> HiveTableScan(table=[[sub, date_dim]],
> table:alias=[date_dim])
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-10144) [LLAP] merge brought in file blocking github sync
[
https://issues.apache.org/jira/browse/HIVE-10144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16718433#comment-16718433
]
Johannes Alberti commented on HIVE-10144:
-
the same issue still exists, the big file
(itests/thirdparty/spark-1.2.0-bin-hadoop2-without-hive.tgz) is still around in
the history of the current github.com/apache/hive repo, the file was introduced
with commit
https://github.com/apache/hive/commit/5feb58db2c99627cb41a747a097a0ec4b019d60c
and is still reachable in the history via branch 'spark' and tag
'master_2015_11_30', a mirror to any github enterprise with binary file limit
of <=100GB fails due to these references
{noformat}
$ git log --all --full-history -- *spark-1.2.0-bin-hadoop2-without-hive*
[...]
$ git branch --all --contains 5feb58db2c99627cb41a747a097a0ec4b019d60c
remotes/origin/spark
$ git tag --contains 5feb58db2c99627cb41a747a097a0ec4b019d60c
master_2015_11_30
{noformat}
I was able to create a partial mirror by excluding the spark branch and the old
tag
{noformat}
$ git clone https://github.com/apache/hive.git
$ cd hive
$ git remote set-url --push origin https://github.foo.bar/foobar/hive
$ git branch -r -d origin/spark
$ git tag master_2015_11_30 -d
$ git reflog expire --expire=now --all && git gc --prune=now --aggressive
$ git push --prune origin +refs/remotes/origin/*:refs/heads/*
+refs/tags/*:refs/tags/*
{noformat}
> [LLAP] merge brought in file blocking github sync
> -
>
> Key: HIVE-10144
> URL: https://issues.apache.org/jira/browse/HIVE-10144
> Project: Hive
> Issue Type: Bug
> Components: Build Infrastructure
>Reporter: Szehon Ho
>Assignee: Gunther Hagleitner
>Priority: Major
>
> r1669718 brought in a file that is not in source control on llap branch:
> [http://svn.apache.org/repos/asf/hive/branches/llap/itests/thirdparty/|http://svn.apache.org/repos/asf/hive/branches/llap/itests/thirdparty/]
> It is a file downloaded during test build and should not be in source
> control. It is actually blocking the github sync as its too large. See
> INFRA-9360
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21015) HCatLoader can't provide statistics for tables not in default DB
[
https://issues.apache.org/jira/browse/HIVE-21015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adam Szita updated HIVE-21015:
--
Summary: HCatLoader can't provide statistics for tables not in default DB
(was: HCatLoader can't provide statistics for tables no in default DB)
> HCatLoader can't provide statistics for tables not in default DB
>
>
> Key: HIVE-21015
> URL: https://issues.apache.org/jira/browse/HIVE-21015
> Project: Hive
> Issue Type: Bug
>Reporter: Adam Szita
>Assignee: Adam Szita
>Priority: Major
> Attachments: HIVE-21015.0.patch, HIVE-21015.1.patch,
> HIVE-21015.2.patch, HIVE-21015.3.patch
>
>
> This is due to a former change (HIVE-20330) that does not take database into
> consideration when retrieving the proper InputJobInfo for the loader.
> Found during testing:
> {code:java}
> 07:52:56 2018-12-05 07:52:16,599 [main] WARN
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.InputSizeReducerEstimator
> - Couldn't get statistics from LoadFunc:
> org.apache.hive.hcatalog.pig.HCatLoader@492fa72a
> 07:52:56 java.io.IOException: java.io.IOException: Could not calculate input
> size for location (table) tpcds_3000_decimal_parquet.date_dim
> 07:52:56 at
> org.apache.hive.hcatalog.pig.HCatLoader.getStatistics(HCatLoader.java:281)
> 07:52:56 at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.InputSizeReducerEstimator.getInputSizeFromLoader(InputSizeReducerEstimator.java:171)
> 07:52:56 at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.InputSizeReducerEstimator.getTotalInputFileSize(InputSizeReducerEstimator.java:118)
> 07:52:56 at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.InputSizeReducerEstimator.getTotalInputFileSize(InputSizeReducerEstimator.java:97)
> 07:52:56 at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.InputSizeReducerEstimator.estimateNumberOfReducers(InputSizeReducerEstimator.java:80)
> 07:52:56 at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.estimateNumberOfReducers(JobControlCompiler.java:1148)
> 07:52:56 at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.calculateRuntimeReducers(JobControlCompiler.java:1115)
> 07:52:56 at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.adjustNumReducers(JobControlCompiler.java:1063)
> 07:52:56 at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler.java:564)
> 07:52:56 at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.compile(JobControlCompiler.java:333)
> 07:52:56 at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:221)
> 07:52:56 at
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.launchPig(HExecutionEngine.java:293)
> 07:52:56 at org.apache.pig.PigServer.launchPlan(PigServer.java:1475)
> 07:52:56 at
> org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1460)
> 07:52:56 at org.apache.pig.PigServer.storeEx(PigServer.java:1119)
> 07:52:56 at org.apache.pig.PigServer.store(PigServer.java:1082)
> 07:52:56 at org.apache.pig.PigServer.openIterator(PigServer.java:995)
> 07:52:56 at
> org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:782)
> 07:52:56 at
> org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:383)
> 07:52:56 at
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:230)
> 07:52:56 at
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:205)
> 07:52:56 at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81)
> 07:52:56 at org.apache.pig.Main.run(Main.java:630)
> 07:52:56 at org.apache.pig.Main.main(Main.java:175)
> 07:52:56 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 07:52:56 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 07:52:56 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 07:52:56 at java.lang.reflect.Method.invoke(Method.java:498)
> 07:52:56 at org.apache.hadoop.util.RunJar.run(RunJar.java:313)
> 07:52:56 at org.apache.hadoop.util.RunJar.main(RunJar.java:227)
> 07:52:56 Caused by: java.io.IOException: Could not calculate input size for
> location (table) tpcds_3000_decimal_parquet.date_dim
> 07:52:56 at
> org.apache.hive.hcatalog.pig.HCatLoader.getStatistics(HCatLoader.java:276)
> 07:52:56 ... 29 more{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21015) HCatLoader can't provide statistics for tables not in default DB
[
https://issues.apache.org/jira/browse/HIVE-21015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16716816#comment-16716816
]
Adam Szita commented on HIVE-21015:
---
Committed to master. Thanks for reviewing, Peter!
> HCatLoader can't provide statistics for tables not in default DB
>
>
> Key: HIVE-21015
> URL: https://issues.apache.org/jira/browse/HIVE-21015
> Project: Hive
> Issue Type: Bug
>Reporter: Adam Szita
>Assignee: Adam Szita
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-21015.0.patch, HIVE-21015.1.patch,
> HIVE-21015.2.patch, HIVE-21015.3.patch
>
>
> This is due to a former change (HIVE-20330) that does not take database into
> consideration when retrieving the proper InputJobInfo for the loader.
> Found during testing:
> {code:java}
> 07:52:56 2018-12-05 07:52:16,599 [main] WARN
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.InputSizeReducerEstimator
> - Couldn't get statistics from LoadFunc:
> org.apache.hive.hcatalog.pig.HCatLoader@492fa72a
> 07:52:56 java.io.IOException: java.io.IOException: Could not calculate input
> size for location (table) tpcds_3000_decimal_parquet.date_dim
> 07:52:56 at
> org.apache.hive.hcatalog.pig.HCatLoader.getStatistics(HCatLoader.java:281)
> 07:52:56 at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.InputSizeReducerEstimator.getInputSizeFromLoader(InputSizeReducerEstimator.java:171)
> 07:52:56 at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.InputSizeReducerEstimator.getTotalInputFileSize(InputSizeReducerEstimator.java:118)
> 07:52:56 at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.InputSizeReducerEstimator.getTotalInputFileSize(InputSizeReducerEstimator.java:97)
> 07:52:56 at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.InputSizeReducerEstimator.estimateNumberOfReducers(InputSizeReducerEstimator.java:80)
> 07:52:56 at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.estimateNumberOfReducers(JobControlCompiler.java:1148)
> 07:52:56 at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.calculateRuntimeReducers(JobControlCompiler.java:1115)
> 07:52:56 at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.adjustNumReducers(JobControlCompiler.java:1063)
> 07:52:56 at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler.java:564)
> 07:52:56 at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.compile(JobControlCompiler.java:333)
> 07:52:56 at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:221)
> 07:52:56 at
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.launchPig(HExecutionEngine.java:293)
> 07:52:56 at org.apache.pig.PigServer.launchPlan(PigServer.java:1475)
> 07:52:56 at
> org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1460)
> 07:52:56 at org.apache.pig.PigServer.storeEx(PigServer.java:1119)
> 07:52:56 at org.apache.pig.PigServer.store(PigServer.java:1082)
> 07:52:56 at org.apache.pig.PigServer.openIterator(PigServer.java:995)
> 07:52:56 at
> org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:782)
> 07:52:56 at
> org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:383)
> 07:52:56 at
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:230)
> 07:52:56 at
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:205)
> 07:52:56 at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81)
> 07:52:56 at org.apache.pig.Main.run(Main.java:630)
> 07:52:56 at org.apache.pig.Main.main(Main.java:175)
> 07:52:56 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 07:52:56 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 07:52:56 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 07:52:56 at java.lang.reflect.Method.invoke(Method.java:498)
> 07:52:56 at org.apache.hadoop.util.RunJar.run(RunJar.java:313)
> 07:52:56 at org.apache.hadoop.util.RunJar.main(RunJar.java:227)
> 07:52:56 Caused by: java.io.IOException: Could not calculate input size for
> location (table) tpcds_3000_decimal_parquet.date_dim
> 07:52:56 at
> org.apache.hive.hcatalog.pig.HCatLoader.getStatistics(HCatLoader.java:276)
> 07:52:56 ... 29 more{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21015) HCatLoader can't provide statistics for tables not in default DB
[
https://issues.apache.org/jira/browse/HIVE-21015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adam Szita updated HIVE-21015:
--
Resolution: Fixed
Fix Version/s: 4.0.0
Status: Resolved (was: Patch Available)
> HCatLoader can't provide statistics for tables not in default DB
>
>
> Key: HIVE-21015
> URL: https://issues.apache.org/jira/browse/HIVE-21015
> Project: Hive
> Issue Type: Bug
>Reporter: Adam Szita
>Assignee: Adam Szita
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-21015.0.patch, HIVE-21015.1.patch,
> HIVE-21015.2.patch, HIVE-21015.3.patch
>
>
> This is due to a former change (HIVE-20330) that does not take database into
> consideration when retrieving the proper InputJobInfo for the loader.
> Found during testing:
> {code:java}
> 07:52:56 2018-12-05 07:52:16,599 [main] WARN
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.InputSizeReducerEstimator
> - Couldn't get statistics from LoadFunc:
> org.apache.hive.hcatalog.pig.HCatLoader@492fa72a
> 07:52:56 java.io.IOException: java.io.IOException: Could not calculate input
> size for location (table) tpcds_3000_decimal_parquet.date_dim
> 07:52:56 at
> org.apache.hive.hcatalog.pig.HCatLoader.getStatistics(HCatLoader.java:281)
> 07:52:56 at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.InputSizeReducerEstimator.getInputSizeFromLoader(InputSizeReducerEstimator.java:171)
> 07:52:56 at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.InputSizeReducerEstimator.getTotalInputFileSize(InputSizeReducerEstimator.java:118)
> 07:52:56 at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.InputSizeReducerEstimator.getTotalInputFileSize(InputSizeReducerEstimator.java:97)
> 07:52:56 at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.InputSizeReducerEstimator.estimateNumberOfReducers(InputSizeReducerEstimator.java:80)
> 07:52:56 at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.estimateNumberOfReducers(JobControlCompiler.java:1148)
> 07:52:56 at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.calculateRuntimeReducers(JobControlCompiler.java:1115)
> 07:52:56 at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.adjustNumReducers(JobControlCompiler.java:1063)
> 07:52:56 at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler.java:564)
> 07:52:56 at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.compile(JobControlCompiler.java:333)
> 07:52:56 at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:221)
> 07:52:56 at
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.launchPig(HExecutionEngine.java:293)
> 07:52:56 at org.apache.pig.PigServer.launchPlan(PigServer.java:1475)
> 07:52:56 at
> org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1460)
> 07:52:56 at org.apache.pig.PigServer.storeEx(PigServer.java:1119)
> 07:52:56 at org.apache.pig.PigServer.store(PigServer.java:1082)
> 07:52:56 at org.apache.pig.PigServer.openIterator(PigServer.java:995)
> 07:52:56 at
> org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:782)
> 07:52:56 at
> org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:383)
> 07:52:56 at
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:230)
> 07:52:56 at
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:205)
> 07:52:56 at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81)
> 07:52:56 at org.apache.pig.Main.run(Main.java:630)
> 07:52:56 at org.apache.pig.Main.main(Main.java:175)
> 07:52:56 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 07:52:56 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 07:52:56 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 07:52:56 at java.lang.reflect.Method.invoke(Method.java:498)
> 07:52:56 at org.apache.hadoop.util.RunJar.run(RunJar.java:313)
> 07:52:56 at org.apache.hadoop.util.RunJar.main(RunJar.java:227)
> 07:52:56 Caused by: java.io.IOException: Could not calculate input size for
> location (table) tpcds_3000_decimal_parquet.date_dim
> 07:52:56 at
> org.apache.hive.hcatalog.pig.HCatLoader.getStatistics(HCatLoader.java:276)
> 07:52:56 ... 29 more{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20760) Reducing memory overhead due to multiple HiveConfs
[
https://issues.apache.org/jira/browse/HIVE-20760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Barnabas Maidics updated HIVE-20760:
Status: Open (was: Patch Available)
> Reducing memory overhead due to multiple HiveConfs
> --
>
> Key: HIVE-20760
> URL: https://issues.apache.org/jira/browse/HIVE-20760
> Project: Hive
> Issue Type: Improvement
> Components: Configuration
>Reporter: Barnabas Maidics
>Assignee: Barnabas Maidics
>Priority: Major
> Attachments: HIVE-20760-1.patch, HIVE-20760-2.patch,
> HIVE-20760-3.patch, HIVE-20760.4.patch, HIVE-20760.5.patch,
> HIVE-20760.6.patch, HIVE-20760.7.patch, HIVE-20760.8.patch,
> HIVE-20760.9.patch, HIVE-20760.patch, hiveconf_interned.html,
> hiveconf_original.html
>
>
> The issue is that every Hive task has to load its own version of
> {{HiveConf}}. When running with a large number of cores per executor (HoS),
> there is a significant (~10%) amount of memory wasted due to this
> duplication.
> I looked into the problem and found a way to reduce the overhead caused by
> the multiple HiveConf objects.
> I've created an implementation of Properties, somewhat similar to
> CopyOnFirstWriteProperties. CopyOnFirstWriteProperties can't be used to solve
> this problem, because it drops the interned Properties right after we add a
> new property.
> So my implementation looks like this:
> * When we create a new HiveConf from an existing one (copy constructor), we
> change the properties object stored by HiveConf to the new Properties
> implementation (HiveConfProperties). We have 2 possible way to do this.
> Either we change the visibility of the properties field in the ancestor class
> (Configuration which comes from hadoop) to protected, or a simpler way is to
> just change the type using reflection.
> * HiveConfProperties instantly intern the given properties. After this,
> every time we add a new property to HiveConf, we add it to an additional
> Properties object. This way if we create multiple HiveConf with the same base
> properties, they will use the same Properties object but each session/task
> can add its own unique properties.
> * Getting a property from HiveConfProperties would look like this: (I stored
> the non-interned properties in super class)
> String property=super.getProperty(key);
> if (property == null) property= interned.getProperty(key);
> return property;
> Running some tests showed that the interning works (with 50 connections to
> HiveServer2, heapdumps created after sessions are created for queries):
> Overall memory:
> original: 34,599K interned: 20,582K
> Retained memory of HiveConfs:
> original: 16,366K interned: 10,804K
> I attach the JXray reports about the heapdumps.
> What are your thoughts about this solution?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20911) External Table Replication for Hive
[
https://issues.apache.org/jira/browse/HIVE-20911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16718531#comment-16718531
]
Hive QA commented on HIVE-20911:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12951461/HIVE-20911.01.patch
{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 20 failed/errored test(s), 15659 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[repl_2_exim_basic]
(batchId=85)
org.apache.hadoop.hive.ql.exec.repl.TestReplDumpTask.removeDBPropertyToPreventRenameWhenBootstrapDumpOfTableFails
(batchId=315)
org.apache.hadoop.hive.ql.parse.TestReplAcidTablesWithJsonMessage.testAcidBootstrapReplLoadRetryAfterFailure
(batchId=248)
org.apache.hadoop.hive.ql.parse.TestReplAcidTablesWithJsonMessage.testAcidTablesBootstrap
(batchId=248)
org.apache.hadoop.hive.ql.parse.TestReplAcidTablesWithJsonMessage.testAcidTablesBootstrapWithConcurrentWrites
(batchId=248)
org.apache.hadoop.hive.ql.parse.TestReplAcidTablesWithJsonMessage.testAcidTablesBootstrapWithOpenTxnsTimeout
(batchId=248)
org.apache.hadoop.hive.ql.parse.TestReplAcidTablesWithJsonMessage.testAcidTablesMoveOptimizationBootStrap
(batchId=248)
org.apache.hadoop.hive.ql.parse.TestReplAcidTablesWithJsonMessage.testDumpAcidTableWithPartitionDirMissing
(batchId=248)
org.apache.hadoop.hive.ql.parse.TestReplAcidTablesWithJsonMessage.testDumpAcidTableWithTableDirMissing
(batchId=248)
org.apache.hadoop.hive.ql.parse.TestReplAcidTablesWithJsonMessage.testMultiDBTxn
(batchId=248)
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcidTables.testAcidBootstrapReplLoadRetryAfterFailure
(batchId=245)
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcidTables.testAcidTablesBootstrap
(batchId=245)
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcidTables.testAcidTablesBootstrapWithConcurrentWrites
(batchId=245)
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcidTables.testAcidTablesBootstrapWithOpenTxnsTimeout
(batchId=245)
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcidTables.testAcidTablesMoveOptimizationBootStrap
(batchId=245)
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcidTables.testDumpAcidTableWithPartitionDirMissing
(batchId=245)
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcidTables.testDumpAcidTableWithTableDirMissing
(batchId=245)
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcidTables.testMultiDBTxn
(batchId=245)
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosIncrementalLoadAcidTables.testMigrationManagedToAcid
(batchId=246)
org.apache.hive.service.TestDFSErrorHandling.testAccessDenied (batchId=254)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15273/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15273/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15273/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 20 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12951461 - PreCommit-HIVE-Build
> External Table Replication for Hive
> ---
>
> Key: HIVE-20911
> URL: https://issues.apache.org/jira/browse/HIVE-20911
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 4.0.0
>Reporter: anishek
>Assignee: anishek
>Priority: Critical
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-20911.01.patch
>
>
> External tables are not replicated currently as part of hive replication. As
> part of this jira we want to enable that.
> Approach:
> * Target cluster will have a top level base directory config that will be
> used to copy all data relevant to external tables. This will be provided via
> the *with* clause in the *repl load* command. This base path will be prefixed
> to the path of the same external table on source cluster.
> * Since changes to directories on the external table can happen without hive
> knowing it, hence we cant capture the relevant events when ever new data is
> added or removed, we will have to copy the data from the source path to
> target path for external tables every time we run incremental replication.
> ** this will require incremental *repl dump* to now create an additional
> file *\_external\_tables\_info* with data in the following form
> {code}
>
[jira] [Commented] (HIVE-21022) Fix remote metastore tests which use ZooKeeper
[
https://issues.apache.org/jira/browse/HIVE-21022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16718563#comment-16718563
]
Hive QA commented on HIVE-21022:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
32s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
17s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
11s{color} | {color:blue} standalone-metastore/metastore-common in master has
29 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
4s{color} | {color:blue} standalone-metastore/metastore-server in master has
188 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
7s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 20m 53s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15274/dev-support/hive-personality.sh
|
| git revision | master / b650083 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: standalone-metastore/metastore-common
standalone-metastore/metastore-server U: standalone-metastore |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15274/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Fix remote metastore tests which use ZooKeeper
> --
>
> Key: HIVE-21022
> URL: https://issues.apache.org/jira/browse/HIVE-21022
> Project: Hive
> Issue Type: Bug
> Components: Tests
>Affects Versions: 4.0.0
>Reporter: Ashutosh Bapat
>Assignee: Ashutosh Bapat
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-21022.01, HIVE-21022.01, HIVE-21022.01,
> HIVE-21022.02, HIVE-21022.02.patch, HIVE-21022.03, HIVE-21022.03
>
>
> Per [~vgarg]'s comment on HIVE-20794 at
> https://issues.apache.org/jira/browse/HIVE-20794?focusedCommentId=16714093=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16714093,
> the remote metatstore tests using ZooKeeper are flaky. They are failing with
> error "Got exception: org.apache.zookeeper.KeeperException$NoNodeException
> KeeperErrorCode = NoNode for /hs2mszktest".
> Both of these tests are using the same root namespace and hence the reason
> for this failure could be that
[jira] [Updated] (HIVE-20733) GenericUDFOPEqualNS may not use = in plan descriptions
[ https://issues.apache.org/jira/browse/HIVE-20733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Lavati updated HIVE-20733: Status: In Progress (was: Patch Available) > GenericUDFOPEqualNS may not use = in plan descriptions > -- > > Key: HIVE-20733 > URL: https://issues.apache.org/jira/browse/HIVE-20733 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Assignee: David Lavati >Priority: Major > Attachments: HIVE-20733.2.patch, HIVE-20733.3.patch, > HIVE-20733.4.patch, HIVE-20733.patch > > > right now GenericUDFOPEqualNS is displayed a "=" in explains; however it > should be "<=>" > this may cause some confusion... > related qtest: is_distinct_from.q > same: GenericUDFOPNotEqualNS -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20733) GenericUDFOPEqualNS may not use = in plan descriptions
[ https://issues.apache.org/jira/browse/HIVE-20733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Lavati updated HIVE-20733: Attachment: HIVE-20733.4.patch Status: Patch Available (was: In Progress) > GenericUDFOPEqualNS may not use = in plan descriptions > -- > > Key: HIVE-20733 > URL: https://issues.apache.org/jira/browse/HIVE-20733 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Assignee: David Lavati >Priority: Major > Attachments: HIVE-20733.2.patch, HIVE-20733.3.patch, > HIVE-20733.4.patch, HIVE-20733.patch > > > right now GenericUDFOPEqualNS is displayed a "=" in explains; however it > should be "<=>" > this may cause some confusion... > related qtest: is_distinct_from.q > same: GenericUDFOPNotEqualNS -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-21029) External table replication: for existing deployments running incremental replication
[
https://issues.apache.org/jira/browse/HIVE-21029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
anishek reassigned HIVE-21029:
--
Assignee: anishek
> External table replication: for existing deployments running incremental
> replication
>
>
> Key: HIVE-21029
> URL: https://issues.apache.org/jira/browse/HIVE-21029
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 3.0.0, 3.1.0, 3.1.1
>Reporter: anishek
>Assignee: anishek
>Priority: Critical
> Fix For: 4.0.0
>
>
> Existing deployments using hive replication do not get external tables
> replicated. For such deployments to enable external table replication they
> will have to provide a specific switch to first bootstrap external tables as
> part of hive incremental replication, following which the incremental
> replication will take care of further changes in external tables.
> The switch will be provided by an additional hive configuration (for ex:
> hive.repl.bootstrap.external.tables) and is to be used in
> {code} WITH {code} clause of
> {code} REPL DUMP {code} command.
> Additionally the existing hive config _hive.repl.include.external.tables_
> will always have to be set to "true" in the above clause.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20959) cbo_rp_limit / cbo_limit are flaky - intermittent whitespace difference
[
https://issues.apache.org/jira/browse/HIVE-20959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Bodor updated HIVE-20959:
Description:
{code:java}
Client Execution succeeded but contained differences (error code = 1) after
executing cbo_rp_limit.q
11c11
< 1 4 2
---
> 1 4 2
{code}
After copying here to jira, cannot found difference, but by checking from
original junit xml, there is a whitespace difference in the lines (represented
as hex values), between 1 (x31) and 4 (x34). See [^diff] . Original golden
file contains horizontal tab (x09), actual output contains space (x20).
The serious thing is that the separator changes to x20, which is wrong, but
then in the same line, it changes back to x09.
{code}
20 31 *20* 34 09 32 <- actual
20 31 *09* 34 09 32 <- expected
{code}
Tried to reproduce it by running the failing batch of qtests locally, but no
luck (maybe it's an environment issue)
{code}
mvn test -T 1C -Dtest.output.overwrite=true -Pitests,hadoop-2 -pl itests/qtest
-pl itests/util -Dtest=TestMiniLlapLocalCliDriver
-Dqfile=dynamic_semijoin_reduction.q,materialized_view_create_rewrite_3.q,vectorization_pushdown.q,correlationoptimizer2.q,cbo_gby_empty.q,schema_evol_text_nonvec_part_all_complex_llap_io.q,vectorization_short_regress.q,mapjoin3.q,cross_product_check_1.q,results_cache_quoted_identifiers.q,unionDistinct_3.q,cbo_join.q,correlationoptimizer6.q,union_remove_26.q,cbo_rp_limit.q,convert_decimal64_to_decimal.q,vector_groupby_cube1.q,union2.q,groupby2.q,dynpart_sort_opt_vectorization.q,constraints_optimization.q,exchgpartition2lel.q,retry_failure.q,schema_evol_text_vecrow_part_llap_io.q,sample10.q,vectorized_timestamp_ints_casts.q,auto_sortmerge_join_2.q,bucketizedhiveinputformat.q,cte_mat_2.q,vectorization_8.q
{code}
was:
{code:java}
Client Execution succeeded but contained differences (error code = 1) after
executing cbo_rp_limit.q
11c11
< 1 4 2
---
> 1 4 2
{code}
After copying here to jira, cannot found difference, but by checking from
original junit xml, there is a whitespace difference in the lines (represented
as hex values), between 1 (x31) and 4 (x34). See [^diff] . Original golden
file contains horizontal tab (x09), actual output contains space (x20).
The serious thing is that the separator changes to x20, which is wrong, but
then in the same line, it changes back to x09.
{code}
20 31 *20* 34 09 32 <- actual
20 31 *09* 34 09 32 <- expected
{code}
> cbo_rp_limit / cbo_limit are flaky - intermittent whitespace difference
> ---
>
> Key: HIVE-20959
> URL: https://issues.apache.org/jira/browse/HIVE-20959
> Project: Hive
> Issue Type: Bug
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments:
> 171-TestMiniLlapLocalCliDriver-dynamic_semijoin_reduction.q-materialized_view_create_rewrite_3.q-vectorization_pushdown.q-and-27-more.txt,
>
> TEST-171-TestMiniLlapLocalCliDriver-dynamic_semijoin_reduction.q-materialized_view_create_rewrite_3.q-vectorization_pushdown.q-and-27-more-TEST-org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.xml,
> diff
>
>
> {code:java}
> Client Execution succeeded but contained differences (error code = 1) after
> executing cbo_rp_limit.q
> 11c11
> < 1 4 2
> ---
> > 1 4 2
> {code}
> After copying here to jira, cannot found difference, but by checking from
> original junit xml, there is a whitespace difference in the lines
> (represented as hex values), between 1 (x31) and 4 (x34). See [^diff] .
> Original golden file contains horizontal tab (x09), actual output contains
> space (x20).
> The serious thing is that the separator changes to x20, which is wrong, but
> then in the same line, it changes back to x09.
> {code}
> 20 31 *20* 34 09 32 <- actual
> 20 31 *09* 34 09 32 <- expected
> {code}
> Tried to reproduce it by running the failing batch of qtests locally, but no
> luck (maybe it's an environment issue)
> {code}
> mvn test -T 1C -Dtest.output.overwrite=true -Pitests,hadoop-2 -pl
> itests/qtest -pl itests/util -Dtest=TestMiniLlapLocalCliDriver
> -Dqfile=dynamic_semijoin_reduction.q,materialized_view_create_rewrite_3.q,vectorization_pushdown.q,correlationoptimizer2.q,cbo_gby_empty.q,schema_evol_text_nonvec_part_all_complex_llap_io.q,vectorization_short_regress.q,mapjoin3.q,cross_product_check_1.q,results_cache_quoted_identifiers.q,unionDistinct_3.q,cbo_join.q,correlationoptimizer6.q,union_remove_26.q,cbo_rp_limit.q,convert_decimal64_to_decimal.q,vector_groupby_cube1.q,union2.q,groupby2.q,dynpart_sort_opt_vectorization.q,constraints_optimization.q,exchgpartition2lel.q,retry_failure.q,schema_evol_text_vecrow_part_llap_io.q,sample10.q,vectorized_timestamp_ints_casts.q,auto_sortmerge_join_2.q,bucketizedhiveinputformat.q,cte_mat_2.q,vectorization_8.q
> {code}
[jira] [Commented] (HIVE-19081) Add partition should prevent loading acid files
[
https://issues.apache.org/jira/browse/HIVE-19081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16717659#comment-16717659
]
Eugene Koifman commented on HIVE-19081:
---
[~ikryvenko], could you take this one please?
> Add partition should prevent loading acid files
> ---
>
> Key: HIVE-19081
> URL: https://issues.apache.org/jira/browse/HIVE-19081
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Vaibhav Gumashta
>Priority: Major
>
> similar to HIVE-19029
> {{Alter Table T add Partition ...} T is acid should check to make sure input
> files were not copied from another Acid table, i.e. make sure the files don't
> have Acid metadata columns.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20936) Allow the Worker thread in the metastore to run outside of it
[ https://issues.apache.org/jira/browse/HIVE-20936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jaume M updated HIVE-20936: --- Status: Open (was: Patch Available) > Allow the Worker thread in the metastore to run outside of it > - > > Key: HIVE-20936 > URL: https://issues.apache.org/jira/browse/HIVE-20936 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Jaume M >Assignee: Jaume M >Priority: Major > Attachments: HIVE-20936.1.patch, HIVE-20936.2.patch, > HIVE-20936.3.patch, HIVE-20936.4.patch, HIVE-20936.5.patch, > HIVE-20936.6.patch, HIVE-20936.7.patch, HIVE-20936.8.patch > > > Currently the Worker thread in the metastore in bounded to the metastore, > mainly because of the TxnHandler that it has. This thread runs some map > reduce jobs which may not only be an option wherever the metastore is > running. A solution for this can be to run this thread in HS2 depending on a > flag. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-16100) Dynamic Sorted Partition optimizer loses sibling operators
[
https://issues.apache.org/jira/browse/HIVE-16100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-16100:
---
Attachment: HIVE-16100.12.patch
> Dynamic Sorted Partition optimizer loses sibling operators
> --
>
> Key: HIVE-16100
> URL: https://issues.apache.org/jira/browse/HIVE-16100
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Affects Versions: 1.2.1, 2.1.1, 2.2.0
>Reporter: Gopal V
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-16100.1.patch, HIVE-16100.10.patch,
> HIVE-16100.11.patch, HIVE-16100.12.patch, HIVE-16100.2.patch,
> HIVE-16100.2.patch, HIVE-16100.3.patch, HIVE-16100.4.patch,
> HIVE-16100.5.patch, HIVE-16100.6.patch, HIVE-16100.7.patch,
> HIVE-16100.8.patch, HIVE-16100.9.patch
>
>
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/SortedDynPartitionOptimizer.java#L173
> {code}
> // unlink connection between FS and its parent
> fsParent = fsOp.getParentOperators().get(0);
> fsParent.getChildOperators().clear();
> {code}
> The optimizer discards any cases where the fsParent has another SEL child
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
