[jira] [Commented] (HIVE-20854) Sensible Defaults: Hive's Zookeeper heartbeat interval is 20 minutes, change to 2
[
https://issues.apache.org/jira/browse/HIVE-20854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16673932#comment-16673932
]
Hive QA commented on HIVE-20854:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
30s{color} | {color:blue} common in master has 65 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 11m 33s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14717/dev-support/hive-personality.sh
|
| git revision | master / f3fab45 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: common U: common |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14717/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Sensible Defaults: Hive's Zookeeper heartbeat interval is 20 minutes, change
> to 2
> -
>
> Key: HIVE-20854
> URL: https://issues.apache.org/jira/browse/HIVE-20854
> Project: Hive
> Issue Type: Bug
>Reporter: Gopal V
>Assignee: Gopal V
>Priority: Major
> Attachments: HIVE-20854.1.patch
>
>
> {code}
> HIVE_ZOOKEEPER_SESSION_TIMEOUT("hive.zookeeper.session.timeout",
> "120ms",
> new TimeValidator(TimeUnit.MILLISECONDS),
> "ZooKeeper client's session timeout (in milliseconds). The client is
> disconnected, and as a result, all locks released, \n" +
> "if a heartbeat is not sent in the timeout."),
> {code}
> That's 1,200,000ms which is too long for all practical purposes - a 20 minute
> outage in case a node has a failure is too long.
> That is too long for the JDBC load-balancing, LLAP failure tolerance and the
> lock manager expiry.
> Change to 2 minutes, as a sensible default
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20835) Interaction between constraints and MV rewriting may create loop in Calcite planner
[
https://issues.apache.org/jira/browse/HIVE-20835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16673930#comment-16673930
]
Hive QA commented on HIVE-20835:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12946158/HIVE-20835.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 26 failed/errored test(s), 15521 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[infer_bucket_sort_reducers_power_two]
(batchId=14)
org.apache.hadoop.hive.cli.TestTezPerfConstraintsCliDriver.testCliDriver[cbo_query11]
(batchId=272)
org.apache.hadoop.hive.cli.TestTezPerfConstraintsCliDriver.testCliDriver[cbo_query23]
(batchId=272)
org.apache.hadoop.hive.cli.TestTezPerfConstraintsCliDriver.testCliDriver[cbo_query2]
(batchId=272)
org.apache.hadoop.hive.cli.TestTezPerfConstraintsCliDriver.testCliDriver[cbo_query31]
(batchId=272)
org.apache.hadoop.hive.cli.TestTezPerfConstraintsCliDriver.testCliDriver[cbo_query4]
(batchId=272)
org.apache.hadoop.hive.cli.TestTezPerfConstraintsCliDriver.testCliDriver[cbo_query58]
(batchId=272)
org.apache.hadoop.hive.cli.TestTezPerfConstraintsCliDriver.testCliDriver[cbo_query59]
(batchId=272)
org.apache.hadoop.hive.cli.TestTezPerfConstraintsCliDriver.testCliDriver[cbo_query61]
(batchId=272)
org.apache.hadoop.hive.cli.TestTezPerfConstraintsCliDriver.testCliDriver[cbo_query74]
(batchId=272)
org.apache.hadoop.hive.cli.TestTezPerfConstraintsCliDriver.testCliDriver[cbo_query75]
(batchId=272)
org.apache.hadoop.hive.cli.TestTezPerfConstraintsCliDriver.testCliDriver[cbo_query83]
(batchId=272)
org.apache.hadoop.hive.cli.TestTezPerfConstraintsCliDriver.testCliDriver[cbo_query90]
(batchId=272)
org.apache.hadoop.hive.cli.TestTezPerfConstraintsCliDriver.testCliDriver[query11]
(batchId=272)
org.apache.hadoop.hive.cli.TestTezPerfConstraintsCliDriver.testCliDriver[query23]
(batchId=272)
org.apache.hadoop.hive.cli.TestTezPerfConstraintsCliDriver.testCliDriver[query2]
(batchId=272)
org.apache.hadoop.hive.cli.TestTezPerfConstraintsCliDriver.testCliDriver[query31]
(batchId=272)
org.apache.hadoop.hive.cli.TestTezPerfConstraintsCliDriver.testCliDriver[query44]
(batchId=272)
org.apache.hadoop.hive.cli.TestTezPerfConstraintsCliDriver.testCliDriver[query4]
(batchId=272)
org.apache.hadoop.hive.cli.TestTezPerfConstraintsCliDriver.testCliDriver[query58]
(batchId=272)
org.apache.hadoop.hive.cli.TestTezPerfConstraintsCliDriver.testCliDriver[query59]
(batchId=272)
org.apache.hadoop.hive.cli.TestTezPerfConstraintsCliDriver.testCliDriver[query61]
(batchId=272)
org.apache.hadoop.hive.cli.TestTezPerfConstraintsCliDriver.testCliDriver[query74]
(batchId=272)
org.apache.hadoop.hive.cli.TestTezPerfConstraintsCliDriver.testCliDriver[query75]
(batchId=272)
org.apache.hadoop.hive.cli.TestTezPerfConstraintsCliDriver.testCliDriver[query83]
(batchId=272)
org.apache.hadoop.hive.cli.TestTezPerfConstraintsCliDriver.testCliDriver[query90]
(batchId=272)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14716/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14716/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14716/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 26 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12946158 - PreCommit-HIVE-Build
> Interaction between constraints and MV rewriting may create loop in Calcite
> planner
> ---
>
> Key: HIVE-20835
> URL: https://issues.apache.org/jira/browse/HIVE-20835
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-20835.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20835) Interaction between constraints and MV rewriting may create loop in Calcite planner
[
https://issues.apache.org/jira/browse/HIVE-20835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16673925#comment-16673925
]
Hive QA commented on HIVE-20835:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
38s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
3s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
27s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
51s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
42s{color} | {color:blue} itests/util in master has 48 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
45s{color} | {color:blue} ql in master has 2315 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
13s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
39s{color} | {color:red} ql: The patch generated 3 new + 170 unchanged - 2
fixed = 173 total (was 172) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 3 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 26m 49s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14716/dev-support/hive-personality.sh
|
| git revision | master / f3fab45 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14716/yetus/diff-checkstyle-ql.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14716/yetus/whitespace-eol.txt
|
| modules | C: itests itests/util ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14716/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Interaction between constraints and MV rewriting may create loop in Calcite
> planner
> ---
>
> Key: HIVE-20835
> URL: https://issues.apache.org/jira/browse/HIVE-20835
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-20835.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20821) Rewrite SUM0 into SUM + COALESCE combination
[
https://issues.apache.org/jira/browse/HIVE-20821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16673920#comment-16673920
]
Hive QA commented on HIVE-20821:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12946164/HIVE-20821.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15520 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14715/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14715/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14715/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12946164 - PreCommit-HIVE-Build
> Rewrite SUM0 into SUM + COALESCE combination
>
>
> Key: HIVE-20821
> URL: https://issues.apache.org/jira/browse/HIVE-20821
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-20821.01.patch, HIVE-20821.patch
>
>
> Since SUM0 is not vectorized, but SUM + COALESCE are.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20821) Rewrite SUM0 into SUM + COALESCE combination
[
https://issues.apache.org/jira/browse/HIVE-20821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16673906#comment-16673906
]
Hive QA commented on HIVE-20821:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
50s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
49s{color} | {color:blue} ql in master has 2315 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 22m 50s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14715/dev-support/hive-personality.sh
|
| git revision | master / f3fab45 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14715/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Rewrite SUM0 into SUM + COALESCE combination
>
>
> Key: HIVE-20821
> URL: https://issues.apache.org/jira/browse/HIVE-20821
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-20821.01.patch, HIVE-20821.patch
>
>
> Since SUM0 is not vectorized, but SUM + COALESCE are.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-16839) Unbalanced calls to openTransaction/commitTransaction when alter the same partition concurrently
[
https://issues.apache.org/jira/browse/HIVE-16839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guang Yang updated HIVE-16839:
--
Attachment: HIVE-16839.01.patch
> Unbalanced calls to openTransaction/commitTransaction when alter the same
> partition concurrently
>
>
> Key: HIVE-16839
> URL: https://issues.apache.org/jira/browse/HIVE-16839
> Project: Hive
> Issue Type: Bug
>Affects Versions: 0.13.1, 1.1.0
>Reporter: Nemon Lou
>Assignee: Guang Yang
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-16839.01.patch
>
>
> SQL to reproduce:
> prepare:
> {noformat}
> hdfs dfs -mkdir -p
> /hzsrc/external/writing_dc/ltgsm/16e7a9b2-21a1-3f4f-8061-bc3395281627
> 1,create external table tb_ltgsm_external (id int) PARTITIONED by (cp
> string,ld string);
> {noformat}
> open one beeline run these two sql many times
> {noformat} 2,ALTER TABLE tb_ltgsm_external ADD IF NOT EXISTS PARTITION
> (cp=2017060513,ld=2017060610);
> 3,ALTER TABLE tb_ltgsm_external PARTITION (cp=2017060513,ld=2017060610) SET
> LOCATION
> 'hdfs://hacluster/hzsrc/external/writing_dc/ltgsm/16e7a9b2-21a1-3f4f-8061-bc3395281627';
> {noformat}
> open another beeline to run this sql many times at the same time.
> {noformat}
> 4,ALTER TABLE tb_ltgsm_external DROP PARTITION (cp=2017060513,ld=2017060610);
> {noformat}
> MetaStore logs:
> {noformat}
> 2017-06-06 21:58:34,213 | ERROR | pool-6-thread-197 | Retrying HMSHandler
> after 2000 ms (attempt 1 of 10) with error:
> javax.jdo.JDOObjectNotFoundException: No such database row
> FailedObject:49[OID]org.apache.hadoop.hive.metastore.model.MStorageDescriptor
> at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:475)
> at
> org.datanucleus.api.jdo.JDOAdapter.getApiExceptionForNucleusException(JDOAdapter.java:1158)
> at
> org.datanucleus.state.JDOStateManager.isLoaded(JDOStateManager.java:3231)
> at
> org.apache.hadoop.hive.metastore.model.MStorageDescriptor.jdoGetcd(MStorageDescriptor.java)
> at
> org.apache.hadoop.hive.metastore.model.MStorageDescriptor.getCD(MStorageDescriptor.java:184)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.convertToStorageDescriptor(ObjectStore.java:1282)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.convertToStorageDescriptor(ObjectStore.java:1299)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.convertToPart(ObjectStore.java:1680)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getPartition(ObjectStore.java:1586)
> at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:98)
> at com.sun.proxy.$Proxy0.getPartition(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.HiveAlterHandler.alterPartitions(HiveAlterHandler.java:538)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_partitions(HiveMetaStore.java:3317)
> at sun.reflect.GeneratedMethodAccessor37.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:102)
> at com.sun.proxy.$Proxy12.alter_partitions(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$alter_partitions.getResult(ThriftHiveMetastore.java:9963)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$alter_partitions.getResult(ThriftHiveMetastore.java:9947)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:110)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:106)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1673)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:118)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
>
[jira] [Updated] (HIVE-16839) Unbalanced calls to openTransaction/commitTransaction when alter the same partition concurrently
[
https://issues.apache.org/jira/browse/HIVE-16839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guang Yang updated HIVE-16839:
--
Attachment: (was:
0001-HIVE-16839-Fix-a-race-condidtion-during-concurrent-p.patch)
> Unbalanced calls to openTransaction/commitTransaction when alter the same
> partition concurrently
>
>
> Key: HIVE-16839
> URL: https://issues.apache.org/jira/browse/HIVE-16839
> Project: Hive
> Issue Type: Bug
>Affects Versions: 0.13.1, 1.1.0
>Reporter: Nemon Lou
>Assignee: Guang Yang
>Priority: Major
> Labels: pull-request-available
>
> SQL to reproduce:
> prepare:
> {noformat}
> hdfs dfs -mkdir -p
> /hzsrc/external/writing_dc/ltgsm/16e7a9b2-21a1-3f4f-8061-bc3395281627
> 1,create external table tb_ltgsm_external (id int) PARTITIONED by (cp
> string,ld string);
> {noformat}
> open one beeline run these two sql many times
> {noformat} 2,ALTER TABLE tb_ltgsm_external ADD IF NOT EXISTS PARTITION
> (cp=2017060513,ld=2017060610);
> 3,ALTER TABLE tb_ltgsm_external PARTITION (cp=2017060513,ld=2017060610) SET
> LOCATION
> 'hdfs://hacluster/hzsrc/external/writing_dc/ltgsm/16e7a9b2-21a1-3f4f-8061-bc3395281627';
> {noformat}
> open another beeline to run this sql many times at the same time.
> {noformat}
> 4,ALTER TABLE tb_ltgsm_external DROP PARTITION (cp=2017060513,ld=2017060610);
> {noformat}
> MetaStore logs:
> {noformat}
> 2017-06-06 21:58:34,213 | ERROR | pool-6-thread-197 | Retrying HMSHandler
> after 2000 ms (attempt 1 of 10) with error:
> javax.jdo.JDOObjectNotFoundException: No such database row
> FailedObject:49[OID]org.apache.hadoop.hive.metastore.model.MStorageDescriptor
> at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:475)
> at
> org.datanucleus.api.jdo.JDOAdapter.getApiExceptionForNucleusException(JDOAdapter.java:1158)
> at
> org.datanucleus.state.JDOStateManager.isLoaded(JDOStateManager.java:3231)
> at
> org.apache.hadoop.hive.metastore.model.MStorageDescriptor.jdoGetcd(MStorageDescriptor.java)
> at
> org.apache.hadoop.hive.metastore.model.MStorageDescriptor.getCD(MStorageDescriptor.java:184)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.convertToStorageDescriptor(ObjectStore.java:1282)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.convertToStorageDescriptor(ObjectStore.java:1299)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.convertToPart(ObjectStore.java:1680)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getPartition(ObjectStore.java:1586)
> at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:98)
> at com.sun.proxy.$Proxy0.getPartition(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.HiveAlterHandler.alterPartitions(HiveAlterHandler.java:538)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_partitions(HiveMetaStore.java:3317)
> at sun.reflect.GeneratedMethodAccessor37.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:102)
> at com.sun.proxy.$Proxy12.alter_partitions(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$alter_partitions.getResult(ThriftHiveMetastore.java:9963)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$alter_partitions.getResult(ThriftHiveMetastore.java:9947)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:110)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:106)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1673)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:118)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
>
[jira] [Commented] (HIVE-20822) Improvements to push computation to JDBC from Calcite
[
https://issues.apache.org/jira/browse/HIVE-20822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16673896#comment-16673896
]
Hive QA commented on HIVE-20822:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12946160/HIVE-20822.01.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 102 failed/errored test(s), 15521 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_partition_coltype]
(batchId=28)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ambiguitycheck]
(batchId=79)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_rp_simple_select]
(batchId=49)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_simple_select]
(batchId=18)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[decimal_udf] (batchId=10)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dynamic_partition_skip_default]
(batchId=87)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[external_jdbc_table_perf]
(batchId=89)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[infer_join_preds]
(batchId=26)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[input8] (batchId=9)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[list_bucket_query_multiskew_2]
(batchId=75)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[macro] (batchId=4)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[order3] (batchId=69)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_0]
(batchId=17)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[pcr] (batchId=64)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[pcs] (batchId=53)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[pointlookup2]
(batchId=88)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[pointlookup3] (batchId=7)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[pointlookup4]
(batchId=80)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[pointlookup] (batchId=4)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_constant_expr]
(batchId=62)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_isops_simplify]
(batchId=35)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[union_offcbo]
(batchId=48)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[windowing_duplicate]
(batchId=33)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_mv]
(batchId=194)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez2]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucketpruning1]
(batchId=182)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cbo_simple_select]
(batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[count]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[current_date_timestamp]
(batchId=170)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dec_str]
(batchId=179)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[lineage2]
(batchId=173)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[mapjoin_hint]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_notin]
(batchId=177)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_scalar]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_views]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorization_0]
(batchId=181)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[bucket_map_join_tez2]
(batchId=115)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[cbo_simple_select]
(batchId=118)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[count] (batchId=122)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[parquet_vectorization_0]
(batchId=117)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[pcr] (batchId=137)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_notin]
(batchId=142)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_scalar]
(batchId=128)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_views]
(batchId=116)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_0]
(batchId=148)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query11]
(batchId=274)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query31]
(batchId=274)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query42]
(batchId=274)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query4]
(batchId=274)
[jira] [Commented] (HIVE-20822) Improvements to push computation to JDBC from Calcite
[
https://issues.apache.org/jira/browse/HIVE-20822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16673894#comment-16673894
]
Hive QA commented on HIVE-20822:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
42s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
26s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
58s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
58s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
41s{color} | {color:blue} ql in master has 2315 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
21s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
4s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
38s{color} | {color:red} ql: The patch generated 6 new + 175 unchanged - 0
fixed = 181 total (was 175) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 552 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
6s{color} | {color:red} The patch 142 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
19s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 55m 32s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc xml compile findbugs
checkstyle |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14714/dev-support/hive-personality.sh
|
| git revision | master / f3fab45 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14714/yetus/diff-checkstyle-ql.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14714/yetus/whitespace-eol.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14714/yetus/whitespace-tabs.txt
|
| modules | C: . ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14714/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Improvements to push computation to JDBC from Calcite
> -
>
> Key: HIVE-20822
> URL: https://issues.apache.org/jira/browse/HIVE-20822
> Project: Hive
> Issue Type: Improvement
> Components: StorageHandler
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-20822.01.patch, HIVE-20822.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20863) remove dead code
[ https://issues.apache.org/jira/browse/HIVE-20863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16673874#comment-16673874 ] Eugene Koifman commented on HIVE-20863: --- [~vgumashta] could you review please > remove dead code > > > Key: HIVE-20863 > URL: https://issues.apache.org/jira/browse/HIVE-20863 > Project: Hive > Issue Type: Improvement > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Minor > Attachments: HIVE-20863.01.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20862) QueryId no longer shows up in the logs
[ https://issues.apache.org/jira/browse/HIVE-20862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16673875#comment-16673875 ] Eugene Koifman commented on HIVE-20862: --- [~vgumashta] could you review please > QueryId no longer shows up in the logs > -- > > Key: HIVE-20862 > URL: https://issues.apache.org/jira/browse/HIVE-20862 > Project: Hive > Issue Type: Bug > Components: Logging >Affects Versions: 4.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Major > Attachments: HIVE-20862.01.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20823) Make Compactor run in a transaction
[
https://issues.apache.org/jira/browse/HIVE-20823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-20823:
--
Attachment: HIVE-20823.03.patch
> Make Compactor run in a transaction
> ---
>
> Key: HIVE-20823
> URL: https://issues.apache.org/jira/browse/HIVE-20823
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
> Attachments: HIVE-20823.01.patch, HIVE-20823.03.patch
>
>
> Have compactor open a transaction and run the job in that transaction.
> # make compactor produced base/delta include this txn id in the folder name,
> e.g. base_7_c17 where 17 is the txnid.
> # add {{CQ_TXN_ID bigint}} to COMPACTION_QUEUE and COMPLETED_COMPACTIONS to
> record this txn id
> # make sure {{AcidUtils.getAcidState()}} pays attention to this transaction
> on read and ignores this dir if this txn id is not committed in the current
> snapshot
> ## this means not only validWriteIdList but ValidTxnIdList should be passed
> along in config (if it isn't yet)
> # once this is done, {{CompactorMR.createCompactorMarker()}} can be
> eliminated and {{AcidUtils.isValidBase}} modified accordingly
> # modify Cleaner so that it doesn't clean old files until new file is visible
> to all readers
> #
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20863) remove dead code
[ https://issues.apache.org/jira/browse/HIVE-20863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-20863: -- Attachment: HIVE-20863.01.patch > remove dead code > > > Key: HIVE-20863 > URL: https://issues.apache.org/jira/browse/HIVE-20863 > Project: Hive > Issue Type: Improvement > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Minor > Attachments: HIVE-20863.01.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20863) remove dead code
[ https://issues.apache.org/jira/browse/HIVE-20863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-20863: -- Status: Patch Available (was: Open) > remove dead code > > > Key: HIVE-20863 > URL: https://issues.apache.org/jira/browse/HIVE-20863 > Project: Hive > Issue Type: Improvement > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Minor > Attachments: HIVE-20863.01.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20862) QueryId no longer shows up in the logs
[ https://issues.apache.org/jira/browse/HIVE-20862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-20862: -- Status: Patch Available (was: Open) > QueryId no longer shows up in the logs > -- > > Key: HIVE-20862 > URL: https://issues.apache.org/jira/browse/HIVE-20862 > Project: Hive > Issue Type: Bug > Components: Logging >Affects Versions: 4.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Major > Attachments: HIVE-20862.01.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20775) Factor cost of each SJ reduction when costing a follow-up reduction
[
https://issues.apache.org/jira/browse/HIVE-20775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16673872#comment-16673872
]
Hive QA commented on HIVE-20775:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12946310/HIVE-20775.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 15515 tests
executed
*Failed tests:*
{noformat}
TestMiniDruidCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=195)
[druidmini_dynamic_partition.q,druidmini_test_ts.q,druidmini_expressions.q,druidmini_test_alter.q,druidmini_test_insert.q]
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[dynamic_semijoin_user_level]
(batchId=155)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_dynamic_semijoin_reduction]
(batchId=159)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[cbo_query23]
(batchId=272)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[cbo_query54]
(batchId=272)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query64]
(batchId=272)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query95]
(batchId=272)
org.apache.hadoop.hive.cli.TestTezPerfConstraintsCliDriver.testCliDriver[cbo_query54]
(batchId=272)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14713/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14713/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14713/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12946310 - PreCommit-HIVE-Build
> Factor cost of each SJ reduction when costing a follow-up reduction
> ---
>
> Key: HIVE-20775
> URL: https://issues.apache.org/jira/browse/HIVE-20775
> Project: Hive
> Issue Type: Bug
> Components: Physical Optimizer
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-20775.01.patch, HIVE-20775.patch
>
>
> Currently, while costing the SJ in a plan, the stats of the a TS that is
> reduced by a SJ are not adjusted after we have decided to keep a SJ in the
> tree. Ideally, we could adjust the stats to take into account decisions that
> have already been made.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20862) QueryId no longer shows up in the logs
[ https://issues.apache.org/jira/browse/HIVE-20862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-20862: -- Attachment: HIVE-20862.01.patch > QueryId no longer shows up in the logs > -- > > Key: HIVE-20862 > URL: https://issues.apache.org/jira/browse/HIVE-20862 > Project: Hive > Issue Type: Bug > Components: Logging >Affects Versions: 4.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Major > Attachments: HIVE-20862.01.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20862) QueryId no longer shows up in the logs
[ https://issues.apache.org/jira/browse/HIVE-20862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman reassigned HIVE-20862: - > QueryId no longer shows up in the logs > -- > > Key: HIVE-20862 > URL: https://issues.apache.org/jira/browse/HIVE-20862 > Project: Hive > Issue Type: Bug > Components: Logging >Affects Versions: 4.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20863) remove dead code
[ https://issues.apache.org/jira/browse/HIVE-20863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman reassigned HIVE-20863: - > remove dead code > > > Key: HIVE-20863 > URL: https://issues.apache.org/jira/browse/HIVE-20863 > Project: Hive > Issue Type: Improvement > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Minor > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20775) Factor cost of each SJ reduction when costing a follow-up reduction
[
https://issues.apache.org/jira/browse/HIVE-20775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16673851#comment-16673851
]
Hive QA commented on HIVE-20775:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
38s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
49s{color} | {color:blue} ql in master has 2315 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
39s{color} | {color:red} ql: The patch generated 3 new + 124 unchanged - 3
fixed = 127 total (was 127) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
1s{color} | {color:red} ql generated 2 new + 2313 unchanged - 2 fixed = 2315
total (was 2315) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 23m 31s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | Dead store to tsRowSize in
org.apache.hadoop.hive.ql.parse.TezCompiler.getBloomFilterBenefit(SelectOperator,
ExprNodeDesc, Statistics, ExprNodeDesc) At
TezCompiler.java:org.apache.hadoop.hive.ql.parse.TezCompiler.getBloomFilterBenefit(SelectOperator,
ExprNodeDesc, Statistics, ExprNodeDesc) At TezCompiler.java:[line 1451] |
| | Should org.apache.hadoop.hive.ql.parse.TezCompiler$SemijoinOperatorInfo
be a _static_ inner class? At TezCompiler.java:inner class? At
TezCompiler.java:[lines 1629-1643] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14713/dev-support/hive-personality.sh
|
| git revision | master / f3fab45 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14713/yetus/diff-checkstyle-ql.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14713/yetus/new-findbugs-ql.html
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14713/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Factor cost of each SJ reduction when costing a follow-up reduction
> ---
>
> Key: HIVE-20775
> URL: https://issues.apache.org/jira/browse/HIVE-20775
> Project: Hive
> Issue Type: Bug
> Components: Physical Optimizer
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-20775.01.patch, HIVE-20775.patch
>
>
> Currently, while costing the SJ in a plan, the stats of the a TS that is
> reduced by a SJ are not adjusted after we have decided to keep a SJ in the
> tree. Ideally, we could adjust the stats to take into account decisions that
> have already been made.
--
This message was sent by Atlassian JIRA
[jira] [Updated] (HIVE-20842) Fix logic introduced in HIVE-20660 to estimate statistics for group by
[ https://issues.apache.org/jira/browse/HIVE-20842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-20842: --- Attachment: HIVE-20842.3.patch > Fix logic introduced in HIVE-20660 to estimate statistics for group by > -- > > Key: HIVE-20842 > URL: https://issues.apache.org/jira/browse/HIVE-20842 > Project: Hive > Issue Type: Bug > Components: Statistics >Affects Versions: 4.0.0 >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-20842.1.patch, HIVE-20842.2.patch, > HIVE-20842.3.patch > > > HIVE-20660 introduced better estimation for group by operator. But the logic > did not account for Partial and Full group by separately. > For partial group by parallelism (i.e. number of tasks) should be taken into > account. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20842) Fix logic introduced in HIVE-20660 to estimate statistics for group by
[ https://issues.apache.org/jira/browse/HIVE-20842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-20842: --- Status: Patch Available (was: Open) > Fix logic introduced in HIVE-20660 to estimate statistics for group by > -- > > Key: HIVE-20842 > URL: https://issues.apache.org/jira/browse/HIVE-20842 > Project: Hive > Issue Type: Bug > Components: Statistics >Affects Versions: 4.0.0 >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-20842.1.patch, HIVE-20842.2.patch, > HIVE-20842.3.patch > > > HIVE-20660 introduced better estimation for group by operator. But the logic > did not account for Partial and Full group by separately. > For partial group by parallelism (i.e. number of tasks) should be taken into > account. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20842) Fix logic introduced in HIVE-20660 to estimate statistics for group by
[ https://issues.apache.org/jira/browse/HIVE-20842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-20842: --- Status: Open (was: Patch Available) > Fix logic introduced in HIVE-20660 to estimate statistics for group by > -- > > Key: HIVE-20842 > URL: https://issues.apache.org/jira/browse/HIVE-20842 > Project: Hive > Issue Type: Bug > Components: Statistics >Affects Versions: 4.0.0 >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-20842.1.patch, HIVE-20842.2.patch, > HIVE-20842.3.patch > > > HIVE-20660 introduced better estimation for group by operator. But the logic > did not account for Partial and Full group by separately. > For partial group by parallelism (i.e. number of tasks) should be taken into > account. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20842) Fix logic introduced in HIVE-20660 to estimate statistics for group by
[ https://issues.apache.org/jira/browse/HIVE-20842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-20842: --- Attachment: (was: HIVE-20842.2.patch) > Fix logic introduced in HIVE-20660 to estimate statistics for group by > -- > > Key: HIVE-20842 > URL: https://issues.apache.org/jira/browse/HIVE-20842 > Project: Hive > Issue Type: Bug > Components: Statistics >Affects Versions: 4.0.0 >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-20842.1.patch, HIVE-20842.2.patch > > > HIVE-20660 introduced better estimation for group by operator. But the logic > did not account for Partial and Full group by separately. > For partial group by parallelism (i.e. number of tasks) should be taken into > account. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20842) Fix logic introduced in HIVE-20660 to estimate statistics for group by
[ https://issues.apache.org/jira/browse/HIVE-20842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-20842: --- Attachment: HIVE-20842.2.patch > Fix logic introduced in HIVE-20660 to estimate statistics for group by > -- > > Key: HIVE-20842 > URL: https://issues.apache.org/jira/browse/HIVE-20842 > Project: Hive > Issue Type: Bug > Components: Statistics >Affects Versions: 4.0.0 >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-20842.1.patch, HIVE-20842.2.patch > > > HIVE-20660 introduced better estimation for group by operator. But the logic > did not account for Partial and Full group by separately. > For partial group by parallelism (i.e. number of tasks) should be taken into > account. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20853) Expose ShuffleHandler.registerDag in the llap daemon API
[ https://issues.apache.org/jira/browse/HIVE-20853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jaume M updated HIVE-20853: --- Attachment: HIVE-20853.1.patch Status: Patch Available (was: Open) > Expose ShuffleHandler.registerDag in the llap daemon API > > > Key: HIVE-20853 > URL: https://issues.apache.org/jira/browse/HIVE-20853 > Project: Hive > Issue Type: Improvement > Components: llap >Affects Versions: 3.1.0 >Reporter: Jaume M >Assignee: Jaume M >Priority: Critical > Attachments: HIVE-20853.1.patch > > > Currently DAGs are only registered when a submitWork is called for that DAG. > At this point the crendentials are added to the ShuffleHandler and it can > start serving. > However Tez might (and will) schedule tasks to fetch from the ShuffleHandler > before anything of this happens and all this tasks will fail which may > results in the query failing. > This happens in the scenario in which a LlapDaemon just comes up and tez > fetchers try to open a connection before a DAG has been registered. > Adding this API will allow to register the DAG against the Daemon when the AM > notices that a new Daemon is up. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HIVE-16839) Unbalanced calls to openTransaction/commitTransaction when alter the same partition concurrently
[
https://issues.apache.org/jira/browse/HIVE-16839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16673840#comment-16673840
]
Vihang Karajgaonkar edited comment on HIVE-16839 at 11/3/18 12:02 AM:
--
The patch name should be {{HIVE-16839.01.patch}} so that automated job can run
tests on the correct branch.
was (Author: vihangk1):
The patch name should be {{HIVE-16839.01.patch}}
> Unbalanced calls to openTransaction/commitTransaction when alter the same
> partition concurrently
>
>
> Key: HIVE-16839
> URL: https://issues.apache.org/jira/browse/HIVE-16839
> Project: Hive
> Issue Type: Bug
>Affects Versions: 0.13.1, 1.1.0
>Reporter: Nemon Lou
>Assignee: Guang Yang
>Priority: Major
> Labels: pull-request-available
> Attachments:
> 0001-HIVE-16839-Fix-a-race-condidtion-during-concurrent-p.patch
>
>
> SQL to reproduce:
> prepare:
> {noformat}
> hdfs dfs -mkdir -p
> /hzsrc/external/writing_dc/ltgsm/16e7a9b2-21a1-3f4f-8061-bc3395281627
> 1,create external table tb_ltgsm_external (id int) PARTITIONED by (cp
> string,ld string);
> {noformat}
> open one beeline run these two sql many times
> {noformat} 2,ALTER TABLE tb_ltgsm_external ADD IF NOT EXISTS PARTITION
> (cp=2017060513,ld=2017060610);
> 3,ALTER TABLE tb_ltgsm_external PARTITION (cp=2017060513,ld=2017060610) SET
> LOCATION
> 'hdfs://hacluster/hzsrc/external/writing_dc/ltgsm/16e7a9b2-21a1-3f4f-8061-bc3395281627';
> {noformat}
> open another beeline to run this sql many times at the same time.
> {noformat}
> 4,ALTER TABLE tb_ltgsm_external DROP PARTITION (cp=2017060513,ld=2017060610);
> {noformat}
> MetaStore logs:
> {noformat}
> 2017-06-06 21:58:34,213 | ERROR | pool-6-thread-197 | Retrying HMSHandler
> after 2000 ms (attempt 1 of 10) with error:
> javax.jdo.JDOObjectNotFoundException: No such database row
> FailedObject:49[OID]org.apache.hadoop.hive.metastore.model.MStorageDescriptor
> at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:475)
> at
> org.datanucleus.api.jdo.JDOAdapter.getApiExceptionForNucleusException(JDOAdapter.java:1158)
> at
> org.datanucleus.state.JDOStateManager.isLoaded(JDOStateManager.java:3231)
> at
> org.apache.hadoop.hive.metastore.model.MStorageDescriptor.jdoGetcd(MStorageDescriptor.java)
> at
> org.apache.hadoop.hive.metastore.model.MStorageDescriptor.getCD(MStorageDescriptor.java:184)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.convertToStorageDescriptor(ObjectStore.java:1282)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.convertToStorageDescriptor(ObjectStore.java:1299)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.convertToPart(ObjectStore.java:1680)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getPartition(ObjectStore.java:1586)
> at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:98)
> at com.sun.proxy.$Proxy0.getPartition(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.HiveAlterHandler.alterPartitions(HiveAlterHandler.java:538)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_partitions(HiveMetaStore.java:3317)
> at sun.reflect.GeneratedMethodAccessor37.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:102)
> at com.sun.proxy.$Proxy12.alter_partitions(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$alter_partitions.getResult(ThriftHiveMetastore.java:9963)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$alter_partitions.getResult(ThriftHiveMetastore.java:9947)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:110)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:106)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1673)
> at
>
[jira] [Commented] (HIVE-16839) Unbalanced calls to openTransaction/commitTransaction when alter the same partition concurrently
[
https://issues.apache.org/jira/browse/HIVE-16839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16673840#comment-16673840
]
Vihang Karajgaonkar commented on HIVE-16839:
The patch name should be {{HIVE-16839.01.patch}}
> Unbalanced calls to openTransaction/commitTransaction when alter the same
> partition concurrently
>
>
> Key: HIVE-16839
> URL: https://issues.apache.org/jira/browse/HIVE-16839
> Project: Hive
> Issue Type: Bug
>Affects Versions: 0.13.1, 1.1.0
>Reporter: Nemon Lou
>Assignee: Guang Yang
>Priority: Major
> Labels: pull-request-available
> Attachments:
> 0001-HIVE-16839-Fix-a-race-condidtion-during-concurrent-p.patch
>
>
> SQL to reproduce:
> prepare:
> {noformat}
> hdfs dfs -mkdir -p
> /hzsrc/external/writing_dc/ltgsm/16e7a9b2-21a1-3f4f-8061-bc3395281627
> 1,create external table tb_ltgsm_external (id int) PARTITIONED by (cp
> string,ld string);
> {noformat}
> open one beeline run these two sql many times
> {noformat} 2,ALTER TABLE tb_ltgsm_external ADD IF NOT EXISTS PARTITION
> (cp=2017060513,ld=2017060610);
> 3,ALTER TABLE tb_ltgsm_external PARTITION (cp=2017060513,ld=2017060610) SET
> LOCATION
> 'hdfs://hacluster/hzsrc/external/writing_dc/ltgsm/16e7a9b2-21a1-3f4f-8061-bc3395281627';
> {noformat}
> open another beeline to run this sql many times at the same time.
> {noformat}
> 4,ALTER TABLE tb_ltgsm_external DROP PARTITION (cp=2017060513,ld=2017060610);
> {noformat}
> MetaStore logs:
> {noformat}
> 2017-06-06 21:58:34,213 | ERROR | pool-6-thread-197 | Retrying HMSHandler
> after 2000 ms (attempt 1 of 10) with error:
> javax.jdo.JDOObjectNotFoundException: No such database row
> FailedObject:49[OID]org.apache.hadoop.hive.metastore.model.MStorageDescriptor
> at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:475)
> at
> org.datanucleus.api.jdo.JDOAdapter.getApiExceptionForNucleusException(JDOAdapter.java:1158)
> at
> org.datanucleus.state.JDOStateManager.isLoaded(JDOStateManager.java:3231)
> at
> org.apache.hadoop.hive.metastore.model.MStorageDescriptor.jdoGetcd(MStorageDescriptor.java)
> at
> org.apache.hadoop.hive.metastore.model.MStorageDescriptor.getCD(MStorageDescriptor.java:184)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.convertToStorageDescriptor(ObjectStore.java:1282)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.convertToStorageDescriptor(ObjectStore.java:1299)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.convertToPart(ObjectStore.java:1680)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getPartition(ObjectStore.java:1586)
> at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:98)
> at com.sun.proxy.$Proxy0.getPartition(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.HiveAlterHandler.alterPartitions(HiveAlterHandler.java:538)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_partitions(HiveMetaStore.java:3317)
> at sun.reflect.GeneratedMethodAccessor37.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:102)
> at com.sun.proxy.$Proxy12.alter_partitions(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$alter_partitions.getResult(ThriftHiveMetastore.java:9963)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$alter_partitions.getResult(ThriftHiveMetastore.java:9947)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:110)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:106)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1673)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:118)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
> at
>
[jira] [Commented] (HIVE-20748) Disable materialized view rewriting when plan pattern is not allowed
[
https://issues.apache.org/jira/browse/HIVE-20748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16673836#comment-16673836
]
Hive QA commented on HIVE-20748:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12946299/HIVE-20748.03.patch
{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 15505 tests
executed
*Failed tests:*
{noformat}
TestAlterTableMetadata - did not produce a TEST-*.xml file (likely timed out)
(batchId=250)
TestAutoPurgeTables - did not produce a TEST-*.xml file (likely timed out)
(batchId=250)
TestCopyUtils - did not produce a TEST-*.xml file (likely timed out)
(batchId=250)
TestLocationQueries - did not produce a TEST-*.xml file (likely timed out)
(batchId=250)
TestReplAcidTablesWithJsonMessage - did not produce a TEST-*.xml file (likely
timed out) (batchId=250)
TestReplIncrementalLoadAcidTablesWithJsonMessage - did not produce a TEST-*.xml
file (likely timed out) (batchId=250)
TestSemanticAnalyzerHookLoading - did not produce a TEST-*.xml file (likely
timed out) (batchId=250)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14712/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14712/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14712/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12946299 - PreCommit-HIVE-Build
> Disable materialized view rewriting when plan pattern is not allowed
>
>
> Key: HIVE-20748
> URL: https://issues.apache.org/jira/browse/HIVE-20748
> Project: Hive
> Issue Type: Bug
> Components: Materialized views
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-20748.01.patch, HIVE-20748.01.patch,
> HIVE-20748.02.patch, HIVE-20748.02.patch, HIVE-20748.03.patch,
> HIVE-20748.patch
>
>
> For instance, currently rewriting algorithm does not support some operators.
> Or we cannot have non-deterministic function in the MV definition. In those
> cases, we should fail either when we try to create the MV with rewriting
> enabled, or when when we enable the rewriting for a MV already created.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-20861) Pass queryId as the client CallerContext to Spark
[ https://issues.apache.org/jira/browse/HIVE-20861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aihua Xu reassigned HIVE-20861: --- > Pass queryId as the client CallerContext to Spark > -- > > Key: HIVE-20861 > URL: https://issues.apache.org/jira/browse/HIVE-20861 > Project: Hive > Issue Type: Improvement >Reporter: Aihua Xu >Assignee: Aihua Xu >Priority: Major > > SPARK-16759 exposes a way for the client to pass the client CallerContext > such as QueryId. For better debug, hive should pass queryId to spark. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-16839) Unbalanced calls to openTransaction/commitTransaction when alter the same partition concurrently
[
https://issues.apache.org/jira/browse/HIVE-16839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guang Yang updated HIVE-16839:
--
Affects Version/s: 0.13.1
> Unbalanced calls to openTransaction/commitTransaction when alter the same
> partition concurrently
>
>
> Key: HIVE-16839
> URL: https://issues.apache.org/jira/browse/HIVE-16839
> Project: Hive
> Issue Type: Bug
>Affects Versions: 0.13.1, 1.1.0
>Reporter: Nemon Lou
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Labels: pull-request-available
> Attachments:
> 0001-HIVE-16839-Fix-a-race-condidtion-during-concurrent-p.patch
>
>
> SQL to reproduce:
> prepare:
> {noformat}
> hdfs dfs -mkdir -p
> /hzsrc/external/writing_dc/ltgsm/16e7a9b2-21a1-3f4f-8061-bc3395281627
> 1,create external table tb_ltgsm_external (id int) PARTITIONED by (cp
> string,ld string);
> {noformat}
> open one beeline run these two sql many times
> {noformat} 2,ALTER TABLE tb_ltgsm_external ADD IF NOT EXISTS PARTITION
> (cp=2017060513,ld=2017060610);
> 3,ALTER TABLE tb_ltgsm_external PARTITION (cp=2017060513,ld=2017060610) SET
> LOCATION
> 'hdfs://hacluster/hzsrc/external/writing_dc/ltgsm/16e7a9b2-21a1-3f4f-8061-bc3395281627';
> {noformat}
> open another beeline to run this sql many times at the same time.
> {noformat}
> 4,ALTER TABLE tb_ltgsm_external DROP PARTITION (cp=2017060513,ld=2017060610);
> {noformat}
> MetaStore logs:
> {noformat}
> 2017-06-06 21:58:34,213 | ERROR | pool-6-thread-197 | Retrying HMSHandler
> after 2000 ms (attempt 1 of 10) with error:
> javax.jdo.JDOObjectNotFoundException: No such database row
> FailedObject:49[OID]org.apache.hadoop.hive.metastore.model.MStorageDescriptor
> at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:475)
> at
> org.datanucleus.api.jdo.JDOAdapter.getApiExceptionForNucleusException(JDOAdapter.java:1158)
> at
> org.datanucleus.state.JDOStateManager.isLoaded(JDOStateManager.java:3231)
> at
> org.apache.hadoop.hive.metastore.model.MStorageDescriptor.jdoGetcd(MStorageDescriptor.java)
> at
> org.apache.hadoop.hive.metastore.model.MStorageDescriptor.getCD(MStorageDescriptor.java:184)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.convertToStorageDescriptor(ObjectStore.java:1282)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.convertToStorageDescriptor(ObjectStore.java:1299)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.convertToPart(ObjectStore.java:1680)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getPartition(ObjectStore.java:1586)
> at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:98)
> at com.sun.proxy.$Proxy0.getPartition(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.HiveAlterHandler.alterPartitions(HiveAlterHandler.java:538)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_partitions(HiveMetaStore.java:3317)
> at sun.reflect.GeneratedMethodAccessor37.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:102)
> at com.sun.proxy.$Proxy12.alter_partitions(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$alter_partitions.getResult(ThriftHiveMetastore.java:9963)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$alter_partitions.getResult(ThriftHiveMetastore.java:9947)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:110)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:106)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1673)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:118)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
>
[jira] [Assigned] (HIVE-16839) Unbalanced calls to openTransaction/commitTransaction when alter the same partition concurrently
[
https://issues.apache.org/jira/browse/HIVE-16839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guang Yang reassigned HIVE-16839:
-
Assignee: Guang Yang (was: Vihang Karajgaonkar)
> Unbalanced calls to openTransaction/commitTransaction when alter the same
> partition concurrently
>
>
> Key: HIVE-16839
> URL: https://issues.apache.org/jira/browse/HIVE-16839
> Project: Hive
> Issue Type: Bug
>Affects Versions: 0.13.1, 1.1.0
>Reporter: Nemon Lou
>Assignee: Guang Yang
>Priority: Major
> Labels: pull-request-available
> Attachments:
> 0001-HIVE-16839-Fix-a-race-condidtion-during-concurrent-p.patch
>
>
> SQL to reproduce:
> prepare:
> {noformat}
> hdfs dfs -mkdir -p
> /hzsrc/external/writing_dc/ltgsm/16e7a9b2-21a1-3f4f-8061-bc3395281627
> 1,create external table tb_ltgsm_external (id int) PARTITIONED by (cp
> string,ld string);
> {noformat}
> open one beeline run these two sql many times
> {noformat} 2,ALTER TABLE tb_ltgsm_external ADD IF NOT EXISTS PARTITION
> (cp=2017060513,ld=2017060610);
> 3,ALTER TABLE tb_ltgsm_external PARTITION (cp=2017060513,ld=2017060610) SET
> LOCATION
> 'hdfs://hacluster/hzsrc/external/writing_dc/ltgsm/16e7a9b2-21a1-3f4f-8061-bc3395281627';
> {noformat}
> open another beeline to run this sql many times at the same time.
> {noformat}
> 4,ALTER TABLE tb_ltgsm_external DROP PARTITION (cp=2017060513,ld=2017060610);
> {noformat}
> MetaStore logs:
> {noformat}
> 2017-06-06 21:58:34,213 | ERROR | pool-6-thread-197 | Retrying HMSHandler
> after 2000 ms (attempt 1 of 10) with error:
> javax.jdo.JDOObjectNotFoundException: No such database row
> FailedObject:49[OID]org.apache.hadoop.hive.metastore.model.MStorageDescriptor
> at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:475)
> at
> org.datanucleus.api.jdo.JDOAdapter.getApiExceptionForNucleusException(JDOAdapter.java:1158)
> at
> org.datanucleus.state.JDOStateManager.isLoaded(JDOStateManager.java:3231)
> at
> org.apache.hadoop.hive.metastore.model.MStorageDescriptor.jdoGetcd(MStorageDescriptor.java)
> at
> org.apache.hadoop.hive.metastore.model.MStorageDescriptor.getCD(MStorageDescriptor.java:184)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.convertToStorageDescriptor(ObjectStore.java:1282)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.convertToStorageDescriptor(ObjectStore.java:1299)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.convertToPart(ObjectStore.java:1680)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getPartition(ObjectStore.java:1586)
> at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:98)
> at com.sun.proxy.$Proxy0.getPartition(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.HiveAlterHandler.alterPartitions(HiveAlterHandler.java:538)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_partitions(HiveMetaStore.java:3317)
> at sun.reflect.GeneratedMethodAccessor37.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:102)
> at com.sun.proxy.$Proxy12.alter_partitions(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$alter_partitions.getResult(ThriftHiveMetastore.java:9963)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$alter_partitions.getResult(ThriftHiveMetastore.java:9947)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:110)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:106)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1673)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:118)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
[jira] [Updated] (HIVE-16839) Unbalanced calls to openTransaction/commitTransaction when alter the same partition concurrently
[
https://issues.apache.org/jira/browse/HIVE-16839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guang Yang updated HIVE-16839:
--
Attachment: 0001-HIVE-16839-Fix-a-race-condidtion-during-concurrent-p.patch
Status: Patch Available (was: Open)
> Unbalanced calls to openTransaction/commitTransaction when alter the same
> partition concurrently
>
>
> Key: HIVE-16839
> URL: https://issues.apache.org/jira/browse/HIVE-16839
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.1.0, 0.13.1
>Reporter: Nemon Lou
>Assignee: Guang Yang
>Priority: Major
> Labels: pull-request-available
> Attachments:
> 0001-HIVE-16839-Fix-a-race-condidtion-during-concurrent-p.patch
>
>
> SQL to reproduce:
> prepare:
> {noformat}
> hdfs dfs -mkdir -p
> /hzsrc/external/writing_dc/ltgsm/16e7a9b2-21a1-3f4f-8061-bc3395281627
> 1,create external table tb_ltgsm_external (id int) PARTITIONED by (cp
> string,ld string);
> {noformat}
> open one beeline run these two sql many times
> {noformat} 2,ALTER TABLE tb_ltgsm_external ADD IF NOT EXISTS PARTITION
> (cp=2017060513,ld=2017060610);
> 3,ALTER TABLE tb_ltgsm_external PARTITION (cp=2017060513,ld=2017060610) SET
> LOCATION
> 'hdfs://hacluster/hzsrc/external/writing_dc/ltgsm/16e7a9b2-21a1-3f4f-8061-bc3395281627';
> {noformat}
> open another beeline to run this sql many times at the same time.
> {noformat}
> 4,ALTER TABLE tb_ltgsm_external DROP PARTITION (cp=2017060513,ld=2017060610);
> {noformat}
> MetaStore logs:
> {noformat}
> 2017-06-06 21:58:34,213 | ERROR | pool-6-thread-197 | Retrying HMSHandler
> after 2000 ms (attempt 1 of 10) with error:
> javax.jdo.JDOObjectNotFoundException: No such database row
> FailedObject:49[OID]org.apache.hadoop.hive.metastore.model.MStorageDescriptor
> at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:475)
> at
> org.datanucleus.api.jdo.JDOAdapter.getApiExceptionForNucleusException(JDOAdapter.java:1158)
> at
> org.datanucleus.state.JDOStateManager.isLoaded(JDOStateManager.java:3231)
> at
> org.apache.hadoop.hive.metastore.model.MStorageDescriptor.jdoGetcd(MStorageDescriptor.java)
> at
> org.apache.hadoop.hive.metastore.model.MStorageDescriptor.getCD(MStorageDescriptor.java:184)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.convertToStorageDescriptor(ObjectStore.java:1282)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.convertToStorageDescriptor(ObjectStore.java:1299)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.convertToPart(ObjectStore.java:1680)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getPartition(ObjectStore.java:1586)
> at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:98)
> at com.sun.proxy.$Proxy0.getPartition(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.HiveAlterHandler.alterPartitions(HiveAlterHandler.java:538)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_partitions(HiveMetaStore.java:3317)
> at sun.reflect.GeneratedMethodAccessor37.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:102)
> at com.sun.proxy.$Proxy12.alter_partitions(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$alter_partitions.getResult(ThriftHiveMetastore.java:9963)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$alter_partitions.getResult(ThriftHiveMetastore.java:9947)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:110)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:106)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1673)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:118)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
> at
>
[jira] [Updated] (HIVE-16839) Unbalanced calls to openTransaction/commitTransaction when alter the same partition concurrently
[
https://issues.apache.org/jira/browse/HIVE-16839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HIVE-16839:
--
Labels: pull-request-available (was: )
> Unbalanced calls to openTransaction/commitTransaction when alter the same
> partition concurrently
>
>
> Key: HIVE-16839
> URL: https://issues.apache.org/jira/browse/HIVE-16839
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.1.0
>Reporter: Nemon Lou
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Labels: pull-request-available
>
> SQL to reproduce:
> prepare:
> {noformat}
> hdfs dfs -mkdir -p
> /hzsrc/external/writing_dc/ltgsm/16e7a9b2-21a1-3f4f-8061-bc3395281627
> 1,create external table tb_ltgsm_external (id int) PARTITIONED by (cp
> string,ld string);
> {noformat}
> open one beeline run these two sql many times
> {noformat} 2,ALTER TABLE tb_ltgsm_external ADD IF NOT EXISTS PARTITION
> (cp=2017060513,ld=2017060610);
> 3,ALTER TABLE tb_ltgsm_external PARTITION (cp=2017060513,ld=2017060610) SET
> LOCATION
> 'hdfs://hacluster/hzsrc/external/writing_dc/ltgsm/16e7a9b2-21a1-3f4f-8061-bc3395281627';
> {noformat}
> open another beeline to run this sql many times at the same time.
> {noformat}
> 4,ALTER TABLE tb_ltgsm_external DROP PARTITION (cp=2017060513,ld=2017060610);
> {noformat}
> MetaStore logs:
> {noformat}
> 2017-06-06 21:58:34,213 | ERROR | pool-6-thread-197 | Retrying HMSHandler
> after 2000 ms (attempt 1 of 10) with error:
> javax.jdo.JDOObjectNotFoundException: No such database row
> FailedObject:49[OID]org.apache.hadoop.hive.metastore.model.MStorageDescriptor
> at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:475)
> at
> org.datanucleus.api.jdo.JDOAdapter.getApiExceptionForNucleusException(JDOAdapter.java:1158)
> at
> org.datanucleus.state.JDOStateManager.isLoaded(JDOStateManager.java:3231)
> at
> org.apache.hadoop.hive.metastore.model.MStorageDescriptor.jdoGetcd(MStorageDescriptor.java)
> at
> org.apache.hadoop.hive.metastore.model.MStorageDescriptor.getCD(MStorageDescriptor.java:184)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.convertToStorageDescriptor(ObjectStore.java:1282)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.convertToStorageDescriptor(ObjectStore.java:1299)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.convertToPart(ObjectStore.java:1680)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getPartition(ObjectStore.java:1586)
> at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:98)
> at com.sun.proxy.$Proxy0.getPartition(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.HiveAlterHandler.alterPartitions(HiveAlterHandler.java:538)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_partitions(HiveMetaStore.java:3317)
> at sun.reflect.GeneratedMethodAccessor37.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:102)
> at com.sun.proxy.$Proxy12.alter_partitions(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$alter_partitions.getResult(ThriftHiveMetastore.java:9963)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$alter_partitions.getResult(ThriftHiveMetastore.java:9947)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:110)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:106)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1673)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:118)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at
[jira] [Commented] (HIVE-16839) Unbalanced calls to openTransaction/commitTransaction when alter the same partition concurrently
[
https://issues.apache.org/jira/browse/HIVE-16839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16673816#comment-16673816
]
ASF GitHub Bot commented on HIVE-16839:
---
GitHub user guangyy opened a pull request:
https://github.com/apache/hive/pull/484
HIVE-16839: Fix a race condidtion during concurrent partition drops
We have seen a leaked lock on hive metastore DB which caused all
PARTITION insertion failed on timeout waiting for lock until the
metastore service is restarted.
A transaction dump on the DB shows there is a thread that is Sleep which
potentiall holds the the lock, like:
```
trx_id: 33603171058
trx_state: RUNNING
trx_started: 2018-10-23 06:43:22
trx_requested_lock_id: NULL
trx_wait_started: NULL
trx_weight: 70298
trx_mysql_thread_id: 275402202
trx_query: NULL
trx_operation_state: NULL
trx_tables_in_use: 0
trx_tables_locked: 0
trx_lock_structs: 21286
trx_lock_memory_bytes: 2881064
trx_rows_locked: 98810
trx_rows_modified: 49012
trx_concurrency_tickets: 0
trx_isolation_level: READ COMMITTED
trx_unique_checks: 1
trx_foreign_key_checks: 1
trx_last_foreign_key_error: NULL
trx_adaptive_hash_latched: 0
trx_adaptive_hash_timeout: 0
trx_is_read_only: 0
trx_autocommit_non_locking: 0
ID: 275402202
USER: metastore_gold
HOST: 10.37.182.82:36684
DB: metastoregold
COMMAND: Sleep
TIME: 1
STATE:
INFO: NULL
duration: 1316
Given the HOST ip, we trace back to the hive metastore instance and found
the following exceptions:
No such database row
org.datanucleus.exceptions.NucleusObjectNotFoundException: No such database
row
at
org.datanucleus.store.rdbms.request.FetchRequest.execute(FetchRequest.java:357)
at
org.datanucleus.store.rdbms.RDBMSPersistenceHandler.fetchObject(RDBMSPersistenceHandler.java:324)
at
org.datanucleus.state.AbstractStateManager.loadFieldsFromDatastore(AbstractStateManager.java:1120)
at
org.datanucleus.state.JDOStateManager.loadSpecifiedFields(JDOStateManager.java:2916)
at
org.datanucleus.state.JDOStateManager.isLoaded(JDOStateManager.java:3219)
```
The problem is that the caller expects a NULL if the partition does not
exist, however, the convertToPart function would throw
an exception which lead to the leak.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/guangyy/hive HIVE-16839
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/hive/pull/484.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #484
commit 5137027ee658990dd1503c09c13a73e2848d8deb
Author: Guang Yang
Date: 2018-11-02T23:21:35Z
HIVE-16839: Fix a race condidtion during concurrent partition drops
> Unbalanced calls to openTransaction/commitTransaction when alter the same
> partition concurrently
>
>
> Key: HIVE-16839
> URL: https://issues.apache.org/jira/browse/HIVE-16839
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.1.0
>Reporter: Nemon Lou
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Labels: pull-request-available
>
> SQL to reproduce:
> prepare:
> {noformat}
> hdfs dfs -mkdir -p
> /hzsrc/external/writing_dc/ltgsm/16e7a9b2-21a1-3f4f-8061-bc3395281627
> 1,create external table tb_ltgsm_external (id int) PARTITIONED by (cp
> string,ld string);
> {noformat}
> open one beeline run these two sql many times
> {noformat} 2,ALTER TABLE tb_ltgsm_external ADD IF NOT EXISTS PARTITION
> (cp=2017060513,ld=2017060610);
> 3,ALTER TABLE tb_ltgsm_external PARTITION (cp=2017060513,ld=2017060610) SET
> LOCATION
> 'hdfs://hacluster/hzsrc/external/writing_dc/ltgsm/16e7a9b2-21a1-3f4f-8061-bc3395281627';
> {noformat}
> open another beeline to run this sql many times at the same time.
> {noformat}
> 4,ALTER TABLE tb_ltgsm_external DROP PARTITION (cp=2017060513,ld=2017060610);
> {noformat}
> MetaStore logs:
> {noformat}
> 2017-06-06 21:58:34,213 | ERROR | pool-6-thread-197 | Retrying HMSHandler
> after 2000 ms (attempt 1 of 10) with error:
> javax.jdo.JDOObjectNotFoundException: No such database row
>
[jira] [Commented] (HIVE-20748) Disable materialized view rewriting when plan pattern is not allowed
[
https://issues.apache.org/jira/browse/HIVE-20748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16673802#comment-16673802
]
Hive QA commented on HIVE-20748:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
46s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
46s{color} | {color:blue} ql in master has 2315 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green} ql: The patch generated 0 new + 792 unchanged - 3
fixed = 792 total (was 795) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m
58s{color} | {color:red} ql generated 1 new + 2314 unchanged - 1 fixed = 2315
total (was 2315) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 23m 1s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | Load of known null value in
org.apache.hadoop.hive.ql.parse.CalcitePlanner.canCBOHandleAst(ASTNode, QB,
CalcitePlanner$PreCboCtx) At CalcitePlanner.java:in
org.apache.hadoop.hive.ql.parse.CalcitePlanner.canCBOHandleAst(ASTNode, QB,
CalcitePlanner$PreCboCtx) At CalcitePlanner.java:[line 860] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14712/dev-support/hive-personality.sh
|
| git revision | master / f3fab45 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14712/yetus/new-findbugs-ql.html
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14712/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Disable materialized view rewriting when plan pattern is not allowed
>
>
> Key: HIVE-20748
> URL: https://issues.apache.org/jira/browse/HIVE-20748
> Project: Hive
> Issue Type: Bug
> Components: Materialized views
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-20748.01.patch, HIVE-20748.01.patch,
> HIVE-20748.02.patch, HIVE-20748.02.patch, HIVE-20748.03.patch,
> HIVE-20748.patch
>
>
> For instance, currently rewriting algorithm does not support some operators.
> Or we cannot have non-deterministic function in the MV definition. In those
> cases, we should fail either when we try to create the MV with rewriting
> enabled, or when when we enable the rewriting for a MV already created.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-16839) Unbalanced calls to openTransaction/commitTransaction when alter the same partition concurrently
[
https://issues.apache.org/jira/browse/HIVE-16839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16673797#comment-16673797
]
Guang Yang commented on HIVE-16839:
---
Thanks [~vihangk1], I am happy to take on this.
> Unbalanced calls to openTransaction/commitTransaction when alter the same
> partition concurrently
>
>
> Key: HIVE-16839
> URL: https://issues.apache.org/jira/browse/HIVE-16839
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.1.0
>Reporter: Nemon Lou
>Assignee: Vihang Karajgaonkar
>Priority: Major
>
> SQL to reproduce:
> prepare:
> {noformat}
> hdfs dfs -mkdir -p
> /hzsrc/external/writing_dc/ltgsm/16e7a9b2-21a1-3f4f-8061-bc3395281627
> 1,create external table tb_ltgsm_external (id int) PARTITIONED by (cp
> string,ld string);
> {noformat}
> open one beeline run these two sql many times
> {noformat} 2,ALTER TABLE tb_ltgsm_external ADD IF NOT EXISTS PARTITION
> (cp=2017060513,ld=2017060610);
> 3,ALTER TABLE tb_ltgsm_external PARTITION (cp=2017060513,ld=2017060610) SET
> LOCATION
> 'hdfs://hacluster/hzsrc/external/writing_dc/ltgsm/16e7a9b2-21a1-3f4f-8061-bc3395281627';
> {noformat}
> open another beeline to run this sql many times at the same time.
> {noformat}
> 4,ALTER TABLE tb_ltgsm_external DROP PARTITION (cp=2017060513,ld=2017060610);
> {noformat}
> MetaStore logs:
> {noformat}
> 2017-06-06 21:58:34,213 | ERROR | pool-6-thread-197 | Retrying HMSHandler
> after 2000 ms (attempt 1 of 10) with error:
> javax.jdo.JDOObjectNotFoundException: No such database row
> FailedObject:49[OID]org.apache.hadoop.hive.metastore.model.MStorageDescriptor
> at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:475)
> at
> org.datanucleus.api.jdo.JDOAdapter.getApiExceptionForNucleusException(JDOAdapter.java:1158)
> at
> org.datanucleus.state.JDOStateManager.isLoaded(JDOStateManager.java:3231)
> at
> org.apache.hadoop.hive.metastore.model.MStorageDescriptor.jdoGetcd(MStorageDescriptor.java)
> at
> org.apache.hadoop.hive.metastore.model.MStorageDescriptor.getCD(MStorageDescriptor.java:184)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.convertToStorageDescriptor(ObjectStore.java:1282)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.convertToStorageDescriptor(ObjectStore.java:1299)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.convertToPart(ObjectStore.java:1680)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getPartition(ObjectStore.java:1586)
> at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:98)
> at com.sun.proxy.$Proxy0.getPartition(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.HiveAlterHandler.alterPartitions(HiveAlterHandler.java:538)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_partitions(HiveMetaStore.java:3317)
> at sun.reflect.GeneratedMethodAccessor37.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:102)
> at com.sun.proxy.$Proxy12.alter_partitions(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$alter_partitions.getResult(ThriftHiveMetastore.java:9963)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$alter_partitions.getResult(ThriftHiveMetastore.java:9947)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:110)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:106)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1673)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:118)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at
[jira] [Commented] (HIVE-20617) Fix type of constants in IN expressions to have correct type
[
https://issues.apache.org/jira/browse/HIVE-20617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16673794#comment-16673794
]
Jesus Camacho Rodriguez commented on HIVE-20617:
+1 (pending tests)
> Fix type of constants in IN expressions to have correct type
>
>
> Key: HIVE-20617
> URL: https://issues.apache.org/jira/browse/HIVE-20617
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-20617.01.patch, HIVE-20617.02.patch,
> HIVE-20617.03.patch, HIVE-20617.05.patch, HIVE-20617.06.patch,
> HIVE-20617.07.patch, HIVE-20617.08.patch, HIVE-20617.08.patch,
> HIVE-20617.08.patch, HIVE-20617.08.patch, HIVE-20617.08.patch,
> HIVE-20617.08.patch, HIVE-20617.08.patch, HIVE-20617.08.patch,
> HIVE-20617.09.patch, HIVE-20617.10.patch, HIVE-20617.10.patch,
> HIVE-20617.11.patch, HIVE-20617.11.patch, HIVE-20617.12.patch,
> HIVE-20617.13.patch, HIVE-20617.14.patch, HIVE-20617.15.patch,
> HIVE-20617.16.patch, HIVE-20617.16.patch, HIVE-20617.17.patch,
> HIVE-20617.18.patch
>
>
> In statements like {{struct(a,b) IN (const struct('x','y'), ... )}} the
> comparision in UDFIn may fail because if a or b is of char/varchar type the
> constants will retain string type - especially after PointlookupOptimizer
> compaction.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20850) Push case conditional from projections to dimension tables if possible
[
https://issues.apache.org/jira/browse/HIVE-20850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-20850:
Attachment: HIVE-20850.01.patch
> Push case conditional from projections to dimension tables if possible
> --
>
> Key: HIVE-20850
> URL: https://issues.apache.org/jira/browse/HIVE-20850
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-20850.01.patch, HIVE-20850.01.patch,
> HIVE-20850.01.patch
>
>
> noticed by [~gopalv]: If there is a project which could be only evaluated
> after the join; but the condition references only a single column from a
> small dimension table; hive will end up evaluating the same thing over and
> over again...
> {code}
> explain
> select s_store_name, s_store_id,
> sum(case when (d_day_name='Sunday') then ss_sales_price else null
> end) sun_sales,
> sum(case when (d_day_name='Monday') then ss_sales_price else null
> end) mon_sales,
> sum(case when (d_day_name='Tuesday') then ss_sales_price else null
> end) tue_sales,
> sum(case when (d_day_name='Wednesday') then ss_sales_price else null
> end) wed_sales,
> sum(case when (d_day_name='Thursday') then ss_sales_price else null
> end) thu_sales,
> sum(case when (d_day_name='Friday') then ss_sales_price else null
> end) fri_sales,
> sum(case when (d_day_name='Saturday') then ss_sales_price else null
> end) sat_sales
> from date_dim, store_sales, store
> where d_date_sk = ss_sold_date_sk and
>s_store_sk = ss_store_sk and
>s_gmt_offset = -6 and
>d_year = 1998
> group by s_store_name, s_store_id
> order by s_store_name,
> s_store_id,sun_sales,mon_sales,tue_sales,wed_sales,thu_sales,fri_sales,sat_sales
> limit 100;
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20744) Use SQL constraints to improve join reordering algorithm
[
https://issues.apache.org/jira/browse/HIVE-20744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16673768#comment-16673768
]
Hive QA commented on HIVE-20744:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12946288/HIVE-20744.02.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 10 failed/errored test(s), 15520 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestTezPerfConstraintsCliDriver.testCliDriver[cbo_query17]
(batchId=272)
org.apache.hadoop.hive.cli.TestTezPerfConstraintsCliDriver.testCliDriver[cbo_query24]
(batchId=272)
org.apache.hadoop.hive.cli.TestTezPerfConstraintsCliDriver.testCliDriver[cbo_query25]
(batchId=272)
org.apache.hadoop.hive.cli.TestTezPerfConstraintsCliDriver.testCliDriver[cbo_query64]
(batchId=272)
org.apache.hadoop.hive.cli.TestTezPerfConstraintsCliDriver.testCliDriver[cbo_query72]
(batchId=272)
org.apache.hadoop.hive.cli.TestTezPerfConstraintsCliDriver.testCliDriver[query17]
(batchId=272)
org.apache.hadoop.hive.cli.TestTezPerfConstraintsCliDriver.testCliDriver[query24]
(batchId=272)
org.apache.hadoop.hive.cli.TestTezPerfConstraintsCliDriver.testCliDriver[query25]
(batchId=272)
org.apache.hadoop.hive.cli.TestTezPerfConstraintsCliDriver.testCliDriver[query64]
(batchId=272)
org.apache.hadoop.hive.cli.TestTezPerfConstraintsCliDriver.testCliDriver[query72]
(batchId=272)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14711/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14711/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14711/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 10 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12946288 - PreCommit-HIVE-Build
> Use SQL constraints to improve join reordering algorithm
>
>
> Key: HIVE-20744
> URL: https://issues.apache.org/jira/browse/HIVE-20744
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-20744.01.patch, HIVE-20744.02.patch,
> HIVE-20744.patch
>
>
> Till now, it was all based on stats stored for the base tables and their
> columns. Now the optimizer can rely on constraints. Hence, this patch is for
> the join reordering costing to use constraints, and if it does not find any,
> rely on old code path.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20740) Remove global lock in ObjectStore.setConf method
[
https://issues.apache.org/jira/browse/HIVE-20740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vihang Karajgaonkar updated HIVE-20740:
---
Attachment: HIVE-20740.09.patch
> Remove global lock in ObjectStore.setConf method
>
>
> Key: HIVE-20740
> URL: https://issues.apache.org/jira/browse/HIVE-20740
> Project: Hive
> Issue Type: Improvement
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Attachments: HIVE-20740.01.patch, HIVE-20740.02.patch,
> HIVE-20740.04.patch, HIVE-20740.05.patch, HIVE-20740.06.patch,
> HIVE-20740.08.patch, HIVE-20740.09.patch
>
>
> The ObjectStore#setConf method has a global lock which can block other
> clients in concurrent workloads.
> {code}
> @Override
> @SuppressWarnings("nls")
> public void setConf(Configuration conf) {
> // Although an instance of ObjectStore is accessed by one thread, there
> may
> // be many threads with ObjectStore instances. So the static variables
> // pmf and prop need to be protected with locks.
> pmfPropLock.lock();
> try {
> isInitialized = false;
> this.conf = conf;
> this.areTxnStatsSupported = MetastoreConf.getBoolVar(conf,
> ConfVars.HIVE_TXN_STATS_ENABLED);
> configureSSL(conf);
> Properties propsFromConf = getDataSourceProps(conf);
> boolean propsChanged = !propsFromConf.equals(prop);
> if (propsChanged) {
> if (pmf != null){
> clearOutPmfClassLoaderCache(pmf);
> if (!forTwoMetastoreTesting) {
> // close the underlying connection pool to avoid leaks
> pmf.close();
> }
> }
> pmf = null;
> prop = null;
> }
> assert(!isActiveTransaction());
> shutdown();
> // Always want to re-create pm as we don't know if it were created by
> the
> // most recent instance of the pmf
> pm = null;
> directSql = null;
> expressionProxy = null;
> openTrasactionCalls = 0;
> currentTransaction = null;
> transactionStatus = TXN_STATUS.NO_STATE;
> initialize(propsFromConf);
> String partitionValidationRegex =
> MetastoreConf.getVar(this.conf,
> ConfVars.PARTITION_NAME_WHITELIST_PATTERN);
> if (partitionValidationRegex != null &&
> !partitionValidationRegex.isEmpty()) {
> partitionValidationPattern =
> Pattern.compile(partitionValidationRegex);
> } else {
> partitionValidationPattern = null;
> }
> // Note, if metrics have not been initialized this will return null,
> which means we aren't
> // using metrics. Thus we should always check whether this is non-null
> before using.
> MetricRegistry registry = Metrics.getRegistry();
> if (registry != null) {
> directSqlErrors =
> Metrics.getOrCreateCounter(MetricsConstants.DIRECTSQL_ERRORS);
> }
> this.batchSize = MetastoreConf.getIntVar(conf,
> ConfVars.RAWSTORE_PARTITION_BATCH_SIZE);
> if (!isInitialized) {
> throw new RuntimeException(
> "Unable to create persistence manager. Check dss.log for details");
> } else {
> LOG.debug("Initialized ObjectStore");
> }
> } finally {
> pmfPropLock.unlock();
> }
> }
> {code}
> The {{pmfPropLock}} is a static object and it disallows any other new
> connection to HMS which is trying to instantiate ObjectStore. We should
> either remove the lock or reduce the scope of the lock so that it is held for
> a very small amount of time.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20860) Fix or disable TestMiniLlapLocalCliDriver.testCliDriver[cbo_limit]
[
https://issues.apache.org/jira/browse/HIVE-20860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vihang Karajgaonkar updated HIVE-20860:
---
Attachment:
182-TestMiniLlapLocalCliDriver-vector_udf_adaptor_1.q-schema_evol_text_vec_part_llap_io.q-join_is_not_distinct_from.q-and-27-more.txt
> Fix or disable TestMiniLlapLocalCliDriver.testCliDriver[cbo_limit]
> --
>
> Key: HIVE-20860
> URL: https://issues.apache.org/jira/browse/HIVE-20860
> Project: Hive
> Issue Type: Test
>Reporter: Vihang Karajgaonkar
>Priority: Minor
> Attachments:
> 182-TestMiniLlapLocalCliDriver-vector_udf_adaptor_1.q-schema_evol_text_vec_part_llap_io.q-join_is_not_distinct_from.q-and-27-more.txt,
> hive.log.gz, maven-test.txt
>
>
> Test failed in one of the precommit job. Looks like there is some case where
> there is additonal space in the diff
> {noformat}
> Error Message
> Client Execution succeeded but contained differences (error code = 1) after
> executing cbo_limit.q
> 11c11
> < 1 4 2
> ---
> > 1 4 2
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20860) Fix or disable TestMiniLlapLocalCliDriver.testCliDriver[cbo_limit]
[
https://issues.apache.org/jira/browse/HIVE-20860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vihang Karajgaonkar updated HIVE-20860:
---
Attachment: maven-test.txt
> Fix or disable TestMiniLlapLocalCliDriver.testCliDriver[cbo_limit]
> --
>
> Key: HIVE-20860
> URL: https://issues.apache.org/jira/browse/HIVE-20860
> Project: Hive
> Issue Type: Test
>Reporter: Vihang Karajgaonkar
>Priority: Minor
> Attachments: hive.log.gz, maven-test.txt
>
>
> Test failed in one of the precommit job. Looks like there is some case where
> there is additonal space in the diff
> {noformat}
> Error Message
> Client Execution succeeded but contained differences (error code = 1) after
> executing cbo_limit.q
> 11c11
> < 1 4 2
> ---
> > 1 4 2
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20860) Fix or disable TestMiniLlapLocalCliDriver.testCliDriver[cbo_limit]
[
https://issues.apache.org/jira/browse/HIVE-20860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vihang Karajgaonkar updated HIVE-20860:
---
Attachment: hive.log.gz
> Fix or disable TestMiniLlapLocalCliDriver.testCliDriver[cbo_limit]
> --
>
> Key: HIVE-20860
> URL: https://issues.apache.org/jira/browse/HIVE-20860
> Project: Hive
> Issue Type: Test
>Reporter: Vihang Karajgaonkar
>Priority: Minor
> Attachments: hive.log.gz
>
>
> Test failed in one of the precommit job. Looks like there is some case where
> there is additonal space in the diff
> {noformat}
> Error Message
> Client Execution succeeded but contained differences (error code = 1) after
> executing cbo_limit.q
> 11c11
> < 1 4 2
> ---
> > 1 4 2
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20740) Remove global lock in ObjectStore.setConf method
[
https://issues.apache.org/jira/browse/HIVE-20740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16673729#comment-16673729
]
Vihang Karajgaonkar commented on HIVE-20740:
Test is working for me locally. I will create a JIRA to disable/fix this test.
[INFO] ---
[INFO] T E S T S
[INFO] ---
[INFO] Running org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 31.367 s
- in org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
> Remove global lock in ObjectStore.setConf method
>
>
> Key: HIVE-20740
> URL: https://issues.apache.org/jira/browse/HIVE-20740
> Project: Hive
> Issue Type: Improvement
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Attachments: HIVE-20740.01.patch, HIVE-20740.02.patch,
> HIVE-20740.04.patch, HIVE-20740.05.patch, HIVE-20740.06.patch,
> HIVE-20740.08.patch
>
>
> The ObjectStore#setConf method has a global lock which can block other
> clients in concurrent workloads.
> {code}
> @Override
> @SuppressWarnings("nls")
> public void setConf(Configuration conf) {
> // Although an instance of ObjectStore is accessed by one thread, there
> may
> // be many threads with ObjectStore instances. So the static variables
> // pmf and prop need to be protected with locks.
> pmfPropLock.lock();
> try {
> isInitialized = false;
> this.conf = conf;
> this.areTxnStatsSupported = MetastoreConf.getBoolVar(conf,
> ConfVars.HIVE_TXN_STATS_ENABLED);
> configureSSL(conf);
> Properties propsFromConf = getDataSourceProps(conf);
> boolean propsChanged = !propsFromConf.equals(prop);
> if (propsChanged) {
> if (pmf != null){
> clearOutPmfClassLoaderCache(pmf);
> if (!forTwoMetastoreTesting) {
> // close the underlying connection pool to avoid leaks
> pmf.close();
> }
> }
> pmf = null;
> prop = null;
> }
> assert(!isActiveTransaction());
> shutdown();
> // Always want to re-create pm as we don't know if it were created by
> the
> // most recent instance of the pmf
> pm = null;
> directSql = null;
> expressionProxy = null;
> openTrasactionCalls = 0;
> currentTransaction = null;
> transactionStatus = TXN_STATUS.NO_STATE;
> initialize(propsFromConf);
> String partitionValidationRegex =
> MetastoreConf.getVar(this.conf,
> ConfVars.PARTITION_NAME_WHITELIST_PATTERN);
> if (partitionValidationRegex != null &&
> !partitionValidationRegex.isEmpty()) {
> partitionValidationPattern =
> Pattern.compile(partitionValidationRegex);
> } else {
> partitionValidationPattern = null;
> }
> // Note, if metrics have not been initialized this will return null,
> which means we aren't
> // using metrics. Thus we should always check whether this is non-null
> before using.
> MetricRegistry registry = Metrics.getRegistry();
> if (registry != null) {
> directSqlErrors =
> Metrics.getOrCreateCounter(MetricsConstants.DIRECTSQL_ERRORS);
> }
> this.batchSize = MetastoreConf.getIntVar(conf,
> ConfVars.RAWSTORE_PARTITION_BATCH_SIZE);
> if (!isInitialized) {
> throw new RuntimeException(
> "Unable to create persistence manager. Check dss.log for details");
> } else {
> LOG.debug("Initialized ObjectStore");
> }
> } finally {
> pmfPropLock.unlock();
> }
> }
> {code}
> The {{pmfPropLock}} is a static object and it disallows any other new
> connection to HMS which is trying to instantiate ObjectStore. We should
> either remove the lock or reduce the scope of the lock so that it is held for
> a very small amount of time.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20744) Use SQL constraints to improve join reordering algorithm
[
https://issues.apache.org/jira/browse/HIVE-20744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16673719#comment-16673719
]
Hive QA commented on HIVE-20744:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
42s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
39s{color} | {color:blue} ql in master has 2315 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
38s{color} | {color:red} ql: The patch generated 21 new + 52 unchanged - 12
fixed = 73 total (was 64) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m
54s{color} | {color:red} ql generated 1 new + 2315 unchanged - 0 fixed = 2316
total (was 2315) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 22m 25s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | Redundant nullcheck of leftInputResult, which is known to be non-null in
org.apache.hadoop.hive.ql.optimizer.calcite.stats.HiveRelMdRowCount.constraintsBasedAnalyzeJoinForPKFK(Join,
RelMetadataQuery) Redundant null check at HiveRelMdRowCount.java:is known to
be non-null in
org.apache.hadoop.hive.ql.optimizer.calcite.stats.HiveRelMdRowCount.constraintsBasedAnalyzeJoinForPKFK(Join,
RelMetadataQuery) Redundant null check at HiveRelMdRowCount.java:[line 360] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14711/dev-support/hive-personality.sh
|
| git revision | master / f3fab45 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14711/yetus/diff-checkstyle-ql.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14711/yetus/new-findbugs-ql.html
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14711/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Use SQL constraints to improve join reordering algorithm
>
>
> Key: HIVE-20744
> URL: https://issues.apache.org/jira/browse/HIVE-20744
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-20744.01.patch, HIVE-20744.02.patch,
> HIVE-20744.patch
>
>
> Till now, it was all based on stats stored for the base tables and their
> columns. Now the optimizer can rely on constraints. Hence, this patch is for
> the join reordering costing to use constraints, and if it does not find any,
> rely on old code path.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20804) Further improvements to group by optimization with constraints
[
https://issues.apache.org/jira/browse/HIVE-20804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16673687#comment-16673687
]
Hive QA commented on HIVE-20804:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12946087/HIVE-20804.3.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 15520 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[schemeAuthority]
(batchId=190)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query1]
(batchId=274)
org.apache.hive.jdbc.TestJdbcDriver2.testSelectExecAsync2 (batchId=259)
org.apache.hive.jdbc.TestJdbcDriver2.testYarnATSGuid (batchId=259)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14710/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14710/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14710/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12946087 - PreCommit-HIVE-Build
> Further improvements to group by optimization with constraints
> --
>
> Key: HIVE-20804
> URL: https://issues.apache.org/jira/browse/HIVE-20804
> Project: Hive
> Issue Type: Improvement
> Components: Query Planning
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-20804.1.patch, HIVE-20804.2.patch,
> HIVE-20804.3.patch
>
>
> Continuation of HIVE-17043
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20617) Fix type of constants in IN expressions to have correct type
[
https://issues.apache.org/jira/browse/HIVE-20617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-20617:
Attachment: HIVE-20617.18.patch
> Fix type of constants in IN expressions to have correct type
>
>
> Key: HIVE-20617
> URL: https://issues.apache.org/jira/browse/HIVE-20617
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-20617.01.patch, HIVE-20617.02.patch,
> HIVE-20617.03.patch, HIVE-20617.05.patch, HIVE-20617.06.patch,
> HIVE-20617.07.patch, HIVE-20617.08.patch, HIVE-20617.08.patch,
> HIVE-20617.08.patch, HIVE-20617.08.patch, HIVE-20617.08.patch,
> HIVE-20617.08.patch, HIVE-20617.08.patch, HIVE-20617.08.patch,
> HIVE-20617.09.patch, HIVE-20617.10.patch, HIVE-20617.10.patch,
> HIVE-20617.11.patch, HIVE-20617.11.patch, HIVE-20617.12.patch,
> HIVE-20617.13.patch, HIVE-20617.14.patch, HIVE-20617.15.patch,
> HIVE-20617.16.patch, HIVE-20617.16.patch, HIVE-20617.17.patch,
> HIVE-20617.18.patch
>
>
> In statements like {{struct(a,b) IN (const struct('x','y'), ... )}} the
> comparision in UDFIn may fail because if a or b is of char/varchar type the
> constants will retain string type - especially after PointlookupOptimizer
> compaction.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20804) Further improvements to group by optimization with constraints
[
https://issues.apache.org/jira/browse/HIVE-20804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16673640#comment-16673640
]
Hive QA commented on HIVE-20804:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
10s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
50s{color} | {color:blue} ql in master has 2315 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
39s{color} | {color:red} ql: The patch generated 1 new + 6 unchanged - 0 fixed
= 7 total (was 6) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m
56s{color} | {color:red} ql generated 1 new + 2315 unchanged - 0 fixed = 2316
total (was 2315) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 23m 9s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | Dead store to mapInToOutPos in
org.apache.hadoop.hive.ql.optimizer.calcite.HiveRelOptUtil$CardinalityChange.isCardinalitySameAsSource(HiveProject,
ImmutableBitSet) At
HiveRelOptUtil.java:org.apache.hadoop.hive.ql.optimizer.calcite.HiveRelOptUtil$CardinalityChange.isCardinalitySameAsSource(HiveProject,
ImmutableBitSet) At HiveRelOptUtil.java:[line 823] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14710/dev-support/hive-personality.sh
|
| git revision | master / f3fab45 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14710/yetus/diff-checkstyle-ql.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14710/yetus/new-findbugs-ql.html
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14710/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Further improvements to group by optimization with constraints
> --
>
> Key: HIVE-20804
> URL: https://issues.apache.org/jira/browse/HIVE-20804
> Project: Hive
> Issue Type: Improvement
> Components: Query Planning
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-20804.1.patch, HIVE-20804.2.patch,
> HIVE-20804.3.patch
>
>
> Continuation of HIVE-17043
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20842) Fix logic introduced in HIVE-20660 to estimate statistics for group by
[
https://issues.apache.org/jira/browse/HIVE-20842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16673613#comment-16673613
]
Hive QA commented on HIVE-20842:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12946328/HIVE-20842.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 142 failed/errored test(s), 15520 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[annotate_stats_groupby]
(batchId=52)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_rp_annotate_stats_groupby]
(batchId=92)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_rp_auto_join1]
(batchId=4)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[empty_dir_in_table]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[except_distinct]
(batchId=155)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[explainuser_2]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[intersect_all]
(batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[intersect_distinct]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[intersect_merge]
(batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_llap_counters1]
(batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_llap_counters]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_ppd_basic]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_ppd_schema_evol_3a]
(batchId=155)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[parallel_colstats]
(batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[reduce_deduplicate]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[reduce_deduplicate_distinct]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[unionDistinct_1]
(batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[auto_smb_mapjoin_14]
(batchId=174)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[auto_sortmerge_join_10]
(batchId=177)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[auto_sortmerge_join_13]
(batchId=175)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[auto_sortmerge_join_14]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[auto_sortmerge_join_15]
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[auto_sortmerge_join_6]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[auto_sortmerge_join_9]
(batchId=176)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket2]
(batchId=172)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket4]
(batchId=178)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_groupby]
(batchId=182)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez2]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_num_reducers2]
(batchId=178)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_num_reducers_acid]
(batchId=173)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucketmapjoin6]
(batchId=173)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[check_constraint]
(batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[constraints_optimization]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[correlationoptimizer1]
(batchId=173)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[correlationoptimizer6]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[database]
(batchId=166)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[disable_merge_for_bucketing]
(batchId=179)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dynpart_sort_opt_vectorization]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dynpart_sort_optimization2]
(batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[enforce_constraint_notnull]
(batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[enforce_order]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[explainanalyze_2]
(batchId=177)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[explainuser_1]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[explainuser_4]
(batchId=169)
[jira] [Commented] (HIVE-20842) Fix logic introduced in HIVE-20660 to estimate statistics for group by
[
https://issues.apache.org/jira/browse/HIVE-20842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16673583#comment-16673583
]
Hive QA commented on HIVE-20842:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
53s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
49s{color} | {color:blue} ql in master has 2315 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
39s{color} | {color:red} ql: The patch generated 2 new + 29 unchanged - 0 fixed
= 31 total (was 29) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m
57s{color} | {color:red} ql generated 1 new + 2315 unchanged - 0 fixed = 2316
total (was 2315) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 22m 52s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | mGop could be null and is guaranteed to be dereferenced in
org.apache.hadoop.hive.ql.optimizer.stats.annotation.StatsRulesProcFactory$GroupByStatsRule.process(Node,
Stack, NodeProcessorCtx, Object[]) Dereferenced at
StatsRulesProcFactory.java:is guaranteed to be dereferenced in
org.apache.hadoop.hive.ql.optimizer.stats.annotation.StatsRulesProcFactory$GroupByStatsRule.process(Node,
Stack, NodeProcessorCtx, Object[]) Dereferenced at
StatsRulesProcFactory.java:[line 1430] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14709/dev-support/hive-personality.sh
|
| git revision | master / f3fab45 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14709/yetus/diff-checkstyle-ql.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14709/yetus/new-findbugs-ql.html
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14709/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Fix logic introduced in HIVE-20660 to estimate statistics for group by
> --
>
> Key: HIVE-20842
> URL: https://issues.apache.org/jira/browse/HIVE-20842
> Project: Hive
> Issue Type: Bug
> Components: Statistics
>Affects Versions: 4.0.0
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-20842.1.patch, HIVE-20842.2.patch
>
>
> HIVE-20660 introduced better estimation for group by operator. But the logic
> did not account for Partial and Full group by separately.
> For partial group by parallelism (i.e. number of tasks) should be taken into
> account.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20855) Clean up checkstyle & other warnings for CachedStore
[ https://issues.apache.org/jira/browse/HIVE-20855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Francke updated HIVE-20855: Status: Patch Available (was: Open) Trying again. I forgot about itests. > Clean up checkstyle & other warnings for CachedStore > > > Key: HIVE-20855 > URL: https://issues.apache.org/jira/browse/HIVE-20855 > Project: Hive > Issue Type: Improvement >Reporter: Lars Francke >Assignee: Lars Francke >Priority: Minor > Attachments: HIVE-20855.1.patch, HIVE-20855.2.patch > > > While documenting the CachedStore in the Wiki I found a bunch of warnings, > inconsistencies etc. in CachedStore, RawStore and a few other related classes. > This is a patch that cleans them up. It's almost whitespace-only. > It does not contain semantic code changes, it does contain two or three > deprecations in favor of renamed methods or fields. > This is going to be boring to review so according to the bylaws I'll ask for > feedback on the mailing list and will commit after a week or so. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20855) Clean up checkstyle & other warnings for CachedStore
[ https://issues.apache.org/jira/browse/HIVE-20855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Francke updated HIVE-20855: Attachment: HIVE-20855.2.patch > Clean up checkstyle & other warnings for CachedStore > > > Key: HIVE-20855 > URL: https://issues.apache.org/jira/browse/HIVE-20855 > Project: Hive > Issue Type: Improvement >Reporter: Lars Francke >Assignee: Lars Francke >Priority: Minor > Attachments: HIVE-20855.1.patch, HIVE-20855.2.patch > > > While documenting the CachedStore in the Wiki I found a bunch of warnings, > inconsistencies etc. in CachedStore, RawStore and a few other related classes. > This is a patch that cleans them up. It's almost whitespace-only. > It does not contain semantic code changes, it does contain two or three > deprecations in favor of renamed methods or fields. > This is going to be boring to review so according to the bylaws I'll ask for > feedback on the mailing list and will commit after a week or so. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20855) Clean up checkstyle & other warnings for CachedStore
[ https://issues.apache.org/jira/browse/HIVE-20855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Francke updated HIVE-20855: Status: Open (was: Patch Available) > Clean up checkstyle & other warnings for CachedStore > > > Key: HIVE-20855 > URL: https://issues.apache.org/jira/browse/HIVE-20855 > Project: Hive > Issue Type: Improvement >Reporter: Lars Francke >Assignee: Lars Francke >Priority: Minor > Attachments: HIVE-20855.1.patch, HIVE-20855.2.patch > > > While documenting the CachedStore in the Wiki I found a bunch of warnings, > inconsistencies etc. in CachedStore, RawStore and a few other related classes. > This is a patch that cleans them up. It's almost whitespace-only. > It does not contain semantic code changes, it does contain two or three > deprecations in favor of renamed methods or fields. > This is going to be boring to review so according to the bylaws I'll ask for > feedback on the mailing list and will commit after a week or so. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20815) JdbcRecordReader.next shall not eat exception
[ https://issues.apache.org/jira/browse/HIVE-20815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-20815: -- Attachment: HIVE-20815.4.patch > JdbcRecordReader.next shall not eat exception > - > > Key: HIVE-20815 > URL: https://issues.apache.org/jira/browse/HIVE-20815 > Project: Hive > Issue Type: Bug > Components: StorageHandler >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-20815.1.patch, HIVE-20815.2.patch, > HIVE-20815.3.patch, HIVE-20815.4.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20833) package.jdo needs to be updated to conform with HIVE-20221 changes
[
https://issues.apache.org/jira/browse/HIVE-20833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-20833:
---
Status: Patch Available (was: Open)
> package.jdo needs to be updated to conform with HIVE-20221 changes
> --
>
> Key: HIVE-20833
> URL: https://issues.apache.org/jira/browse/HIVE-20833
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-20833.1.patch, HIVE-20833.2.patch,
> HIVE-20833.3.patch, HIVE-20833.4.patch
>
>
> Following test if run with TestMiniLlapLocalCliDriver will fail:
> {code:sql}
> CREATE TABLE `alterPartTbl`(
>`po_header_id` bigint,
>`vendor_num` string,
>`requester_name` string,
>`approver_name` string,
>`buyer_name` string,
>`preparer_name` string,
>`po_requisition_number` string,
>`po_requisition_id` bigint,
>`po_requisition_desc` string,
>`rate_type` string,
>`rate_date` date,
>`rate` double,
>`blanket_total_amount` double,
>`authorization_status` string,
>`revision_num` bigint,
>`revised_date` date,
>`approved_flag` string,
>`approved_date` timestamp,
>`amount_limit` double,
>`note_to_authorizer` string,
>`note_to_vendor` string,
>`note_to_receiver` string,
>`vendor_order_num` string,
>`comments` string,
>`acceptance_required_flag` string,
>`acceptance_due_date` date,
>`closed_date` timestamp,
>`user_hold_flag` string,
>`approval_required_flag` string,
>`cancel_flag` string,
>`firm_status_lookup_code` string,
>`firm_date` date,
>`frozen_flag` string,
>`closed_code` string,
>`org_id` bigint,
>`reference_num` string,
>`wf_item_type` string,
>`wf_item_key` string,
>`submit_date` date,
>`sap_company_code` string,
>`sap_fiscal_year` bigint,
>`po_number` string,
>`sap_line_item` bigint,
>`closed_status_flag` string,
>`balancing_segment` string,
>`cost_center_segment` string,
>`base_amount_limit` double,
>`base_blanket_total_amount` double,
>`base_open_amount` double,
>`base_ordered_amount` double,
>`cancel_date` timestamp,
>`cbc_accounting_date` date,
>`change_requested_by` string,
>`change_summary` string,
>`confirming_order_flag` string,
>`document_creation_method` string,
>`edi_processed_flag` string,
>`edi_processed_status` string,
>`enabled_flag` string,
>`encumbrance_required_flag` string,
>`end_date` date,
>`end_date_active` date,
>`from_header_id` bigint,
>`from_type_lookup_code` string,
>`global_agreement_flag` string,
>`government_context` string,
>`interface_source_code` string,
>`ledger_currency_code` string,
>`open_amount` double,
>`ordered_amount` double,
>`pay_on_code` string,
>`payment_term_name` string,
>`pending_signature_flag` string,
>`po_revision_num` double,
>`preparer_id` bigint,
>`price_update_tolerance` double,
>`print_count` double,
>`printed_date` date,
>`reply_date` date,
>`reply_method_lookup_code` string,
>`rfq_close_date` date,
>`segment2` string,
>`segment3` string,
>`segment4` string,
>`segment5` string,
>`shipping_control` string,
>`start_date` date,
>`start_date_active` date,
>`summary_flag` string,
>`supply_agreement_flag` string,
>`usd_amount_limit` double,
>`usd_blanket_total_amount` double,
>`usd_exchange_rate` double,
>`usd_open_amount` double,
>`usd_order_amount` double,
>`ussgl_transaction_code` string,
>`xml_flag` string,
>`purchasing_organization_id` bigint,
>`purchasing_group_code` string,
>`last_updated_by_name` string,
>`created_by_name` string,
>`incoterms_1` string,
>`incoterms_2` string,
>`ame_approval_id` double,
>`ame_transaction_type` string,
>`auto_sourcing_flag` string,
>`cat_admin_auth_enabled_flag` string,
>`clm_document_number` string,
>`comm_rev_num` double,
>`consigned_consumption_flag` string,
>`consume_req_demand_flag` string,
>`conterms_articles_upd_date` timestamp,
>`conterms_deliv_upd_date` timestamp,
>`conterms_exist_flag` string,
>`cpa_reference` double,
>`created_language` string,
>`email_address` string,
>`enable_all_sites` string,
>`fax` string,
>`lock_owner_role` string,
>`lock_owner_user_id` double,
>`min_release_amount` double,
>`mrc_rate` string,
>`mrc_rate_date` string,
>`mrc_rate_type` string,
>`otm_recovery_flag` string,
>`otm_status_code` string,
>`pay_when_paid` string,
>`pcard_id` bigint,
>`program_update_date` timestamp,
>`quotation_class_code`
[jira] [Updated] (HIVE-20833) package.jdo needs to be updated to conform with HIVE-20221 changes
[
https://issues.apache.org/jira/browse/HIVE-20833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-20833:
---
Attachment: HIVE-20833.4.patch
> package.jdo needs to be updated to conform with HIVE-20221 changes
> --
>
> Key: HIVE-20833
> URL: https://issues.apache.org/jira/browse/HIVE-20833
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-20833.1.patch, HIVE-20833.2.patch,
> HIVE-20833.3.patch, HIVE-20833.4.patch
>
>
> Following test if run with TestMiniLlapLocalCliDriver will fail:
> {code:sql}
> CREATE TABLE `alterPartTbl`(
>`po_header_id` bigint,
>`vendor_num` string,
>`requester_name` string,
>`approver_name` string,
>`buyer_name` string,
>`preparer_name` string,
>`po_requisition_number` string,
>`po_requisition_id` bigint,
>`po_requisition_desc` string,
>`rate_type` string,
>`rate_date` date,
>`rate` double,
>`blanket_total_amount` double,
>`authorization_status` string,
>`revision_num` bigint,
>`revised_date` date,
>`approved_flag` string,
>`approved_date` timestamp,
>`amount_limit` double,
>`note_to_authorizer` string,
>`note_to_vendor` string,
>`note_to_receiver` string,
>`vendor_order_num` string,
>`comments` string,
>`acceptance_required_flag` string,
>`acceptance_due_date` date,
>`closed_date` timestamp,
>`user_hold_flag` string,
>`approval_required_flag` string,
>`cancel_flag` string,
>`firm_status_lookup_code` string,
>`firm_date` date,
>`frozen_flag` string,
>`closed_code` string,
>`org_id` bigint,
>`reference_num` string,
>`wf_item_type` string,
>`wf_item_key` string,
>`submit_date` date,
>`sap_company_code` string,
>`sap_fiscal_year` bigint,
>`po_number` string,
>`sap_line_item` bigint,
>`closed_status_flag` string,
>`balancing_segment` string,
>`cost_center_segment` string,
>`base_amount_limit` double,
>`base_blanket_total_amount` double,
>`base_open_amount` double,
>`base_ordered_amount` double,
>`cancel_date` timestamp,
>`cbc_accounting_date` date,
>`change_requested_by` string,
>`change_summary` string,
>`confirming_order_flag` string,
>`document_creation_method` string,
>`edi_processed_flag` string,
>`edi_processed_status` string,
>`enabled_flag` string,
>`encumbrance_required_flag` string,
>`end_date` date,
>`end_date_active` date,
>`from_header_id` bigint,
>`from_type_lookup_code` string,
>`global_agreement_flag` string,
>`government_context` string,
>`interface_source_code` string,
>`ledger_currency_code` string,
>`open_amount` double,
>`ordered_amount` double,
>`pay_on_code` string,
>`payment_term_name` string,
>`pending_signature_flag` string,
>`po_revision_num` double,
>`preparer_id` bigint,
>`price_update_tolerance` double,
>`print_count` double,
>`printed_date` date,
>`reply_date` date,
>`reply_method_lookup_code` string,
>`rfq_close_date` date,
>`segment2` string,
>`segment3` string,
>`segment4` string,
>`segment5` string,
>`shipping_control` string,
>`start_date` date,
>`start_date_active` date,
>`summary_flag` string,
>`supply_agreement_flag` string,
>`usd_amount_limit` double,
>`usd_blanket_total_amount` double,
>`usd_exchange_rate` double,
>`usd_open_amount` double,
>`usd_order_amount` double,
>`ussgl_transaction_code` string,
>`xml_flag` string,
>`purchasing_organization_id` bigint,
>`purchasing_group_code` string,
>`last_updated_by_name` string,
>`created_by_name` string,
>`incoterms_1` string,
>`incoterms_2` string,
>`ame_approval_id` double,
>`ame_transaction_type` string,
>`auto_sourcing_flag` string,
>`cat_admin_auth_enabled_flag` string,
>`clm_document_number` string,
>`comm_rev_num` double,
>`consigned_consumption_flag` string,
>`consume_req_demand_flag` string,
>`conterms_articles_upd_date` timestamp,
>`conterms_deliv_upd_date` timestamp,
>`conterms_exist_flag` string,
>`cpa_reference` double,
>`created_language` string,
>`email_address` string,
>`enable_all_sites` string,
>`fax` string,
>`lock_owner_role` string,
>`lock_owner_user_id` double,
>`min_release_amount` double,
>`mrc_rate` string,
>`mrc_rate_date` string,
>`mrc_rate_type` string,
>`otm_recovery_flag` string,
>`otm_status_code` string,
>`pay_when_paid` string,
>`pcard_id` bigint,
>`program_update_date` timestamp,
>`quotation_class_code` string,
>
[jira] [Updated] (HIVE-20833) package.jdo needs to be updated to conform with HIVE-20221 changes
[
https://issues.apache.org/jira/browse/HIVE-20833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-20833:
---
Status: Open (was: Patch Available)
> package.jdo needs to be updated to conform with HIVE-20221 changes
> --
>
> Key: HIVE-20833
> URL: https://issues.apache.org/jira/browse/HIVE-20833
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-20833.1.patch, HIVE-20833.2.patch,
> HIVE-20833.3.patch, HIVE-20833.4.patch
>
>
> Following test if run with TestMiniLlapLocalCliDriver will fail:
> {code:sql}
> CREATE TABLE `alterPartTbl`(
>`po_header_id` bigint,
>`vendor_num` string,
>`requester_name` string,
>`approver_name` string,
>`buyer_name` string,
>`preparer_name` string,
>`po_requisition_number` string,
>`po_requisition_id` bigint,
>`po_requisition_desc` string,
>`rate_type` string,
>`rate_date` date,
>`rate` double,
>`blanket_total_amount` double,
>`authorization_status` string,
>`revision_num` bigint,
>`revised_date` date,
>`approved_flag` string,
>`approved_date` timestamp,
>`amount_limit` double,
>`note_to_authorizer` string,
>`note_to_vendor` string,
>`note_to_receiver` string,
>`vendor_order_num` string,
>`comments` string,
>`acceptance_required_flag` string,
>`acceptance_due_date` date,
>`closed_date` timestamp,
>`user_hold_flag` string,
>`approval_required_flag` string,
>`cancel_flag` string,
>`firm_status_lookup_code` string,
>`firm_date` date,
>`frozen_flag` string,
>`closed_code` string,
>`org_id` bigint,
>`reference_num` string,
>`wf_item_type` string,
>`wf_item_key` string,
>`submit_date` date,
>`sap_company_code` string,
>`sap_fiscal_year` bigint,
>`po_number` string,
>`sap_line_item` bigint,
>`closed_status_flag` string,
>`balancing_segment` string,
>`cost_center_segment` string,
>`base_amount_limit` double,
>`base_blanket_total_amount` double,
>`base_open_amount` double,
>`base_ordered_amount` double,
>`cancel_date` timestamp,
>`cbc_accounting_date` date,
>`change_requested_by` string,
>`change_summary` string,
>`confirming_order_flag` string,
>`document_creation_method` string,
>`edi_processed_flag` string,
>`edi_processed_status` string,
>`enabled_flag` string,
>`encumbrance_required_flag` string,
>`end_date` date,
>`end_date_active` date,
>`from_header_id` bigint,
>`from_type_lookup_code` string,
>`global_agreement_flag` string,
>`government_context` string,
>`interface_source_code` string,
>`ledger_currency_code` string,
>`open_amount` double,
>`ordered_amount` double,
>`pay_on_code` string,
>`payment_term_name` string,
>`pending_signature_flag` string,
>`po_revision_num` double,
>`preparer_id` bigint,
>`price_update_tolerance` double,
>`print_count` double,
>`printed_date` date,
>`reply_date` date,
>`reply_method_lookup_code` string,
>`rfq_close_date` date,
>`segment2` string,
>`segment3` string,
>`segment4` string,
>`segment5` string,
>`shipping_control` string,
>`start_date` date,
>`start_date_active` date,
>`summary_flag` string,
>`supply_agreement_flag` string,
>`usd_amount_limit` double,
>`usd_blanket_total_amount` double,
>`usd_exchange_rate` double,
>`usd_open_amount` double,
>`usd_order_amount` double,
>`ussgl_transaction_code` string,
>`xml_flag` string,
>`purchasing_organization_id` bigint,
>`purchasing_group_code` string,
>`last_updated_by_name` string,
>`created_by_name` string,
>`incoterms_1` string,
>`incoterms_2` string,
>`ame_approval_id` double,
>`ame_transaction_type` string,
>`auto_sourcing_flag` string,
>`cat_admin_auth_enabled_flag` string,
>`clm_document_number` string,
>`comm_rev_num` double,
>`consigned_consumption_flag` string,
>`consume_req_demand_flag` string,
>`conterms_articles_upd_date` timestamp,
>`conterms_deliv_upd_date` timestamp,
>`conterms_exist_flag` string,
>`cpa_reference` double,
>`created_language` string,
>`email_address` string,
>`enable_all_sites` string,
>`fax` string,
>`lock_owner_role` string,
>`lock_owner_user_id` double,
>`min_release_amount` double,
>`mrc_rate` string,
>`mrc_rate_date` string,
>`mrc_rate_type` string,
>`otm_recovery_flag` string,
>`otm_status_code` string,
>`pay_when_paid` string,
>`pcard_id` bigint,
>`program_update_date` timestamp,
>`quotation_class_code`
[jira] [Commented] (HIVE-20826) Enhance HiveSemiJoin rule to convert join + group by on left side to Left Semi Join
[
https://issues.apache.org/jira/browse/HIVE-20826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16673541#comment-16673541
]
Hive QA commented on HIVE-20826:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12946469/HIVE-20826.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 15520 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[semijoin]
(batchId=120)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[cbo_query14]
(batchId=272)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query14]
(batchId=272)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14708/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14708/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14708/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12946469 - PreCommit-HIVE-Build
> Enhance HiveSemiJoin rule to convert join + group by on left side to Left
> Semi Join
> ---
>
> Key: HIVE-20826
> URL: https://issues.apache.org/jira/browse/HIVE-20826
> Project: Hive
> Issue Type: Improvement
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-20826.1.patch
>
>
> Currently HiveSemiJoin rule looks for pattern where group by is on right side.
> We can convert joins which have group by on left side (assuming group by keys
> are same as join keys and none of the columns are being projected from left
> side) to LEFT SEMI JOIN by swapping the inputs. e.g. queries such as:
> {code:sql}
> explain select pp.p_partkey from (select distinct p_name from part) p join
> part pp on pp.p_name = p.p_name;
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20617) Fix type of constants in IN expressions to have correct type
[
https://issues.apache.org/jira/browse/HIVE-20617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-20617:
Attachment: HIVE-20617.17.patch
> Fix type of constants in IN expressions to have correct type
>
>
> Key: HIVE-20617
> URL: https://issues.apache.org/jira/browse/HIVE-20617
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-20617.01.patch, HIVE-20617.02.patch,
> HIVE-20617.03.patch, HIVE-20617.05.patch, HIVE-20617.06.patch,
> HIVE-20617.07.patch, HIVE-20617.08.patch, HIVE-20617.08.patch,
> HIVE-20617.08.patch, HIVE-20617.08.patch, HIVE-20617.08.patch,
> HIVE-20617.08.patch, HIVE-20617.08.patch, HIVE-20617.08.patch,
> HIVE-20617.09.patch, HIVE-20617.10.patch, HIVE-20617.10.patch,
> HIVE-20617.11.patch, HIVE-20617.11.patch, HIVE-20617.12.patch,
> HIVE-20617.13.patch, HIVE-20617.14.patch, HIVE-20617.15.patch,
> HIVE-20617.16.patch, HIVE-20617.16.patch, HIVE-20617.17.patch
>
>
> In statements like {{struct(a,b) IN (const struct('x','y'), ... )}} the
> comparision in UDFIn may fail because if a or b is of char/varchar type the
> constants will retain string type - especially after PointlookupOptimizer
> compaction.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20778) Join reordering may not be triggered if all joins in plan are created by decorrelation logic
[ https://issues.apache.org/jira/browse/HIVE-20778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-20778: --- Status: Patch Available (was: Open) > Join reordering may not be triggered if all joins in plan are created by > decorrelation logic > > > Key: HIVE-20778 > URL: https://issues.apache.org/jira/browse/HIVE-20778 > Project: Hive > Issue Type: Bug > Components: CBO >Reporter: Jesus Camacho Rodriguez >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-20778.1.patch, HIVE-20778.2.patch, > HIVE-20778.3.patch, HIVE-20778.4.patch, HIVE-20778.5.patch, HIVE-20778.6.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20778) Join reordering may not be triggered if all joins in plan are created by decorrelation logic
[ https://issues.apache.org/jira/browse/HIVE-20778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-20778: --- Attachment: HIVE-20778.6.patch > Join reordering may not be triggered if all joins in plan are created by > decorrelation logic > > > Key: HIVE-20778 > URL: https://issues.apache.org/jira/browse/HIVE-20778 > Project: Hive > Issue Type: Bug > Components: CBO >Reporter: Jesus Camacho Rodriguez >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-20778.1.patch, HIVE-20778.2.patch, > HIVE-20778.3.patch, HIVE-20778.4.patch, HIVE-20778.5.patch, HIVE-20778.6.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20778) Join reordering may not be triggered if all joins in plan are created by decorrelation logic
[ https://issues.apache.org/jira/browse/HIVE-20778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-20778: --- Status: Open (was: Patch Available) > Join reordering may not be triggered if all joins in plan are created by > decorrelation logic > > > Key: HIVE-20778 > URL: https://issues.apache.org/jira/browse/HIVE-20778 > Project: Hive > Issue Type: Bug > Components: CBO >Reporter: Jesus Camacho Rodriguez >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-20778.1.patch, HIVE-20778.2.patch, > HIVE-20778.3.patch, HIVE-20778.4.patch, HIVE-20778.5.patch, HIVE-20778.6.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20826) Enhance HiveSemiJoin rule to convert join + group by on left side to Left Semi Join
[
https://issues.apache.org/jira/browse/HIVE-20826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16673500#comment-16673500
]
Hive QA commented on HIVE-20826:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
52s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
49s{color} | {color:blue} ql in master has 2315 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
39s{color} | {color:red} ql: The patch generated 5 new + 182 unchanged - 0
fixed = 187 total (was 182) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 23m 0s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14708/dev-support/hive-personality.sh
|
| git revision | master / f3fab45 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14708/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14708/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Enhance HiveSemiJoin rule to convert join + group by on left side to Left
> Semi Join
> ---
>
> Key: HIVE-20826
> URL: https://issues.apache.org/jira/browse/HIVE-20826
> Project: Hive
> Issue Type: Improvement
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-20826.1.patch
>
>
> Currently HiveSemiJoin rule looks for pattern where group by is on right side.
> We can convert joins which have group by on left side (assuming group by keys
> are same as join keys and none of the columns are being projected from left
> side) to LEFT SEMI JOIN by swapping the inputs. e.g. queries such as:
> {code:sql}
> explain select pp.p_partkey from (select distinct p_name from part) p join
> part pp on pp.p_name = p.p_name;
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20833) package.jdo needs to be updated to conform with HIVE-20221 changes
[
https://issues.apache.org/jira/browse/HIVE-20833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16673463#comment-16673463
]
Hive QA commented on HIVE-20833:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12946470/HIVE-20833.3.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 15520 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_partition_change_col]
(batchId=27)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cbo_limit]
(batchId=182)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14707/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14707/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14707/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12946470 - PreCommit-HIVE-Build
> package.jdo needs to be updated to conform with HIVE-20221 changes
> --
>
> Key: HIVE-20833
> URL: https://issues.apache.org/jira/browse/HIVE-20833
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-20833.1.patch, HIVE-20833.2.patch,
> HIVE-20833.3.patch
>
>
> Following test if run with TestMiniLlapLocalCliDriver will fail:
> {code:sql}
> CREATE TABLE `alterPartTbl`(
>`po_header_id` bigint,
>`vendor_num` string,
>`requester_name` string,
>`approver_name` string,
>`buyer_name` string,
>`preparer_name` string,
>`po_requisition_number` string,
>`po_requisition_id` bigint,
>`po_requisition_desc` string,
>`rate_type` string,
>`rate_date` date,
>`rate` double,
>`blanket_total_amount` double,
>`authorization_status` string,
>`revision_num` bigint,
>`revised_date` date,
>`approved_flag` string,
>`approved_date` timestamp,
>`amount_limit` double,
>`note_to_authorizer` string,
>`note_to_vendor` string,
>`note_to_receiver` string,
>`vendor_order_num` string,
>`comments` string,
>`acceptance_required_flag` string,
>`acceptance_due_date` date,
>`closed_date` timestamp,
>`user_hold_flag` string,
>`approval_required_flag` string,
>`cancel_flag` string,
>`firm_status_lookup_code` string,
>`firm_date` date,
>`frozen_flag` string,
>`closed_code` string,
>`org_id` bigint,
>`reference_num` string,
>`wf_item_type` string,
>`wf_item_key` string,
>`submit_date` date,
>`sap_company_code` string,
>`sap_fiscal_year` bigint,
>`po_number` string,
>`sap_line_item` bigint,
>`closed_status_flag` string,
>`balancing_segment` string,
>`cost_center_segment` string,
>`base_amount_limit` double,
>`base_blanket_total_amount` double,
>`base_open_amount` double,
>`base_ordered_amount` double,
>`cancel_date` timestamp,
>`cbc_accounting_date` date,
>`change_requested_by` string,
>`change_summary` string,
>`confirming_order_flag` string,
>`document_creation_method` string,
>`edi_processed_flag` string,
>`edi_processed_status` string,
>`enabled_flag` string,
>`encumbrance_required_flag` string,
>`end_date` date,
>`end_date_active` date,
>`from_header_id` bigint,
>`from_type_lookup_code` string,
>`global_agreement_flag` string,
>`government_context` string,
>`interface_source_code` string,
>`ledger_currency_code` string,
>`open_amount` double,
>`ordered_amount` double,
>`pay_on_code` string,
>`payment_term_name` string,
>`pending_signature_flag` string,
>`po_revision_num` double,
>`preparer_id` bigint,
>`price_update_tolerance` double,
>`print_count` double,
>`printed_date` date,
>`reply_date` date,
>`reply_method_lookup_code` string,
>`rfq_close_date` date,
>`segment2` string,
>`segment3` string,
>`segment4` string,
>`segment5` string,
>`shipping_control` string,
>`start_date` date,
>`start_date_active` date,
>`summary_flag` string,
>`supply_agreement_flag` string,
>`usd_amount_limit` double,
>`usd_blanket_total_amount` double,
>`usd_exchange_rate` double,
>`usd_open_amount` double,
>`usd_order_amount` double,
>
[jira] [Commented] (HIVE-20833) package.jdo needs to be updated to conform with HIVE-20221 changes
[
https://issues.apache.org/jira/browse/HIVE-20833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16673410#comment-16673410
]
Hive QA commented on HIVE-20833:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
35s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
21s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
46s{color} | {color:blue} ql in master has 2315 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
0s{color} | {color:blue} standalone-metastore/metastore-server in master has
185 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
10s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 26m 51s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14707/dev-support/hive-personality.sh
|
| git revision | master / f3fab45 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: itests ql standalone-metastore/metastore-server U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14707/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> package.jdo needs to be updated to conform with HIVE-20221 changes
> --
>
> Key: HIVE-20833
> URL: https://issues.apache.org/jira/browse/HIVE-20833
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-20833.1.patch, HIVE-20833.2.patch,
> HIVE-20833.3.patch
>
>
> Following test if run with TestMiniLlapLocalCliDriver will fail:
> {code:sql}
> CREATE TABLE `alterPartTbl`(
>`po_header_id` bigint,
>`vendor_num` string,
>`requester_name` string,
>`approver_name` string,
>`buyer_name` string,
>`preparer_name` string,
>`po_requisition_number` string,
>`po_requisition_id` bigint,
>`po_requisition_desc` string,
>`rate_type` string,
>`rate_date` date,
>`rate` double,
>`blanket_total_amount` double,
>`authorization_status` string,
>`revision_num` bigint,
>`revised_date` date,
>`approved_flag` string,
>`approved_date` timestamp,
>`amount_limit` double,
>
[jira] [Updated] (HIVE-20834) Hive QueryResultCache entries keeping reference to SemanticAnalyzer from cached query
[ https://issues.apache.org/jira/browse/HIVE-20834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-20834: -- Attachment: HIVE-20834.3.patch > Hive QueryResultCache entries keeping reference to SemanticAnalyzer from > cached query > - > > Key: HIVE-20834 > URL: https://issues.apache.org/jira/browse/HIVE-20834 > Project: Hive > Issue Type: Bug >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Attachments: HIVE-20834.1.patch, HIVE-20834.2.patch, > HIVE-20834.3.patch, dominator_tree.png > > > QueryResultCache.LookupInfo ends up keeping a reference to the > SemanticAnalyzer from the cached query, for as long as the cached entry is in > the cache. We should not be keeping the SemanticAnalyzer around after the > query is done executing since they can hold on to quite a bit of memory. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20778) Join reordering may not be triggered if all joins in plan are created by decorrelation logic
[
https://issues.apache.org/jira/browse/HIVE-20778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16673332#comment-16673332
]
Hive QA commented on HIVE-20778:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12946238/HIVE-20778.5.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 15518 tests
executed
*Failed tests:*
{noformat}
TestMiniDruidCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=196)
[druidmini_masking.q,druidmini_test1.q,druidkafkamini_basic.q,druidmini_joins.q,druid_timestamptz.q]
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_notin]
(batchId=142)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_select]
(batchId=128)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14706/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14706/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14706/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12946238 - PreCommit-HIVE-Build
> Join reordering may not be triggered if all joins in plan are created by
> decorrelation logic
>
>
> Key: HIVE-20778
> URL: https://issues.apache.org/jira/browse/HIVE-20778
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-20778.1.patch, HIVE-20778.2.patch,
> HIVE-20778.3.patch, HIVE-20778.4.patch, HIVE-20778.5.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20512) Improve record and memory usage logging in SparkRecordHandler
[
https://issues.apache.org/jira/browse/HIVE-20512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16673291#comment-16673291
]
Bharathkrishna Guruvayoor Murali commented on HIVE-20512:
-
Tests run successfully. [~stakiar] , can you please push this patch to master
if there are no further comments.
> Improve record and memory usage logging in SparkRecordHandler
> -
>
> Key: HIVE-20512
> URL: https://issues.apache.org/jira/browse/HIVE-20512
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Major
> Attachments: HIVE-20512.1.patch, HIVE-20512.2.patch,
> HIVE-20512.3.patch, HIVE-20512.4.patch, HIVE-20512.5.patch, HIVE-20512.6.patch
>
>
> We currently log memory usage and # of records processed in Spark tasks, but
> we should improve the methodology for how frequently we log this info.
> Currently we use the following code:
> {code:java}
> private long getNextLogThreshold(long currentThreshold) {
> // A very simple counter to keep track of number of rows processed by the
> // reducer. It dumps
> // every 1 million times, and quickly before that
> if (currentThreshold >= 100) {
> return currentThreshold + 100;
> }
> return 10 * currentThreshold;
> }
> {code}
> The issue is that after a while, the increase by 10x factor means that you
> have to process a huge # of records before this gets triggered.
> A better approach would be to log this info at a given interval. This would
> help in debugging tasks that are seemingly hung.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20855) Clean up checkstyle & other warnings for CachedStore
[ https://issues.apache.org/jira/browse/HIVE-20855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16673282#comment-16673282 ] Lars Francke commented on HIVE-20855: - This compiles on my end so I must have uploaded an outdated patch or done something different wrong. Either way I'll look at it. > Clean up checkstyle & other warnings for CachedStore > > > Key: HIVE-20855 > URL: https://issues.apache.org/jira/browse/HIVE-20855 > Project: Hive > Issue Type: Improvement >Reporter: Lars Francke >Assignee: Lars Francke >Priority: Minor > Attachments: HIVE-20855.1.patch > > > While documenting the CachedStore in the Wiki I found a bunch of warnings, > inconsistencies etc. in CachedStore, RawStore and a few other related classes. > This is a patch that cleans them up. It's almost whitespace-only. > It does not contain semantic code changes, it does contain two or three > deprecations in favor of renamed methods or fields. > This is going to be boring to review so according to the bylaws I'll ask for > feedback on the mailing list and will commit after a week or so. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20778) Join reordering may not be triggered if all joins in plan are created by decorrelation logic
[
https://issues.apache.org/jira/browse/HIVE-20778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16673254#comment-16673254
]
Hive QA commented on HIVE-20778:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
48s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
38s{color} | {color:blue} ql in master has 2315 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 22m 38s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14706/dev-support/hive-personality.sh
|
| git revision | master / f3fab45 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14706/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Join reordering may not be triggered if all joins in plan are created by
> decorrelation logic
>
>
> Key: HIVE-20778
> URL: https://issues.apache.org/jira/browse/HIVE-20778
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-20778.1.patch, HIVE-20778.2.patch,
> HIVE-20778.3.patch, HIVE-20778.4.patch, HIVE-20778.5.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20855) Clean up checkstyle & other warnings for CachedStore
[
https://issues.apache.org/jira/browse/HIVE-20855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16673184#comment-16673184
]
Hive QA commented on HIVE-20855:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12946606/HIVE-20855.1.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14705/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14705/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14705/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL
https://issues.apache.org/jira/secure/attachment/12946606/HIVE-20855.1.patch
was found in seen patch url's cache and a test was probably run already on it.
Aborting...
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12946606 - PreCommit-HIVE-Build
> Clean up checkstyle & other warnings for CachedStore
>
>
> Key: HIVE-20855
> URL: https://issues.apache.org/jira/browse/HIVE-20855
> Project: Hive
> Issue Type: Improvement
>Reporter: Lars Francke
>Assignee: Lars Francke
>Priority: Minor
> Attachments: HIVE-20855.1.patch
>
>
> While documenting the CachedStore in the Wiki I found a bunch of warnings,
> inconsistencies etc. in CachedStore, RawStore and a few other related classes.
> This is a patch that cleans them up. It's almost whitespace-only.
> It does not contain semantic code changes, it does contain two or three
> deprecations in favor of renamed methods or fields.
> This is going to be boring to review so according to the bylaws I'll ask for
> feedback on the mailing list and will commit after a week or so.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20855) Clean up checkstyle & other warnings for CachedStore
[
https://issues.apache.org/jira/browse/HIVE-20855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16673183#comment-16673183
]
Hive QA commented on HIVE-20855:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12946606/HIVE-20855.1.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14704/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14704/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14704/
Messages:
{noformat}
This message was trimmed, see log for full details
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/eclipse/jetty/jetty-server/9.3.25.v20180904/jetty-server-9.3.25.v20180904.jar(org/eclipse/jetty/server/Server.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/eclipse/jetty/jetty-server/9.3.25.v20180904/jetty-server-9.3.25.v20180904.jar(org/eclipse/jetty/server/ServerConnector.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/eclipse/jetty/jetty-server/9.3.25.v20180904/jetty-server-9.3.25.v20180904.jar(org/eclipse/jetty/server/handler/HandlerList.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/eclipse/jetty/jetty-servlet/9.3.25.v20180904/jetty-servlet-9.3.25.v20180904.jar(org/eclipse/jetty/servlet/FilterHolder.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/eclipse/jetty/jetty-servlet/9.3.25.v20180904/jetty-servlet-9.3.25.v20180904.jar(org/eclipse/jetty/servlet/ServletContextHandler.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/eclipse/jetty/jetty-servlet/9.3.25.v20180904/jetty-servlet-9.3.25.v20180904.jar(org/eclipse/jetty/servlet/ServletHolder.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/eclipse/jetty/jetty-xml/9.3.25.v20180904/jetty-xml-9.3.25.v20180904.jar(org/eclipse/jetty/xml/XmlConfiguration.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/slf4j/jul-to-slf4j/1.7.10/jul-to-slf4j-1.7.10.jar(org/slf4j/bridge/SLF4JBridgeHandler.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/javax/servlet/javax.servlet-api/3.1.0/javax.servlet-api-3.1.0.jar(javax/servlet/DispatcherType.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/javax/servlet/javax.servlet-api/3.1.0/javax.servlet-api-3.1.0.jar(javax/servlet/Filter.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/javax/servlet/javax.servlet-api/3.1.0/javax.servlet-api-3.1.0.jar(javax/servlet/FilterChain.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/javax/servlet/javax.servlet-api/3.1.0/javax.servlet-api-3.1.0.jar(javax/servlet/FilterConfig.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/javax/servlet/javax.servlet-api/3.1.0/javax.servlet-api-3.1.0.jar(javax/servlet/ServletException.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/javax/servlet/javax.servlet-api/3.1.0/javax.servlet-api-3.1.0.jar(javax/servlet/ServletRequest.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/javax/servlet/javax.servlet-api/3.1.0/javax.servlet-api-3.1.0.jar(javax/servlet/ServletResponse.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/javax/servlet/javax.servlet-api/3.1.0/javax.servlet-api-3.1.0.jar(javax/servlet/annotation/WebFilter.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/javax/servlet/javax.servlet-api/3.1.0/javax.servlet-api-3.1.0.jar(javax/servlet/http/HttpServletRequest.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/javax/servlet/javax.servlet-api/3.1.0/javax.servlet-api-3.1.0.jar(javax/servlet/http/HttpServletResponse.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/apache-github-source-source/classification/target/hive-classification-4.0.0-SNAPSHOT.jar(org/apache/hadoop/hive/common/classification/InterfaceAudience$LimitedPrivate.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/apache-github-source-source/classification/target/hive-classification-4.0.0-SNAPSHOT.jar(org/apache/hadoop/hive/common/classification/InterfaceStability$Unstable.class)]]
[loading
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/io/ByteArrayOutputStream.class)]]
[loading
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/io/OutputStream.class)]]
[loading
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/io/Closeable.class)]]
[loading
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/lang/AutoCloseable.class)]]
[loading
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/io/Flushable.class)]]
[loading
[jira] [Updated] (HIVE-20838) Timestamps with timezone are set to null when using the streaming API
[
https://issues.apache.org/jira/browse/HIVE-20838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jaume M updated HIVE-20838:
---
Attachment: HIVE-20838.5.patch
Status: Patch Available (was: Open)
> Timestamps with timezone are set to null when using the streaming API
> -
>
> Key: HIVE-20838
> URL: https://issues.apache.org/jira/browse/HIVE-20838
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.1.0
>Reporter: Jaume M
>Assignee: Jaume M
>Priority: Major
> Attachments: HIVE-20838.1.patch, HIVE-20838.2.patch,
> HIVE-20838.3.patch, HIVE-20838.4.patch, HIVE-20838.5.patch
>
>
> For example:
> {code}
> beeline> create table default.timest (a TIMESTAMP) stored as orc " +
> "TBLPROPERTIES('transactional'='true')
> # And then:
> connection.write("2018-10-19 10:35:00 America/Los_Angeles".getBytes());
> {code}
> inserts NULL.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20838) Timestamps with timezone are set to null when using the streaming API
[
https://issues.apache.org/jira/browse/HIVE-20838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jaume M updated HIVE-20838:
---
Status: Open (was: Patch Available)
> Timestamps with timezone are set to null when using the streaming API
> -
>
> Key: HIVE-20838
> URL: https://issues.apache.org/jira/browse/HIVE-20838
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.1.0
>Reporter: Jaume M
>Assignee: Jaume M
>Priority: Major
> Attachments: HIVE-20838.1.patch, HIVE-20838.2.patch,
> HIVE-20838.3.patch, HIVE-20838.4.patch, HIVE-20838.5.patch
>
>
> For example:
> {code}
> beeline> create table default.timest (a TIMESTAMP) stored as orc " +
> "TBLPROPERTIES('transactional'='true')
> # And then:
> connection.write("2018-10-19 10:35:00 America/Los_Angeles".getBytes());
> {code}
> inserts NULL.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20682) Async query execution can potentially fail if shared sessionHive is closed by master thread.
[
https://issues.apache.org/jira/browse/HIVE-20682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16673161#comment-16673161
]
Hive QA commented on HIVE-20682:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12946589/HIVE-20682.05.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15520 tests
executed
*Failed tests:*
{noformat}
org.apache.hive.service.cli.session.TestSessionManagerMetrics.testActiveSessionTimeMetrics
(batchId=235)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14703/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14703/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14703/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12946589 - PreCommit-HIVE-Build
> Async query execution can potentially fail if shared sessionHive is closed by
> master thread.
>
>
> Key: HIVE-20682
> URL: https://issues.apache.org/jira/browse/HIVE-20682
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 3.1.0, 4.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-20682.01.patch, HIVE-20682.02.patch,
> HIVE-20682.03.patch, HIVE-20682.04.patch, HIVE-20682.05.patch
>
>
> *Problem description:*
> The master thread initializes the *sessionHive* object in *HiveSessionImpl*
> class when we open a new session for a client connection and by default all
> queries from this connection shares the same sessionHive object.
> If the master thread executes a *synchronous* query, it closes the
> sessionHive object (referred via thread local hiveDb) if
> {{Hive.isCompatible}} returns false and sets new Hive object in thread local
> HiveDb but doesn't change the sessionHive object in the session. Whereas,
> *asynchronous* query execution via async threads never closes the sessionHive
> object and it just creates a new one if needed and sets it as their thread
> local hiveDb.
> So, the problem can happen in the case where an *asynchronous* query is being
> executed by async threads refers to sessionHive object and the master thread
> receives a *synchronous* query that closes the same sessionHive object.
> Also, each query execution overwrites the thread local hiveDb object to
> sessionHive object which potentially leaks a metastore connection if the
> previous synchronous query execution re-created the Hive object.
> *Possible Fix:*
> The *sessionHive* object could be shared my multiple threads and so it
> shouldn't be allowed to be closed by any query execution threads when they
> re-create the Hive object due to changes in Hive configurations. But the Hive
> objects created by query execution threads should be closed when the thread
> exits.
> So, it is proposed to have an *isAllowClose* flag (default: *true*) in Hive
> object which should be set to *false* for *sessionHive* and would be
> forcefully closed when the session is closed or released.
> Also, when we reset *sessionHive* object with new one due to changes in
> *sessionConf*, the old one should be closed when no async thread is referring
> to it. This can be done using "*finalize*" method of Hive object where we can
> close HMS connection when Hive object is garbage collected.
> cc [~pvary]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20682) Async query execution can potentially fail if shared sessionHive is closed by master thread.
[
https://issues.apache.org/jira/browse/HIVE-20682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16673127#comment-16673127
]
Hive QA commented on HIVE-20682:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
29s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
10s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
38s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
40s{color} | {color:blue} ql in master has 2315 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
36s{color} | {color:blue} service in master has 48 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
31s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s{color} | {color:green} The patch hive-unit passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} ql: The patch generated 0 new + 218 unchanged - 2
fixed = 218 total (was 220) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} The patch service passed checkstyle {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m
49s{color} | {color:red} ql generated 1 new + 2315 unchanged - 0 fixed = 2316
total (was 2315) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 31m 14s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | org.apache.hadoop.hive.ql.metadata.Hive.finalize() is public; should be
protected At Hive.java:protected At Hive.java:[lines 419-421] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14703/dev-support/hive-personality.sh
|
| git revision | master / f3fab45 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14703/yetus/new-findbugs-ql.html
|
| modules | C: itests/hive-unit ql service U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14703/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Async query execution can potentially fail if shared sessionHive is closed by
> master thread.
>
>
> Key: HIVE-20682
> URL: https://issues.apache.org/jira/browse/HIVE-20682
> Project: Hive
> Issue Type: Bug
>
[jira] [Resolved] (HIVE-20287) Document the differences between managed and external tables
[ https://issues.apache.org/jira/browse/HIVE-20287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Francke resolved HIVE-20287. - Resolution: Fixed > Document the differences between managed and external tables > > > Key: HIVE-20287 > URL: https://issues.apache.org/jira/browse/HIVE-20287 > Project: Hive > Issue Type: Improvement > Components: Documentation >Reporter: Lars Francke >Assignee: Lars Francke >Priority: Major > > We should document all the differences between managed and external tables. I > need everyone's help for that though. > This is what I know: > * ARCHIVE/UNARCHIVE - DDLTask - Only works for managed tables > * TRUNCATE - DDLSemanticAnalyzer - Only works for managed tables > * MERGE/CONCATENATE - HiveRelOpMaterializationValidator - Only works for > managed tables > * Constraints - DDLSemanticAnalyzer - (NOT NULL, DEFAULT, CHECK, only RELY > ist allowed) > * IMPORT - ImportSemanticAnalyzer - This has some wild restrictions I didn't > follow for external tables > * Query Results Caching - https://issues.apache.org/jira/browse/HIVE-18513 > SemanticAnalyzer > > Hortonworks has extra documentation listing these things: > * Query cache > * Materialized views, except in a limited way > * Default statistics gathering > * Compute queries using statistics > * Automatic runtime filtering > * File merging after insert > > It'd be great if someone (from Hortonworks or otherwise) could elaborate on > those. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20287) Document the differences between managed and external tables
[ https://issues.apache.org/jira/browse/HIVE-20287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16673028#comment-16673028 ] Lars Francke commented on HIVE-20287: - I have created https://cwiki.apache.org/confluence/display/Hive/Managed+vs.+External+Tables Any additions are still welcome but I'll close this issue as fixed for now. > Document the differences between managed and external tables > > > Key: HIVE-20287 > URL: https://issues.apache.org/jira/browse/HIVE-20287 > Project: Hive > Issue Type: Improvement > Components: Documentation >Reporter: Lars Francke >Assignee: Lars Francke >Priority: Major > > We should document all the differences between managed and external tables. I > need everyone's help for that though. > This is what I know: > * ARCHIVE/UNARCHIVE - DDLTask - Only works for managed tables > * TRUNCATE - DDLSemanticAnalyzer - Only works for managed tables > * MERGE/CONCATENATE - HiveRelOpMaterializationValidator - Only works for > managed tables > * Constraints - DDLSemanticAnalyzer - (NOT NULL, DEFAULT, CHECK, only RELY > ist allowed) > * IMPORT - ImportSemanticAnalyzer - This has some wild restrictions I didn't > follow for external tables > * Query Results Caching - https://issues.apache.org/jira/browse/HIVE-18513 > SemanticAnalyzer > > Hortonworks has extra documentation listing these things: > * Query cache > * Materialized views, except in a limited way > * Default statistics gathering > * Compute queries using statistics > * Automatic runtime filtering > * File merging after insert > > It'd be great if someone (from Hortonworks or otherwise) could elaborate on > those. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20740) Remove global lock in ObjectStore.setConf method
[
https://issues.apache.org/jira/browse/HIVE-20740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16673025#comment-16673025
]
Hive QA commented on HIVE-20740:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12946583/HIVE-20740.08.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15521 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cbo_limit]
(batchId=182)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14702/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14702/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14702/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12946583 - PreCommit-HIVE-Build
> Remove global lock in ObjectStore.setConf method
>
>
> Key: HIVE-20740
> URL: https://issues.apache.org/jira/browse/HIVE-20740
> Project: Hive
> Issue Type: Improvement
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Attachments: HIVE-20740.01.patch, HIVE-20740.02.patch,
> HIVE-20740.04.patch, HIVE-20740.05.patch, HIVE-20740.06.patch,
> HIVE-20740.08.patch
>
>
> The ObjectStore#setConf method has a global lock which can block other
> clients in concurrent workloads.
> {code}
> @Override
> @SuppressWarnings("nls")
> public void setConf(Configuration conf) {
> // Although an instance of ObjectStore is accessed by one thread, there
> may
> // be many threads with ObjectStore instances. So the static variables
> // pmf and prop need to be protected with locks.
> pmfPropLock.lock();
> try {
> isInitialized = false;
> this.conf = conf;
> this.areTxnStatsSupported = MetastoreConf.getBoolVar(conf,
> ConfVars.HIVE_TXN_STATS_ENABLED);
> configureSSL(conf);
> Properties propsFromConf = getDataSourceProps(conf);
> boolean propsChanged = !propsFromConf.equals(prop);
> if (propsChanged) {
> if (pmf != null){
> clearOutPmfClassLoaderCache(pmf);
> if (!forTwoMetastoreTesting) {
> // close the underlying connection pool to avoid leaks
> pmf.close();
> }
> }
> pmf = null;
> prop = null;
> }
> assert(!isActiveTransaction());
> shutdown();
> // Always want to re-create pm as we don't know if it were created by
> the
> // most recent instance of the pmf
> pm = null;
> directSql = null;
> expressionProxy = null;
> openTrasactionCalls = 0;
> currentTransaction = null;
> transactionStatus = TXN_STATUS.NO_STATE;
> initialize(propsFromConf);
> String partitionValidationRegex =
> MetastoreConf.getVar(this.conf,
> ConfVars.PARTITION_NAME_WHITELIST_PATTERN);
> if (partitionValidationRegex != null &&
> !partitionValidationRegex.isEmpty()) {
> partitionValidationPattern =
> Pattern.compile(partitionValidationRegex);
> } else {
> partitionValidationPattern = null;
> }
> // Note, if metrics have not been initialized this will return null,
> which means we aren't
> // using metrics. Thus we should always check whether this is non-null
> before using.
> MetricRegistry registry = Metrics.getRegistry();
> if (registry != null) {
> directSqlErrors =
> Metrics.getOrCreateCounter(MetricsConstants.DIRECTSQL_ERRORS);
> }
> this.batchSize = MetastoreConf.getIntVar(conf,
> ConfVars.RAWSTORE_PARTITION_BATCH_SIZE);
> if (!isInitialized) {
> throw new RuntimeException(
> "Unable to create persistence manager. Check dss.log for details");
> } else {
> LOG.debug("Initialized ObjectStore");
> }
> } finally {
> pmfPropLock.unlock();
> }
> }
> {code}
> The {{pmfPropLock}} is a static object and it disallows any other new
> connection to HMS which is trying to instantiate ObjectStore. We should
> either remove the lock or reduce the scope of the lock so that it is held for
> a very small amount of time.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20740) Remove global lock in ObjectStore.setConf method
[
https://issues.apache.org/jira/browse/HIVE-20740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16673002#comment-16673002
]
Hive QA commented on HIVE-20740:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
4s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
38s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
39s{color} | {color:blue} ql in master has 2315 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
3s{color} | {color:blue} standalone-metastore/metastore-server in master has
185 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
38s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
35s{color} | {color:red} hive-unit in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
29s{color} | {color:red} ql in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
0s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
20s{color} | {color:red} itests/hive-unit: The patch generated 1 new + 608
unchanged - 0 fixed = 609 total (was 608) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
13s{color} | {color:red} standalone-metastore/metastore-server generated 2 new
+ 183 unchanged - 2 fixed = 185 total (was 185) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
34s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 31m 36s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:standalone-metastore/metastore-server |
| | Dead store to thislimit in
org.apache.hadoop.hive.metastore.ObjectStore.getCallStack() At
ObjectStore.java:org.apache.hadoop.hive.metastore.ObjectStore.getCallStack()
At ObjectStore.java:[line 9522] |
| |
org.apache.hadoop.hive.metastore.PersistenceManagerProvider.updatePmfProperties(Configuration)
does not release lock on all paths At PersistenceManagerProvider.java:on all
paths At PersistenceManagerProvider.java:[line 151] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14702/dev-support/hive-personality.sh
|
| git revision | master / f3fab45 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| mvninstall |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14702/yetus/patch-mvninstall-itests_hive-unit.txt
|
| mvninstall |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14702/yetus/patch-mvninstall-ql.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14702/yetus/diff-checkstyle-itests_hive-unit.txt
|
| findbugs |
[jira] [Commented] (HIVE-20782) Cleaning some unused code
[
https://issues.apache.org/jira/browse/HIVE-20782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16672955#comment-16672955
]
Teddy Choi commented on HIVE-20782:
---
As we don't have any feedbacks, I think it would be okay to delete them. If we
need them, we can restore them from VCS history.
> Cleaning some unused code
> -
>
> Key: HIVE-20782
> URL: https://issues.apache.org/jira/browse/HIVE-20782
> Project: Hive
> Issue Type: Improvement
>Reporter: slim bouguerra
>Assignee: Teddy Choi
>Priority: Major
> Attachments: HIVE-20782.2.patch, HIVE-20782.2.patch,
> HIVE-20782.3.patch, HIVE-20782.patch
>
>
> Am making my way into the vectorize code and trying understand the APIs. Ran
> into this unused one, i guess it is not used anymore.
> [~ashutoshc] maybe can explain as you are the main contributor to this file
> {code}
> a/ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizedSerde.java{code}
> ?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-16913) Support per-session S3 credentials
[
https://issues.apache.org/jira/browse/HIVE-16913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16672912#comment-16672912
]
Steve Loughran commented on HIVE-16913:
---
DTs aren't sufficient here as Hive uses its granted superuser rights to request
DTs as a specific user from HDFS and YARN; you can't do this with object
stores. Instead users will somehow have to be able to submit DTs with their
queries
> Support per-session S3 credentials
> --
>
> Key: HIVE-16913
> URL: https://issues.apache.org/jira/browse/HIVE-16913
> Project: Hive
> Issue Type: Improvement
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
>
> Currently, the credentials needed to support Hive-on-S3 (or any other
> cloud-storage) need to be to the hive-site.xml. Either using a hadoop
> credential provider or by adding the keys in the hive-site.xml in plain text
> (unsecure)
> This limits the usecase to using a single S3 key. If we configure per bucket
> s3 keys like described [here |
> http://hadoop.apache.org/docs/current/hadoop-aws/tools/hadoop-aws/index.html#Configurations_different_S3_buckets]
> it exposes the access to all the buckets to all the hive users.
> It is possible that there are different sets of users who would not like to
> share there buckets and still be able to process the data using Hive.
> Enabling session level credentials will help solve such use-cases. For
> example, currently this doesn't work
> {noformat}
> set fs.s3a.secret.key=my_secret_key;
> set fs.s3a.access.key=my_access.key;
> {noformat}
> Because metastore is unaware of the the keys. This doesn't work either
> {noformat}
> set fs.s3a.secret.key=my_secret_key;
> set fs.s3a.access.key=my_access.key;
> set metaconf:fs.s3a.secret.key=my_secret_key;
> set metaconf:fs.s3a.access.key=my_access_key;
> {noformat}
> This is because only a certain metastore configurations defined in
> {{HiveConf.MetaVars}} are allowed to be set by the user. If we enable the
> above approaches we could potentially allow multiple S3 credentials on a
> per-session level basis.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20512) Improve record and memory usage logging in SparkRecordHandler
[
https://issues.apache.org/jira/browse/HIVE-20512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16672786#comment-16672786
]
Hive QA commented on HIVE-20512:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12946581/HIVE-20512.6.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15520 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14701/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14701/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14701/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12946581 - PreCommit-HIVE-Build
> Improve record and memory usage logging in SparkRecordHandler
> -
>
> Key: HIVE-20512
> URL: https://issues.apache.org/jira/browse/HIVE-20512
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Major
> Attachments: HIVE-20512.1.patch, HIVE-20512.2.patch,
> HIVE-20512.3.patch, HIVE-20512.4.patch, HIVE-20512.5.patch, HIVE-20512.6.patch
>
>
> We currently log memory usage and # of records processed in Spark tasks, but
> we should improve the methodology for how frequently we log this info.
> Currently we use the following code:
> {code:java}
> private long getNextLogThreshold(long currentThreshold) {
> // A very simple counter to keep track of number of rows processed by the
> // reducer. It dumps
> // every 1 million times, and quickly before that
> if (currentThreshold >= 100) {
> return currentThreshold + 100;
> }
> return 10 * currentThreshold;
> }
> {code}
> The issue is that after a while, the increase by 10x factor means that you
> have to process a huge # of records before this gets triggered.
> A better approach would be to log this info at a given interval. This would
> help in debugging tasks that are seemingly hung.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20512) Improve record and memory usage logging in SparkRecordHandler
[
https://issues.apache.org/jira/browse/HIVE-20512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16672716#comment-16672716
]
Hive QA commented on HIVE-20512:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
34s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
44s{color} | {color:blue} ql in master has 2315 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} ql: The patch generated 0 new + 4 unchanged - 5
fixed = 4 total (was 9) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m
52s{color} | {color:red} ql generated 1 new + 2315 unchanged - 0 fixed = 2316
total (was 2315) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 22m 17s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | Increment of volatile field
org.apache.hadoop.hive.ql.exec.spark.SparkRecordHandler.rowNumber in
org.apache.hadoop.hive.ql.exec.spark.SparkRecordHandler.incrementRowNumber()
At SparkRecordHandler.java:in
org.apache.hadoop.hive.ql.exec.spark.SparkRecordHandler.incrementRowNumber()
At SparkRecordHandler.java:[line 110] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14701/dev-support/hive-personality.sh
|
| git revision | master / f3fab45 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14701/yetus/new-findbugs-ql.html
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14701/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Improve record and memory usage logging in SparkRecordHandler
> -
>
> Key: HIVE-20512
> URL: https://issues.apache.org/jira/browse/HIVE-20512
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Major
> Attachments: HIVE-20512.1.patch, HIVE-20512.2.patch,
> HIVE-20512.3.patch, HIVE-20512.4.patch, HIVE-20512.5.patch, HIVE-20512.6.patch
>
>
> We currently log memory usage and # of records processed in Spark tasks, but
> we should improve the methodology for how frequently we log this info.
> Currently we use the following code:
> {code:java}
> private long getNextLogThreshold(long currentThreshold) {
> // A very simple counter to keep track of number of rows processed by the
> // reducer. It dumps
> // every 1 million times, and quickly before that
> if (currentThreshold >= 100) {
> return currentThreshold + 100;
> }
> return 10 *
[jira] [Commented] (HIVE-20834) Hive QueryResultCache entries keeping reference to SemanticAnalyzer from cached query
[
https://issues.apache.org/jira/browse/HIVE-20834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16672691#comment-16672691
]
Hive QA commented on HIVE-20834:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12946580/HIVE-20834.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 15515 tests
executed
*Failed tests:*
{noformat}
TestMiniDruidCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=195)
[druidmini_dynamic_partition.q,druidmini_test_ts.q,druidmini_expressions.q,druidmini_test_alter.q,druidmini_test_insert.q]
org.apache.hive.minikdc.TestJdbcWithMiniKdc.testTokenAuth (batchId=273)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14700/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14700/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14700/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12946580 - PreCommit-HIVE-Build
> Hive QueryResultCache entries keeping reference to SemanticAnalyzer from
> cached query
> -
>
> Key: HIVE-20834
> URL: https://issues.apache.org/jira/browse/HIVE-20834
> Project: Hive
> Issue Type: Bug
>Reporter: Jason Dere
>Assignee: Jason Dere
>Priority: Major
> Attachments: HIVE-20834.1.patch, HIVE-20834.2.patch,
> dominator_tree.png
>
>
> QueryResultCache.LookupInfo ends up keeping a reference to the
> SemanticAnalyzer from the cached query, for as long as the cached entry is in
> the cache. We should not be keeping the SemanticAnalyzer around after the
> query is done executing since they can hold on to quite a bit of memory.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18661) CachedStore: Use metastore notification log events to update cache
[ https://issues.apache.org/jira/browse/HIVE-18661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16672685#comment-16672685 ] Daniel Dai commented on HIVE-18661: --- Note with current patch, it could happen: 1. Metastore A update table t1 2. Metastore B update table t1 3. Metastore B read notification about A's update, overwrite B's newer change 4. Metastore B will eventually read B's update from notification, bring back B's newer change I think it might be ok with this eventual consistent behavior. Otherwise, we need some more complex mechanism to track which change is newer. cc [~thejas] > CachedStore: Use metastore notification log events to update cache > -- > > Key: HIVE-18661 > URL: https://issues.apache.org/jira/browse/HIVE-18661 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Reporter: Vaibhav Gumashta >Assignee: mahesh kumar behera >Priority: Major > Labels: pull-request-available > Attachments: HIVE-18661.02.patch > > > Currently, a background thread updates the entire cache which is pretty > inefficient. We capture the updates to metadata in NOTIFICATION_LOG table > which is getting used in the Replication work. We should have the background > thread apply these notifications to incrementally update the cache. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20834) Hive QueryResultCache entries keeping reference to SemanticAnalyzer from cached query
[
https://issues.apache.org/jira/browse/HIVE-20834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16672656#comment-16672656
]
Hive QA commented on HIVE-20834:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
50s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
48s{color} | {color:blue} ql in master has 2315 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
40s{color} | {color:red} ql: The patch generated 2 new + 466 unchanged - 1
fixed = 468 total (was 467) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 22m 45s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14700/dev-support/hive-personality.sh
|
| git revision | master / f3fab45 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14700/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14700/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Hive QueryResultCache entries keeping reference to SemanticAnalyzer from
> cached query
> -
>
> Key: HIVE-20834
> URL: https://issues.apache.org/jira/browse/HIVE-20834
> Project: Hive
> Issue Type: Bug
>Reporter: Jason Dere
>Assignee: Jason Dere
>Priority: Major
> Attachments: HIVE-20834.1.patch, HIVE-20834.2.patch,
> dominator_tree.png
>
>
> QueryResultCache.LookupInfo ends up keeping a reference to the
> SemanticAnalyzer from the cached query, for as long as the cached entry is in
> the cache. We should not be keeping the SemanticAnalyzer around after the
> query is done executing since they can hold on to quite a bit of memory.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20839) "Cannot find field" error during dynamically partitioned hash join
[
https://issues.apache.org/jira/browse/HIVE-20839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16672616#comment-16672616
]
Hive QA commented on HIVE-20839:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12946579/HIVE-20839.3.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 15520 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cbo_limit]
(batchId=182)
org.apache.hadoop.hive.metastore.TestCatalogNonDefaultClient.getTableMeta
(batchId=221)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14699/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14699/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14699/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12946579 - PreCommit-HIVE-Build
> "Cannot find field" error during dynamically partitioned hash join
> --
>
> Key: HIVE-20839
> URL: https://issues.apache.org/jira/browse/HIVE-20839
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Reporter: Jason Dere
>Assignee: Jason Dere
>Priority: Major
> Attachments: HIVE-20839.1.patch, HIVE-20839.2.patch,
> HIVE-20839.3.patch
>
>
> Occurs in some cases in the non-CBO optimized queries, either if CBO is
> disabled or has failed due to error.
> {noformat}
> 2018-10-11T04:40:22,724 ERROR [TezTR-85144_8944_1085_28_996_2
> (1539092085144_8944_1085_28_000996_2)] tez.ReduceRecordProcessor: Hit error
> while closing operators - failing tree
> 2018-10-11T04:40:22,724 ERROR [TezTR-85144_8944_1085_28_996_2
> (1539092085144_8944_1085_28_000996_2)] tez.TezProcessor:
> java.lang.RuntimeException: cannot find field _col304 from [0:_col0, 1:_col1,
> 2:_col2, 3:_col3, 4:_col4, 5:_col5, 6:_col6, 7:_col7, 8:_col8, 9:_col9,
> 10:_col10, 11:_col11, 12:_col12, 13:_col13, 14:_col15, 15:_col16, 16:_col17,
> 17:_col18, 18:_col19, 19:_col20, 20:_col21, 21:_col22, 22:_col23, 23:_col24,
> 24:_col25, 25:_col26, 26:_col27, 27:_col28, 28:_col29, 29:_col30, 30:_col31,
> 31:_col32, 32:_col33, 33:_col34, 34:_col35, 35:_col36, 36:_col37, 37:_col38,
> 38:_col39, 39:_col40, 40:_col41, 41:_col42, 42:_col43, 43:_col44, 44:_col45,
> 45:_col46, 46:_col47, 47:_col48, 48:_col49, 49:_col50, 50:_col51, 51:_col52,
> 52:_col53, 53:_col54, 54:_col55, 55:_col56, 56:_col57, 57:_col58, 58:_col59,
> 59:_col60, 60:_col61, 61:_col62, 62:_col63, 63:_col64, 64:_col65, 65:_col66,
> 66:_col67, 67:_col68, 68:_col70, 69:_col72, 70:_col73, 71:_col74, 72:_col75,
> 73:_col76, 74:_col77, 75:_col78, 76:_col79, 77:_col80, 78:_col81, 79:_col82,
> 80:_col83, 81:_col84, 82:_col85, 83:_col86, 84:_col87, 85:_col88, 86:_col89,
> 87:_col90, 88:_col91, 89:_col92, 90:_col93, 91:_col94, 92:_col95, 93:_col96,
> 94:_col97, 95:_col98, 96:_col99, 97:_col100, 98:_col101, 99:_col102,
> 100:_col103, 101:_col104, 102:_col105, 103:_col106, 104:_col107, 105:_col108,
> 106:_col109, 107:_col110, 108:_col111, 109:_col112, 110:_col113, 111:_col114,
> 112:_col115, 113:_col116, 114:_col117, 115:_col118, 116:_col119, 117:_col120,
> 118:_col121, 119:_col122, 120:_col123, 121:_col124, 122:_col125, 123:_col126,
> 124:_col127, 125:_col128, 126:_col129, 127:_col130, 128:_col131, 129:_col132,
> 130:_col133, 131:_col134, 132:_col135, 133:_col136, 134:_col137, 135:_col138,
> 136:_col139, 137:_col140, 138:_col141, 139:_col142, 140:_col143, 141:_col144,
> 142:_col145, 143:_col146, 144:_col147, 145:_col148, 146:_col149, 147:_col150,
> 148:_col151, 149:_col152, 150:_col153, 151:_col154, 152:_col155, 153:_col156,
> 154:_col157, 155:_col158, 156:_col159, 157:_col160, 158:_col161, 159:_col162,
> 160:_col163, 161:_col164, 162:_col165, 163:_col166, 164:_col167, 165:_col168,
> 166:_col169, 167:_col170, 168:_col171, 169:_col318]
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.getStandardStructFieldRef(ObjectInspectorUtils.java:485)
> at
> org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector.getStructFieldRef(StandardStructObjectInspector.java:153)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator.initialize(ExprNodeColumnEvaluator.java:80)
> at
>
